Lead Data Engineer

Summary:

Greenfield Opportunity: Are you ready to shape the future Our data warehouse isn't just a project; it's a blank canvas, waiting for your expertise. Imagine the thrill of creating from scratch, drawing upon the wisdom gained from your past experiences. Join our small team of IT professionals and be part of something extraordinary. Let's build a data landscape that inspires innovation!

Reporting to the CIO, you will own the continuous improvement culture for designing, building, and maintaining the infrastructure and systems required for collecting, storing, processing, and analyzing large volumes of data. As a Data Engineer, you will play a crucial role in ensuring that data is readily available, accessible, and usable by other data professionals, analysts, and stakeholders within an organization. You will be responsible for managing a data Lake / warehouse and implementing efficient data integration processes with internal and external systems.

Duties/Responsibilities:

• Design and maintain a scalable and secure Azure data warehouse.
• Build data pipelines: Create and manage data integration processes, including data extraction, transformation, and loading (ETL) from various sources into the data warehouse.
• Develop and implement data management strategies to ensure data quality, consistency, and accuracy.
• Collaborate with cross-functional teams, including to identify data requirements and develop data models.
• Develop, adhere, and enforce data governance policies and procedures to ensure compliance.
• Monitor data quality and develop response mechanisms to address problems.
• Collaborate with IT teams to ensure the availability, reliability, and performance of data systems and infrastructure.
• Develop documentation related to data management processes, procedures, and data dictionaries.
• Stay up to date with industry trends and advancements in data management technologies and techniques.
• Maintain the current state legacy Microsoft environment and processing, which includes troubleshooting production issues and implementing tactical enhancements where needed.
• As needed, collaborate with external contractors to enhance data solutions. Coordinate their work, provide guidance, and ensure alignment with project goals.

Qualifications:

Education/Experience

1. Education and Experience :

• Bachelor's or Master's degree in Computer Science, Information Systems, or a related field.
• Proven experience working as a Data Engineer or in a similar role, preferably within the insurance or financial sector.

1. Data Management Proficiency :

• Strong understanding of data management principles, including data modeling, schema design, and data lineage.
• Knowledge of data quality assurance techniques and best practices.
• Familiarity with data governance frameworks and policies.

1. Azure Technologies :

• Proficiency in data warehousing concepts and hands-on experience with Azure services (e.g., Azure Data Factory, Azure Synapse Analytics, Azure SQL Database).
• Ability to design and implement ETL processes using Azure tools.

1. Technical Skills :

• Proficiency in SQL for data manipulation and querying.
• Experience with Python for data transformation and scripting.
• Familiarity with ETL frameworks (e.g., Apache Spark, Talend).

1. Data Privacy and Security :

• Solid understanding of regulatory requirements such as GDPR and HIPAA.
• Ability to implement data privacy controls and ensure secure data handling.

1. Problem-Solving and Analytical Skills :

• Excellent ability to analyze complex data sets, troubleshoot issues, and optimize data workflows.
• Creative problem-solving mindset to address data-related challenges.

1. Effective Communication :

• Strong interpersonal skills to collaborate with cross-functional teams.
• Ability to convey complex technical ideas clearly to both technical and non-technical stakeholders.

1. Positive Attitude and Adaptability :

• A can-do mindset and willingness to tackle challenges head-on.
• Adaptability to evolving technologies and business needs.

Technical Competencies

1. Microsoft Azure Services Proficiency :

• Azure Data Factory: Ability to design and implement data integration workflows using Azure Data Factory.
• Azure Data Lake Storage: Familiarity with managing data in Azure Data Lake Storage.
• Azure SQL Data Warehouse: Expertise in designing and optimizing data warehousing solutions using Azure SQL Data Warehouse.

1. Data Processing and Analytics :

• Azure Data Lake Analytics: Understanding of data processing using Azure Data Lake Analytics.
• Big Data Technologies: Knowledge of big data frameworks (e.g., Apache Spark, Hadoop) within Azure.

1. Data Governance and Security :

• Data Governance Principles: Awareness of data governance best practices within Azure.
• Access Control: Understanding of role-based access control (RBAC) and permissions management.
• Data Privacy Compliance: Knowledge of compliance requirements (e.g., GDPR, HIPAA) related to data privacy and security.

1. Monitoring and Troubleshooting :

• Azure Data Factory Monitoring: Skills in monitoring, optimizing, and troubleshooting data pipelines within Azure Data Factory.
• Diagnostic Logging: Familiarity with diagnostic logs and metrics for Azure services.

1. Scripting and Automation :

• PowerShell or Python: Proficiency in scripting languages (PowerShell or Python) for automating tasks within Azure.

1. Effective Communication and Collaboration :

• Cross-Functional Teams: Strong communication skills to collaborate effectively with data scientists, analysts, and business stakeholders.
• Technical Documentation: Ability to document data solutions clearly.

1. Continuous Learning and Adaptability :

• Stay Updated: Proactively keep up with the latest Azure data services, features, and advancements.
• Adaptability: Be ready to embrace new technologies and adapt to evolving business needs.

1. Support and maintenance for legacy technologies

• Microsoft SQL Server: Knowledge of SQL Server for data warehousing and relational databases.
• T-SQL: Proficiency in Transact-SQL (T-SQL) for querying and managing data.
• Understanding of operating systems such as:

1. Microsoft Windows
2. Linux
3. UNIX

Physical Demands and Work Environment

This job description in no way states or implies that these are the only duties to be performed by the employee(s) incumbent in this position. Employees will be required to follow any other job-related instructions and to perform any other job-related duties requested by any person authorized to give instructions or assignments. All duties and responsibilities are essential functions and requirements and are subject to possible modification to reasonably accommodate individuals with disabilities. To perform this job successfully, the incumbents will possess the skills, aptitudes, and abilities to perform each duty proficiently. The company is an Equal Opportunity Employer, drug free workplace, and complies with ADA regulations as applicable.