One North Consulting is currently hiring Senior Big Data Engineers with about 5 - 8 years of experience as per details given below.
Skills & Responsibilities
As Senior Big Data Engineer, you will focus on managing the Hadoop cluster, implementing data ingestion framework designing data models.
Your primary role will be to implement data lake & transform the data for business use.
- Experience with at least one Cloud Infra provider (Azure/AWS)
- Experience in building data pipelines using batch processing with Apache Spark (Spark SQL, DataSet / Dataframe API) or Hive query language (HQL)
- Knowledge of Big data ETL processing tools
- Experience with Hive and Hadoop file formats (Avro / Parquet / ORC)
- Basic knowledge of scripting (shell / bash)
- Experience with CI CD tools such as Jenkins, JIRA, Bitbucket, Artifactory, Bamboo and Azure Dev-ops.
- understanding of DevOps practices using Git version control
- Debug, fine tune and optimize large scale data processing jobs
- Deliver big data solutions based on premise Hadoop or cloud-based systems like AWS.
- Manage Hadoop cluster, participate in scale out planning & implementation.
- Design ingestion layer for structured & unstructured data (text, voice, xml etc)
- Implement specific data model for business & analytics use.
- Deliver ELT solution including data extraction, transformation, cleansing, data integration and data management.
- Augment with new sources of data including internal/external untapped data.
- Contribute to the establishment and maintenance of cloud computing platform and big data services.
- Provide support for analytics tools & environment like RServer etc & debug performance issues.
COMPETENCIES
- Databases: RDBMS, SQL programming
- ETL and Data Integration Tools: Azure Data Factory (ADF), Microsoft SQL Server Integration Service(SSIS), SAS Data Integration
- Big Data: Hadoop (Hortonworks), Hive, Spark, Sqoop, etc
- Programming and Scripting: Linux/Unix Shell Scripting, Java, Scala, Hive QL
- BI/Dashboarding: SAS Visual Analytics, Qliksense
Working Experience:
- 58 years in data engineering and modelling.
- Hands-on in managing data mapping, data quality and integrity, performance in data processing.
- Experience in Agile software development.
If Interested, please email us your CV to [Confidential Information]
