Data Engineer with Azure DataBricks job vacancy

Vacancy expired!

Position:

Lead Data Engineer

Location: Columbus, IN

Minimum years of experience: 10+ Years (Mandatory)

Job Details: • Min Experience should have 10+ years of IT Experience, with

at least 3 years of Python and DataBricks experience• We need a Data Engineer/Lead to develop reusable ETL framework on

Azure DataBricks using Spark 3.0.• This custom framework will be used to create a template driven data load and transformation system on Data Lake and Big Databases. The Framework shall use components from Azure and DataBricks (Spark).• The developer is expected to write Python Test cases and Data Quality checks for all code produced as a precondition for CI/CD and higher environments.

Must Have Skills:• Must have Azure Cloud experience and understand Azure components like ADF, Azure SQL and Azure DataBricks.• Very Strong DataBricks, Spark, Pyspark, DataBricks SQL with Azure• Must have Strong ETL and ELT experience • Must have strong Python, DataBricks SQL skills beyond just calling Spark Api, must be fluent in Python programming language• Must have Relational Database knowledge for optimum loading of Data from on premise system and data lake• Must have experience with Data Lake and Data Bases• Must have knowledge of OOP and functional programming to create reusable framework for ETL• Must understand encryption and security required for PII, Financial and other sensitive data.• Must understand Delta Lake and other big data file formats• Good to have exposure to DevOps and CICD skills in Big Data Space• Good to have Airflow or AppWorx.• Good to have exposure to Manufacturing Domain• Good to have SQL as well as No SQL databases• Good to have AD security experience

Job Description:• Work closely with Architect and Business units in understanding technical requirement and implement independently reusable code. • Develop ETL Framework for template driven ETL• Develop Databricks code that can call Scala and other required libraries• Work with offshore and on shore teams and mentor team members on ETL and do KT on framework and design• Implement transformations and aggregations as requirement • Work in Agile manner and resolve ambiguous requirements and communicate effectively with his peers

Contact:

Anuradha Mahanand

IT Recruiter

Desk: Direct:

Email:

Vacancy expired!

ID	#19898121
State	Indiana
City	Columbus
Job type	Contract
Salary	USD Depends on Experience Depends on Experience
Source	IT-SCIENT
Showed	2021-09-19
Date	2021-09-16
Deadline	2021-11-14
Category	Et cetera
Create resume