Vacancy expired!
Responsibilities- Design, develop and maintain scalable data pipelines using Azure Data Factory and Azure Databricks.- Responsible for day-to-day collection and ingestion of raw data.- Work with the team to formalize data flows and standards.- Implement processes and solutions that ensure pipeline efficiency and monitor the data quality and pipeline status.- Collaborate with engineers, data scientists, and other team members to identify and transform data for ingestion, exploration, and modeling.- Performance tune and optimize all data ingestion and data integration processes.- Ensures governance and compliance to architectural standards and implementation practices.- Design, build, deploy and maintain Infrastructure as Code.
Required Skills and Experience- Bachelor’s degree in Computer Science or a similar field.- 10+ years of overall experience.- Hands-on experience with Azure Stack - Azure Data Factory, Azure Databricks, ADSL, Synapse, SQL Server.- Experience in building event driven Azure Function based pipeline solutions.- Experience in managing pipelines into a heterogeneous data architecture/technology ecosystem.- Comfortable setting up and overseeing batch and API based data pipelines.- Experience in designing data solutions in Azure including data distributions and partitions, scalability, disaster recovery and high availability.- Hands on experience with python or spark.- Microsoft Azure certification is a plus.- Effectively handle multiple projects simultaneously and pay close attention to detail.- Have excellent interpersonal and written/verbal communication skills.- Experience working in agile/scrum environment.Vacancy expired!