Vacancy expired!
We are seeking a skilled Azure Databricks Engineer to design, develop, and optimize large-scale data processing systems using Azure Databricks. The ideal candidate will have expertise in Apache Spark, data engineering pipelines, and Azure cloud technologies. You will collaborate with cross-functional teams to build robust, secure, and scalable data solutions.Typical Duties and Responsibilities:Design and implement data ingestion and transformation pipelines using Azure Databricks and other Azure data services.Develop ETL/ELT processes for structured, semi-structured, and unstructured data.Optimize and tune Apache Spark jobs for performance and cost efficiency.Build and manage scalable data lakehouse solutions using Delta Lake and Azure Data Lake Storage.Integrate Databricks with Azure Synapse Analytics, Data Factory, and other Azure resources.Implement security best practices: role-based access control, encryption, and data masking.Collaborate with data scientists and analysts to operationalize machine learning models using MLflow.Automate workflows with Databricks Jobs and CI/CD pipelines.Monitor and troubleshoot performance issues in Databricks clusters and Spark applications.
Vacancy expired!