Sr Big Data Engineer / Lead (Spark, Scala/Java)

Job Description

(12+ Years of experience required with lead /Architect who is willing to do hands-on development work )Key skills expected : Spark Streaming, Kafka, Scala, AWS, with strong development experience

Note : Profiles from Data Background will not work , Please look for profiles from Java background

Responsibilities

Build and optimize performance of Scala-Spark batch jobs using AWS EMR Build and optimize performance of Kafka, NIFI and other components of real-time pipelines Participate and contribute to the design, architecture and development of high quality data-lake data models Build data pipeline orchestration Ensuring code quality and conformance of code to applicable rules, norms and relevant best practices Coaching, mentoring and developing new hires and less experienced developers to work on Big Data projectsSkills Strong understanding of distributed systems and distributed computation. Strong working knowledge in Scala and Spark Streaming, Kafka Good knowledge in AWS services like S3, EMR, Glue, SageMaker, Lambda, ECS, DMW and Athena. Demonstrated working knowledge in Spark, Kafka, NiFi, Demonstrated working knowledge of data modeling Experience in designing and running Unit and Integration testing Hands-on exposure to data store technologies like MongoDB, DynamoDB, Postgres and Redshift. Knowledge of TDD & BDD methodologies, tools and practices

Qualifications

Spark, Scala/Java

Additional Information