Vacancy expired!
- Google Cloud Platform Data Engineer Looking for a Big Data Engineer on Google Cloud Platform to help design, develop and maintain data engineering solutions to migrate data hosted on our on-prem platform to Google cloud Platform (Google Cloud Platform).
- The engineer will design and program PB sized scalable data lake interfaces, micro services and web technologies that support ingesting and querying structured data.
- Architect and program sophisticated distributed systems and high performance compute and data pipelines.
- Enable mining and analyzing data to help AI leaders and researchers make data driven decisions for data collection, diversity, training, and evaluation.
- Build and implement support for versioned, traceable, and immutable datasets in a data lake in a distributed and scalable manner.
- Spend a majority of the time hands-on writing code and peer reviewing high performance, high quality, and well tested and well architected code.
- Certified Google Cloud Data Engineer/Architect Bachelor’s degree with 3-5+ years of experience in Data Engineering/BI areas with at least 2 years data engineering on Google Cloud Platform·
- Experience working in Google Cloud Platform based Big Data deployments (Batch/Real-Time) leveraging Big Query, Big Table, Google Cloud Storage, PubSub, Data Fusion, Dataflow, Dataproc, etc.
- Experience developing and deploying ETL / ELT processes and documentation including physical data model, source to target mappings, ETL / ELT packages (Matillion, Fivetran, Spark, Google Data Fusion, etc.)
- Demonstrated mastery in Google BigQuery
- Strong knowledge of Hadoop, HDFS, Hive, Spark, Spark Streaming and Presto Strong knowledge on Google cloud storage Data lifecycle management
- Strong knowledge on BIGQuery Slots management
- Cost optimization for Dataproc workload management
- Demonstrated mastery in cloud database concepts and large-scale cloud data warehouse and lake implementations using Big Data Tools: BigQuery, Cloud Dataflow, Cloud Proc, Cloud Pub/Sub, Cloud Composer, Google Data Studio, Cloud functions, Google Cloud Storage Implement solutions for structured, semi-structured, and unstructured data sources, relational and non-relational databases.
- Advanced Java/Python coding skills Experience with CI-CD pipelines for promoting big data release deployments and designing log monitoring features ((e.g. JIRA, GitHub, Jenkins, Nexus, Artifactory)
- Experience in data visualization tools like Kibana, Grafana, Tableau and associated architectures.
- Design and build data engineering solutions using Google Cloud Platform (Google Cloud Platform) services: BigQuery, DataFlow, Pub/Sub, BigTable, Data Fusion, DataProc.
- Data modeling and schema design that will range across multiple business domains within the cloud for large enterprise data warehouse and data lakes solutions.
- Extracting, Loading, Transforming, cleaning, and validating data using cloud ETL/ELT tools
- Designing data engineering pipelines and architectures for data processing Partner with multiple client stakeholders including partners, business users, BI and Analytics teams.
- Work with teams to conduct training workshops to identify data sources, flows, and requirements.
- Periodically update senior management with the status of the project with excellent written and verbal communication skills
Vacancy expired!