Job Details

ID #19722050
State Texas
City Plano
Job type Contract
Salary USD $DOE DOE
Source Lorven Technologies, Inc.
Showed 2021-09-16
Date 2021-09-16
Deadline 2021-11-14
Category Et cetera
Create resume

Big Data Engineer - Plano, TX - Contract

Texas, Plano, 75094 Plano USA

Vacancy expired!

Job Title: Big Data Engineer Location: Plano, TX Duration: 12 + Months Job Description:

  • This project will implement a 3-step process including the following:
  • First to identify the contacts, source of information and availability of data.
  • Second, using the requirements, develop a pipeline for the data into a normalized structure called the BRD (Business Ready Dataset).
  • Third will be to automate the process of data acquisition and BRD creation into a production environment with checks and balances (monitoring, etc.).
Responsibilities:
  • Development of a self-service platform to be used by our customers driving cost savings initiatives.
  • Contribute to an Agile team of developers focused on data ingestion across multiple sources.
  • Operate in a CI/CD environment.
  • Implement data extraction from various/distributed sources utilizing scripting and storage in a SQL/HSQL environment.
  • Strong analytical, planning, and organizational skills with an ability to manage competing demands
  • Strong experience in data exploration with large data sets
Mandatory Skills Description:
  • Strong technical background with high proficiency in SQL, Python and PySpark.
  • Experience in Big Data Technologies like Hadoop (HDFS, Hive, Spark)
  • Research, learn & adapt new technologies to solve problems & improve existing solutions
  • Experience delivering ETL, data warehouse and data analytics capabilities on big-data architecture such as Hadoop
  • Solid understanding of distributed computing and/or massively parallel processing concepts in Spark
  • Experience with schema design and dimensional data modeling
  • Deep expertise in the design, creation, management, and business use of significantly large datasets
  • Ability to analyze the legacy reporting applications, queries and be able to migrate them over to Spark based reporting
  • Perform data analysis required to troubleshoot data related issues and assist in the resolution of data issues.
  • Develop and maintain scalable data pipelines and build out new integrations to support continuing increases in data volume and complexity.
Great to have:
  • Java scripting experience to develop UI for customized reporting
  • Creativity in taking on complex tasks head on and generating solutions to complex issues
Nice-to-Have Skills:
  • Palantir experience
  • Azure experience
  • Databricks experience
  • Experience using dashboarding software such as MS Power BI

Vacancy expired!

Subscribe Report job