Vacancy expired!
Job Title: Big Data Engineer Location: Plano, TX Duration: 12 + Months Job Description:
- This project will implement a 3-step process including the following:
- First to identify the contacts, source of information and availability of data.
- Second, using the requirements, develop a pipeline for the data into a normalized structure called the BRD (Business Ready Dataset).
- Third will be to automate the process of data acquisition and BRD creation into a production environment with checks and balances (monitoring, etc.).
- Development of a self-service platform to be used by our customers driving cost savings initiatives.
- Contribute to an Agile team of developers focused on data ingestion across multiple sources.
- Operate in a CI/CD environment.
- Implement data extraction from various/distributed sources utilizing scripting and storage in a SQL/HSQL environment.
- Strong analytical, planning, and organizational skills with an ability to manage competing demands
- Strong experience in data exploration with large data sets
- Strong technical background with high proficiency in SQL, Python and PySpark.
- Experience in Big Data Technologies like Hadoop (HDFS, Hive, Spark)
- Research, learn & adapt new technologies to solve problems & improve existing solutions
- Experience delivering ETL, data warehouse and data analytics capabilities on big-data architecture such as Hadoop
- Solid understanding of distributed computing and/or massively parallel processing concepts in Spark
- Experience with schema design and dimensional data modeling
- Deep expertise in the design, creation, management, and business use of significantly large datasets
- Ability to analyze the legacy reporting applications, queries and be able to migrate them over to Spark based reporting
- Perform data analysis required to troubleshoot data related issues and assist in the resolution of data issues.
- Develop and maintain scalable data pipelines and build out new integrations to support continuing increases in data volume and complexity.
- Java scripting experience to develop UI for customized reporting
- Creativity in taking on complex tasks head on and generating solutions to complex issues
- Palantir experience
- Azure experience
- Databricks experience
- Experience using dashboarding software such as MS Power BI
Vacancy expired!