Data Lake Architect

Data Lake Architect at Weehawken,NJ (Day 1 onsite -Hybrid Model ) Job Description : Design a data storage structure: design an Azure Data Lake solution | recommend file types for storage | recommend file types for analytical queries | design for efficient querying | design for data pruning | design a folder structure that represents the levels of data transformation | design a distribution strategy Design a partition strategy: design a partition strategy for files | design a partition strategy for analytical workloads Implement physical data storage structures compression | partitioning | shading | different table geometries with Azure Synapse Analytics pools | data redundancy | implement distributions | implement data archiving Implement the serving layer: deliver data in a relational star schema | deliver data in Parquet files | maintain metadata |implement a dimensional hierarch Design and Develop Data Processing Ingest and transform data using Spark, T-SQL, Data Factory, Synapse Pipelines Implement stream and batch pipelines. Design and Implement Data Security Data policies and standards: masking, encryption, row level and column level security, RBAC, Data retention, auditing Manage sensitive information Monitor and Optimize Data Storage and Data Processing Implement logging used by Azure Monitor, configure monitoring service Measure and improve data pipeline performance, cluster performance, query performance Manage storage related optimizations like compaction, handling data skews, tune queries using indexing, cache, trouble failed spark jobs Vimerse is a fast growing IT Company having a team of well trained professionals and with a huge spectrum of services to offer. It s having Innovation and Reliability as its driving force. We are proud to say that with our fleeting yet diverse experiences we are able to become a partner of choice with an astonishing accomplishment and customer satisfaction for our clients. Our main motive is to provide professional support system to our clients aligned to their core business activity with an aim to provide cost effectiveness and optimum utilization of available resource while keeping pace with dynamic market conditions. We are an equal opportunity employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, disability, or veteran status