Lead Consultant(Product/Domain)

Technology, Data & Digital · Data, AI & Analytics · Data Engineering · DevOps · Cloud Engineering

In short

Lead Consultant (Product/Domain) needed to architect end-to-end data products using Medallion Architecture on AWS with Databricks. Responsibilities include designing and scaling data pipelines using PySpark and SQL, optimizing Lakehouse storage, and implementing data governance. Requires 6+ years of Data Engineering experience, with a deep focus on Databricks within an AWS environment.

Responsibilities

Architect end-to-end data products using the Medallion Architecture (Bronze, Silver, Gold layers) on Amazon S3 and Delta Lake.
Design and scale production pipelines using PySpark, SQL, and Delta Live Tables (DLT) for batch and real-time streaming data.
Ingest diverse data streams by integrating Databricks with AWS messaging systems like Amazon Kinesis or MSK.
Optimize Lakehouse storage layouts using Delta features (Liquid Clustering, Z-Ordering, data compaction).
Implement data governance and role-based access control using Unity Catalog.
Manage infrastructure security by configuring security roles and personas.
Optimize cloud spend by managing Databricks clusters and selecting appropriate EC2 instance types.
Integrate security frameworks using AWS Secrets Manager.
Utilize Azure DevOps via Git integration, automated unit testing, and custom CI/CD pipelines (GitHub Actions, Databricks DAB).
Orchestrate data workflows using Databricks Workflows or Apache Airflow.
Deploy Infrastructure as Code (IaC) using Terraform.
Mentor and guide mid-level engineers, establishing coding standards and running peer code reviews.

Requirements

6+ years in Data Engineering, with 4+ years of deep focus on Databricks within an AWS environment.
Expert-level Python (PySpark) and complex, analytical SQL.
Deep technical mastery of Delta Lake, Amazon S3, and advanced Spark optimization techniques.
Strong hands-on proficiency with EC2, IAM, S3, Secrets Manager, and VPC networking fundamentals.
Proven experience with Terraform for data platform provisioning and automated deployment tools.
Ability to lead technical design discussions and choose cost-efficient cloud patterns.
Clear documentation skills to map complex data lineage and infrastructure dependencies.

Desired Qualifications

Databricks Certified Data Engineer Associate/Professional (optional).

Benefits

Supercharge your potential at HCLTech.
Find your career and your spark at HCLTech.
HCLTech puts its people first.
Global technology company with over 223,000 people across 60 countries.
Industry-leading capabilities in digital, engineering, cloud, and AI.
Works with clients across all major verticals including Financial Services, Manufacturing, Life Sciences, Technology, Telecom, Media, Retail, CPG, and Public Services.

Skills

PythonPySparkSQLDatabricksAWSDelta LakeAmazon S3EC2IAMSecrets ManagerVPCTerraformAzure DevOpsGitGitHub ActionsDatabricks DABDatabricks WorkflowsApache AirflowUnity CatalogDelta Live TablesAmazon KinesisMSK

#Data Engineering#Databricks#AWS#Medallion Architecture#PySpark#SQL#Delta Live Tables#Batch#Streaming#Delta Lake#Amazon S3#EC2#IAM#Secrets Manager#VPC#Terraform#Azure DevOps#Git#CI/CD#GitHub Actions#Databricks DAB#Databricks Workflows#Apache Airflow#Unity Catalog#Data Governance#Infrastructure as Code#Python

About HCLTech

Supercharging progress through technology and innovation

HCLTech is a global technology company, home to 219,000+ people across 54 countries, delivering industry-leading capabilities centered around digital, engineering and cloud, powered by a broad portfolio of technology services and products. We work with clients across all major verticals, providing industry solutions for Financial Services, Manufacturing, Life Sciences and Healthcare, Technology and Services, Telecom and Media, Retail and CPG, and Public Services. Consolidated revenues as of 12 months ending September 2022 totaled $12.1 billion. To learn how we can supercharge progress for…

See all open jobs at HCLTech Read more about HCLTech Website