The Job

Senior Technical Specialist

Technology, Data & Digital · Data, AI & Analytics · Data Engineering

Join for updates

In short

Core Data Engineer (AWS) role in Bengaluru, Karnataka, responsible for designing, building, and operating secure and efficient data pipelines on AWS. This includes batch, streaming, and CDC ingestion, with a focus on embedding testing, governance, observability, and CI/CD throughout the data lifecycle.

Responsibilities

  • Implement ingestion and CDC pipelines using AWS Lambda, EMR, and manage schema evolution and backfills.
  • Build orchestration workflows with Step Functions and/or MWAA, including retries, alerts, and SLAs.
  • Model datasets on S3 (Parquet/Iceberg), Glue Data Catalog, and Athena, with a focus on partitioning, compaction, and performance tuning.
  • Implement automated data quality tests (Great Expectations/Deequ, PySpark unit tests), data contracts, and CI gates.
  • Implement security and governance measures including IAM, KMS, VPC endpoints, Lake Formation policies, PII handling, audit trails, and RBAC/RLS.
  • Manage observability and FinOps aspects such as CloudWatch metrics/alerts, lineage, usage dashboards, and cost optimization.
  • Provision infrastructure with Terraform/CloudFormation/CDK and build CI/CD pipelines with GitHub Actions for environment promotion and rollbacks.
  • Maintain pipeline diagrams, SLAs/SLOs, and incident playbooks.
  • Adopt and contribute to capabilities from other teams instead of building duplicate functionalities.

Requirements

  • 10+ years of experience building data platforms on AWS.
  • Strong proficiency in Python (including PySpark) and SQL.
  • Hands-on experience with DMS, Kinesis/MSK, Glue/EMR, Lambda, Step Functions/MWAA, S3/Parquet/Iceberg, Glue Catalog, Athena, and Redshift/Serverless.
  • Experience with Infrastructure as Code (IaC) using Terraform.
  • Experience with CI/CD tools like GitHub Actions.
  • Familiarity with data quality frameworks such as Great Expectations or Deequ.
  • Knowledge of schema evolution and backfills.
  • Understanding of security and compliance measures: IAM/KMS, private networking, Lake Formation, audit/lineage.
  • Experience with performance and cost tuning across storage and compute.

Desired Qualifications

  • Experience with Snowflake.
  • Experience with Apache Flink.
  • Experience with Redshift Spectrum.
  • Experience with OpenLineage.
  • Experience with OPA/policy as code.

Benefits

  • Supercharge your potential at HCLTech.
  • Find your career and spark at a company that puts its people first.
  • Work for a global technology company with over 223,000 employees across 60 countries.
  • Deliver industry-leading capabilities in digital, engineering, cloud, and AI.
  • Serve clients across major verticals including Financial Services, Manufacturing, Life Sciences and Healthcare, Technology and Services, Telecom and Media, Retail and CPG, and Public Services.
#Data Engineering#AWS#Data Pipelines#Batch Processing#Streaming#CDC#Analytics#Testing#Governance#Observability#CI/CD#Orchestration#Lakehouse#Data Warehouse#Data Quality#Security#Cloud#FinOps#IaC
HCLTech Logo

Company

HCLTech

Job Posted

1 month ago

Employment Type

Full Time

WorkMode

On Site

Experience Level

Senior

Locations

Bengaluru, India

Applicants

Be an early applicant