The Job

Senior Technical Specialist

Teknik, data och digitalt · Data, AI och analys · Data engineering

Gå med för uppdateringar

I korthet

Core Data Engineer (AWS) role in Bengaluru, Karnataka, responsible for designing, building, and operating secure and efficient data pipelines on AWS. This includes batch, streaming, and CDC ingestion, with a focus on embedding testing, governance, observability, and CI/CD throughout the data lifecycle.

Ansvarsområden

  • Implement ingestion and CDC pipelines using AWS Lambda, EMR, and manage schema evolution and backfills.
  • Build orchestration workflows with Step Functions and/or MWAA, including retries, alerts, and SLAs.
  • Model datasets on S3 (Parquet/Iceberg), Glue Data Catalog, and Athena, with a focus on partitioning, compaction, and performance tuning.
  • Implement automated data quality tests (Great Expectations/Deequ, PySpark unit tests), data contracts, and CI gates.
  • Implement security and governance measures including IAM, KMS, VPC endpoints, Lake Formation policies, PII handling, audit trails, and RBAC/RLS.
  • Manage observability and FinOps aspects such as CloudWatch metrics/alerts, lineage, usage dashboards, and cost optimization.
  • Provision infrastructure with Terraform/CloudFormation/CDK and build CI/CD pipelines with GitHub Actions for environment promotion and rollbacks.
  • Maintain pipeline diagrams, SLAs/SLOs, and incident playbooks.
  • Adopt and contribute to capabilities from other teams instead of building duplicate functionalities.

Krav

  • 10+ years of experience building data platforms on AWS.
  • Strong proficiency in Python (including PySpark) and SQL.
  • Hands-on experience with DMS, Kinesis/MSK, Glue/EMR, Lambda, Step Functions/MWAA, S3/Parquet/Iceberg, Glue Catalog, Athena, and Redshift/Serverless.
  • Experience with Infrastructure as Code (IaC) using Terraform.
  • Experience with CI/CD tools like GitHub Actions.
  • Familiarity with data quality frameworks such as Great Expectations or Deequ.
  • Knowledge of schema evolution and backfills.
  • Understanding of security and compliance measures: IAM/KMS, private networking, Lake Formation, audit/lineage.
  • Experience with performance and cost tuning across storage and compute.

Önskade kvalifikationer

  • Experience with Snowflake.
  • Experience with Apache Flink.
  • Experience with Redshift Spectrum.
  • Experience with OpenLineage.
  • Experience with OPA/policy as code.

Förmåner

  • Supercharge your potential at HCLTech.
  • Find your career and spark at a company that puts its people first.
  • Work for a global technology company with over 223,000 employees across 60 countries.
  • Deliver industry-leading capabilities in digital, engineering, cloud, and AI.
  • Serve clients across major verticals including Financial Services, Manufacturing, Life Sciences and Healthcare, Technology and Services, Telecom and Media, Retail and CPG, and Public Services.
#Data Engineering#AWS#Data Pipelines#Batch Processing#Streaming#CDC#Analytics#Testing#Governance#Observability#CI/CD#Orchestration#Lakehouse#Data Warehouse#Data Quality#Security#Cloud#FinOps#IaC
HCLTech Logo

Företag

HCLTech

Publicerade jobb

för 1 månad sedan

Anställningstyp

Heltid

Arbetsform

På plats

Erfarenhetsnivå

Senior

Platser

Bengaluru, India

Sökande

Ansök tidigt