Senior Group Technical Architect

Teknik, data och digitalt · Data, AI och analys · Data engineering · Dataanalys · Mjukvaruutveckling

I korthet

The Senior Group Technical Architect will design and build scalable data platforms on Microsoft Azure using Azure Databricks. Responsibilities include architecting data pipelines, ensuring data governance, and integrating advanced analytics, AI, and ML solutions.

Ansvarsområden

Design and implement scalable data architectures using Azure services
Build and optimize ETL/ELT pipelines using Azure Databricks and Apache Spark
Architect data lake and data warehouse solutions using Azure Data Lake and Azure Synapse Analytics
Define data modeling strategies (batch & real-time processing)
Ensure data security, governance, and compliance standards
Collaborate with stakeholders to understand business requirements and translate them into technical solutions
Optimize performance and cost efficiency of data workloads
Implement CI/CD pipelines for data engineering workflows
Mentor and guide data engineers and development teams
Integrate advanced analytics, AI, and ML solutions when required

Krav

Strong experience with Azure Databricks and Spark (PySpark/Scala)
Hands-on experience with Azure services (ADF, ADLS, Synapse, Event Hub)
Expertise in big data architecture and distributed systems
Strong knowledge of SQL, Python, and data engineering concepts
Experience with data modeling techniques (star schema, dimensional modeling)
Understanding of real-time streaming (Kafka/Event Hub)
Knowledge of DevOps and CI/CD practices
Analytical Skills & Problem Solving: Break down data problems, root-cause incidents, propose alternatives, estimate complexity, communicate clearly with stakeholders
Stake Holder Management: Interaction with Client Stakeholders, Communication

Önskade kvalifikationer

Data Engineering Foundations (Batch vs streaming; lakehouse concepts; medallion; file formats; partitioning & clustering; schema evolution; data governance basics; DevOps/CI-CD for data)
SQL (Joins, subqueries, CTEs, window functions, set operations; aggregation & rollups; MERGE/UPSERT; analytic functions; performance optimization)
Apache Spark (Core architecture; RDD vs DataFrame/Dataset; transformations; shuffle mechanics; caching; partitioning; broadcast joins; skew handling; checkpointing; job tuning)
PySpark (DataFrame API, Spark SQL; UDF vs pandas UDF; windowing; incremental loads; structured streaming; handling semi-structured data; optimizing with predicates, pushdown, join strategies; error handling; unit testing)
Databricks Platform (Workspace basics; clusters; DBR/LTS; notebooks & Repos; Jobs & Workflows; Delta Lake & Delta Live Tables; Unity Catalog; MLflow; secret scopes; DBFS; REST APIs; Databricks Connect)
Delta Lake / Lakehouse Patterns (ACID transactions; schema enforcement/evolution; time travel; OPTIMIZE/ZORDER; VACUUM; CDC patterns; streaming vs batch Delta; expectations/constraints; table maintenance)
Orchestration & Scheduling (Databricks Workflows; Azure Data Factory/Synapse pipelines; triggers; parameter passing; fail/retry; alerts; integration with AutoSys/Control-M/Jenkins/GitHub Actions; event-driven patterns)
Python (Core syntax; typing & data structures; file I/O; logging; virtual environments; packaging; testing; common data libs; error handling; performance considerations)
Data Quality & Testing (Great Expectations/dbx expectations or custom checks; unit/integration tests; reconciliation; anomaly detection; contract testing for schemas; data observability)
Security & Governance (Unity Catalog permissions, row/column-level security; secrets management; PII handling; audit logs; token management; compliance basics)
Cost & Performance Optimization (Cluster sizing, autoscaling; DBU awareness; spot/preemptible instances; storage formats; caching; efficient joins; job scheduling; monitoring; cost tagging and chargeback)

Förmåner

Opportunity to work with cutting-edge cloud technologies
Career growth and development opportunities
Competitive salary and benefits package

Färdigheter

Azure DatabricksApache SparkPySparkScalaAzureAzure Data FactoryAzure Data LakeAzure Synapse AnalyticsSQLPythonETLELTData ModelingKafkaEvent HubDevOpsCI/CDDelta LakeUnity CatalogMLflow

#Data Engineering#Big Data#Azure#Databricks#Spark#Cloud#Architecture

Om HCLTech

Supercharging progress through technology and innovation

HCLTech is a global technology company, home to 219,000+ people across 54 countries, delivering industry-leading capabilities centered around digital, engineering and cloud, powered by a broad portfolio of technology services and products. We work with clients across all major verticals, providing industry solutions for Financial Services, Manufacturing, Life Sciences and Healthcare, Technology and Services, Telecom and Media, Retail and CPG, and Public Services. Consolidated revenues as of 12 months ending September 2022 totaled $12.1 billion. To learn how we can supercharge progress for…

Se alla lediga jobb hos HCLTech Läs mer om HCLTech Webbplats