The Job

Data Pipeline Engineer

Technology, Data & Digital · Data, AI & Analytics · Data Engineering · Technical Support

Join for updates

In short

Microsoft is seeking a Data Pipeline Engineer to design, develop, and operate reliable and scalable data platforms for business intelligence and analytics. This role involves building ETL/ELT solutions, optimizing data models, and ensuring data quality and security within modern cloud environments, utilizing skills in SQL, Python/Scala, and various cloud data technologies.

Responsibilities

  • Design, develop, and maintain batch and real-time data pipelines for enterprise data ingestion, transformation, and delivery.
  • Build scalable ETL/ELT solutions with monitoring, alerting, and data quality controls.
  • Ensure data pipelines are reliable, performant, and cost-efficient.
  • Troubleshoot and resolve pipeline failures, performance bottlenecks, and data quality issues.
  • Design and optimize data models, data warehouses, lakehouses, and storage solutions.
  • Support migration of legacy data environments to modern cloud-based architectures.
  • Apply data security recommended approaches and ensure compliance with organizational policies.
  • Protect enterprise data assets and support secure operational processes.
  • Partner with data scientists, analysts, engineers, and business stakeholders to deliver trusted datasets.

Requirements

  • Bachelor's Degree in Computer Science, Information Technology, Engineering, Data Science, or related technical field AND 3+ years of technical support, technical consulting, data engineering, or information technology experience OR 5+ years of technical support, data engineering, technical consulting experience, or information technology experience OR equivalent experience.
  • Strong SQL and proficiency in a programming language such as Python or Scala.
  • Experience building ETL/ELT pipelines and data modeling for analytics.
  • Familiarity with orchestration tools (e.g., Airflow, Azure Data Factory) and cloud data platforms.
  • Experience with Microsoft Fabric, especially migrations of existing warehouses into Fabric.
  • Experience with big-data and streaming tools (e.g., Spark, Kafka).
  • Experience with cloud data warehouses (e.g., Synapse, OneLake, Lakehouse).
  • Knowledge of data quality frameworks, CI/CD, and infrastructure-as-code.
  • Experience with Near Real Time (NRT) event streams like Azure Event Hubs + Azure Synapse.
  • Experience with Azure SQL Managed Instances, Power BI Gateways, and DataFlows.

Desired Qualifications

  • Ability to meet Microsoft, customer and / or government security screening requirements.
  • This position will be required to pass the Microsoft Cloud Background Check upon hire / transfer and every two years thereafter.

Benefits

  • Typical base pay range for this role across the U.S. is USD $86,100 - $169,800 per year.
  • Different base pay range applicable to specific work locations, within the San Francisco Bay area and New York City metropolitan area: USD $113,300 - $187,400 per year.
  • Certain roles may be eligible for benefits and other compensation.
#Data Pipeline#Data Engineering#Cloud#ETL#ELT#SQL#Python#Scala#Microsoft Fabric#Spark#Kafka#Synapse#Lakehouse#Azure
Microsoft Logo

Company

Microsoft

Job Posted

1 month ago

Employment Type

Full Time

WorkMode

Hybrid

Experience Level

Mid-Senior

Locations

USA

Qualification

Bachelor

Applicants

Be an early applicant