Azure Data Engineer

Position Description:

We are seeking a highly skilled Senior Azure Data Engineer to join our growing data team. The role involves designing, developing, and operationalizing large-scale data processing systems on the Azure cloud platform. The ideal candidate will have strong expertise in Azure Databricks, Azure Data Lake, PySpark, and Python, with a proven ability to build efficient, scalable, and reliable data pipelines.

Your future duties and responsibilities:

• Design and maintain scalable data pipelines for ingesting, processing, and transforming structured and unstructured data.
• Develop and optimize Spark jobs using PySpark and Spark SQL within Azure Databricks.
• Implement Azure Data Lake Storage (Gen2) solutions following medallion architecture (Bronze, Silver, Gold layers).
• Collaborate with stakeholders (data architects, analysts, business teams) to translate requirements into technical solutions.
• Ensure data security and compliance with access controls, encryption, and masking.
• Perform data modeling for batch and real-time analytics (star schema, snowflake, data vault).
• Monitor and troubleshoot pipelines for performance and cost optimization.
• Automate deployments using CI/CD pipelines (Azure DevOps/GitHub Actions) and IaC tools (Terraform/Bicep).
• Establish data quality checks and governance standards.
• Mentor junior engineers and promote best practices in data engineering.

Required qualifications to be successful in this role:

• 5+ years of professional experience in data engineering with enterprise-grade solutions.
• 3+ years of hands-on expertise in Microsoft Azure data services, including:
• Azure Databricks: Spark clusters, notebooks, tuning, debugging.
• Medallion Architecture (Bronze, Silver, Gold).
• Experience with Azure Data Factory or Apache Airflow for orchestration.
• Azure Data Lake Storage (Gen2): Directory structure, RBAC/ACLs, performance optimization.
• Strong programming skills in Python (Pandas, APIs, unit testing).
• Expert-level proficiency in PySpark (DataFrame API, Spark SQL, Catalyst Optimizer).
• Advanced SQL development and optimization.
• Solid understanding of data modeling concepts.
• Familiarity with Git and collaborative development practices.

Technical Skills
• Microsoft Azure Data Engineer Associate (DP-) certification.
• Experience with Delta Lake (ACID transactions, schema enforcement, time travel).
• Real-time data processing with Azure Stream Analytics or Spark Streaming.
• Knowledge of Synapse Analytics, Event Hubs, Azure SQL DB, Cosmos DB, Purview.
• Exposure to DevOps/DataOps principles and tools (CI/CD, Terraform, Azure DevOps).
• Familiarity with Power BI or other visualization tools.

Skills:

Azure Data Lake

Python