Senior Data Engineer

Role: Senior Data Engineer - Azure

Experience Level: 4 to 6 Years

Work location: Mumbai, Bangalore, Trivandrum (Hybrid)

Notice Perid: 0-30 days

Key Responsibilities:

● Design and build scalable ETL/ELT pipelines using Azure Data Factory (ADF), Azure Databricks (Spark), and Azure Synapse Analytics.

● Mandatory hands-on experience with Microsoft Azure and Databricks.

● Understanding of the Medallion Architecture (specifically the Bronze/Raw layer) and data modeling concepts for data lakes.

● Experience building robust Orchestration pipelines (scheduling, error handling, logging) and managing full/incremental data loads.

● Knowledge of Delta format best practices, including partitioning, Z-ordering, vacuuming, and storage tiering for cost efficiency.

● Familiarity with CI/CD pipelines, automated deployment scripts, version control (Git), and Unit/Integration testing frameworks.

● Experience implementing Data Quality rules, PII handling/masking, and metadata management.

● Develop and optimize PySpark/Spark SQL jobs for large-scale batch and streaming data transformations.

● Ingest data from various sources including Apache Kafka, REST APIs, and RDBMS, ensuring real-time or near-real-time availability.

● Implement data modeling strategies (star schema, snowflake schema) for analytics consumption layers in Synapse or ADLS.

● Collaborate with DevOps teams to automate deployment using CI/CD pipelines (Azure DevOps, GitHub Actions, etc.).

● Monitor, troubleshoot, and optimize data workflows for performance, cost-efficiency, and reliability.

● Follow coding standards, participate in peer reviews, and maintain version-controlled code in Git repositories.

● Support data quality checks, logging, alerting, and observability mechanisms for production workloads.

● Participate in sprint ceremonies and contribute to task estimation and delivery planning.

Must have skills:

● 3+ years of experience in data engineering roles.

● Hands-on experience with:

○ Azure Data Factory (ADF) – building pipelines, triggers, linked services.

○ Azure Databricks – building and managing Spark jobs in PySpark.

○ Azure Synapse Analytics – data warehousing, SQL queries, workspace orchestration.

○ Apache Kafka – consuming and processing real-time data streams.

● Strong in SQL, Python, and Spark for data manipulation and transformation.

● Exposure to CI/CD practices (Azure DevOps, Git workflows, build/release pipelines).

● Understanding of data lake architecture and modern data warehousing principles.