Job Description

Azure Data Engineer (Lead) with skills Data Engineering, Python, Apache Hadoop, Apache Hive, Apache Airflow, synapse, Databricks, SQL, Apache Spark, Azure Data Factory, Pyspark, GenAI Fundamentals, Cloud Pub/Sub, BigQuery for location Gurugram, India

Posted on: January 28, Share on Linkedin Share on Twitter Share on Facebook

ROLES & RESPONSIBILITIES

Key Responsibilities

  • Lead design and execution of Dataproc ? Databricks PySpark migration roadmap.

  • Define modernization strategy, including data ingestion, transformation, orchestration, and governance.

  • Architect scalable Delta Lake and Unity Catalog–based solutions.

  • Manage and guide teams on code conversion, dependency mapping, and data validation.

  • Collaborate with platform, infra, and DevOps teams to optimize compute costs and performance.

  • Own the automation & GenAI acceleration layer, integrating code parsers, lineage tools, and validation utilities.

  • Conduct performance benchmarking, cost optimization, and platform tuning (Photon, Auto-scaling, Delta Caching).

  • Mentor senior and mid-level developers, ensuring quality standards, documentation, and delivery timelines.

    • Technical Skills

  • Languages: Python, PySpark, SQL

  • Platforms: Databricks (Jobs, Workflows, Delta Live Tables, Unity Catalog), GCP Dataproc

  • Data Tools: Hadoop, Hive, Pig, Spark (RDD & DataFrame APIs), Delta Lake

  • Cloud & Integration: GCS, BigQuery, Pub/Sub, Cloud Composer, Airflow

  • Automation: GenAI-powered migration tools, custom Python utilities for code conversion

  • Version Control & DevOps: Git, Terraform, Jenkins, CI/CD pipelines

  • Other: Performance tuning, cost optimization, and lineage tracking with Unity Catalog

    • Preferred Experience

  • 10–14 years of data engineering experience with at least 3 years leading Databricks or Spark modernization programs.

  • Proven success in migration or replatforming projects from Hadoop or Dataproc to Databricks.

  • Exposure to AI/GenAI in code transformation or data engineering automation.

  • Strong stakeholder management and technical leadership skills.

    • EXPERIENCE

  • 11-12 Years

    • SKILLS

  • Primary Skill: Data Engineering
  • Sub Skill(s): Data Engineering
  • Additional Skill(s): Python, Apache Hadoop, Apache Hive, Apache Airflow, synapse, Databricks, SQL, Apache Spark, Azure Data Factory, Pyspark, GenAI Fundamentals, Cloud Pub/Sub, BigQuery

    • ABOUT THE COMPANY

    Infogain is a human-centered digital platform and software engineering company based out of Silicon Valley. We engineer business outcomes for Fortune companies and digital natives in the technology, healthcare, insurance, travel, telecom, and retail & CPG industries using technologies such as cloud, microservices, automation, IoT, and artificial intelligence. We accelerate experience-led transformation in the delivery of digital platforms. Infogain is also a Microsoft (NASDAQ: MSFT) Gold Partner and Azure Expert Managed Services Provider (MSP).


    Infogain, an Apax Funds portfolio company, has offices in California, Washington, Texas, the UK, the UAE, and Singapore, with delivery centers in Seattle, Houston, Austin, Kraków, Noida, Gurgaon, Mumbai, Pune, and Bengaluru.

    Apply for this Position

    Ready to join ? Click the button below to submit your application.

    Submit Application