Job Description
Lead Data Engineer – AWS & Python
Client: EGEN
Role: AWS Data Engineer (C2 H)
Experience: 6–12 Years
- L3: 6–9 Years (Budget: 17 LPA)
- L4: 9–12 Years (Budget: 23 LPA)
- Location: Hyderabad
- Notice Period: Immediate to 7 Days Preferred
- Mode of Interview: Face-to-Face ( Wednesday, Thursday)
Job Overview
We are looking for an experienced Lead Data Engineer to design, build, and optimize scalable, high-performance data platforms using AWS cloud services and Python. This role involves architecting end-to-end data pipelines, ensuring data quality, enabling analytics and AI workloads, and providing technical leadership to data engineering teams.
Key Responsibilities
- Architect and implement fault-tolerant data pipelines using AWS Glue, Lambda, EMR, Step Functions, Redshift
- Build and optimize data lakes and data warehouses on S3, Redshift, Athena
- Develop Python-based ETL/ELT frameworks and reusable transformation modules
- Integrate multiple data sources (RDBMS, APIs, Kafka/Kinesis, Saa S systems) into unified models
- Lead data modeling, schema design, partitioning strategies for performance and cost optimization
- Drive data quality, observability, and lineage using AWS Data Catalog, Glue Data Quality, or third-party tools
- Define and enforce data governance, security, and compliance best practices
- Collaborate with Data Science, Analytics, Product, and Dev Ops teams
- Implement CI/CD pipelines for data workflows using AWS Code Pipeline, Git Hub Actions, or Cloud Build
- Provide technical leadership, code reviews, and mentoring to junior engineers
- Monitor infrastructure performance, troubleshoot issues, and lead capacity planning
Mandatory Skills
- 6–12 years of experience in data engineering / data platform development
- Expert in Python (pandas, Py Spark, boto3, SQLAlchemy)
- Strong experience with AWS Data Services: Glue, Lambda, EMR, Step Functions, Redshift, Athena, S3, Kinesis, Dynamo DB
- Proficiency in SQL, data modeling, and performance tuning
- Hands-on experience with ETL/ELT pipelines, data lakes, data warehouses, and streaming solutions
- Version control (Git) and CI/CD for data pipelines
- Experience with containerization (Docker/Kubernetes) is a plus
- Strong analytical, debugging, and communication skills
Preferred / Nice-to-Have Skills
- Experience with Apache Spark or Py Spark on EMR or Glue
- Familiarity with Airflow, dbt, or Dagster
- Real-time data streaming (Kafka, Kinesis)
- Exposure to Lake Formation, Glue Studio, Data Brew
- Integration with ML/Analytics platforms (Sage Maker, Quick Sight)
- AWS Certifications: Data Analytics – Specialty or Solutions Architect
Soft Skills
- Ownership mindset with focus on reliability and automation
- Ability to mentor and guide junior engineers
- Effective communication with technical and non-technical stakeholders
- Comfortable working in agile, cross-functional teams
Client: EGEN
Role: AWS Data Engineer (C2 H)
Experience: 6–12 Years
- L3: 6–9 Years (Budget: 17 LPA)
- L4: 9–12 Years (Budget: 23 LPA)
- Location: Hyderabad
- Notice Period: Immediate to 7 Days Preferred
- Mode of Interview: Face-to-Face ( Wednesday, Thursday)
Job Overview
We are looking for an experienced Lead Data Engineer to design, build, and optimize scalable, high-performance data platforms using AWS cloud services and Python. This role involves architecting end-to-end data pipelines, ensuring data quality, enabling analytics and AI workloads, and providing technical leadership to data engineering teams.
Key Responsibilities
- Architect and implement fault-tolerant data pipelines using AWS Glue, Lambda, EMR, Step Functions, Redshift
- Build and optimize data lakes and data warehouses on S3, Redshift, Athena
- Develop Python-based ETL/ELT frameworks and reusable transformation modules
- Integrate multiple data sources (RDBMS, APIs, Kafka/Kinesis, Saa S systems) into unified models
- Lead data modeling, schema design, partitioning strategies for performance and cost optimization
- Drive data quality, observability, and lineage using AWS Data Catalog, Glue Data Quality, or third-party tools
- Define and enforce data governance, security, and compliance best practices
- Collaborate with Data Science, Analytics, Product, and Dev Ops teams
- Implement CI/CD pipelines for data workflows using AWS Code Pipeline, Git Hub Actions, or Cloud Build
- Provide technical leadership, code reviews, and mentoring to junior engineers
- Monitor infrastructure performance, troubleshoot issues, and lead capacity planning
Mandatory Skills
- 6–12 years of experience in data engineering / data platform development
- Expert in Python (pandas, Py Spark, boto3, SQLAlchemy)
- Strong experience with AWS Data Services: Glue, Lambda, EMR, Step Functions, Redshift, Athena, S3, Kinesis, Dynamo DB
- Proficiency in SQL, data modeling, and performance tuning
- Hands-on experience with ETL/ELT pipelines, data lakes, data warehouses, and streaming solutions
- Version control (Git) and CI/CD for data pipelines
- Experience with containerization (Docker/Kubernetes) is a plus
- Strong analytical, debugging, and communication skills
Preferred / Nice-to-Have Skills
- Experience with Apache Spark or Py Spark on EMR or Glue
- Familiarity with Airflow, dbt, or Dagster
- Real-time data streaming (Kafka, Kinesis)
- Exposure to Lake Formation, Glue Studio, Data Brew
- Integration with ML/Analytics platforms (Sage Maker, Quick Sight)
- AWS Certifications: Data Analytics – Specialty or Solutions Architect
Soft Skills
- Ownership mindset with focus on reliability and automation
- Ability to mentor and guide junior engineers
- Effective communication with technical and non-technical stakeholders
- Comfortable working in agile, cross-functional teams
Apply for this Position
Ready to join ? Click the button below to submit your application.
Submit Application