Lead Data Pipeline Engineer (GCP)

Location: Chennai

Experience: 8-10 years

Job Description:

• Architect and implement enterprise-grade data migration solutions using Java and Python, enabling seamless data transfers from on-premises to GCP (Cloud Storage, BigQuery, Pub/Sub) using Apache Airflow and Google Cloud Composer.
• Build secure, scalable, and optimized data architectures leveraging GCP services such as Cloud Storage, Pub/Sub, Dataproc, Dataflow, and BigQuery.
• Design and implement automated frameworks for data delivery, monitoring, and troubleshooting.
• Develop data observability frameworks to ensure quality, lineage, and reliability across pipelines.
• Proactively monitor system performance, identify bottlenecks, and optimize pipelines for efficiency, scalability, and cost.
• Troubleshoot and resolve complex technical issues in distributed systems and cloud environments.
• Drive best practices in documentat...