Data Engineer

Position : Data Engineer

Location: India- Bangalore ( WFO)

Objective:

The objectives of Data Engineer typically revolve around designing, implementing, and managing robust data architectures and infrastructures. The Data Engineer will play a key role in driving the organization's data strategy, ensuring the efficient flow and storage of data, and supporting data-driven decision-making processes.

Duties and Responsibilities:

1. Data Model Design:

· Develop and maintain a scalable and efficient data models and schemas that support the storage, processing, and retrieval of data.

2. Data Integration:

· Implement robust ELT (Extract, Load, Transform) processes to integrate data from various sources into a unified and consistent format.

· Ensure seamless data flow between different systems, databases, and applications.

3. Data Quality:

· Implement data validation and cleansing processes to ensure the accuracy and reliability of data.

4. Emerging Technologies:

· Stay abreast of emerging technologies and trends in the field of data engineering.

5. Database Management:

· Manage and optimize database systems, including relational databases (e.g., MySQL, PostgreSQL) and NoSQL databases (e.g., MongoDB, Cassandra).

6. Performance Optimization:

· Identify and address performance bottlenecks in data processing and storage systems.

· Optimize queries, indexing strategies, and data partitioning for improved performance.

7. Scalability and Reliability:

· Ensure that data systems are scalable to handle growing volumes of data.

· Implement redundancy and fault-tolerance measures to enhance data system reliability.

Mandatory Skills Required:

· Strong hands-on expertise with AWS services such as S3, Glue, Lambda, Athena, Redshift, Step Functions, Streaming tools like Apache Kafka \ Amazon Kinesis.

· Proficiency in Python, PySpark, experience with ELT and orchestration tools

· Strong hands-on expertise on Apache Hudi \ Apache Iceberg

· Strong hands-on expertise in SQL

· Experience using GitHub for version control and collaboration

· Develop, optimize, and maintain ELT processes for data integration.

· Strong communication skills to interface with technical and business stakeholders.

Secondary Skills Required:

· Knowledge of cloud infrastructure automation (e.g., Terraform, CloudFormation).

· Experience with DevOps CICD pipelines and source code management.

· AWS Certification, experience in platform automation and monitoring frameworks.

Qualifications Required:

· Bachelor’s degree from four-year college or university, or equivalent combination of education and experience.

· 5+ years of experience in data engineering or a related role

· Proficient in data modeling, ELT development, and database management.

· Experience with big data technologies and distributed computing frameworks