Data Engineer

Job Description

Key Responsibilities

• Pipeline Development: Design, build, and optimize scalable data ingestion pipelines to extract, transform, and load (ETL) data from source systems (Oracle, SQL MI, MongoDB, file feeds) into Databricks.
• Medallion Architecture Implementation: Lead the adoption and implementation of Medallion architecture (Bronze, Silver, Gold layers) for data processing and reporting in Databricks.
• Data Modeling on Silver Layer: Develop and maintain effective data models within the Silver layer to support standardized, cleansed, and business-ready datasets. Ensure that data relationships, normalization, and transformation logic are well-defined to facilitate downstream analytics and reporting.
• Testing & Validation: Develop automated tests for data pipelines and transformations to ensure reliability and correctness.
• Data Quality & Governance: Ensure data quality, inte...