Senior Software Engineer

Job Description - Senior Software Engineer (CREQ )

Senior Software Engineer - ( CREQ )

• Kafka, Confluent
• Real-Time, Streaming Data Ingestion
• Producers, Consumers, Topics
• Data Lake Integration, Lakehouse Integration

Databricks Engineer Positions: 2

Overview

Duration: 3 Months | Location: Dubai. Bank is building a scalable data ingestion and streaming platform that ingests change data capture (CDC) events from diverse source systems (databases and applications), processes them in real time and lands curated data into our analytics lake.

Responsibilities

• Design and develop streaming ingestion pipelines.
• Use Apache Spark (Structured Streaming) and Databricks Auto Loader to consume files from cloud storage or messages from Kafka/RabbitMQ/Confluent Cloud and ingest them into Delta Lake, ensuring schema evolution and exactly‑once semantics.
• Implement CDC and deduplication...