AI Data Engineer

𝐏𝐨𝐬𝐢𝐭𝐢𝐨𝐧: AI Data Engineer

𝐄𝐦𝐩𝐥𝐨𝐲𝐦𝐞𝐧𝐭 𝐓𝐲𝐩𝐞: Full-Time with Cogent IBS

𝐖𝐨𝐫𝐤 𝐌𝐨𝐝𝐞: Remote

𝐍𝐞𝐞𝐝 𝐫𝐞𝐬𝐨𝐮𝐫𝐜𝐞𝐬 𝐟𝐫𝐨𝐦 𝐈𝐧𝐝𝐢𝐚 (primarily) 𝐚𝐧𝐝 𝐋𝐀𝐓𝐀𝐌 𝐜𝐨𝐮𝐧𝐭𝐫𝐢𝐞𝐬 (secondary)

𝐑𝐞𝐬𝐩𝐨𝐧𝐬𝐢𝐛𝐢𝐥𝐢𝐭𝐢𝐞𝐬:

• Design and implement the Agentic RAG architecture for large-scale data standardization.

• Build scalable ETL pipelines using PySpark and Databricks for processing 1.2B+ records.

• Design, develop, and optimize Databricks jobs, workflows, and clusters.

• Leverage Delta Lake for reliable, scalable data storage and processing.

• Integrate vector databases, LLM orchestration, and external catalog APIs (MTP, PCdb, VCdb).

• Implement confidence scoring, retry logic, and human-in-the-loop workflows.

• Optimize performance, cost, and scalability across distributed pipelines.

• Ensure auditability, lineage tracking, and data governance compliance.

𝐑𝐞𝐪𝐮𝐢𝐫𝐞𝐦𝐞𝐧𝐭𝐬/𝐒𝐤𝐢𝐥𝐥𝐬:

• Strong Python, PySpark, and SQL expertise.

• Hands-on Databricks experience (notebooks, workflows, jobs, Delta Lake).

• Experience with large-scale ETL and distributed data processing.

• Working knowledge of LLMs, RAG, embeddings, and vector databases.

• Cloud experience on AWS, Azure, or GCP.

• Experience integrating REST APIs and external data catalogs.

• Familiarity with data governance, monitoring, and logging frameworks.

Interested candidates, please send your resume to 𝒉𝒆𝒍𝒍𝒐@𝒄𝒐𝒈𝒆𝒏𝒕𝒊𝒃𝒔.𝒄𝒐𝒎 and mention “𝐀𝐈 𝐃𝐚𝐭𝐚 𝐄𝐧𝐠𝐢𝐧𝐞𝐞𝐫” in the subject line.