AI Data Engineer

𝐏𝐨𝐬𝐢𝐭𝐢𝐨𝐧: AI Data Engineer

𝐄𝐦𝐩𝐥𝐨𝐲𝐦𝐞𝐧𝐭 𝐓𝐲𝐩𝐞: Full-Time with Cogent IBS

𝐖𝐨𝐫𝐤 𝐌𝐨𝐝𝐞: Remote

𝐍𝐞𝐞𝐝 𝐫𝐞𝐬𝐨𝐮𝐫𝐜𝐞𝐬 𝐟𝐫𝐨𝐦 𝐈𝐧𝐝𝐢𝐚 (primarily) 𝐚𝐧𝐝 𝐋𝐀𝐓𝐀𝐌 𝐜𝐨𝐮𝐧𝐭𝐫𝐢𝐞𝐬 (secondary)

𝐑𝐞𝐬𝐩𝐨𝐧𝐬𝐢𝐛𝐢𝐥𝐢𝐭𝐢𝐞𝐬:

• Design and implement the Agentic RAG architecture for large-scale data standardization.

• Build scalable ETL pipelines using PySpark and Databricks for processing 1.2B+ records.

• Design, develop, and optimize Databricks jobs, workflows, and clusters.

• Leverage Delta Lake for reliable, scalable data storage and processing.

• Integrate vector databases, LLM orchestration, and external catalog APIs (MTP, PCdb, VCdb).

• Implement confidence scoring, retry logic, and human-in-the-loop workflows.

• Optimize performance, cost, and scalability across distributed pipelines.

• Ensure auditability, lineage tracking, and data governance compliance.

𝐑𝐞𝐪𝐮𝐢𝐫𝐞𝐦𝐞𝐧𝐭𝐬/𝐒𝐤𝐢𝐥𝐥𝐬:

• Strong Python, PySpark, and SQL expertise.<...