Principal Engineer

What Youll Be Doing (position summary):

The Principal Engineer Databricks & Python is responsible for designing, building, and optimizing data pipelines within a large-scale Capital markets Databricks Lakehouse environment. You will design and implement the bronze, silver, gold medallion pipelines that power analytics, reporting, RAG, semantic layers, and downstream AI agents. The role transforms complex capital markets data into high-quality, governed, and query-ready datasets. This position ensures robust data engineering practices to support business and technology objectives.

At CIBC India we enable the work environment most optimal for you to thrive in your role. Details on your work arrangement (including on-site and remote work) will be discussed at the time of your interview.

How Youll Succeed (responsibilities):

• Build robust ETL/ELT pipelines using Databricks, Delta Lake, PySpark, Spark SQL, and Python to enable high-quality data processing. Ingest complex datasets into Bronze tables, apply data validation and transformations into Silver, and model Gold-layer datasets for analytics and business consumption.
• Implement schema evolution, data quality enforcement, expectations, and automated lineage tracking.
• Optimize Spark jobs for performance and cost efficiency, ensuring Service Level Agreements (SLAs) are consistently met.

Collaboration & Integration:

• Work closely with Quant, Risk, and Trading Technology teams to integrate pricing, risk, RFQ, and PnL data.
• Collaborate with AI teams to ensure data is prepared for retrieval-augmented generation (RAG), semantic models, and agent workflows.

Governance & Orchestration:

• Apply Unity Catalog for governance, lineage, classification, and entitlements.
• Use Delta Live Tables, Workflows, or Databricks Jobs for production orchestration.
• Maintain and enhance metadata, documentation, and data contracts.

DevOps & Automation:

• Build and maintain CI/CD pipelines for Databricks code using GitHub Actions or Azure DevOps.

Experience: 12+ years of experience

Who You Are (skills/qualifications):

Must Have Skills:

• Strong experience with Databricks Lakehouse, Delta Lake, and PySpark.
• Deep expertise in Python for data transformations, UDFs, notebooks, and workflow automation.
• Advanced SQL skills with experience modeling Gold-layer datasets for analytics.
• Proficiency with data quality frameworks (Expectations, Delta Live Tables tests, custom validators).
• Solid understanding of medallion architecture and best practices for data reliability.
• Experience with ETL/ELT patterns and structured/unstructured data ingestion.
• Experience with Azure (Data Lake, Blob, Key Vault, Data Factory) or AWS equivalents.

Good to Have:

• Financial markets experience (risk data, PnL, trades, positions, yield curves, vol surfaces).
• Experience with real-time ingestion (Kafka, Solace, EventHub).
• Experience building semantic models (Cube.js, dbt, AtScale, Microsoft Fabric).
• Knowledge of feature stores, MLflow, or model training on Databricks.

What CIBC India Offers:

At CIBC India, your goals are a priority. We start with your strengths and ambitions and strive to create opportunities to tap into your potential. We aspire to give you a career that goes well beyond your compensation.

• We work to recognize you in meaningful, personalized ways including competitive fixed pay, performance-based incentive programs, a family-first benefits program, employer contributions for your retirement, attractive time off and leave offerings, wellbeing support, and MomentMakers, our social, points-based recognition program
• Our spaces and technological toolkit will make it simple to bring together great minds to create innovative solutions that make a difference for our clients worldwide
• We cultivate a culture where you can express your ambition through initiatives like Purpose Day, a paid day off dedicated for you to use to invest in your growth and development

Privacy:

By proceeding with the creation of a profile and submitting an application in response to this job posting, you acknowledge and agree that your personal data will be collected, used, retained and disclosed by CIBC India and its affiliates, as described in theCandidate Privacy Statement.