Azure Data Engineer

<p><b>Job Description:</b></p> <div> </div> <div> <ul> <li> <p>Experience in leading development of Data and Analytics products, from Requirement Gathering State to Driving User Adoption </p> </li> <li> <p>Develop and optimize ETL processes using Databricks and related tools like Apache Spark </p> </li> <li> <p>Design efficient data processing systems and pipelines using Databricks, APIs, and other cloud services </p> </li> <li> <p>Candidate with strong data transformation experience on Unity Catalog, Delta Tables, DLT </p> </li> <li> <p>Strong proficiency in writing and optimizing SQL queries and working with databases </p> </li> <li> <p>Ability to acquire specialized domain knowledge required to be more effective in all work activities BI & Data-warehousing concepts are a must. </p> </li> <li> <p>Design, develop, and maintain scalable ETL/ELT pipelines using <b>PySpark</b> on <b>Databricks</b>. </p> </li> <li> <p>Ingest and transform data from multiple structured and unstructured sources including cloud storage (Azure Data Lake, AWS S3, etc.). </p> </li> <li> <p>Optimize Spark jobs for performance and cost-efficiency on the Databricks platform. </p> </li> <li> <p>Collaborate with data scientists, analysts, and stakeholders to understand data requirements and deliver high-quality solutions. </p> </li> <li> <p>Implement best practices in data engineering, including modular coding, unit testing, and version control (e.g., Git). </p> </li> <li> <p>Automate data workflows and schedule jobs using <b>Databricks Workflows</b> or external orchestration tools (e.g., Airflow, Azure Data Factory). </p> </li> <li> <p>Ensure data quality, integrity, and governance in all data pipelines. </p> </li> <li> <p>Participate in code reviews, performance tuning, and system monitoring. </p> </li> <li> <p>Document solutions, processes, and configurations. </p> </li> <li> <p>Responsible for data management activities related to the migration of on-prem sources to cloud system using Microsoft Azure PaaS Services and Azure Cloud Technologies, including the creation and use of ingestion pipelines, data lakes, cloud-based data marts & data warehouses, cloud-based semantic data services layer. </p> </li> </ul> </div>