Job Description
As a Data Engineer with expertise in PySpark, Databricks, and Microsoft Azure, you will be responsible for designing, developing, and maintaining robust and scalable data pipelines and processing systems. You will work closely with data scientists, analysts, and other stakeholders to ensure our data solutions are efficient, reliable, and scalable.
Responsibilities:
• Design, develop, and optimize ETL pipelines using PySpark and Databricks to process large-scale data on the Azure cloud platform.
• Implement data ingestion processes from various data sources into Azure Data Lake and Azure SQL Data Warehouse.
• Develop and maintain data models, data schemas, and data transformation logic tailored for Azure.
• Collaborate with data scientists and analysts to understand data requirements and deliver high-quality datasets.
• Ensure data quality and integrity through robust testing, validation, and monitoring procedures.
• Optimize and tune PySpark jobs for performance an...
Responsibilities:
• Design, develop, and optimize ETL pipelines using PySpark and Databricks to process large-scale data on the Azure cloud platform.
• Implement data ingestion processes from various data sources into Azure Data Lake and Azure SQL Data Warehouse.
• Develop and maintain data models, data schemas, and data transformation logic tailored for Azure.
• Collaborate with data scientists and analysts to understand data requirements and deliver high-quality datasets.
• Ensure data quality and integrity through robust testing, validation, and monitoring procedures.
• Optimize and tune PySpark jobs for performance an...
Apply for this Position
Ready to join Coffeee.io? Click the button below to submit your application.
Submit Application