Data Quality Analyst

The client is integrating a second-layer SCADA system for renewable assets into the Databricks environment. The Tester will ensure data integrity and reliability by designing and executing automated checks on incoming data streams and recording results in Azure DevOps.

• Develop and execute automated data and integration tests in Databricks (Python, PySpark, SQL)
• Conduct end-to-end pipeline testing from SCADA through AMQP into Databricks, including verification of transformation logic and compliance with defined business rules
• Capture, store, and prepare test results using Azure services (Data Lake, Blob Storage, SQL)
• Integrate automated tests into CI/CD pipelines with Azure DevOps, collaborating closely with DevOps and Cloud teams to ensure continuous validation and monitoring
• Work closely with Data Engineers, SCADA specialists, and Cloud architects, ensuring full documentation of all test cases, ...