Lead Data QA Engineer

Role Overview

We are looking for a QA Lead Engineer who can provide both technical leadership and hands-on programming expertise to drive the testing strategy and quality assurance for our data and analytics initiatives. This role is ideal for someone who thrives in a dual capacity — designing processes and actively developing, implementing, and optimizing testing frameworks, tools, and automation scripts. You will work closely with Data Engineers and business stakeholders to ensure the highest quality of data pipelines, ETL processes, and analytics deliverables.

Key Responsibilities

• Design Solutions: Act as the QA Lead Engineer designing and optimizing testing processes for data and analytics functions.
• Hands-On Development: Code, and maintain automated test scripts, frameworks, and utilities in Python, PySpark, and Spark SQL to validate data pipelines and transformations.
• Support Data Engineering: Actively assist the data engineering team during peak periods by contributing to data transformation tasks and ensuring smooth execution of data pipelines.
• Testing Strategy Ownership: Take ownership of the testing strategy, driving the design, execution, and reporting of test cases using various testing frameworks and tools for new development requirements or bug fixes.
• Framework Implementation: Lead and actively develop/customize testing frameworks and methodologies to meet business objectives and testing requirements.
• Technical Leadership: Provide end-to-end technical leadership for implementing testing processes, while maintaining and enhancing existing test frameworks and applications.
• Best Practices: Implement and promote best practices in test automation, manual testing, performance testing, and integration testing to ensure consistency and quality across all testing activities.
• Test Optimization: Write efficient, maintainable, and scalable test scripts that maximize testing coverage, accuracy, and issue detection.
• Documentation: Create and maintain detailed documentation for testing processes, test cases, and results for future reference and training.
• Stakeholder Collaboration: Work closely with business users and key stakeholders to understand testing needs and recommend solutions aligned with business goals.
• Data Validation: Partner with Data Engineers to validate data pipelines, ETL processes, and data integrity during testing phases.
• Reporting: Build simple Power BI reports and dashboards to track and visualize test results and quality metrics.

Technical Skills & Qualifications

• Hands-on expertise in Python for test automation, scripting, and data validation.
• Strong skills in PySpark and Spark SQL for validating and testing large-scale data transformations.
• Experience with Azure Data Factory to orchestrate and validate ETL workflows.
• Proficiency in Azure Databricks for data processing and distributed testing of data pipelines.
• Ability to create simple Power BI reports for test result visualization or reporting.
• Proven experience implementing, customizing, and maintaining testing frameworks in data and analytics environments.
• Deep understanding of test automation, performance testing, integration testing, and QA best practices for data-centric systems.
• Excellent communication, and stakeholder