Senior Data Quality Engineer
We are seeking a skilled Senior Data Quality Engineer to join our team. The successful candidate will play a crucial role in ensuring the integrity, reliability, and quality of our data engineering pipelines and platforms. Responsibilities Design and develop a robust automated testing framework and strategy for data engineering pipelines, with a focus on integration with Databricks, PySpark, Scala, Spark SQL, and other relevant technologies Implement automated tests to ensure data integrity, quality, and reliability across all data pipelines and platforms Integrate automated testing within Azure DevOps CI/CD pipelines to ensure seamless testing, build, and deployment processes Conduct and execute performance tests to assess the efficiency and scalability of data processing systems Work closely with data engineers and other stakeholders to understand requirements and ensure testing strategies are aligned with business objectives Establish monitoring for automated tests, providing regular reports on test coverage, defects, and data quality issues Requirements Minimum of 3 years of experience in Data Quality Engineering or a related field Solid understanding of automated testing frameworks and data engineering principles Proficiency with Databricks, PySpark, Scala, Spark SQL, Terraform, and Azure Event Hubs. Solid understanding of automated testing frameworks and data engineering principles Experience in integrating testing frameworks with Azure DevOps and managing source control with Git Knowledge of pipeline configuration and deployments utilizing Terraform Familiarity with Jira Strong analytical abilities with a proven track record in troubleshooting and resolving complex data and testing challenges Fluency in English with a B2+ level Nice to have Experience with Power BI, Azure Data Lake, Spark Streaming, and other data processing and visualization tools Prior experience in a data-intensive industry or with large-scale data environments Proficient in implementing solutions through Infrastructure as Code (IaC)