Senior/Lead Data Quality Engineer

Our team is looking to hire a talented Senior/Lead Data Quality Engineer . In this role, you will be instrumental in safeguarding the integrity, reliability, and quality of our data engineering platforms and pipelines. Responsibilities Build and architect a strong automated testing strategy and framework for data engineering pipelines, emphasizing integration with Databricks, PySpark, Scala, Spark SQL, and other pertinent technologies Deploy automated tests that guarantee data quality, integrity, and reliability across every data pipeline and platform Embed automated testing into Azure DevOps CI/CD pipelines to enable smooth testing, build, and deployment workflows Carry out performance testing to evaluate the scalability and efficiency of data processing systems Collaborate closely with data engineers and additional stakeholders to gather requirements and align testing strategies with business goals Set up monitoring for automated tests and deliver routine reports covering test coverage, defects, and data quality concerns Requirements At least 3 years of experience working in Data Quality Engineering or an adjacent field Strong grasp of data engineering principles and automated testing frameworks Hands-on expertise with Databricks, PySpark, Scala, Spark SQL, Terraform, and Azure Event Hubs Background in connecting testing frameworks to Azure DevOps and handling source control through Git Understanding of pipeline configuration and deployment using Terraform Working knowledge of Jira Excellent analytical skills with a demonstrated history of diagnosing and resolving complex testing and data issues English proficiency at a B2+ level Nice to have Exposure to Power BI, Azure Data Lake, Spark Streaming, and additional data visualization and processing tools Previous work in a data-heavy industry or within large-scale data environments Skill in delivering solutions via Infrastructure as Code (IaC)