DE&A - Core - Data Quality Management - Data Quality Management (Other)

We are seeking a hands on QA Lead to drive quality assurance for a scalable, enterprise-wide data platform for an insurance client. The role involves validating batch and real-time data pipelines, ensuring data accuracy across Raw, Silver, and Gold layers, and supporting report rationalization and self-service analytics (Power BI).

The QA Lead will define and implement end-to-end data testing strategies, covering ingestion (AWS Glue, Kinesis), transformation (DBT), and consumption layers, while ensuring data quality, integrity, and performance optimization.

Key Responsibilities

1. QA Strategy & Leadership

• Define and implement end-to-end QA strategy for the enterprise data platform
• Establish test frameworks, standards, and governance for data validation
• Lead QA planning, estimation, and execution across multiple data streams

2. Data Validation & Testing

• Validate data across Raw, Silver, and Gold layers ensuring accuracy, completeness, and consistency
• Perform source-to-target reconciliation for batch and real-time pipelines
• Design and execute:
  • Data quality checks
  • Transformation validation (DBT models)
  • Aggregation and KPI validation

3. Batch & Real-Time Pipeline Testing

• Test batch ingestion pipelines using AWS Glue
• Validate real-time streaming data pipelines using Amazon Kinesis
• Ensure data latency, sequencing, and event consistency in streaming pipelines

4. Reporting & Rationalization QA

• Validate datasets powering Power BI self-service reports
• Support report rationalization initiatives by ensuring consistency of KPIs and eliminating redundant data sources
• Perform report/data reconciliation testing across legacy vs new platform

5. Automation & Tools

• Develop and implement automated data testing frameworks
• Leverage SQL, Python, and testing tools (e.g., Great Expectations, DBT tests, custom frameworks)
• Enable continuous testing integration within CI/CD pipelines

6. Performance & Optimization Testing

• Validate performance of:
  • Data pipelines
  • Queries in Snowflake
• Identify bottlenecks and work with engineering teams to optimize pipelines and queries
• Ensure scalability for large data volumes and concurrent workloads

7. Data Quality & Governance

• Define and enforce data quality rules, thresholds, and monitoring
• Implement data anomaly detection and alerting mechanisms
• Ensure compliance with audit, reconciliation, and governance standards

Required Skills & Experience

Core Technical Skills

• Strong experience in data testing / ETL testing / data QA
• Hands-on expertise with:
  • Snowflake (data validation, SQL testing)
  • DBT (testing, model validation)
  • AWS Glue (batch pipeline validation)
  • Amazon Kinesis (real-time pipeline testing)
• Advanced proficiency in SQL for data validation and reconciliation
• Programming skills in Python (preferred)

Testing Expertise

• Experience in:
  • Data reconciliation (source vs target)
  • Data quality frameworks and validation techniques
  • Automated data testing tools
• Understanding of medallion architecture (Raw, Silver, Gold layers)

Analytics & Reporting

• Experience validating Power BI reports and datasets
• Strong understanding of business KPIs and reporting consistency

Domain Expertise (Preferred)

• Experience in Insurance domain (Policy, Claims, Billing data)
• Familiarity with regulatory reporting, audit, and reconciliation requirements

Experience

• 8–12 years in QA / Data Testing / ETL Testing
• 3+ years in QA leadership or lead role
• Experience working on enterprise-scale data platforms

QA Lead

QA Lead