Verisk

Lead Data Engineer

Show

We are looking for an experienced Lead Data Engineer to lead the architecture, evolution, and delivery of our modern AWS-native data platform. This platform powers underwriting intelligence, GenAI-driven insights, and large-scale content processing, serving as the foundation for data products across the organization.

You will own the full data lifecycle—from ingestion and transformation to storage, retrieval, and AI consumption—working across Python-based processing pipelines, .NET services, Aurora PostgreSQL, OpenSearch, and LangGraph-powered agentic workflows.

This is a senior individual contributor role with significant technical leadership responsibilities. You will define engineering standards, drive architectural decisions, mentor engineers, and collaborate closely with product, platform, and AI teams to build reliable, scalable, and high-performing data systems.

  • Design, build, and evolve scalable, fault-tolerant data pipelines across ingestion, transformation, storage, and serving layers on AWS.
  • Own the delivery of cloud-native data solutions, including AWS Batch, Lambda, Python, .NET APIs (REST/GraphQL), Aurora PostgreSQL, and OpenSearch.
  • Define data architecture, schema design, indexing strategies, and data contracts to support analytics and GenAI-powered applications.
  • Establish and enforce data quality, testing, monitoring, and operational standards using CloudWatch, Splunk, Pentaho, and ThoughtSpot.
  • Partner with AI/ML and product teams to enable LangGraph-based agentic RAG solutions, ensuring reliable, high-quality, and retrieval-optimized data.
  • Develop and optimize data pipelines supporting LLM evaluation, prompt engineering, ground-truth datasets, and synthetic test data generation.
  • Review and validate AI-assisted engineering outputs, ensuring production readiness and technical excellence.
  • Drive technical design through architecture documents, RFCs, and engineering best practices, balancing scalability, maintainability, and business needs.
  • Lead deployment strategies, incident investigations, and continuous improvements to platform reliability and operational resilience.
  • Mentor engineers, establish data engineering standards, and provide technical leadership through design reviews and hands-on guidance.
  • Collaborate across engineering teams to manage data dependencies, align technical direction, and communicate delivery plans, risks, and architectural decisions.
  • Solid software and data engineering experience building and operating scalable, production-grade data platforms.
  • Strong Python skills for data processing and transformation, including structured and unstructured data.
  • Experience building data services and APIs using .NET (C#), REST, and GraphQL.
  • Hands-on expertise with AWS services, including S3, Batch, Lambda, Aurora PostgreSQL, EKS, SNS, and CloudWatch.
  • Strong SQL and PostgreSQL skills, including schema design, query optimization, and data modeling.
  • Experience designing distributed, fault-tolerant systems with robust testing, observability, and operational excellence.
  • Proficiency troubleshooting complex data pipelines using logs, metrics, and monitoring tools.
  • Experience with CI/CD, modern DevOps practices, and Agile software delivery.
  • Excellent communication skills, with the ability to document technical designs and influence architectural decisions.

Nice to have:

  • Experience building GenAI or Retrieval-Augmented Generation (RAG) applications using LangGraph, LangChain, or similar frameworks.
  • Familiarity with OpenSearch, vector search, or AI retrieval architectures.
  • Experience with API gateways (e.g., Kong) or Model Context Protocol (MCP).
  • Experience processing unstructured documents and content transformation pipelines.
  • Exposure to BI and analytics platforms such as ThoughtSpot or Pentaho.

We offer:

  • The opportunity to own and shape a cloud-native data platform powering AI, analytics, and intelligent underwriting solutions.
  • The chance to work with modern technologies, including AWS, GenAI, agentic workflows, and large-scale distributed data systems.
  • A high-impact technical leadership role with the autonomy to influence architecture, engineering standards, and platform evolution.
  • A collaborative, international engineering environment that values innovation, knowledge sharing, and continuous learning.
  • A hybrid work model with flexible working hours.
  • A benefits package, including private health insurance, medical care, and a Multisport card.

#LI-AA1

#LI-Hybrid