Senior Databricks Engineer

Key Responsibilities:
Databricks Platform Engineering
•
Lakehouse Architecture: Design and implement scalable Lakehouse architectures using Databricks, Delta Lake, and Unity Catalog as the primary platform.
•
Delta Live Tables: Build and manage DLT pipelines for declarative, reliable, and maintainable data transformation workflows.
•
Unity Catalog: Govern data assets — tables, volumes, and models — through Unity Catalog, enforcing fine-grained access control and lineage tracking.
•
Performance Optimization: Tune Spark jobs and Delta tables using Z-ordering, liquid clustering, auto-optimize, and partition strategies to maximize query performance.
•
Databricks Workflows: Orchestrate multi-task pipelines using Databricks Workflows, including job scheduling, dependency management, and alerting.
Data Pipeline & Integration
•
ELT/ETL Pipelines: Design, develop, and maintain robust ELT/ETL pipelines ingesting data from APIs, databases, streaming sources, and flat files into the Lakehouse.
•
Azure Integration: Integrate Databricks with Azure Data Factory, ADLS Gen2, Azure Synapse, Event Hubs, and other Azure-native services.
•
Streaming: Build real-time data processing pipelines using Structured Streaming and Delta Live Tables for low-latency analytics use cases.
Software Engineering & DevOps
•
Code Quality: Write clean, efficient, and reusable PySpark, Python, and SQL code following SOLID principles and team coding standards.
•
CI/CD: Implement and manage CI/CD pipelines in Azure DevOps for automated testing, deployment, and release of Databricks notebooks and jobs.
•
Infrastructure as Code: Provision and manage Databricks workspaces, clusters, and supporting Azure infrastructure using Terraform or Bicep.
•
Databricks Asset Bundles: Package and deploy Databricks projects using DABs for reproducible, environment-consistent deployments.
Data Governance & Quality
•
Data Quality Frameworks: Implement automated data validation, expectation checks, and quality monitoring using Great Expectations or DLT expectations.
•
Data Lineage: Maintain end-to-end data lineage visibility through Unity Catalog and Microsoft Purview integration.
•
Documentation: Produce and maintain thorough technical documentation for pipelines, data models, runbooks, and architectural decisions.
Collaboration & Leadership
•
Mentorship: Coach and mentor junior and mid-level engineers, conducting code reviews and sharing best practices.
•
Stakeholder Engagement: Translate business requirements into robust technical solutions; communicate trade-offs clearly to non-technical stakeholders.
•
Standards & Governance: Contribute to and enforce team engineering standards, SDLC processes, and platform governance policies.
Required Qualifications
•
Databricks Certified Data Engineer Professional or Associate (mandatory).
•
Microsoft Certified: Azure Data Engineer Associate (mandatory).
•
10+ years of hands-on experience in Data Engineering roles.
•
Deep expertise in Databricks — Delta Lake, Unity Catalog, Delta Live Tables, Workflows, Clusters.
•
Advanced PySpark and Python skills for large-scale distributed data processing.
•
Strong SQL skills: query authoring, performance tuning, and data modeling.
•
Experience with Azure Data Factory, ADLS Gen2, Azure Synapse Analytics.
•
Hands-on experience with Azure DevOps CI/CD and Infrastructure as Code (Terraform or Bicep).
•
Strong understanding of Data Warehousing concepts, dimensional modeling, and Data Lakehouse patterns (Medallion architecture).
•
Experience with Structured Streaming or other real-time data processing frameworks. Preferred Qualifications
•
Experience with Microsoft Purview for data cataloguing and governance.
•
Familiarity with dbt (data build tool) for SQL-based transformation workflows.
•
Knowledge of MLflow and Databricks Feature Store for ML engineering integration.
•
Exposure to Databricks Asset Bundles (DABs) for project packaging and deployment.
•
Experience with Azure Function Apps, Logic Apps, or Event Hubs for event-driven architectures.
•
Background in data mesh or data product thinking. Key Skills at a Glance Core Platform Databricks, Delta Lake, Unity Catalog, DLT Languages PySpark, Python, SQL
Cloud & Integration
Azure Data Factory, ADLS Gen2, Synapse, Event Hubs
DevOps & IaC
Azure DevOps, Terraform, Bicep, DABs
Data Quality & Governance Great Expectations, DLT Expectations, Purview Architecture Lakehouse, Medallion Architecture, Data Mesh Why Join Us?
•
Lead the build-out of a modern, Databricks-first data platform at scale.
•
Work with a collaborative, forward-thinking Data & Analytics team.
•
Access to cutting-edge tooling across Azure, Databricks, and the broader modern data stack.
•
Competitive salary, benefits, and a strong commitment to your professional development and certifications.
•
Opportunity to shape team standards, architecture decisions, and engineering culture.
•
Contribute to high-impact, global data initiatives.

Additional Information

About Us, Our Culture & What We Can Offer You

Established over 200 years ago, Chubb Fire & Security are a global business driven by keeping people and the world safe. We provide essential systems, equipment and services, from digital CCTV surveillance and intruder alarms, to fire detection and suppression systems. Offering a full range of innovative products and services to a broad range of customers, from local independent business, to many of the FTSE 100 companies

We believe we offer a unique working culture, where you are as important to us as our customers, and we want you to feel that everyday. We are proud to offer extensive training to all of our new Advisors, fully supporting, and enabling you to thrive in your new role and beyond, with ongoing career development opportunities throughout your career journey with us.

#BR-CB