Databricks Developer

We are seeking a skilled Databricks Developer<\/b> to design, develop, and optimize scalable data pipelines and analytics solutions<\/b> using Apache Spark, Python (PySpark), and SQL<\/b> within modern cloud environments. The ideal candidate will have hands\-on experience working with Databricks, Delta Lake, and cloud data platforms<\/b>, and will be responsible for building high\-performance data processing workflows that support data\-driven decision\-making.
<\/p>

Key Responsibilities
<\/h3>

Pipeline Development<\/b>
<\/p>

• Design, develop, and maintain ETL/ELT data pipelines<\/b> using PySpark and SQL<\/b> in Databricks notebooks.
  <\/p><\/li>

• Process large\-scale datasets and ensure reliable and efficient data transformations.
  <\/p><\/li><\/ul>

  Architecture Design<\/b>
  <\/p>

  • Implement data lake and data warehouse architectures<\/b> using Databricks Delta Lake<\/b> and Delta Live Tables<\/b>.
    <\/p><\/li>

  • Build and manage Medallion Architecture (Bronze, Silver, Gold layers)<\/b> for structured data processing.
    <\/p><\/li><\/ul>

    Performance Optimization<\/b>
    <\/p>

    • Optimize Spark jobs and queries<\/b> for performance, scalability, and cost\-efficiency.
      <\/p><\/li>

    • Manage cluster configurations, partitioning strategies, and caching mechanisms<\/b>.
      <\/p><\/li><\/ul>

      Cloud Integration<\/b>
      <\/p>

      • Integrate Databricks solutions with cloud services<\/b> such as:
        <\/p>

        • Azure Data Factory (ADF)
          <\/p><\/li>

        • Azure Data Lake Storage (ADLS) Gen2
          <\/p><\/li>

        • AWS S3 or other cloud storage platforms
          <\/p><\/li><\/ul><\/li><\/ul>

          Data Governance & Quality<\/b>
          <\/p>

          • Implement data quality checks, validation frameworks, and monitoring processes<\/b>.
            <\/p><\/li>

          • Ensure data security, encryption, masking, and lineage tracking<\/b>.
            <\/p><\/li><\/ul>

            Workflow Automation<\/b>
            <\/p>

            • Build and manage automated workflows and scheduling<\/b> using Databricks Jobs, Airflow, or CI/CD pipelines.
              <\/p><\/li>

            • Integrate with DevOps tools such as Azure DevOps or Jenkins<\/b> for continuous integration and deployment.
              <\/p><\/li><\/ul>

              Collaboration<\/b>
              <\/p>

              • Work closely with data engineers, analysts, and business stakeholders<\/b> to translate business requirements into scalable data solutions.
                <\/p><\/li>

              • Participate in Agile development processes<\/b> including sprint planning and technical discussions.
                <\/p><\/li><\/ul>

                Required Skills & Qualifications
                <\/h3>

                Core Technologies<\/b>
                <\/p>

                • 3\u20135+ years of experience<\/b> with Databricks, Apache Spark, and Python (PySpark)<\/b>.
                  <\/p><\/li>

                • Strong experience building scalable ETL/ELT pipelines<\/b>.
                  <\/p><\/li><\/ul>

                  SQL Expertise<\/b>
                  <\/p>

                  • Advanced knowledge of Spark SQL or Databricks SQL<\/b> for data transformation and analysis.
                    <\/p><\/li><\/ul>

                    Cloud Platforms<\/b>
                    <\/p>

                    • Hands\-on experience with Azure Databricks, AWS, or Google Cloud Platform (GCP)<\/b>.
                      <\/p><\/li><\/ul>

                      Data Modeling<\/b>
                      <\/p>

                      • Experience with Delta Lake<\/b> and Medallion Architecture (Bronze/Silver/Gold layers)<\/b>.
                        <\/p><\/li>

                      • Strong understanding of data modeling and data warehouse concepts<\/b>.
                        <\/p><\/li><\/ul>

                        Version Control & CI/CD<\/b>
                        <\/p>

                        • Proficiency with Git<\/b> and CI/CD tools such as Azure DevOps or Jenkins<\/b>.
                          <\/p><\/li><\/ul>

                          Preferred Qualifications
                          <\/h3>

                          • Experience with Data Build Tool (dbt)<\/b>.
                            <\/p><\/li>

                          • Knowledge of workflow orchestration tools such as Apache Airflow<\/b>.
                            <\/p><\/li>

                          • Experience with Machine Learning workflows using MLflow<\/b>.
                            <\/p><\/li>

                          • Familiarity with data governance and enterprise data platforms<\/b>.
                            <\/p><\/li><\/ul>

                            Education
                            <\/h3>

                            • Bachelor\u2019s or Master\u2019s degree in Computer Science, Information Technology, Data Engineering, or a related field<\/b>.
                              <\/p><\/li><\/ul>

                              Key Competencies
                              <\/h3>

                              • Strong problem\-solving and analytical skills<\/b>
                                <\/p><\/li>

                              • Ability to handle large\-scale data processing environments<\/b>
                                <\/p><\/li>

                              • Good communication and collaboration skills<\/b> in Agile teams
                                <\/p><\/li><\/ul>

                                
                                <\/div><\/span>