Lead Data Engineer
Job Title: Lead Data Engineer
Experience: 10–14 Years
Location: Bengaluru
Notice Period: Immediate or 15days
Job Summary
We are seeking an experienced and visionary Lead Data Engineer to lead enterprise-scale data engineering initiatives and drive the design, implementation, and modernization of cloud-native data platforms. The ideal candidate will have extensive experience in defining enterprise data architecture, leading high-performing engineering teams, and delivering scalable data solutions using modern cloud technologies. This role requires strong technical leadership, architectural expertise, and the ability to collaborate with cross-functional teams and business stakeholders.
Key Responsibilities
- Define and implement enterprise data architecture aligned with business and technology strategies.
- Lead multiple data engineering teams across complex enterprise projects and cloud transformation initiatives.
- Design, build, and optimize scalable cloud-native data platforms and modern data ecosystems.
- Architect and implement Lakehouse solutions using industry best practices.
- Drive enterprise-wide data governance, data quality, metadata management, and security initiatives.
- Review solution architecture, technical designs, and code to ensure adherence to engineering standards and best practices.
- Collaborate with Solution Architects, Product Owners, Business Analysts, and stakeholders to translate business requirements into scalable technical solutions.
- Ensure platform reliability, scalability, performance, and operational excellence.
- Champion automation, CI/CD, Infrastructure as Code (IaC), and DataOps best practices.
- Mentor and coach data engineers, fostering technical excellence and continuous learning.
- Identify opportunities for process improvements and technology modernization.
- Provide technical leadership throughout the software development lifecycle, from design through deployment and support.
Required Qualifications
- Bachelor's or Master's degree in Computer Science, Information Technology, Engineering, or a related discipline.
- 10–14 years of experience in Data Engineering, Data Platform Development, or Cloud Data Architecture.
- Proven experience leading enterprise-scale data engineering programs and cloud migration initiatives.
- Strong expertise in designing scalable, secure, and high-performance data platforms.
- Demonstrated experience leading and mentoring technical teams.
- Strong understanding of data modeling, data warehousing, Lakehouse architecture, and distributed data processing.
- Excellent analytical, problem-solving, and decision-making skills.
- Outstanding communication, stakeholder management, and client-facing skills.
Preferred Qualifications
- Experience with enterprise data governance frameworks and metadata management solutions.
- Knowledge of real-time streaming and event-driven architectures.
- Experience with Infrastructure as Code (Terraform), containerization, and orchestration technologies.
- Azure certifications or cloud architecture certifications are highly desirable.
- Familiarity with Agile, Scrum, and modern software engineering practices.
Requirements
Required Technical Skills
- Strong programming expertise in Python and Scala
- Extensive experience with Apache Spark
- Hands-on experience with Databricks
- Strong knowledge of Snowflake
- Experience with Apache Kafka and event-driven data architectures
- Deep expertise in Microsoft Azure cloud services
- Hands-on experience with:
Azure Data Factory (ADF)
Azure Synapse Analytics
Azure Data Lake Storage
- Strong understanding of CI/CD pipelines and DevOps practices
- Experience implementing DataOps methodologies
- Knowledge of Terraform or other Infrastructure as Code (IaC) tools is highly preferred
- Strong programming expertise in Python and Scala