Data Engineer/Developer<\/span><\/span><\/b>
<\/p>Role Summary<\/span><\/span><\/b>
Design, build, and optimize Azure data pipelines and lakehouse solutions.\nDeliver secure, reliable datasets with strong governance, automation, and\ndocumentation. Collaborate across teams and contribute to standards in an Agile\nsetting.<\/span><\/span><\/p>Must\-Have (Day 1)<\/span><\/span><\/b>
<\/p>- Experience:\n 4\u20136 years in data engineering<\/span>
<\/li>- Core\n Platform: Databricks with Python, Spark, Pandas (notebooks and modular\n code)<\/span>
<\/li>- Orchestration:\n Azure Data Factory (pipelines, integration runtimes); ingest from diverse\n sources<\/span>
<\/li>- Lakehouse:\n Delta Lake fundamentals; Medallion architecture (bronze/silver/gold) in\n production<\/span>
<\/li>- Storage/SQL/Performance:\n Azure Data Lake Storage (ADLS); strong SQL; performance\-aware design<\/span>
<\/li>- Data\n Patterns: ETL/ELT; data modeling (e.g., dimensional/star schema)<\/span>
<\/li>- DevOps\n & Security: CI/CD for data projects (Azure DevOps or GitHub\n Enterprise); familiarity with Azure Entra ID for SSO/RBAC; secure\n workspace/data access<\/span>
<\/li>- Quality\n & Observability: Data validation/testing, code reviews, and basic\n monitoring/alerting for jobs/pipelines<\/span>
<\/li>- Ways of\n Working: Agile/Scrum (Jira/Confluence); clear pipeline and data contract\n documentation<\/span>
<\/li>- Collaboration:\n Effective stakeholder engagement; support/mentor junior team members;\n clear communication<\/span>
<\/li>- Generative\n AI (Day 1):<\/span>
<\/li>- Prompt\n design for data tasks (ingestion, transformations, documentation) with\n clear objectives and constraints<\/span>
<\/li>- Use of\n Copilot/ChatGPT to scaffold notebooks/jobs, generate tests, and optimize\n SQL/Spark\u2014validates outputs before merging<\/span>
<\/li><\/ul><\/ul>Nice\-to\-Have (Train within 60\u201390 days)<\/span><\/span><\/b>
<\/p>- Unity\n Catalog migration (Hive to Unity) and permissions/governance<\/span>
<\/li>- Databricks\n DevOps (cluster configuration, secret management, workspace automation)<\/span>
<\/li>- Azure\n Functions (C# or Python) for orchestration/integration<\/span>
<\/li>- Synapse\n dedicated SQL pools or dbt; Delta Live Tables<\/span>
<\/li>- Financial\n services domain exposure<\/span>
<\/li><\/ul>
<\/span><\/span>
<\/div>Shared Expectations<\/span><\/span><\/b>
<\/p>- Work\n independently with minimal supervision while contributing to team outcomes<\/span>
<\/li>- Commitment\n to secure practices and production\-grade reliability<\/span>
<\/li>- Continuous\n improvement mindset and willingness to learn new tools/technologies<\/span>
<\/li>- Willingness\n to work within regulated environment controls and policies<\/span>
<\/li>- Use\n Generative AI responsibly to improve velocity and quality (simple,\n structured prompts; guardrails; validate AI\-assisted outputs before\n adoption)<\/span>
<\/li><\/ul>
<\/span><\/span>
<\/div><\/span>
