Azure Data Platform Engineer

Key Responsibilities Azure & Databricks Platform Engineering

• Engineer, operate, and evolve enterprise‑grade Azure Databricks platforms across multiple subscriptions and environments (dev, test, prod).
• Design and implement Databricks workspace configurations, cluster policies, pools, and job orchestration with a strong focus on scalability, reliability, and cost efficiency.
• Engineer and maintain Azure platform components supporting Databricks, including networking, identity, storage, and security services.
• Monitor platform health, performance, and usage; proactively design improvements and resolve operational issues.

Databricks‑Centric Data & Analytics Enablement

• Engineer and optimize Databricks integrations with Azure services, including:
  • Azure Data Lake Storage (ADLS Gen2)
  • Delta Lake and Unity Catalog
  • Azure SQL, Azure Synapse, Event Hubs, and other data sources
• Support batch and streaming data pipelines and analytics workloads running on Databricks.
• Partner with Data Engineering, Analytics, and AI teams to ensure performant, reliable, and secure production workloads.

Security, Governance & Compliance Engineering

• Design and implement security and governance controls across Databricks and Azure, including:
  • Azure Active Directory integration, RBAC, and SCIM provisioning
  • Unity Catalog, data access policies, and cluster governance
• Engineer secure integrations using Azure Key Vault, Private Endpoints, and network security constructs.
• Ensure platform designs comply with organizational security, governance, and audit requirements.

Infrastructure as Code & Platform Automation

• Design, develop, and maintain Terraform‑based IaC for Databricks and supporting Azure infrastructure (workspaces, permissions, networking, storage).
• Build and evolve CI/CD pipelines using Azure DevOps YAML pipelines for platform and infrastructure deployments.
• Promote engineering best practices for IaC, including modularization, versioning, automated testing, and environment consistency.

Collaboration & Platform Leadership

• Act as a subject matter expert for Databricks on Azure, guiding teams on platform usage, architecture, and operational best practices.
• Work cross‑functionally with Cloud, Architecture, CICD, Data, AI, and Application teams.
• Create and maintain platform documentation, reference architectures, and operational runbooks.

Required Qualifications

• 5+ years of hands‑on experience engineering solutions on Microsoft Azure, with a strong focus on Azure Databricks platforms.
• Deep expertise in:
  • Azure Databricks (clusters, jobs, pools, Delta Lake, Unity Catalog, workspace engineering)
  • Azure core services (Storage, Networking, Identity & Access Management)
  • Cost optimization, monitoring, and observability for data platforms
• Strong experience with Terraform, including multi‑subscription and multi‑environment deployments.
• Proficiency with CI/CD pipelines, preferably Azure DevOps.
• Solid understanding of cloud security, governance, and platform engineering practices.

Preferred Certifications Azure & Databricks Certifications

• Microsoft Certified: Azure Administrator Associate
• Microsoft Certified: Azure Solutions Architect Expert
• Databricks Certified Data Platform Engineer or Administrator

HashiCorp Terraform Certifications

• HashiCorp Terraform Associate (003 or later)
• HashiCorp Terraform Engineer

Soft Skills

• Strong communication and documentation skills.
• Ability to explain complex Azure and Databricks platform designs to diverse technical audiences.
• Engineering mindset with a strong focus on automation, reliability, and continuous improvement.
• Collaborative, proactive, and adaptable in a fast‑evolving cloud and data landscape.