Azure Databricks Administrator

Job Description:

Databricks Subject Matter Expert (SME) plays a pivotal role in admin, security best practices, platform sustain support, new tools adoption, cost optimization, supporting new patterns/design solutions using the Databricks platform. Here’s a breakdown of typical responsibilities:

Core Technical Responsibilities

• Architect and optimize big data pipelines using Apache Spark, Delta Lake, and Databricks-native tools.
• Design scalable data ingestion and transformation workflows, including batch and streaming (e.g., Kafka, Spark Structured Streaming).
• Create integration guidelines to configure and integrate Databricks with other existing security tools relevant to data access control.
• Implement data security and governance using Unity Catalog, access controls, and data classification techniques.
• Support migration of legacy systems to Databricks on cloud platforms like Azure, AWS, or GCP.