Consulting Lead - Databricks

We’re seeking a Consulting Lead-Databricks with strong experience in Databricks, Python, and PySpark, along with hands-on knowledge of Unity Catalog (UC) and Lakehouse architecture. In this role, you’ll design and optimize large-scale data pipelines, implement robust governance and security through UC, and build scalable solutions in a modern cloud data environment. The ideal candidate must have 5+ years in data engineering, expertise with any one of the cloud platforms (Azure/AWS/GCP), and proven ability to deliver high-performing ETL/ELT pipelines that support business-critical analytics.

Qualification

• Hands-on expertise in Databricks, Python, PySpark, and Delta Lake/Lakehouse architecture.
• Experience implementing and managing Unity Catalog in a production environment.
• Strong SQL skills and experience with cloud platforms (Azure, AWS, or GCP).
• Proven ability to design scalable data pipelines and optimize performance.
• Strong communication skills and ability to work closely with stakeholders.

Key Responsibilities:

• Design, develop, and maintain large-scale ETL pipelines using Databricks and PySpark.
• Optimize data workflows for performance, scalability, and reliability.
• Collaborate with data scientists, analysts, and business stakeholders to deliver high-quality data solutions.
• Implement best practices for data modeling, governance, and security in a cloud data environment.
• Migrate existing AWS Glu/Redshift based data pipelines to Databricks