Databricks Data Engineer

About the job

One of our clients is seeking a Databricks engineer to modernize enterprise data platforms and deliver governed lakehouse data products.

Responsibilities

• Design, develop, test and maintain scalable pipelines and data products for enterprise analytics and reporting.
• Build batch and streaming processing with Spark and Delta Lake, using Delta Live Tables, Auto Loader, Structured Streaming, Databricks Workflows and Airflow.
• Create curated data products using metadata-driven ingestion, PySpark, incremental processing, quality frameworks and third-normal-form or dimensional models, with Unity Catalog controls.
• Implement governance, security and compliance through fine-grained Unity Catalog permissions.
• Tune code, jobs, clusters and pipelines to improve reliability, throughput and cost efficiency.
• Automate delivery with CI/CD tools such as Azure DevOps, AWS CodePipeline, Jenkins, TFS and PowerShell.
• Explain architectural decisions, tradeoffs and business value to technical and business stakeholders, and maintain delivery plans, progress updates and client deliverables.
• Document reusable architecture, integration, modeling and engineering patterns.

Required Qualifications

• A bachelor’s degree in computer science, engineering or a related field.
• At least two years of data engineering experience with strong Databricks delivery on AWS, Azure or Google Cloud.
• At least two years working with Databricks, cloud storage and compute, and distributed processing.
• At least two years applying lakehouse architecture with Spark and Delta Lake.
• At least two years working with enterprise data lakes and warehouse models, including third normal form and dimensional modeling.
• At least two years with Delta Live Tables, Auto Loader, Structured Streaming and Databricks Workflows orchestration.
• At least two years developing PySpark, incremental processing, metadata-driven ingestion and data quality capabilities.
• At least two years implementing Unity Catalog and granular access controls.
• Demonstrated experience automating deployments through CI/CD.
• Experience optimizing pipeline performance, code and compute resources.
• Ability to travel when project delivery requires it, potentially averaging up to 50%.

Preferred Qualifications

• A master’s degree in a related discipline.
• Broader experience with AWS, Azure or Google Cloud data services and big-data ecosystems.
• Experience enabling or supporting AI and machine-learning workloads.
• Relevant Databricks certifications.