We are looking for a Data Developer with a specialization in Databricks, who will be responsible for developing, optimizing and managing data pipelines on Spark, Delta Lake and Unity Catalog. The role is focused on working with very large datasets (hundreds of millions of rows, TB volumes), tuning performance and efficiently using cluster resources.

• Develop and optimize data pipeline in Apache Spark on Databricks platform
• Work with Unity Catalog and manage data objects within the governance model
• Optimize pipeline performance over tables of 500+ million rows and >1 TB of data
• Tune and configure Databricks clusters for maximum efficiency and resource utilization
• Optimize Delta Lake tables (layout, partitioning, Z-ordering, vacuum, compaction)
• Collaborate with architects, data engineers and analysts on data platform development

• At least 1 year of experience with Unity Catalog
• At least 2 years of experience on min. 2 projects with:
  • tuning and optimizing Spark pipeline
  • working with tables of min. 500 million rows
  • working with data of at least 1 TB**
  • optimizing cluster settings for maximum resource utilization
• At least 2 years of experience with Delta Lake debugging on tables > 1 TB

Advantage

• Experience with Azure (ADF, Synapse, Data Lake Storage)
• Knowledge of CI/CD for data pipeline
• Experience with Spark job performance monitoring
• Experience from the financial sector

• Great colleagues and fully flexible work policy
• Career coaching and development
• Flexible working hours
• Technical training and workshops
• Technical equipment for work (Mac / Windows)
• Company parties
• Company psychologist for mental well-being
• Multisport card