Data Engineers often have to juggle both data processing and DevOps pipelines. I’ve worked with many skilled data professionals, but they tend to shy away from the DevOps side of things. Often, I get asked to take over this aspect due to my background.
For Databricks, resource deployment has always been a combination of Terraform and Databricks Asset Bundles. Terraform likely needs no introduction, as it's prevalent in infrastructure deployment nowadays. Other options like Pulumi or even Ansible is a distant second. While I’m no expert, I have a working understanding of how to implement changes with it.
Databricks Asset Bundles (DABs), however, are not as widely known. Before DABs, deploying jobs and workflows in Databricks was converging using DBX. Eventually, Databricks added the bundle command to the Databricks CLI, and it’s now being pushed more and more.
When I first heard about DABs from our Solutions Architect, it seemed like we could use it for all resource deployments. I was disappointed to find out later that it only deployed Workflows and DLT Pipelines.
What's Missing?
All the platform-related components, of course. Our project involved setting up Databricks on AWS from scratch, and we aimed to minimize manual configurations (clickops) while adhering to strict security protocols. Both Account Admin and workspace admin privileges were tightly controlled, as they should be.
We needed to deploy everything from the metastore, workspaces, groups, and service principals to catalogs, schemas, tables, views, and grants. However, aside from Workflows and DLT Pipelines, DABs couldn't create any of these components—not even tables and views. While we could write code to create these using the Databricks API, it required running them through notebooks, leading to less elegant, more cumbersome code compared to the declarative style of Terraform.
What we did eventually
Terraform continues to be our primary tool for provisioning most resources, including Catalogs, Schemas, External Locations, and Volumes. It also manages Shared Databricks Clusters and SQL Warehouses effectively. To maintain simplicity and security, we limit grants to the Schema level, avoiding individual object grants.
Alas, Terraform is not very common in our current crew of data engineers. This equated to a higher platform engineer to data engineer ratio for us.
We still rely on Databricks Asset Bundles for deploying Databricks workflows, as they offer a straightforward method to deploy the same code across different environments.
For creating tables and views, we utilize DBT (Data Build Tool), which provides a declarative approach to object creation. As for User-Defined Functions (UDFs), they might still require the less elegant notebook approach, but fortunately, we haven't needed to create any yet.
I am hopeful that Databricks Asset Bundles will eventually support a broader range of resources. The tool is rapidly evolving, and I remain optimistic about its future capabilities.