Three-tier architecture was the norm when I started my IT career. It consisted of the presentation layer (the web servers), the application layer (the app servers), and the data layer (the databases).
We could spin up as many environments as needed for the presentation and application layers — most engineers typically had their own local setups for these. But the database layer? That was usually shared.
This setup works fine when the schema is stable and the only operations are data manipulation (DML: inserts, updates, deletes). But in my experience, that’s rarely the case.
Data-heavy applications are always evolving — new tables here, an extra column there, another index for that slow join. Sharing a database across multiple developers and environments quickly becomes a source of friction.
Modern Workarounds for Database Isolation
Today, there are far more elegant ways to manage this. I've worked on systems where each developer spun up their own isolated data layer using Docker containers for PostgreSQL or MySQL. For Oracle, we found it more practical to use PDBs (Pluggable Databases) instead. In AWS-heavy environments, some teams spun up databases from RDS snapshots using CloudFormation or Terraform.
There are also broader solutions involving full-stack virtualization, infrastructure-as-code, and automation tools. These approaches help bring the data layer up to the same level of flexibility as the app and presentation layers — and reduce cross-team collisions along the way.
Databricks and the Environment Problem
But what happens when your “database layer” isn’t a traditional RDBMS at all?
What if it’s Databricks, where data lives in cloud-managed storage, and the interface revolves around notebooks and distributed compute jobs? In this world, environment isolation takes a different form — and requires different strategies.
What Is an Environment in Databricks?
The first important question is: how do you define an environment in Databricks? There are several valid approaches.
A common pattern is to define an environment as a Databricks workspace — e.g., kurdapyo-prd, kurdapyo-uat, kurdapyo-sit, kurdapyo-dev. This gives you clean isolation. Pre–Unity Catalog, you could even reuse catalog names across workspaces. But with Unity Catalog, catalog names must be globally unique, so it's now common to prefix them — e.g., gold_uat, gold_prd.
The downside? Spinning up a new workspace environment isn’t cheap. On AWS, it often involves provisioning a new account and VPC, setting up DNS, configuring SCIM and user permissions — the kind of setup best left to platform teams or DevOps experts.
Splitting a Workspace into Logical Environments
The next logical approach is to carve out logical environments within a single Databricks workspace. We explored two main options:
I personally prefer the schema-level split. I even advocated for building automation to create schemas on demand and using zero-copy cloning to generate like-for-like environments — perfect for safe, isolated testing.
However, that approach wasn’t prioritized at the time. Instead, we went with a catalog-level split, managed through Terraform to keep things clean and avoid ad-hoc changes by engineers.
Naming Your Logical Environments
Once you've defined your environment boundaries, the next step is naming. While I prefer a “treat them as cattle, not pets” mindset, stakeholders usually prefer meaningful names.
A practical trick I’ve learned: always include a numeric suffix, like dev-001 or uat-02. It sounds simple, but it's incredibly effective — because no matter how many environments you start with, you'll always need more later. This naming convention makes it easy to scale your environments without resorting to naming gymnastics.
Aspirations: Automation and Production Parity
Once your environments and naming conventions are in place, the long-term goal is automation:
Automate environment creation and teardown
Make it easy to spin up production-like environments
Enable regression testing in consistent, isolated setups
These efforts can be time-consuming upfront but pay off significantly in delivery speed and engineering confidence. Of course, it’s essential to balance these aspirations with business priorities. Delivering value to users comes first — but carving out time for automation is what sets teams up for long-term success.
One tool I’m particularly curious about is SQLMesh. It promises branch-based environments, automated model testing, and declarative change tracking — all of which seem like a natural fit for solving the isolation problem at the data transformation layer. I haven’t tried it in anger yet, but it’s definitely on my list to explore as we continue refining our approach to Databricks environments.
Final Thoughts
The challenges of environment isolation aren’t new — they just evolve with the tools we use. Whether it's a traditional database in a three-tier architecture or a modern platform like Databricks, the underlying principles remain the same: avoid shared mutable state, automate what you can, and build with repeatability in mind.