Elements of Style and Substance: Databricks Edition

Databricks development has accelerated significantly in our organisation. Engineers from a wide range of backgrounds are now contributing code—often with AI assistance—to solve an equally wide range of business problems.

As adoption has grown, we've started to see clear patterns in what leads to maintainable, supportable production code, and what tends to create operational headaches.

While Databricks notebooks are a long way from spacecraft flight software, I found myself repeatedly returning to NASA's Power of 10 Rules for Developing Safety Critical Code. The principles are surprisingly transferable: write code that is easy to understand, easy to test, and easy to troubleshoot under pressure.

These are the guidelines I use when reviewing pull requests. If I wouldn't want to debug it let alone make sense of it at 3AM during a production incident, it probably needs another pass.

Notebook etiquette

Notebooks remain controversial in some engineering circles, particularly among developers who prefer packaged Python applications. In an ideal world, every production workload would live in a well-structured Python package with proper testing and deployment pipelines. This is actually Rule #10 which starts with:

All code must be compiled, from the first day of development, with all compiler warnings enabled at the most pedantic setting available. All code must compile without warnings. All code must also be checked daily with at least one, but preferably more than one, strong static source code analyzer and should pass all analyses with zero warnings.

However, our reality is different. Many teams use notebooks because they allow rapid delivery and lower operational overhead. If notebooks are going to be part of the production platform, they should be treated as production code.

Use the notebook structure intentionally:

• Divide code into logical sections.

• Give cells meaningful titles.

• Use markdown to explain non-obvious business logic.

• Keep setup, transformation, validation, and output stages clearly separated.

Simple Flow

Restrict all code to very simple control flow constructs—do not use goto statements, setjmp or longjmp constructs, or direct or indirect recursion.

This was NASA's first rule. Though these goto statements don't really apply, the message is still simple and applicable. The procedural flow should be simple to understand.

Deeply nested logic is often a sign that responsibilities are mixed together. Instead of adding another level of indentation, consider:

• Extracting logic into a helper function.

• Performing validation up front and exiting early.

• Splitting complex transformations into separate stages.

A notebook should read like a pipeline, not a maze.

Function Construction

The Don't Repeat Yourself(DRY) principle is well known and I've seen many functions or procedures done which scare me. The common theme for these are they try to do too much. They don't follow the 4th NASA rule, which states:

No function should be longer than what can be printed on a single sheet of paper in a standard format with one line per statement and one line per declaration. Typically, this means no more than about 60 lines of code per function.

This is strict but brilliant. Large functions are particularly problematic in Spark workloads because they often combine business logic, data transformation, validation, logging, and persistence into a single block of code.

Prefer small functions with a single responsibility. A reader should be able to understand what a function does without scrolling through multiple screens of code.

Error Handling

Catch only exceptions that you understand and can meaningfully recover from.

Avoid patterns such as:

try:
    do_work()
except Exception as e:
    pass

Or worse:

try:
    do_work()
except Exception as e:
    logger.error(e)

These patterns hide failures and make troubleshooting significantly harder.

Unexpected exceptions should generally be allowed to fail the workflow so monitoring, retry policies, and alerting mechanisms can do their job.

If an exception is handled, there should be a clear business or operational reason for doing so.

This is a common anti-pattern in Databricks projects and worth calling out explicitly.

Idempotency

Idempotency is one of those concepts every data engineer agrees is important, yet it is often only applied to individual Spark operations rather than the workflow as a whole.

Many pipelines are built with the assumption that they will run once, succeed once, and never need to be revisited. Reality is messier. Clusters terminate unexpectedly, upstream systems resend data, workflows are retried automatically, and operators rerun jobs manually during incidents.

A well-designed workflow should be able to fail halfway through, be rerun, and still produce the same end result.

When building a workflow, ask yourself:

• Can this notebook be executed multiple times without creating duplicates?

• What happens if the workflow fails after writing some, but not all, of its outputs?

• What happens if an upstream job retries and sends the same data again?

• Can an operator safely rerun the workflow during an incident without needing manual cleanup?

Delta Lake gives us powerful tools such as MERGE, transactional writes, and change data capture patterns, but those features alone do not make a workflow idempotent. Idempotency is a design principle, not a technology feature.

Design for retries from the start. Assume failures will happen. The goal is not to prevent every failure, but to make recovery predictable and safe.

Final Thoughts

None of these guidelines are particularly revolutionary. Most are simply attempts to keep code understandable when the original author is unavailable, the pressure is high, and a production issue needs to be resolved quickly.

The best Databricks code is rarely the cleverest. It is the code that another engineer can read, understand, and safely modify months later with minimal context.

When reviewing a pull request, I often ask myself a simple question:

Would I be comfortable when I'm tired, sleepy, or maybe even a little hung over to troubleshoot this code?

If the answer is no, the code probably needs another pass.

Write notebooks that are easy to follow. Keep functions focused. Handle errors deliberately. Design for retries. Optimise for the engineer who has to support the workload, not the engineer writing it.

That engineer may very well be you.