The goal was straightforward: run the same full batch through two identical, production-like environments using identical inputs, compare every output, and ensure the batch completed without errors. This was parallel testing at scale. Parallel testing is the approach; regression testing is the purpose. And to put the scale into perspective, the batch consisted of around 3,000 individual jobs and produced roughly 1,000 extracts.

Back then, we solved the comparison problem with dynamically generated SQL, using row count comparisons and two-way MINUS queries to identify differences between the outputs.

We then built a reporting layer to turn those differences into something people could actually read and investigate. It worked, but it wasn't exactly elegant. You had to build the SQL, handle different tables and keys, interpret the results, and deal with all the edge cases that come with comparing datasets.

Fast forward a decade, and I still find myself solving similar problems in the Databricks world. This time, instead of wrangling SQL statements and building our own comparison reports, I reached for DataComPy.

DataComPy

DataComPy is a Python package for comparing datasets. Originally created for pandas, it also supports Spark DataFrames, making it a natural fit for Databricks.

Using it is quite simple. Set up two DataFrames, specify the join columns, then run the report.

Basic Example

Let's say we have the same account output generated in two environments and want to verify that they are identical.

# SparkSQLCompare
env01_df = spark.table("env01.account")
env02_df = spark.table("env02.account")

compare = SparkSQLCompare(
    spark,
    env01_df,
    env02_df,
    join_columns='acct_id',
    df1_name='env01',  # friendly name on report
    df2_name='env02',  # friendly name on report
)
compare.matches()

print(compare.report())

matches() gives you a quick boolean answer, while report() provides the detailed explanation of where the differences are.

The report

This is where DataComPy starts to feel familiar to me. In the old system, we had to build our own reporting layer to turn raw comparison results into something investigators could work with. We also had to compare row counts because MINUS doesn't account for duplicates.

DataComPy gives you much of that out of the box.

DataFrame Summary
-----------------
  DataFrame  Columns  Rows
0  env01        5     5
1  env02        4     5

Column Summary
--------------
Number of columns in common: 4
Number of columns in env01 but not in env02: 1
Number of columns in env02 but not in env01: 0

Row Summary
-----------
Matched on: id
Any duplicates on match values: No
Absolute Tolerance: 0
Relative Tolerance: 0
Number of rows in common: 4
Number of rows in env01 but not in env02: 1
Number of rows in env02 but not in env01: 1

Number of rows with some compared columns unequal: 4
Number of rows with all compared columns equal: 0

Column Comparison
-----------------
Number of columns compared with some values unequal: 3
Number of columns compared with all values equal: 1
Total number of values which compare unequal: 6

Inspecting the data

The report tells you that something is different. The next question is what exactly is different?

This is where the other attributes and methods of the SparkSQLCompare class come in handy. It exposes DataFrames that are useful for investigating the differences. In Databricks, these can be displayed using the notebook's display() function, which makes the investigation much easier.

• df1_unq_rows and df2_unq_rows let me inspect rows that exist in only one of the DataFrames.

• all_mismatch() gives me a DataFrame containing the join key and the columns values for each DataFrame.

The hard yards

DataComPy makes the comparison part surprisingly easy. But anyone who has built a serious regression-testing framework knows that comparison is only one piece of the puzzle.

The real work is in:

• Setting up the parallel environments, inputs and automated output collection

• Investigating differences

• Reducing noise

I remember the first few iterations of our regression testing. The reports were full of differences. It took weeks to work through them.

The challenge wasn't simply finding differences. It was determining which differences mattered.

A difference is not necessarily a defect.

Some were genuine defects. Others were noise:

• Batch timestamps

• Run-specific identifiers

• Control columns

• Null vs default values

• Schema or datatype differences

• Floating-point and rounding differences

• Non-deterministic processing

• Differences caused by jobs running in a different sequence

• Legitimate differences introduced by an intentional change

The goal wasn't to get the number of differences to zero. It was to get the number of unexpected differences to zero.

As the process became more automated and routine, we started to see the payoff. We saw fewer unexpected defects make it into production. Developers spent less time diagnosing problems after deployment, and significant migrations and platform changes could be made with much greater confidence.

Regression testing became less of a safety net we hoped we wouldn't need and more of an enabler for making changes safely.

Why I'm at it again

The temptation with regression testing is to see it as another testing framework to build, another pipeline to maintain, and another set of reports for developers to work through.

And honestly, getting it started can feel like a lot of work.

But once you have it running, something interesting happens.

You stop being afraid of making big changes.

Want to upgrade the Databricks runtime? Compare it.

Want to refactor a complicated pipeline? Compare it.

Want to migrate workloads or introduce a performance optimisation? Compare them.

Run the same inputs through the old and new implementations. Reconcile the outputs, investigate the differences and reduce the noise. What you end up with is much more valuable than a collection of unit tests.

You have evidence.

Evidence that the new implementation behaves like the old one. Evidence that a migration hasn't silently changed the numbers. Evidence that a performance improvement hasn't introduced a subtle business logic regression.

That's the real value of parallel testing.

It's not about proving that two systems are identical. They're often not supposed to be. It's about making the differences visible, explainable and intentional.

The investment is not in building another testing framework.

The investment is in buying your team the confidence to change things.

And for systems that matter, that can be worth a lot.

As adoption has grown, we've started to see clear patterns in what leads to maintainable, supportable production code, and what tends to create operational headaches.

While Databricks notebooks are a long way from spacecraft flight software, I found myself repeatedly returning to NASA's Power of 10 Rules for Developing Safety Critical Code. The principles are surprisingly transferable: write code that is easy to understand, easy to test, and easy to troubleshoot under pressure.

These are the guidelines I use when reviewing pull requests. If I wouldn't want to debug it let alone make sense of it at 3AM during a production incident, it probably needs another pass.

Notebook etiquette

Notebooks remain controversial in some engineering circles, particularly among developers who prefer packaged Python applications. In an ideal world, every production workload would live in a well-structured Python package with proper testing and deployment pipelines. This is actually Rule #10 which starts with:

All code must be compiled, from the first day of development, with all compiler warnings enabled at the most pedantic setting available. All code must compile without warnings. All code must also be checked daily with at least one, but preferably more than one, strong static source code analyzer and should pass all analyses with zero warnings.

However, our reality is different. Many teams use notebooks because they allow rapid delivery and lower operational overhead. If notebooks are going to be part of the production platform, they should be treated as production code.

Use the notebook structure intentionally:

• Divide code into logical sections.

• Give cells meaningful titles.

• Use markdown to explain non-obvious business logic.

• Keep setup, transformation, validation, and output stages clearly separated.

Simple Flow

Restrict all code to very simple control flow constructs—do not use goto statements, setjmp or longjmp constructs, or direct or indirect recursion.

This was NASA's first rule. Though these goto statements don't really apply, the message is still simple and applicable. The procedural flow should be simple to understand.

Deeply nested logic is often a sign that responsibilities are mixed together. Instead of adding another level of indentation, consider:

• Extracting logic into a helper function.

• Performing validation up front and exiting early.

• Splitting complex transformations into separate stages.

A notebook should read like a pipeline, not a maze.

Function Construction

The Don't Repeat Yourself(DRY) principle is well known and I've seen many functions or procedures done which scare me. The common theme for these are they try to do too much. They don't follow the 4th NASA rule, which states:

No function should be longer than what can be printed on a single sheet of paper in a standard format with one line per statement and one line per declaration. Typically, this means no more than about 60 lines of code per function.

This is strict but brilliant. Large functions are particularly problematic in Spark workloads because they often combine business logic, data transformation, validation, logging, and persistence into a single block of code.

Prefer small functions with a single responsibility. A reader should be able to understand what a function does without scrolling through multiple screens of code.

Error Handling

Catch only exceptions that you understand and can meaningfully recover from.

Avoid patterns such as:

try:
    do_work()
except Exception as e:
    pass

Or worse:

try:
    do_work()
except Exception as e:
    logger.error(e)

These patterns hide failures and make troubleshooting significantly harder.

Unexpected exceptions should generally be allowed to fail the workflow so monitoring, retry policies, and alerting mechanisms can do their job.

If an exception is handled, there should be a clear business or operational reason for doing so.

This is a common anti-pattern in Databricks projects and worth calling out explicitly.

Idempotency

Idempotency is one of those concepts every data engineer agrees is important, yet it is often only applied to individual Spark operations rather than the workflow as a whole.

Many pipelines are built with the assumption that they will run once, succeed once, and never need to be revisited. Reality is messier. Clusters terminate unexpectedly, upstream systems resend data, workflows are retried automatically, and operators rerun jobs manually during incidents.

A well-designed workflow should be able to fail halfway through, be rerun, and still produce the same end result.

When building a workflow, ask yourself:

• Can this notebook be executed multiple times without creating duplicates?

• What happens if the workflow fails after writing some, but not all, of its outputs?

• What happens if an upstream job retries and sends the same data again?

• Can an operator safely rerun the workflow during an incident without needing manual cleanup?

Delta Lake gives us powerful tools such as MERGE, transactional writes, and change data capture patterns, but those features alone do not make a workflow idempotent. Idempotency is a design principle, not a technology feature.

Design for retries from the start. Assume failures will happen. The goal is not to prevent every failure, but to make recovery predictable and safe.

Final Thoughts

None of these guidelines are particularly revolutionary. Most are simply attempts to keep code understandable when the original author is unavailable, the pressure is high, and a production issue needs to be resolved quickly.

The best Databricks code is rarely the cleverest. It is the code that another engineer can read, understand, and safely modify months later with minimal context.

When reviewing a pull request, I often ask myself a simple question:

Would I be comfortable when I'm tired, sleepy, or maybe even a little hung over to troubleshoot this code?

If the answer is no, the code probably needs another pass.

Write notebooks that are easy to follow. Keep functions focused. Handle errors deliberately. Design for retries. Optimise for the engineer who has to support the workload, not the engineer writing it.

That engineer may very well be you.

• replacing count-based existence checks with is_empty

• swapping single-task loops for foreach tasks

• upgrading Databricks runtimes

The code is shared across five datasets, so we expected broad performance gains.

During testing, everything looked promising. Cluster runtimes dropped, DBU usage improved, and the jobs consistently finished faster.

After a long regression-testing cycle, we released to production.

Four of the five datasets behaved exactly as expected: runtimes dropped from roughly an hour down to 20–30 minutes.

Then the fifth dataset did the opposite.

A workflow that normally completed in ~45 minutes suddenly started taking over an hour.

Initial Investigations

Looking at the foreach task, two iterations stood out immediately. While most iterations completed in under five minutes, these two were each taking close to an hour.

The odd part was that they didn’t correspond to unusually large partitions or datasets — at least not based on the metrics we initially checked.

Digging deeper into the slow iterations, we found two non-lazy operations taking significant runtimes:

• a query retrieving the max timestamp

• an is_empty existence check

Both queries were taking around 15 minutes each.

That was unexpected, especially given how lightweight those operations normally are.

The not so successful attempts at tuning

We tried several tuning approaches that seemed likely to help:

• Changed the sequence of the foreach iterations so tasks wouldn't run concurrently and contend for resources.

• Bumped up driver/executor resources to see if it was resource starvation.

• Tweaked Spark parallelism and shuffle partitioning.

• Looked for skew and tried redistributing keys.

• Considered setting the concurrency to 1 but decided it might make the job run for 2-3 hours.

• Considered gathering stats but saw the full stats were already there.

• Enabled (and disabled) a few runtime optimizations.

Those only cut off less then 5 minutes. Not enough. How can the serialized operation run faster than the foreach task in this scenario. The problem remained stubbornly intermittent.

What finally helped (the fix)

The problem kept bothering me over the weekend. It started to feel very similar to the old days of wrestling with the Oracle optimizer and comparing execution plans.

Back on Monday, I dug deeper into the Spark UI, looking for the SQL DataFrames behind the slow queries.

Once I found them, I used AI to help translate the Spark execution plans into something closer to the Oracle-style plans I was more familiar with reading.

It looked like a subtle optimizer change, but it had a massive impact on those two queries under concurrency.

That narrowed the investigation to one major variable: the DBR upgrade from 15.4 to 16.4.

After validating the hypothesis, we rolled the runtime back to 15.4.

The result was immediate:

• total job runtime dropped to 11 minutes

• the two problematic iterations fell from over an hour to roughly two minutes

Interestingly, the refactor itself wasn’t the real problem. The new concurrency model simply exposed an optimizer behaviour change that had never surfaced in the serialized implementation.

Takeaways

After solving that, we could finally celebrate. Four datasets got dramatically faster, while the fifth reminded us that performance tuning is rarely as simple as “newer runtime = better runtime.”

A few things stood out from this exercise:

• Runtime upgrades deserve the same scrutiny as application changes.
  We treated the DBR upgrade as a low-risk improvement bundled into a broader refactor. In reality, it introduced different optimizer behaviour that caused a major regression for a specific workload.

  Runtime upgrades aren’t just infrastructure changes — they can alter execution plans, heuristics, and query strategies in ways that materially affect performance. They deserve the same level of regression and performance testing as application code changes.

• Execution plans still matter.
  Even in managed platforms like Databricks, understanding what the engine is actually doing remains critical.

  The breakthrough only came after digging through the Spark UI and comparing execution plans closely enough to notice that the faster plan was using a bloom filter while the slower one was not.

  The tooling may have changed, but the engineering skill of reading and reasoning about execution plans is still incredibly valuable.

• Parallelism can expose problems that serialized workloads hide.
  The foreach implementation surfaced behaviour that never appeared in the single-task version. Concurrency, optimizer choices, and runtime heuristics can interact in unexpected ways, particularly at production scale.

• Non-production environments rarely behave exactly like production.
  We never reproduced the slowdown outside production, despite extensive testing.

  The issue only surfaced under production concurrency and workload characteristics, which exposed gaps in our validation approach. Dataset shape, partition distribution, cluster sizing, and concurrent execution patterns all matter when testing performance-sensitive changes.

• Observability pays for itself during incidents.
  Better instrumentation would have shortened this investigation significantly.

  Faster access to execution plans, clearer query-level metrics, better task-level visibility, and more targeted logging would have reduced a lot of the guesswork and helped isolate the regression earlier.

  Performance tuning becomes dramatically easier when the system makes its decisions observable.

The silver lining is that these are the kinds of problems that sharpen engineering instincts. There’s something deeply satisfying about tracking down a subtle optimizer regression — especially when the final fix turns an hour-long task back into a two-minute operation.

Addendum

After the typical back-and-forth shuffle with Databricks Support, they identified a new Spark planning optimisation in DBR 16.4 called CollapsePartialAndFinalAggregates as the root cause of the plan change. This feature is enabled by default in this version. To disable it, set the Spark configuration spark.databricks.execution.collapsePartialAndFinalAggregates.enabled to false.

This is 2 of those hard-earned lessons: antipatterns I now watch for and the guidance I share when I see them creeping into projects.

Single Row DMLs

Many engineers coming from a stored-procedure or OLTP background (I was one of them) naturally reach for single-row INSERT/UPDATE/DELETE operations: write a checkpoint row, flip a flag, append a log entry — all from the same SQL client used for development. That feels simple and fast during development, and it usually “just works” at low scale.

Why this becomes a problem on Delta

• Every individual DML tends to produce a tiny file. Tens, then thousands, then millions of those files drive:

  • poor read performance (many small files = high task scheduling and IO overhead),

  • long metadata operations (Delta/Apache Spark must track many files and transactions),

  • higher storage and management costs,

  • increased maintenance (VACUUM, compaction, and longer checkpoint/commit times).

• Single-row DMLs also make compaction and concurrency trickier — your housekeeping jobs have to work harder and run more frequently.

Practical guidance and alternatives

• Avoid single-row transactional writes to Delta tables whenever possible.

• Prefer append-only, batched writes:

  • accumulate events/changes and write them in bulk (micro-batches) rather than per row,

  • use a streaming approach (Structured Streaming) with appropriate micro-batch sizes and checkpointing for incremental, bounded writes.

• For upserts/merges, use MERGE with a source batch (not one-row MERGEs). Build a small staging dataset of changes and MERGE that in one operation.

• Control file count at write time:

  • repartition/coalesce to a reasonable number of output files before writing,

  • avoid writing one tiny file per write operation.

• If you must persist fine-grained events, write them to append-only files (Delta or Parquet) and compact periodically.

• Use Delta compaction tools on a schedule (OPTIMIZE/compaction jobs) to consolidate small files, and monitor file counts so compaction is run before it becomes painful.

• Prefer external logging/monitoring for operational logs: cluster logs, Databricks/job metrics, or a centralized logging system (e.g., ELK, Splunk, cloud logging). Application-level logs in Delta are only justified if you need them for audit or replay — and then treat them as batched event streams, not single-row table DMLs.

When you can’t avoid single-row operations

• If strict transactional single-row updates are unavoidable, isolate that usage:

  • put such records in a dedicated table with a clear compaction policy,

  • limit retention or TTL to keep the table small,

  • schedule frequent, automated compaction and vacuuming,

  • and instrument monitoring to alert when small-file counts grow.

Bottom line Single-row DMLs are easy and familiar, but they don’t scale well on Delta. Design for batched or append-only writes, use MERGE for bulk upserts, and rely on compaction/OPTIMIZE and centralized logging when possible to avoid the small-files and metadata headaches.

dbutils.fs — convenience vs. scalability

dbutils.fs is incredibly handy in notebooks: quick file reads/writes, directory listings, tiny helper copies and deletes. That convenience makes it an easy go-to during development — but several common uses become antipatterns in production.

Common antipatterns

• Using dbutils.fs.put or repeated dbutils.fs.cp/mv/rm in tight loops to emit many small files (e.g., one file per event or checkpoint).

• Relying on file presence / _SUCCESS files as locks or transactional signals (race conditions and non-atomic checks).

• Using dbutils.fs.ls to enumerate very large directories from notebooks (slow, paginated REST calls).

• Using dbutils for large data movement (instead of Spark, cloud SDKs or storage-native tools).

• Storing secrets/credentials in files written via dbutils or using it to persist sensitive info.

• Orchestrating business logic by chaining many dbutils operations synchronously (blocking, brittle).

• Using dbutils.fs for compaction or heavy maintenance in production without retry/monitoring.

Why these are a problem

• Many dbutils.fs calls translate to REST operations against cloud storage; they’re relatively slow and expensive when done at scale.

• They don’t provide the transactional/atomic semantics Delta or Spark offer — leads to races and inconsistent state.

• Repeated small-file writes create the same small-files and metadata explosion problems described earlier.

• Lack of retries, backoff, and backpressure patterns in ad-hoc uses makes jobs brittle and hard to debug.

Practical guidance and alternatives

• For data writes, prefer Spark/DataFrame APIs or Delta write APIs:

  • Use DataFrame.write.format("delta").mode("append").save(path) or Structured Streaming with controlled micro-batch sizing instead of dbutils.fs.put.

• For bulk copy/move of large datasets, prefer:

  • Spark read/write (distributed), cloud storage SDKs (AWS S3 SDK, Azure Data Lake SDK) or storage-native copy tools — not dbutils.fs.cp for terabytes of data.

• For listing/metadata at scale, prefer:

  • Use Spark’s distributed listing (spark.read or cloud SDK with pagination) or storage-native APIs that support efficient pagination.

• For coordination and locking, prefer:

  • Use Delta transactions, a small metadata table (Delta) for control records, or an external lock service (Azure Blob lease, S3 atomic rename patterns, Redis, etc.), not file-existence checks.

• For event/app-level logs, write append-only files in batches and compact (OPTIMIZE) periodically; avoid single-row file writes.

• For secrets, use secret scopes / cloud KMS; never write credentials with dbutils.fs to storage.

• Add robust retry/backoff, idempotency, and alerting around any required dbutils.fs calls to handle transient errors.

Safe uses of dbutils.fs

• Quick inspections in interactive sessions (dbutils.fs.ls of small directories, head for a sample).

• Small administrative tasks in low-volume jobs, with clear TTL and compaction policy.

• One-off notebook utilities, not core production pipelines.

Operational checklist

• Audit notebook/job code for dbutils.fs usage in loops or per-row logic.

• Replace per-row file writes with batched writes or Delta upserts.

• Add metrics: file count, average file size, listing latencies; alert when thresholds exceeded.

• Schedule compaction/OPTIMIZE jobs before file counts get out of hand.

• Use cloud-native locking or Delta where atomicity is required.

Bottom line dbutils.fs is great for development and light administrative tasks, but it’s not a substitute for distributed, transactional, or high-throughput I/O patterns. Replace repetitive or large-scale dbutils.fs usage with Spark/Delta writes, cloud storage SDKs, or dedicated coordination mechanisms — and treat any remaining dbutils operations as small, monitored, and idempotent utilities.

Closing thoughts

Experience taught me that solutions that feel “simple and fast” in development often turn expensive at scale. Small decisions—like doing one-row DMLs from a notebook—compound into operational debt: slower queries, heavier maintenance, and surprising costs. The best defense is awareness plus a few engineering guardrails: batch where possible, use the platform’s compaction and write-optimization features, and add monitoring to catch small-file growth early. Those moves keep your Delta tables healthy and let you scale from a prototype to production without painful rewrites.

Mindset Shift

Coding for Databricks using PySpark comes with a slight paradigm shift. Instead of the usual procedural approach, involving row-by-row processing with cursors and loops, one must think of how to do operations in bulk. How to setup dataframes with data transformations that will be lazily evaluated. Storage is not tightly coupled with the database engine, hence the usual context switching problems with PL/SQL hits extra hard. Understanding the columnar way the data is stored is important. There are no indexes and data is organized into files and partitions, whether explicitly or implicitly defined. Optimizing access paths is about eliminating partitions or files that do not need to be read.

SQL vs PySpark Syntax

Given that background, I present how an SQL statement actually looks like converted to PySpark. For example:

SELECT ename, job, sal
FROM emp
WHERE job IN ('CLERK', 'ANALYST') 
  AND sal > 1000
ORDER BY sal DESC;

This converted to PySpark looks like this:

from PySpark.sql import functions as F

df = spark.table("emp")

result = (
    df.filter(F.col("job").isin("CLERK", "ANALYST"))
    .filter(F.col("sal") > 1000)
    .select("ename", "job", "sal")
    .orderBy(col("sal").desc())
)

Some things stand out immediately. Firstly, we start with the FROM clause which initializes the dataframe. The next part is a set of filters, a select clause, and an order by clause. This introduces the chaining syntax which can be useful for refactoring later on. I would also highlight the fact that with PySpark, the filters should normally be defined before the select clause. This brings me to some key takeaways I've come to know about and want to impart.

Takeaway Boxes

Filtration is key

Think about the filters first. It's a good rule because this makes one think of how to filter a table so that the least amount of rows are subjected to the next steps. Multiple filter functions can be chained as well to avoid the multitude of AND clauses which I can never decide when to cut to a new line and indent for readability.

Be Lazy Until You Can't

Running the code above segues into the concept of lazy evaluation. Data Engineers should be conscious of the operations that are eager actions which is the opposite of .

A df.count() actually executes, do not use it to check for the existence of a row.

A df.collect() executes and fetches all rows and puts it into an array of python rows which can easily be iterated. It should only be used on small data sets for possibly metadata.

A df.show() or a display(df) actually runs the query and displays it on the cell.

A df.write() should almost always be the last operation as this means the data frame has all the lazy transformations it needs and can now be written to a table, or the storage.

Why not spark.sql

Data engineers with insane SQL skills new to PySpark tend to write their queries and transformations in SQL statements when they are in a python notebook. Therein lies the problem, using spark.sql() means building dynamic SQL strings in python, F-strings can only get you so far. You've got to handle the WHERE clauses with multiple AND conditions. You've got to escape special characters. You'll have to handle subqueries and make sure that parenthesis are matching. Simple put, it quickly becomes a string manipulation exercise. Some of the points I raise regularly are:

• SQL Injection - this used to be a mortal sin in database development

• Using spark.sql means everything is a string. Datatypes need to be casted properly from string. Special characters need to be escaped properly to become the correct value for the SQL statement.

• Capitalization and SQL standards are optional and not enforced. Linting tools can easily refactor python PySpark functions, but the strings inside a spark.sql is free form.

Know your (data) types

I can usually tell a very good engineer by how they handle data types. In databases, using the correct native datatype gives significant gains in performance and storage efficiency. Relying on implicit casting or leaving types ambiguous often leads to subtle bugs, data quality issues, and degraded performance.

In PySpark, a strong grasp of data types is essential because you are working across two systems:

• Python data types (e.g. int, float, str, datetime)

• PySpark SQL data types (e.g. IntegerType, DoubleType, StringType, TimestampType)

Misalignment between these can introduce silent errors, especially during ingestion, transformation, and writes. Timestamps (or datetime in python) can also be easily confusing. Using the python datetime variables help in getting all the nuances on timestamps in place rather than converting to specific datetime/timestamp string formats. Databricks also by default does not display all the microseconds when it implicitly converts timestamps to string. Hence loss of precision happens when someone naively just uses strings.

Explicitly cast literals on spark dataframes:

from PySpark.sql import functions as F
from datetime import datetime

current_time = datetime.now()

df.filter(F.col("id") == F.lit(123).cast("int"))
df.withColumn("extract_timestamp", F.lit(current_time).cast("timestamp"))

If needing to actually put datetimes into strings, use the isoformat. That format maintains readability and the option to put the desired time offset that makes comparison of the data easier later on.

Null and None Matters

Nulls are always a fun database concept. In PySpark, nulls actually highlight the importance of explicitly setting data types. In Python, None is the equivalent of the Spark null.

Shuffling Out

Wrapping up, I realize I haven't even discussed shuffling yet. I'll save that for later.

So shuffling out, the biggest shift when moving SQL/PLSQL-minded engineers to PySpark on Databricks isn’t just new syntax — it’s a different way of thinking about data. Instead of procedural, row-by-row logic, you design transformations as lazy, set-based operations on dataframes; instead of relying on indexes you reason about files, partitions and columnar layouts; and instead of micro-optimizations in a procedural loop you focus on eliminating unnecessary reads and choosing the right execution patterns.

For the next article I’ll dig into concrete patterns and anti-patterns — the idioms you’ll reach for most often and the traps you’ll want to avoid. I'll aim for runnable examples, checklist-style rules of thumb, and before/after comparisons so engineers coming from SQL or PL/SQL can map familiar concepts to PySpark best practices.

Frankly speaking, doing a select count just to check the query will return some rows is inefficient and many systems I’ve come across are littered with samples of this. I won’t dwell into that too much here.

What I want to discuss is more on how to get the counts after running doing dataframe.write statements in spark.

The Old Traditions

Let me start with how it used to be done with traditional databases. Across database systems, the method for determining the number of inserted rows differs. In MySQL, the ROW_COUNT() function returns the count of rows affected by the preceding INSERT statement. PostgreSQL offers a RETURNING clause within the INSERT statement itself, which can provide a count or return the inserted data. In SQL Server, the @@ROWCOUNT system variable holds the number of rows impacted by the most recent statement. Snowflake provides a SQLROWCOUNT variable for this purpose, specifically for DML statements within its scripting environment.

For Amazon Redshift, not really a traditional database but is a fork off good old postgres, obtaining an accurate insert count typically involves querying the stl_insert system table for the query ID of the recent INSERT operation, which can be more complex due to the distributed nature of the database. These variations mean developers must use the correct function or variable for their specific database to get an immediate and accurate count of inserted rows.

Simply put, there used to be values returned or special variables that provides a way to count.

Breaking from tradition

Redshift was already hinting a little bit about the counting difficulties for distributed databases.

In my early days using databricks, I tried this:

results = df.write \
  .format("csv") \
  .option("header", "true") \
  .mode("overwrite") \
  .save(output_path)

print(results)

What a surprise to see this back:

> None

A common workaround is to do a df.count() to get the count either before or after the operation to get the counts. This was okay in cases where the data is not changing in between operations. However, it means that the dataframe will actually be executed 2x albeit the count would be optimized to just get a count.

I explored this a bit further and found that accumulators can be used.

sc = spark.sparkContext
rows_written_acc = sc.longAccumulator("RowsWrittenCount")

def increment_counter_and_return(row):
    rows_written_acc.add(1)
    return row

df_rdd = df.rdd.map(increment_counter_and_return)

df.write \
  .format("csv") \
  .option("header", "true") \
  .mode("overwrite") \
  .save(output_path)

print(rows_written_acc)

This code works BUT only if the cluster’s isolation mode is set to Single User. It rules out Unity Catalog so I had to rule it out instead. Back to the drawing board.

Observing counts

My search for a better way to count or not count led me to the Dataframe.observe function. This was introduced on Spark 3.3.1 and it computes the defined aggregates in an dataframe operation. This was exactly what I needed. I ended up with this:

from pyspark.sql import Observation
from pyspark.sql.functions import lit, count

observation = Observation("write-metrics")

df.observe(observation, count(lit("1")).alias("count")) \
  .write \
  .format("csv") \
  .option("header", "true") \
  .mode("overwrite") \
  .save("test.csv")

print(observation.get)

This returned me a dict of the aggregates I put on the observation:

> {'count': 4425435}

More things can be added to the observe function. Things like getting the max timestamps or summing up values. This is basically what I needed for now.

It is important to note though that the Observation class works on batch queries. For streaming a slightly different approach is needed. But that is a topic for a future blog.

The catch

This approach worked well and good for 99.9% of our use case. However, I did observe that .1% of our processes got None returned on observation.get. I’m still getting to the bottom of it but I’ve observed that this happens when I try to overload the cluster with too much workload. I eventually added a check on that observation.get function to count in case it was null. Because I really don’t want to write a big file again. Wish me luck on this investigation and if I ever get to the reasons I’ll put an addendum to this article.

Take Databricks, for example. We create Databricks workflows, but the Databricks workflow itself acts as an orchestrator. The challenge lies in determining what should be executed as a single Databricks workflow and what should be tasks within that workflow.

The Problem and Initial Approach

The problem I wanted to solve was flattening nested JSON data into multiple child tables. This required using the variant datatype to transform and store the data in standard tables with native datatypes. While creating a separate Databricks workflow for each table was possible, it wasn't practical. Instead, we chose to use a single Databricks workflow, triggered by the enterprise orchestration tool (Airflow), to process all the necessary tables. The simplest solution was to go through each table one by one to refine the flattening logic. This was version 1 of the implementation.

The Issue with the Loop

While this approach worked in most cases, the total processing time was the sum of all individual iterations. For example, if we had 10 tables that finished in 1 minute each and 2 tables that took 20 minutes each, the entire workflow would take 50 minutes to complete. Adding more compute power wouldn't significantly reduce this time and would only increase costs exponentially.

For Each to the Rescue

This is where the "for each" Databricks task type becomes helpful. Instead of using a loop, I restructured the iterations into a single generic notebook, which became an individual Databricks workflow task. In the job definition, I can specify the individual tables in a JSON array that is formatted for YAML.

tasks:
  - task_key: "process"
    for_each_task:
      inputs: "[\"table_a\",\"table_b\",\"table_c\"]"
      concurrency: 2
      task:
        task_key: "process_iterator"
        notebook_task:
          base_parameters:
            p_table: "{{input}}"
          notebook_path: "process.py"

A Pre-Task to Make it Dynamic

That's all well and good, but sometimes we don't want to hardcode everything in the job definition. We want things to be dynamic.

This is where introducing a pre-task becomes useful. By using another task, we can set up some Python code that builds an array and passes it to the next tasks.

For example:

# sample logic to build a list for the foreach
tables_to_process = ["table_1", "table_2"]
tables_to_process.append("table_3")

dbutils.jobs.taskValues.set(key="tables_to_process", value=tables_to_process)

How It All Looks

The job definition should look like this:

tasks:
  - task_key: "prepare"
    notebook_task:
      notebook_path: "prepare.py"
  - task_key: "process"
    depends_on:
      - task_key: prepare
    for_each_task:
      inputs: "{{ tasks.prepare.values.tables_to_process }}"
      concurrency: 4 # or whatever
      task:
        task_key: "process_iterator"
        notebook_task:
          base_parameters:
            p_table: "{{input}}"
        notebook_path: "process.py"

Once everything is set up, the Databricks job/workflow definition will have the "for each" task receiving input from the pre-task's output.

After deployment:

In a dynamic "for each" setup, you won't see the actual list of tables or iterations beforehand. This list only appears when the job runs and finishes the pre-step. Once that's done, the "for each" task starts. Clicking on the "for each" task takes you to a page that looks like this:

Interestingly, the iterations were not in alphabetical order, and I haven't needed to make them alphabetical.

Conclusion

So there you have it, the "for each" loop. It's quite useful and can help improve serialized for loops if you have them.

I did encounter one limitation. Currently, the "for each" task can only take inputs from other tasks. It can't pass task values to the next tasks. My workaround was to append the results to a single table which the next tasks can query later.

We could spin up as many environments as needed for the presentation and application layers — most engineers typically had their own local setups for these. But the database layer? That was usually shared.

This setup works fine when the schema is stable and the only operations are data manipulation (DML: inserts, updates, deletes). But in my experience, that’s rarely the case.

Data-heavy applications are always evolving — new tables here, an extra column there, another index for that slow join. Sharing a database across multiple developers and environments quickly becomes a source of friction.

Modern Workarounds for Database Isolation

Today, there are far more elegant ways to manage this. I've worked on systems where each developer spun up their own isolated data layer using Docker containers for PostgreSQL or MySQL. For Oracle, we found it more practical to use PDBs (Pluggable Databases) instead. In AWS-heavy environments, some teams spun up databases from RDS snapshots using CloudFormation or Terraform.

There are also broader solutions involving full-stack virtualization, infrastructure-as-code, and automation tools. These approaches help bring the data layer up to the same level of flexibility as the app and presentation layers — and reduce cross-team collisions along the way.

Databricks and the Environment Problem

But what happens when your “database layer” isn’t a traditional RDBMS at all?

What if it’s Databricks, where data lives in cloud-managed storage, and the interface revolves around notebooks and distributed compute jobs? In this world, environment isolation takes a different form — and requires different strategies.

What Is an Environment in Databricks?

The first important question is: how do you define an environment in Databricks? There are several valid approaches.

A common pattern is to define an environment as a Databricks workspace — e.g., kurdapyo-prd, kurdapyo-uat, kurdapyo-sit, kurdapyo-dev. This gives you clean isolation. Pre–Unity Catalog, you could even reuse catalog names across workspaces. But with Unity Catalog, catalog names must be globally unique, so it's now common to prefix them — e.g., gold_uat, gold_prd.

The downside? Spinning up a new workspace environment isn’t cheap. On AWS, it often involves provisioning a new account and VPC, setting up DNS, configuring SCIM and user permissions — the kind of setup best left to platform teams or DevOps experts.

Splitting a Workspace into Logical Environments

The next logical approach is to carve out logical environments within a single Databricks workspace. We explored two main options:

• Splitting at the catalog level

• Splitting at the schema level

I personally prefer the schema-level split. I even advocated for building automation to create schemas on demand and using zero-copy cloning to generate like-for-like environments — perfect for safe, isolated testing.

However, that approach wasn’t prioritized at the time. Instead, we went with a catalog-level split, managed through Terraform to keep things clean and avoid ad-hoc changes by engineers.

Naming Your Logical Environments

Once you've defined your environment boundaries, the next step is naming. While I prefer a “treat them as cattle, not pets” mindset, stakeholders usually prefer meaningful names.

A practical trick I’ve learned: always include a numeric suffix, like dev-001 or uat-02. It sounds simple, but it's incredibly effective — because no matter how many environments you start with, you'll always need more later. This naming convention makes it easy to scale your environments without resorting to naming gymnastics.

Aspirations: Automation and Production Parity

Once your environments and naming conventions are in place, the long-term goal is automation:

• Automate environment creation and teardown

• Make it easy to spin up production-like environments

• Enable regression testing in consistent, isolated setups

These efforts can be time-consuming upfront but pay off significantly in delivery speed and engineering confidence. Of course, it’s essential to balance these aspirations with business priorities. Delivering value to users comes first — but carving out time for automation is what sets teams up for long-term success.

One tool I’m particularly curious about is SQLMesh. It promises branch-based environments, automated model testing, and declarative change tracking — all of which seem like a natural fit for solving the isolation problem at the data transformation layer. I haven’t tried it in anger yet, but it’s definitely on my list to explore as we continue refining our approach to Databricks environments.

Final Thoughts

The challenges of environment isolation aren’t new — they just evolve with the tools we use. Whether it's a traditional database in a three-tier architecture or a modern platform like Databricks, the underlying principles remain the same: avoid shared mutable state, automate what you can, and build with repeatability in mind.

In an AWS-based Databricks setup, naming conventions extend to AWS resources like VPNs, S3 buckets, IAM Roles, and IAM Policies. Additionally, with SCIM integration, service principals and groups are synced into Databricks, retaining their original names.

In Databricks, you'll encounter a mix of naming styles, including snake_case, kebab-case, camelCase, and PascalCase. This mix can include inconsistently named resources and occasionally misspelled ones that, for various reasons, aren't worth the effort to correct. It can be quite frustrating.

Give to Caesar the things which are Caesar’s

Typically, different teams manage the cloud and Databricks environments. It's essential to adhere to the cloud team's established naming conventions. They service other technological needs aside from the data so they know their world best. Cloud resources often use kebab-case because bucket names become part of URLs. Interestingly, Google advises against using underscores in URLs, as they don't treat them as word separators (e.g., my_site is read as mysite).

Databricks start with the workspace name. It’s a name that is used in a URL and should be in kebab-case.

For groups and users, it is common practice to implement SCIM(System for Cross-domain Identity Management) Integration. What happens there is that the groups and users are synced from ActiveDirectory/IdentityNow and the groups and users are synced into databricks carrying over how it’s named. Service principals, I’ve always seen kebab-case used there. There is also the option of creating databricks groups which are separate from the SCIM integrated groups, these ones often just get the same convention for Service Principals.

Enter Unity Catalog

Unity Catalog introduces additional complexity. Most resources, like Catalogs, Schemas, Tables, and Views, are used in SQL, where snake_case or UPPER_SNAKE_CASE is preferred. SQL syntax generally handles these well, as many databases aren't case-sensitive unless names are enclosed in double quotes. Using kebab-case can be problematic because the dash might be interpreted as a subtraction sign, necessitating double quotes.

PascalCase and camelCase are popular among Java developers and in SQLServer, but using these conventions require double quotes due to case sensitivity and spaces.

Beyond SQL objects, resources like storage credentials and external locations often follow kebab-case, especially when linked to S3 buckets.

ABCs

Ultimately, the key to effective naming is to Always Be Consistent. While smaller projects often achieve this, larger implementations may struggle due to the involvement of many people, time constraints, and limited reviews. Therefore, in addition to striving for consistency, it's important to Also Be Considerate. People generally do their best with the knowledge and resources available, and while names that violate standards can be changed, consistency might not always be prioritized due to other pressing business needs.

For Databricks, resource deployment has always been a combination of Terraform and Databricks Asset Bundles. Terraform likely needs no introduction, as it's prevalent in infrastructure deployment nowadays. Other options like Pulumi or even Ansible is a distant second. While I’m no expert, I have a working understanding of how to implement changes with it.

Databricks Asset Bundles (DABs), however, are not as widely known. Before DABs, deploying jobs and workflows in Databricks was converging using DBX. Eventually, Databricks added the bundle command to the Databricks CLI, and it’s now being pushed more and more.

When I first heard about DABs from our Solutions Architect, it seemed like we could use it for all resource deployments. I was disappointed to find out later that it only deployed Workflows and DLT Pipelines.

What's Missing?

All the platform-related components, of course. Our project involved setting up Databricks on AWS from scratch, and we aimed to minimize manual configurations (clickops) while adhering to strict security protocols. Both Account Admin and workspace admin privileges were tightly controlled, as they should be.

We needed to deploy everything from the metastore, workspaces, groups, and service principals to catalogs, schemas, tables, views, and grants. However, aside from Workflows and DLT Pipelines, DABs couldn't create any of these components—not even tables and views. While we could write code to create these using the Databricks API, it required running them through notebooks, leading to less elegant, more cumbersome code compared to the declarative style of Terraform.

What we did eventually

Terraform continues to be our primary tool for provisioning most resources, including Catalogs, Schemas, External Locations, and Volumes. It also manages Shared Databricks Clusters and SQL Warehouses effectively. To maintain simplicity and security, we limit grants to the Schema level, avoiding individual object grants.

Alas, Terraform is not very common in our current crew of data engineers. This equated to a higher platform engineer to data engineer ratio for us.

We still rely on Databricks Asset Bundles for deploying Databricks workflows, as they offer a straightforward method to deploy the same code across different environments.

For creating tables and views, we utilize DBT (Data Build Tool), which provides a declarative approach to object creation. As for User-Defined Functions (UDFs), they might still require the less elegant notebook approach, but fortunately, we haven't needed to create any yet.

I am hopeful that Databricks Asset Bundles will eventually support a broader range of resources. The tool is rapidly evolving, and I remain optimistic about its future capabilities.

Setting the Scene

In our scenario, our data source is a Snowflake database managed by a SaaS vendor. We decided to use Fivetran to sync data from Snowflake into our target database. The SaaS vendor follows a weekly release cycle for minor updates and a monthly cadence for major releases, with changes occurring in different environments on varying dates.

We set up Fivetran replication from this Snowflake source. Our connector is configured to sync a substantial number of tables, and we have been syncing daily for several days without issues.

Cue in the Schema Change

One day, our sync did not complete successfully. We received a status of “Rescheduled” with the reason “Unsupported schema change requires table resync: ADD_COLUMN” for one of the tables.

As we were still new to analyzing the Fivetran logs (not available on the Fivetran Console), we were unsure of what had happened. We asked some questions within the team, and while it was a learning experience, it was challenging at the time. Here are the questions we faced:

Question 1: What column was added?

This information was not readily available. However, by querying the Fivetran logs for alter_table events, we could identify the table and the column that was added.

SELECT   *
FROM     <destination_db>.<destination_schema>.log
WHERE    conector_id = :connector_id
AND      message_event = 'alter_table'
ORDER BY time_stamp DESC;

Question 2: What happened to the sync of the other tables?

We were initially unaware of the status of the other tables. The Fivetran Console did not provide this information. However, we later learned that the other tables had indeed been synced. This information can also be queried from the logs.

SELECT   message_data:schema AS "schema",
         message_data:table AS "table",
         message_data:count
FROM     <destination_db>.<destination_schema>.log
WHERE    conector_id = :connector_id
AND      message_event = 'records_modified'
ORDER BY time_stamp DESC;

Question 3: Should we just rerun the sync?

If we had relied on the Fivetran scheduler, it would have retried the sync automatically without any intervention from us. So, we decided to rerun the sync.

What we now understand is that Fivetran, on that first sync that ended in a reschedule, had already altered the table with the schema change. At this point, it simply required an acknowledgment to proceed with a full sync for that table.

This realization was surprising. I initially thought that Fivetran would detect the schema change and, upon rerunning the sync, would then alter the table and perform the sync. However, that was not the case.

This issue is somewhat unique to us because we do not own the source. But imagine this happening to a table of multi-terabyte size. Attempting to resync could take days or even weeks. And there was no way of getting around it.

RTFM

The documentation clarified the issue further and the behavior we saw.

https://fivetran.com/docs/connectors/databases/snowflake#automatictableresyncs

Automatic table re-syncs

For tables with a primary key, we support ADD COLUMN DDL operations with null or default static values. All other table or column alterations trigger an automatic table re-sync. For tables without a primary key, any DDL operation triggers an automatic table re-sync.

Apparently the fault was ours. Though we can’t really do anything about it as the snowflake source is vendor owned.

What to do now

Sad to say, not much could be done.

We educated support about this and watch out and just prepare communications on these occurences.

We raised the issue to fivetran and made a feature request.

Now we wait.

My Mission (or so I thought)

I was brought into this project to work on Databricks components. Despite this, I was eager to see Fivetran in action. Architect colleagues I’ve collaborated with for DMS often touted how much easier Fivetran will make such acquisition or migration tasks.

I diligently explored how to orchestrate the data acquisition with Databricks workflows. As expected, Databricks didn’t have direct connectivity to Fivetran. This is a common security practice. It prevents data engineers from being tempted to create code to directly access the Fivetran APIs, potentially leading to messy implementations.

I expressed concerns that orchestration should employ the preferred orchestration tool, despite it being considered outdated technology. Unfortunately, my concerns were ignored. With my well-defined JIRA card languishing without any progress, I decided to focus and proceed with the work regardless.

With no API connectivity, I began examining Fivetran logs, which were being synced by Fivetran to Databricks. BTW, Fivetran doesn’t provide logs on its web console, and the way a customer can access logs is by syncing log data to a destination using a Fivetran connector. (This novel topic might be a future blog.)

The Fivetran log connector had a 5-15 minute delay which was acceptable for now. SLA's will be dealt with much later on.

Initial Signs

We had a job scheduled at 4 p.m. From the logs, I noticed syncs were only starting around 4:11 p.m. This seemed odd, but I didn't think much of it at the time, as it was still early in the project and nothing was stable yet.

After a few more weeks, we were preparing to promote the code to the next higher environment. We deployed the same code (Fivetran was deployed using Terraform, by the way) for a 4 p.m. sync. We waited, but by 4:15, the job hadn't started. By 4:30, still nothing. 4:45, nothing. Finally, the job only started at 4:54 p.m.

RTFM

Digging through the Fivetran documentation, I stumbled upon this warning:

https://fivetran.com/docs/core-concepts/syncoverview

When you add a new destination, Fivetran assigns it a fixed time offset. The offset can be any random value in minutes ranging from 0 to 60. It is derived from the destination ID hash. This offset is shared by every connector in the destination. The offset value remains the same regardless of the set sync frequency.

Let that sink in: a random 0-60 minute offset. And it can’t be modified, even through a support request (trust me, I tried).

Denouement

This discovery threw a wrench in our plans to avoid using Control-M. After losing a moderate amount of effort on creating a log-based solution, we pivoted to developing a simple script to be triggered by Control-M. In terms of code, simpler solution. However, in terms of infrastructure, it was a minor nightmare. We had to sort out an agent for Control-M, open up network connectivity, create a Fivetran service user to trigger the API calls, and manage secrets, among other tasks.

Things I learned

Avoid the fivetran scheduler like the plague, the API is the way to go to trigger fivetran syncs. You won’t be hit by an unreasonably high offset. And you’ll have more options to handle reschedules or errors on syncs. I’ll rant about the Fivetran Scheduler retry logic in another time.

Embrace the orchestration tool as your ally. I still do not like Control-M. While Control-M may not be my preferred choice, I recognize the robust infrastructure this organization has established around it, including change management processes, 24/7 support teams, alerts, and more. It does make me think of what hurdles needs to be scaled to get something like Airflow productionized.