Search for a command to run...
A decade ago, I was fortunate enough to work on a regression testing suite for a critical banking system. The technology stack was a greatest hits collection from another era: Oracle, SQL, PL/SQL, Per
Practical rules for writing maintainable, supportable Databricks code.
Lately we’ve been tuning and refactoring several Databricks workflows. Nothing radical — mostly sensible engineering improvements: replacing count-based existence checks with is_empty swapping singl
Experience is a great teacher. After several years working with Databricks, I’ve noticed patterns that made perfect sense at the start but eventually became costly. At small scale they work fine — we
Lately, I've been working for an enterprise customer with hundreds of data engineers in a chapter. They're new in their Databricks journey, and their language of choice is SQL with a sprinkling of ora
A seasoned data engineer is deliberate on their use of expensive operations such as counts. It’s fairly common to see a misuse of this. Such as using a count to check if a query returns some rows. Or running a count query to figure out the number of ...
Optimizing Databricks Workflows with For Each Task Type
A Practical Approach to Planning Databricks Environments
Naming conventions are often a contentious topic among architects. In Databricks, these debates become even more pronounced due to the extensive platform work required to establish a Databricks implementation. In an AWS-based Databricks setup, naming...