Search for a command to run...
Articles tagged with #databricks
A seasoned data engineer is deliberate on their use of expensive operations such as counts. It’s fairly common to see a misuse of this. Such as using a count to check if a query returns some rows. Or running a count query to figure out the number of ...
Optimizing Databricks Workflows with For Each Task Type
A Practical Approach to Planning Databricks Environments
Naming conventions are often a contentious topic among architects. In Databricks, these debates become even more pronounced due to the extensive platform work required to establish a Databricks implementation. In an AWS-based Databricks setup, naming...
Data Engineers often have to juggle both data processing and DevOps pipelines. I’ve worked with many skilled data professionals, but they tend to shy away from the DevOps side of things. Often, I get asked to take over this aspect due to my backgroun...