Testing is a key component of any data engineering process. It allows us to validate that the data we are generating or transforming meets the expectations defined at each stage. Furthermore, it helps ensure the system’s robustness by facilitating the…
In one of the latest projects we’re working on in Damavis, we began building an ETL process from scratch. On one hand, a team member was setting up the cloud architecture and all its components. Meanwhile, I was analszing the…
In various blog posts, we’ve explored what graphs are and how they work from both theoretical and practical perspectives. In the post Graph databases: Analysis and example, we already discussed how this technology can model relationships that would be impossible…
Apache Spark’s Structured Streaming API is a powerful tool for processing real-time data streams. In this context, there are certain use cases where ensuring the accuracy of the processed data is not trivial due to the time dimension that inherently…
One of the peculiarities of SQL and its many dialects is that they attempt to add features that are not present in ANSI SQL. This phenomenon is so common that it is usual to find that each dialect implements these…
In this practical guide adapted to 2026, we will analyse how DBT integrates with Spark and how this integration can be useful for us. What is DBT and what is it used for? DBT is a framework that facilitates the…
The arrival of Apache Spark version 3.5 has revolutionised the way we work with UDFs. Although there have always been problems with them, as we discussed in Avoiding UDFs in Apache Spark, we can now enjoy the optimisation of our…
In cases where it is necessary to model relationships between entities, it is common to use a graph system. The functioning of this type of element is very simple. Its structure consists of entities, represented as nodes, and edges, which…
Apache Spark Streaming is a widely used technology for real-time data processing. In Theoretical Introduction to Spark Streaming, we learned about the key concepts and how this Apache ecosystem solution works. We also looked at some practical examples with Apache…
The continuous advances in Artificial Intelligence models have greatly facilitated users’ lives. The evolution of generative AI has opened up a wide range of possibilities and paved the way for developers to the point where they can now have a…