It is increasingly common to see job offers requesting candidates with knowledge of managing large volumes of data. This is largely due to the rise of Big Data as one of the major trends for the future. And no only…
When there is a need to explore data or generate visualizations related to geographic entities (the polygon of a country or geographic coordinate points) it is quite reasonable to think about maps. Sometimes it may be valid to use simpler…
Apache Spark’s Structured Streaming API is a powerful tool for processing real-time data streams. In this context, there are certain use cases where ensuring the accuracy of the processed data is not trivial due to the time dimension that inherently…
One of the peculiarities of SQL and its many dialects is that they attempt to add features that are not present in ANSI SQL. This phenomenon is so common that it is usual to find that each dialect implements these…
In this practical guide adapted to 2026, we will analyse how DBT integrates with Spark and how this integration can be useful for us. What is DBT and what is it used for? DBT is a framework that facilitates the…
The arrival of Apache Spark version 3.5 has revolutionised the way we work with UDFs. Although there have always been problems with them, as we discussed in Avoiding UDFs in Apache Spark, we can now enjoy the optimisation of our…
In the digital age we live in, there are millions of websites that generate a massive amount of content every day. That’s why it’s very common to feel overwhelmed by so much information available at the click of a button.…
In sectors such as hospitality and aviation, dynamic pricing systems are the basis of revenue management. What is Dynamic Pricing? A dynamic pricing system is a series of algorithmic rules that adjust prices in real time based on certain factors.…
In cases where it is necessary to model relationships between entities, it is common to use a graph system. The functioning of this type of element is very simple. Its structure consists of entities, represented as nodes, and edges, which…
Apache Spark Streaming is a widely used technology for real-time data processing. In Theoretical Introduction to Spark Streaming, we learned about the key concepts and how this Apache ecosystem solution works. We also looked at some practical examples with Apache…