In the world of data engineering, the use of job orchestrators is key to ensuring that data arrives at the right time and in the right format. One of the most widely used orchestrators—which we’ve already discussed in several posts…
When there is a need to explore data or generate visualizations related to geographic entities (the polygon of a country or geographic coordinate points) it is quite reasonable to think about maps. Sometimes it may be valid to use simpler…
In this practical guide adapted to 2026, we will analyse how DBT integrates with Spark and how this integration can be useful for us. What is DBT and what is it used for? DBT is a framework that facilitates the…
The arrival of Apache Spark version 3.5 has revolutionised the way we work with UDFs. Although there have always been problems with them, as we discussed in Avoiding UDFs in Apache Spark, we can now enjoy the optimisation of our…
Apache Spark Streaming is a widely used technology for real-time data processing. In Theoretical Introduction to Spark Streaming, we learned about the key concepts and how this Apache ecosystem solution works. We also looked at some practical examples with Apache…
The continuous advances in Artificial Intelligence models have greatly facilitated users’ lives. The evolution of generative AI has opened up a wide range of possibilities and paved the way for developers to the point where they can now have a…
When developing code, it is not only important to consider its functionality, but also its readability. This is especially important in working environments where a team of people are working on the same code. Readable code that is unified by…
Today we are going to talk about two ways of testing in Apache Airflow. Historically, testing in Airflow has been something that has been a headache for all users of the famous framework. The coupling of the code with the…
PyCharm is an IDE developed by JetBrains for Python. As an IDE, its goal is to provide the programmer with a set of features that make code development in this language more efficient. For example, it automatically writes import lines…
Function overloading is a very interesting tool that allows several different functions to share the same identifier. This makes the readability and interpretability of our code better in some scenarios. This tool is available by default in many of today’s…