Airflow is an open source framework for developing, orchestrating and monitoring batch workflows. In case you are not familiar with it, and to better understand the content of this article, it is recommended to review the basics of Apache Airflow…
Imagine you are a data scientist working for a company that has decided to merge the databases of two of its departments into one. There is no common index between the two sources and the task is to ‘deduplicate’ the…
Git is a version control system that has become an essential tool for software development. It allows teams of developers to work collaboratively on the same code and efficiently manage change history. In this post we will introduce Git at…
Apache Spark’s Structured Streaming API is a powerful tool for processing real-time data streams. In this context, there are certain use cases where ensuring the accuracy of the processed data is not trivial due to the time dimension that inherently…
When there is a need to explore data or generate visualizations related to geographic entities (the polygon of a country or geographic coordinate points) it is quite reasonable to think about maps. Sometimes it may be valid to use simpler…
A linear regression is a model that is used to approximate the linear relationship between a dependent variable Y and a set of independent variables X. In matrix format it can be expressed as where ε is a vector of…