Apache Spark’s Structured Streaming API is a powerful tool for processing real-time data streams. In this context, there are certain use cases where ensuring the accuracy of the processed data is not trivial due to the time dimension that inherently…
In the field of Data Engineering, efficient database design is essential to handle large volumes of data and provide effective analysis. Throughout my experience as a Data Engineer, I have worked with the main data relationship systems and have observed…
Being a hard working team that develops great projects is not incompatible with the possibility of organizing different and fun activities to promote a good working environment. At Damavis we put this into practice by having the whole team, which…
Introduction This article assumes that there is a basic knowledge about embeddings of objects, either text or images. In case you don’t have any notions on the subject, the post on Text Embeddings: the basis of modern NLP explains this…
In a previous article of our blog we detailed what is RAG (Retrieval Augmented Generation) and how to take advantage of embedding models to extend the knowledge of an LLM with our own document base. In this post, we will…
As a BI tool, Looker Studio is a visualizer as well as a data management platform which allows extracting information while maintaining its governance, security, accessibility and agility in use. The main advantage of its use lies in the fact…
When there is a need to explore data or generate visualizations related to geographic entities (the polygon of a country or geographic coordinate points) it is quite reasonable to think about maps. Sometimes it may be valid to use simpler…
The linear regression model is one of the most useful tools in every data scientist’s kit bag. Although this post is geared towards people with first-hand knowledge of this statistical model, it never hurts to remember that linear regression aims…
When dealing with significant amounts of data, the way you store it can make the difference between success and failure. In this post, we’re going to take a look at a file format that may not be the most popular,…
Today we are going to talk about two ways of testing in Apache Airflow. Historically, testing in Airflow has been something that has been a headache for all users of the famous framework. The coupling of the code with the…