The linear regression model is one of the most useful tools in every data scientist’s kit bag. Although this post is geared towards people with first-hand knowledge of this statistical model, it never hurts to remember that linear regression aims…
When dealing with significant amounts of data, the way you store it can make the difference between success and failure. In this post, we’re going to take a look at a file format that may not be the most popular,…
Today we are going to talk about two ways of testing in Apache Airflow. Historically, testing in Airflow has been something that has been a headache for all users of the famous framework. The coupling of the code with the…
In 2020 Apache Spark released version 3.0.0.0 which introduced some changes to the API for defining custom data sources, known within the Spark environment as Custom Data Source. These were previously used through DatasourceV2, which generated confusion and an unintuitive…
Apache Airflow is an open source tool designed for workflow orchestration especially useful in the field of data engineering. DAGs are defined in Python files and set the relationship and dependencies between the tasks to be executed. You can take…
RAG stands for “retrieval augmented generation”. RAG is intended to allow a “large language model” (LLM) to use additional information provided by a user. LLMs are trained on large amounts of text, usually obtained from the Internet. However, it would…
Usually, when we start working on a new integration where it is necessary to connect to AWS services in the early stages of development, it is easier and faster to work only locally. For this, we can make use of…
The largest international event on technology returns to Fira de Barcelona in a new edition that took place from February 26th to 29th in Barcelona. Damavis has been one of the 45 Spanish companies selected by red.es to be part…
PyCharm is an IDE developed by JetBrains for Python. As an IDE, its goal is to provide the programmer with a set of features that make code development in this language more efficient. For example, it automatically writes import lines…
In the world of data engineering, the efficient organization and structuring of process flows plays a crucial role. At this point, Apache Airflow has positioned itself as one of the most efficient tools to achieve this task. However, to maximize…