In 2020 Apache Spark released version 3.0.0.0 which introduced some changes to the API for defining custom data sources, known within the Spark environment as Custom Data Source. These were previously used through DatasourceV2, which generated confusion and an unintuitive…
Apache Airflow is an open source tool designed for workflow orchestration especially useful in the field of data engineering. DAGs are defined in Python files and set the relationship and dependencies between the tasks to be executed. You can take…
RAG stands for “retrieval augmented generation”. RAG is intended to allow a “large language model” (LLM) to use additional information provided by a user. LLMs are trained on large amounts of text, usually obtained from the Internet. However, it would…
Usually, when we start working on a new integration where it is necessary to connect to AWS services in the early stages of development, it is easier and faster to work only locally. For this, we can make use of…
The largest international event on technology returns to Fira de Barcelona in a new edition that took place from February 26th to 29th in Barcelona. Damavis has been one of the 45 Spanish companies selected by red.es to be part…
PyCharm is an IDE developed by JetBrains for Python. As an IDE, its goal is to provide the programmer with a set of features that make code development in this language more efficient. For example, it automatically writes import lines…
In the world of data engineering, the efficient organization and structuring of process flows plays a crucial role. At this point, Apache Airflow has positioned itself as one of the most efficient tools to achieve this task. However, to maximize…
Function overloading is a very interesting tool that allows several different functions to share the same identifier, improving the readability and interpretability of our code in some scenarios. This tool is available by default in many of today’s most widely…
This post will deal from another point of view with one of the most popular games that can be played on paper, the Sudoku. The objective will be to explain an algorithm for solving it, together with the code that…
In other discussion spaces we elaborated a comparison between DBT, Pentaho and Spark to perform data transformations. In this post, we will look at some of the key concepts of DBT: models, snapshots and materializations. In the context of DBT…