What is Apache Kafka? Apache Kafka is an open source distributed event system. It was originally developed by LinkedIn, in order to cover the needs caused by its rapid growth, and moved to a microservices-based infrastructure. It is also an…
Logistic regression is a statistical methodology that allows modeling the relationships between a binary categorical variable and a set of explanatory variables. Specifically, it models the probability that an observation belongs to one of the categories of that binary variable.…
One of the most outstanding new features of Airflow 2.3.0 is Dynamic Task Mapping. This new feature adds the possibility of creating tasks dynamically at runtime. Thanks to this we can change the number of such tasks in our DAG…
We are all familiar with the concept of Linear Programming or Linear Optimization, that branch of mathematics that is dedicated to optimizing (maximizing or minimizing) a linear objective function subject to constraints in the form of equations and/or inequalities. Although…
Today I would like to deal with a topic that, from my point of view, is very important and is probably the holy grail of data engineering projects. However, we rarely reach the necessary level of maturity to be able…
Last April, the management and human resources team at Damavis got down to work to organize a new meet-up to bring together all the members of the company once again. For this occasion, we changed our desks, computers and our…
In our previous post about Reinforcement Learning, we made an introduction to this area through one of its most popular techniques: Q-learning. We laid the groundwork by talking about Markov decision processes, policies and value functions, and we saw a…
When we deal with data and we want to plot them in graphs, sometimes we need to generate those graphs using a format that allows scaling to any resolution and without losing quality. If we also need some interactivity, being…
DevOps is a set of practices that aim to streamline the software development lifecycle by coordinating the development (Dev) and operations (Ops) departments. Teams that have an integrated DevOps culture are able to continuously integrate and deliver software (CI/CD) by…