When processing data we often find ourselves in a situation where we want to calculatevariables over certain subset of observations. For example, we might be interested in theaverage value per group or the maximum value for each group. The groupBy…
Demand forecasting and suitable pricing are two key factors in any business. In the hotel context in particular, and more specifically in the online segment, these tasks are of vital importance due to the dynamic nature of prices and high…
It is well known that the use of UDFs (User Defined Functions) in Apache Spark, and especially in using the Python API, can compromise our application performace. For this reason, at Damavis we try to avoid their use as much…
Dynamic programming Dynamic programming (DP) is a computer programming method which consists in breaking down a complex problem into smaller problems, usually in a recursive way. Then, these problems are solved one at a time, typically storing the already computed…
In this article we are going to tell you some ways to solve problems related to the complexity of data engineering itself. An Airflow DAG can become very complex if we start including all dependencies in it, and furthermore, this…
Artificial neural networks are one of the main lines of study in the field of artificial intelligence today. This family of algorithms allows solving tasks as complex and diverse as image recognition, natural language processing or music generation. The main…