Damavis Summary of week 24, 2021
The use of Window in Spark, how is it to work at Damavis and news about artificial intelligence in healthcare
The use of Window in Apache Spark
When processing data we often find ourselves in a situation where we want to calculatevariables over certain subset of observations. For example, we might be interested in theaverage value per group or the maximum value for each group. groupBy and…
Damavis Summary of week 21, 2021
Avoiding UDFs in Apache Spark, working in Damavis, and a guide to discovering which open-source tool is the right for your project
Demand forecasting and Dynamic Pricing
Demand forecasting and suitable pricing are two key factors in any business. In the hotel context in particular, and more specifically in the online segment, these tasks are of vital importance due to the dynamic nature of prices and high…
Avoiding UDFs in Apache Spark
It is well known that the use of UDFs (User Defined Functions) in Apache Spark, and especially in using the Python API, can compromise our application performace. For this reason, at Damavis we try to avoid their use as much…
Damavis Summary of week 20, 2021
Demand forecasting model, a dynamic programming application and a job offer at Damavis
Dynamic programming and the Fibonacci series
What is dynamic programming? Dynamic programming (DP) is a computer programming method which consists in breaking down a complex problem into smaller problems, usually in a recursive way. Then, these problems are solved one at a time, typically storing the…
Damavis Summary of week 18, 2021
Advanced Airflow, creation of machine learning pipelines and artificial intelligence in supermarkets
Advanced Airflow: Cross-DAG tasks and sensor dependencies
In this article we are going to tell you some ways to solve problems related to the complexity of data engineering itself. An Airflow DAG can become very complex if we start including all dependencies in it, and furthermore, this…
Damavis Summary of week 17, 2021
Simple perceptron, machine learning at airports, and the artificial intelligence act