Option Pattern in Golang
It has been about a year since I started developing some things in Go. At first, it was an unknown language to me. I only knew that it was a “typed” language, compiled and memory safe through a GC (Garbage…
Cristòfol Torrens
Avoiding UDFs in Apache Spark
It is well known that the use of UDFs (User Defined Functions) in Apache Spark, and especially in using the Python API, can compromise our application performace. For this reason, at Damavis we try to avoid their use as much…
Advanced Airflow: Cross-DAG tasks and sensor dependencies
In this article we are going to tell you some ways to solve problems related to the complexity of data engineering itself. An Airflow DAG can become very complex if we start including all dependencies in it, and furthermore, this…
Pentaho PDI Plugin for Airflow
Schedule, orchestrate and monitor your Kettle tasks with Airflow with this Pentaho plugin. At Damavis we know the importance of data processing. Extracting, cleaning, transforming, aggregating, loading or cross-referencing multiple data sources allows our clients to have Insights or Predictive…