When we imagine a simple programming algorithm, it is logical to think about a succession of instructions that are executed sequentially, where the next instruction will not be executed until the one immediately preceding it has been completed. However, depending…
Batch processing is a methodology widely used in the world of Big Data. Throughout this post, we will analyse how technologies typically used for streaming -such as Apache Kafka and Apache Spark– can ve leveraged for batch data processing. What…
In the world of Data Engineering, it is well known that the use of UDFs (User Defined Functions) in Apache Spark (especially with the Python API) can compromise our application performace. For this reason, at Damavis we try to avoid…
In this article we are going to tell you some ways to solve problems related to the complexity of data engineering itself. An Airflow DAG can become very complex if we start including all dependencies in it, and furthermore, this…
Basic Single Node Configuration Note: the code of this post has been tested using Apache Hadoop 2.10.1. Please check out our previous post, Introduction to Apache Hadoop, to configure this version of Hadoop, in case you have not done it…
At Damavis we are very aware of the importance for our clients to have access to their data in real time. For this reason, one of our strengths is the development of tools and technologies that can move, transform and…
Note: the code of this post has been tested using Apache Hadoop 2.10.1. Please check out our previous post, Introduction to Apache Hadoop, to configure this version of Hadoop, in case you have not done it yet. As explained in…
What is Apache Airflow and how does it work? One of the work processes of a data engineer is called ETL (Extract, Transform, Load), which allows organisations to have the capacity to load data from different sources, apply an appropriate…
Sometimes it might be a bit overwhelming to understand the role of the most common open source technologies used in big data contexts. For example, probably most of you have heard about tools such as Apache Hadoop, Apache Spark, Apache…
In this article, we present the definitive guide to setting up a PySpark development environment in PyCharm, one of the most complete options. Spark has become the Big Data tool par excellence, helping us to process large volumes of data…