Note: the code of this post has been tested using Apache Hadoop 2.10.1. Please check out our previous post, Introduction to Apache Hadoop, to configure this version of Hadoop, in case you have not done it yet. As explained in…
Compilation of the news and news shared this week in our networks
At Damavis, we always like to use the latest tools added to the language. Sometimes, we often have to write code for a project that is in an older version of Java, which can be frustrating. Especially when we know…
Compilation of the news and news shared this week in our networks
What is Apache Airflow and how does it work? One of the work processes of a data engineer is called ETL (Extract, Transform, Load), which allows organisations to have the capacity to load data from different sources, apply an appropriate…
Compilation of the news and news shared this week in our networks
Sometimes it might be a bit overwhelming to understand the role of the most common open source technologies used in big data contexts. For example, probably most of you have heard about tools such as Apache Hadoop, Apache Spark, Apache…
In this article, we present the definitive guide to setting up a PySpark development environment in PyCharm, one of the most complete options. Spark has become the Big Data tool par excellence, helping us to process large volumes of data…
In this article, we will explore what graph theory is and how it works. We will also see how it can help us solve various route and path optimization problems using a practical example. What is a graph system? A…
Schedule, orchestrate and monitor your Kettle tasks with Airflow with this Pentaho plugin. At Damavis we know the importance of data processing. Extracting, cleaning, transforming, aggregating, loading or cross-referencing multiple data sources allows our clients to have Insights or Predictive…