Testing in Apache Airflow

Testing in Apache Airflow

Today we are going to talk about two ways of testing in Apache Airflow. Historically, testing in Airflow has been something that has been a headache for all users of the famous framework. The coupling of the code with the…

Custom Data Source in Spark 3

Custom Data Source in Spark 3

In 2020 Apache Spark released version 3.0.0.0 which introduced some changes to the API for defining custom data sources, known within the Spark environment as Custom Data Source. These were previously used through DatasourceV2, which generated confusion and an unintuitive…

Apache Airflow with LocalStack

Apache Airflow with LocalStack

Usually, when we start working on a new integration where it is necessary to connect to AWS services in the early stages of development, it is easier and faster to work only locally. For this, we can make use of…

Introduction to Pycharm Templates

Introduction to Pycharm Templates

PyCharm is an IDE developed by JetBrains for Python. As an IDE, its goal is to provide the programmer with a set of features that make code development in this language more efficient. For example, it automatically writes import lines…