SQLFluff is an SQL code linting and auto-formatting tool that we’ve already covered in previous articles. In SQL optimization with SQLFluff, we discussed how to install it from scratch and walked through the first steps for using it. Implementing SQLFluff…
Currently, there are more than 400 million code repositories on platforms such as GitHub. With such a large ecosystem, documenting a project well has become crucial for any developer who wants to use or review the code to understand how…
Testing is a key component of any data engineering process. It allows us to validate that the data we are generating or transforming meets the expectations defined at each stage. Furthermore, it helps ensure the system’s robustness by facilitating the…
For years, Airflow has been used as a scheduler that allows tasks to be executed at specific times. This was done using cron expressions that specified the exact time an event was to occur. This approach works well for traditional…
In the digital age we live in, there are millions of websites that generate a massive amount of content every day. That’s why it’s very common to feel overwhelmed by so much information available at the click of a button.…
When developing code, it is not only important to consider its functionality, but also its readability. This is especially important in working environments where a team of people are working on the same code. Readable code that is unified by…
In the world of data engineering, the efficient organization and structuring of process flows plays a crucial role. At this point, Apache Airflow has positioned itself as one of the most efficient tools to achieve this task. However, to maximize…