Introduction to Apache Arrow
In this article we are going to see what this tool is and how it works. Apache Arrow is present in most of the frameworks in our environment, such as Spark, Dask, Drill, Tensorflow, Kudu, etc. All of them are…
APISIX: Authentication, security and traffic
In Introduction to Apache APISIX we studied the different components of this platform and the advantages of using it. In addition, we saw a way to dockerise it and we configured a simple route with the openid-connect plugin, integrating authentication…
Index optimisation in relational databases
When we are working with relational databases, it is usual for some queries to be extremely slow. This can be due to different reasons, but the most common is that the data access pattern that the query demands is not…
New features in DBT 1.9
The developers of DBT have set themselves the goal of releasing a new major release every four months, and so far they are delivering. Last October, version 1.9 of this popular framework was released, and has been undergoing corrections. In…
Sensitive data management in BigQuery – Part II
This is a continuation of what was described in the first part of this series of articles on Sensitive data management in BigQuery. Working with data in this tool can require control over who accesses it and how it is…
Tutorial DataHub 4 – API
In this new part of the DataHub tutorial saga, we are going to work on the connectivity with the platform through the API. As data engineers, the goal is to incorporate DataHub as a Data Governance tool in our ecosystem.…
Sensitive data management in BigQuery – Part I
Among all the tools we can access in Google Cloud, one of the most powerful and widely used is undoubtedly BigQuery. Its versatility as a DWH (Data Warehouse) has positioned it as one of the main cloud solutions for data…
Trigger rules in Apache Airflow
Airflow is an open source framework for developing, orchestrating and monitoring batch workflows. In case you are not familiar with it, and to better understand the content of this article, it is recommended to review the basics of Apache Airflow…
Dynamic Pricing: Avoiding inefficient implementations
The Revenue Management (RM) departments of companies have the objective of maximising revenue (ideally it should be profit, but this is something we will talk about in another article) through optimal pricing management. There are many considerations that these teams…
Team building: Training and team work
At Damavis we continue to add moments and experiences together through team building activities and events. With the aim of promoting the integration of all members, we organise these actions that allow us to get to know our colleagues better…