Posts tagged with data

Viewing run logs for your development work is vital to ensuring your tasks are executing as you expect them to. However this can sometimes be more difficult in Apache Airflow, and the reasons may not be very clear to you, the developer. We will look at one scenario where this happens, and how you can work around it while improving your logging mechanics.

For distributed databases, efficient processing and transfer of records and values is paramount. Semijoins allow for data to be sent back and forth in a way that sounds like more work, but may in fact have less data transfer than an equivalent traditional inner join.