Posts by Eric

It is often still necessary to run certain applications on an EC2 instance, particularly if it is not easily containerised, and you do not want to pay extra for a managed service. In a non-production environment, you will be able to reduce costs by shutting down these instances when they are not needed by your developers or users.

Viewing run logs for your development work is vital to ensuring your tasks are executing as you expect them to. However this can sometimes be more difficult in Apache Airflow, and the reasons may not be very clear to you, the developer. We will look at one scenario where this happens, and how you can work around it while improving your logging mechanics.

Geoscience data has typically been harder to obtain as it comes from disparate source systems and physical media, and the file sizes — particularly for seismic surveys — can be massive. The Geological Survey of Queensland (GSQ) has recently undergone a data modernisation transformation, and terabytes of geoscience data is now available on their new Open Data Portal (ODP).

Accessing rich datasets from public web platforms is a common scenario for clients and the trend looks set to continue. The number of social media users is growing by the millions as more people get access to faster and cheaper smart devices. For one of our clients, getting data from some of these online platforms and matching them with their internal databases seemed like it would be a time consuming process.

For distributed databases, efficient processing and transfer of records and values is paramount. Semijoins allow for data to be sent back and forth in a way that sounds like more work, but may in fact have less data transfer than an equivalent traditional inner join.