Custom monitoring for a 20-hour data pipeline

The analytics data pipeline ran daily and took the better part of a day to complete. A failure discovered late was expensive — in machine time, in engineer stress, and in missed commitments.

I built custom live monitoring that watches the pipeline and alerts the moment something breaks, so problems surface early instead of at hour twenty. To make changes safe, I also constructed a production-like test environment where we could try new versions of the underlying big-data stack and new configurations before rolling them out.

Part of the work was plain data analysis too — checking that results made sense, including comparing down-sampled output against full-data-set results.

The pipeline stopped surprising us, and we could validate changes locally before touching the cluster — saving developer time and machine cost, and removing a recurring source of stress.