A while back I was the engineer responsible for a data pipeline, AWS into Snowflake, only a couple hundred thousand data points a day. And I got tired of the same conversation: my boss, or my boss's boss, telling me that something hadn't loaded. Every time, I was the last to know that reporting was off or broken and it looked poorly on me. I realized this cannot be a singular point of failure for someone in my position.
The problem is that there are always too many tables, columns, and processes to be checked every, single, day. So when things break quietly, it can create real problems.
So I built Pegel - the software that will flag these specific items. You connect a Postgres database, pick which tables you want monitored, how you want it monitored (NULL, freshness, schema drift, row counts) and get notified through your favorite method that something is off.
It's early and I truly want help and support for this one:
What type of failures have hurt you in the past? How did you catch it?
Is there willingness for users to use the software with real purpose?
What is the next step? Other database services?
My site is live and accessible; I would love some comments on how something like this would fit in a data world.