Ask HN: Pondering Why the Social Security Website Shuts Down Nightly
1 comments
I wasn't aware of that, but no I don't think it has anything to do with developers getting paged.
I have worked on these older systems that exist in government offices and I have done work to integrate new systems to them over the years. So much of the VA, SSA, IRS and a ton of other federal, state and local government systems are focused around batch processing that they literally need that maintenance window to update systems, synchronize datasets and do data imports/exports etc. I saw just about 5ish years ago a system that still required 4 hours nightly just to manage the backup and recovery, then another 2 to ingest data from other agencies so that the agents had data to work off the next day. And it wasn't rare that you had to wait 2-3 days for your transaction to process through the system to get included in the update. Newer developers take for granted all the technology and systems we have today and I think sometimes feel it has always been that way, it hasn't. When I started doing work in the early 90's offline backup windows were still normal, batch updates required databases to be offline routinely etc. Yes, Oracle existed and was used some places but most government agencies were still on mini computers or mainframes which didn't generally support that type of software. Even the ISAM databases of the time required compaction and cleanup nearly nightly which generally needed some downtime (offline).
Hell, a couple of years ago there was an article talking about the IRS still running a critical system that is 60 years old and written in a some assembly type language that isn't well understood anymore. So not surprised the SSA does this. lol
I have worked on these older systems that exist in government offices and I have done work to integrate new systems to them over the years. So much of the VA, SSA, IRS and a ton of other federal, state and local government systems are focused around batch processing that they literally need that maintenance window to update systems, synchronize datasets and do data imports/exports etc. I saw just about 5ish years ago a system that still required 4 hours nightly just to manage the backup and recovery, then another 2 to ingest data from other agencies so that the agents had data to work off the next day. And it wasn't rare that you had to wait 2-3 days for your transaction to process through the system to get included in the update. Newer developers take for granted all the technology and systems we have today and I think sometimes feel it has always been that way, it hasn't. When I started doing work in the early 90's offline backup windows were still normal, batch updates required databases to be offline routinely etc. Yes, Oracle existed and was used some places but most government agencies were still on mini computers or mainframes which didn't generally support that type of software. Even the ISAM databases of the time required compaction and cleanup nearly nightly which generally needed some downtime (offline).
Hell, a couple of years ago there was an article talking about the IRS still running a critical system that is 60 years old and written in a some assembly type language that isn't well understood anymore. So not surprised the SSA does this. lol
Thanks for your insight. It's incredibly interesting! I suppose with so much inter-departmental exchanges of information running on legacy HW/SW, I can see why it might need several hours of downtime.
You’re welcome. I definitely get your question, no one would intentionally make a transactional system like that today. With all the advances we have made it just isn’t necessary anymore.
Although there are a few places where I can think that this pattern kinda still exists today for good reason. For example, compliance reporting systems where data is sliced off on a batch basis on some interval and pushed into a different DB for read-only reporting etc. We did that for public accounting compliance reports so the data was locked and non-updatable that produced the reports. But there are other patterns which can be used too.
Although there are a few places where I can think that this pattern kinda still exists today for good reason. For example, compliance reporting systems where data is sliced off on a batch basis on some interval and pushed into a different DB for read-only reporting etc. We did that for public accounting compliance reports so the data was locked and non-updatable that produced the reports. But there are other patterns which can be used too.
https://i.imgur.com/fgkdCTi.jpg