‘They Don’t Make These Anymore’: Maintaining the MBTA’s 100-Year-Old Signals(wbur.org)
wbur.org
‘They Don’t Make These Anymore’: Maintaining the MBTA’s 100-Year-Old Signals
http://www.wbur.org/2016/03/07/mbta-signal-maintenance
7 comments
For most of the (new) equipment procurement the MBTA does now, they actually own the specifications for the equipment, which means that if they want to make more, they put out an RFP for companies to manufacture new equipment based on their spec. The downside to this is that the T likes to over-customize their orders, which drives costs up compared to buying more standard "off-the-shelf" equipment, and if no responses to the RFP come back, they have to relax some of the requirements and put it out again.
While its true that they own the specification, in practice the large company (i.e. Siemens in this example) is the only one capable of producing the product at a non-prohibitive rate.
I suspect what they'll find when they finally rip out all this equipment is that the new system will be unrepairable without huge long term contracts with the original vendor.
These old mechanical systems have their own problems, but repairability is not one of them. Almost any competent machining shop could duplicate one of the switches in the article.
I suspect what they'll find when they finally rip out all this equipment is that the new system will be unrepairable without huge long term contracts with the original vendor.
These old mechanical systems have their own problems, but repairability is not one of them. Almost any competent machining shop could duplicate one of the switches in the article.
Is owning the specification really the same thing as owning the hardware source? Why is it that you need the long-term vendor contracts? If you go beyond the spec and make all the hardware control boards, software, etc. open, could you lower that cost?
Boston has one unique feature that makes public-ish ownership of specifications and the hardware running more plausible than other cities... it has a large number of technically-sophisticated potential stewards for such a program who aren't going anywhere: all the colleges and universities.
MIT and Harvard are something like the largest land owners in cambridge and boston... they're not leaving anytime soon, and ideally they have an interest in maintaining relationships with their community. They're invested in the area, and as academic institutions, inclined to support open source initiatives like this (how many of you use the MIT keyserver for your PGP keys?) Imagine a "civic service" initiative where the area universities are stewards of the open source implementation for the city's infrastructure.
Not at all saying you want the T to run on overworked grad student code -- outsource that in a professional RFP -- but if the concern is that technical stewardship might be out of the realm of a municipal infrastructure agency, Boston's academic population could make for an interesting partnership.
Hairbrained stream-of-consciousness idea, but hey, that's what internet forums are for ; )
Boston has one unique feature that makes public-ish ownership of specifications and the hardware running more plausible than other cities... it has a large number of technically-sophisticated potential stewards for such a program who aren't going anywhere: all the colleges and universities.
MIT and Harvard are something like the largest land owners in cambridge and boston... they're not leaving anytime soon, and ideally they have an interest in maintaining relationships with their community. They're invested in the area, and as academic institutions, inclined to support open source initiatives like this (how many of you use the MIT keyserver for your PGP keys?) Imagine a "civic service" initiative where the area universities are stewards of the open source implementation for the city's infrastructure.
Not at all saying you want the T to run on overworked grad student code -- outsource that in a professional RFP -- but if the concern is that technical stewardship might be out of the realm of a municipal infrastructure agency, Boston's academic population could make for an interesting partnership.
Hairbrained stream-of-consciousness idea, but hey, that's what internet forums are for ; )
As the first example of the money involved in railway signalling, in 2012 "Siemens has entered into an agreement to acquire Invensys Rail, the rail automation business of Invensys for approximately €2.2 billion"
That's presumably not the kind of operation that can be run in someone's spare time, and exactly the kind of work that graduates from these universities are training towards, and would expect to be well-paid to do.
> if the concern is that technical stewardship might be out of the realm of a municipal infrastructure agency
...then they really must employ someone to do that work. Except perhaps for the tunnels, the signalling system is their most important asset.
That's presumably not the kind of operation that can be run in someone's spare time, and exactly the kind of work that graduates from these universities are training towards, and would expect to be well-paid to do.
> if the concern is that technical stewardship might be out of the realm of a municipal infrastructure agency
...then they really must employ someone to do that work. Except perhaps for the tunnels, the signalling system is their most important asset.
That particular problem is easily solved by providing all needed documentation as part of the development contract. I'm a big fan of OSS & OSH, but it doesn't add value in this case.
I'm pretty sure the problem isn't knowing how the devices work, it's the limitations of the 100 year-old technology in the face of modern expectations.
I'm pretty sure the problem isn't knowing how the devices work, it's the limitations of the 100 year-old technology in the face of modern expectations.
The Atlantic article was on HN:
https://news.ycombinator.com/item?id=10566776
102 points by jsomers 114 days ago | 21 comments
https://news.ycombinator.com/item?id=10566776
102 points by jsomers 114 days ago | 21 comments
live in MA and take the T (what we call the subway/bus service here). Public transit is a major problem in this city and surrounding areas. It what make this city OK and expensive vs being great and affordable.
While the transit system here needs upgrading badly, this piece is very political. The quoting of Charles Chieppo is a give away.
"For many systems, Chieppo said, the focus is on improving stations or opening new ones rather than fixing the core infrastructure." The T is working on the first expansion of the subway rail system since the 1980s. Its going into the neighboring city of Somerville and "surprise" its over budget. They might kill the expansion with the focus on "fixing the existing" as kind of justification, instead of doing both at once? They're also reducing late night service for the extra couple hours they tacked on to friday and saturday, while proposing a fair hike. Things cost money, but people don't like to pay more to get less.
(I'm actually usually on one of the MASCO "private" buses that supplement the T for its failure to provide adequate routing to the Longwood Medical area.)
While the transit system here needs upgrading badly, this piece is very political. The quoting of Charles Chieppo is a give away.
"For many systems, Chieppo said, the focus is on improving stations or opening new ones rather than fixing the core infrastructure." The T is working on the first expansion of the subway rail system since the 1980s. Its going into the neighboring city of Somerville and "surprise" its over budget. They might kill the expansion with the focus on "fixing the existing" as kind of justification, instead of doing both at once? They're also reducing late night service for the extra couple hours they tacked on to friday and saturday, while proposing a fair hike. Things cost money, but people don't like to pay more to get less.
(I'm actually usually on one of the MASCO "private" buses that supplement the T for its failure to provide adequate routing to the Longwood Medical area.)
Green line expansion (two stops) initially budgeted at $954M in June 2010.
5 years later, costs estimated at $3.09B and project is terminated.
http://www.mbta.com/uploadedfiles/About_the_T/Board_Meetings...
5 years later, costs estimated at $3.09B and project is terminated.
A Reliable Project Budget is Defined as being
within +10% and -5% of what it would Cost
to Design and Construct the Project
http://www.mbta.com/uploadedfiles/About_the_T/Board_Meetings...
I believe that the housing market in Medford and union Sq for years has been inflated on the speculation of the green line exrension. For years it was taken as a given. Has the Market declined since it came to light that the estimates were off, and the project cancelled ? I imagine a lot of people will lose money over this.
Its not canceled yet..Just scaled back.
Though I think the City of Somerville took some property in union square to make it work.
3.09B!!! That is insane. I would think after the big dig disaster they would have learned their lesson.
In Zürich they just finished a huge expansion with a complex 10km tunnel und a 3rd underground 4 track station. Project cost was 2.0B up from 1.8B. At one point a drill got stuck and they had to abandon it.
[1] https://de.wikipedia.org/wiki/Durchmesserlinie_Altstetten–Zü...
In Zürich they just finished a huge expansion with a complex 10km tunnel und a 3rd underground 4 track station. Project cost was 2.0B up from 1.8B. At one point a drill got stuck and they had to abandon it.
[1] https://de.wikipedia.org/wiki/Durchmesserlinie_Altstetten–Zü...
MBTA has huge project management problems to be sure, of course it should be looked into.
But its really the only subway the city has, plus I think they're required by consent decree to build sometimg (Part of the "big dig" project (putting highway in a tunnel), another fine example of poor project budget control in the commonwealth of Massachusetts)
But its really the only subway the city has, plus I think they're required by consent decree to build sometimg (Part of the "big dig" project (putting highway in a tunnel), another fine example of poor project budget control in the commonwealth of Massachusetts)
Ha. This is too true. If there wasn't some kind of construction project over budget at all times, people would think something was wrong.
Really though, the MBTA has a bunch of major issues. Not just project management. I honestly don't know what it would take to fix it but something has to give sooner or later. People are finally getting fed up.
Really though, the MBTA has a bunch of major issues. Not just project management. I honestly don't know what it would take to fix it but something has to give sooner or later. People are finally getting fed up.
I honestly think that the $954M original price tag is a bigger problem than the 3X overrun.
Wow, $3.09B for 4.3 miles of track and 4 stations? In possibly 7 years, if everything goes according to plan?
Elsewhere in the world, the Delhi Metro Phase 2 cost $2.9B and took 3 years. 77 miles of track, 85 stations, and countdown clocks. The Bangalore Metro (phase 1) is expected to be $2.5B (including a 25% cost overrun) for 26 miles and 40 stations. It also includes countdown clocks. Barcelona spent $200M for a 5.8km rail tunnel.
Where's your money going?
Elsewhere in the world, the Delhi Metro Phase 2 cost $2.9B and took 3 years. 77 miles of track, 85 stations, and countdown clocks. The Bangalore Metro (phase 1) is expected to be $2.5B (including a 25% cost overrun) for 26 miles and 40 stations. It also includes countdown clocks. Barcelona spent $200M for a 5.8km rail tunnel.
Where's your money going?
Have you looked at the wages of Indian labourers versus American construction workers lately?
In fact, most of the cost of train construction is not labor (except in the US). That's why the Egypt, China and India (low labor costs) aren't significantly cheaper than Spain or Korea (high labor costs).
https://pedestrianobservations.wordpress.com/2013/06/03/comp...
If only I thought of this - then I might have cited a Spanish construction project in the very comment you replied to, so that no one could cluelessly bring up labor costs.
In any case, even if your claim were true, there is an easy solution. Import Indians to get the job done. The goal is fixing our crumbling infrastructure rather than union featherbedding, right?
https://pedestrianobservations.wordpress.com/2013/06/03/comp...
If only I thought of this - then I might have cited a Spanish construction project in the very comment you replied to, so that no one could cluelessly bring up labor costs.
In any case, even if your claim were true, there is an easy solution. Import Indians to get the job done. The goal is fixing our crumbling infrastructure rather than union featherbedding, right?
To be fair, he mentioned Barcelona, which is not too different from America hopefully.
I'm going to see if I can google what a Swiss equivalent costs (pretty certain the Swiss should be more expensive than American workers).
Edit: the only metro in Switzerland is in Lausanne, and it cost SFr590 million ($472 million) for about 6 km (3.7 miles) and 14 stations.
Construction is probably much easier, but the American one still has some pretty brutal cost overruns.
I'm going to see if I can google what a Swiss equivalent costs (pretty certain the Swiss should be more expensive than American workers).
Edit: the only metro in Switzerland is in Lausanne, and it cost SFr590 million ($472 million) for about 6 km (3.7 miles) and 14 stations.
Construction is probably much easier, but the American one still has some pretty brutal cost overruns.
To be fair in regards to the late night service cut, it was only a test program when they rolled it out and supposedly ridership was declining since they introduced it. Plus, it was only available on the weekends (two days a week) anyway.
> It’s not as simple as just changing the signal. In order to change it out and modernize, you need to update the entire system.
Does anyone have any idea why this is? What prevents new signals from being backwards compatible with the old system?
And if this were true, wouldn't all of the signals need to be the same hundred year old type? How would line extensions ever be able to be done in that case? Is the system segmented in some sense? If it is segmented, what prevents further segmentation to allow incremental updates of the signals?
Does anyone have any idea why this is? What prevents new signals from being backwards compatible with the old system?
And if this were true, wouldn't all of the signals need to be the same hundred year old type? How would line extensions ever be able to be done in that case? Is the system segmented in some sense? If it is segmented, what prevents further segmentation to allow incremental updates of the signals?
Because the old signals systems are analog or electromechanical, all modern signals systems are computerized. You'd essentially have to build something custom from scratch to do so, which would be way more expensive than simply replacing everything. Since both options sound unappetizing, they end up doing patch repairs like this to keep the system going, but that can only last so long before it just breaks down.
^ that
The Atlantic article linked above covers this in more detail, and is an excellent read all-around.
The Atlantic article linked above covers this in more detail, and is an excellent read all-around.
Further segmenting the existing system probably means a huge amount of changes to the existing system — more expensive than replacing the whole lot.
The old system will be based on detecting trains in "blocks", sections of track insulated from each other where the train's wheels make the circuit. I was about to say this would use relays, but Wikipedia says the first all-relay system was installed in 1929. So it's at least partly mechanical (levers moving in the way of other levers). The logic implemented in this will prevent the signalman from sending trains on a collision path, but only if drivers obey all red lights.
The new system will either be something similar, except implemented in electronics and stopping the train if the driver doesn't. But for a metro system, in order to get higher capacity, it will probably do away with the "blocks" altogether, instead ensuring that every train has a safe distance ahead of it. Maybe they will also automate the trains, so the train drives itself.
Either way, they'll probably upgrade the trains to support the new system, then install the new system in whatever sections allow it, then remove the old system once the new system is working correctly. I'd expect there's a huge amount of sunk cost once the project starts (design, modifications to trains, staff training etc) so it doesn't make any sense to stop halfway and keep the costs of both systems.
https://en.wikipedia.org/wiki/Interlocking
The old system will be based on detecting trains in "blocks", sections of track insulated from each other where the train's wheels make the circuit. I was about to say this would use relays, but Wikipedia says the first all-relay system was installed in 1929. So it's at least partly mechanical (levers moving in the way of other levers). The logic implemented in this will prevent the signalman from sending trains on a collision path, but only if drivers obey all red lights.
The new system will either be something similar, except implemented in electronics and stopping the train if the driver doesn't. But for a metro system, in order to get higher capacity, it will probably do away with the "blocks" altogether, instead ensuring that every train has a safe distance ahead of it. Maybe they will also automate the trains, so the train drives itself.
Either way, they'll probably upgrade the trains to support the new system, then install the new system in whatever sections allow it, then remove the old system once the new system is working correctly. I'd expect there's a huge amount of sunk cost once the project starts (design, modifications to trains, staff training etc) so it doesn't make any sense to stop halfway and keep the costs of both systems.
https://en.wikipedia.org/wiki/Interlocking
Actually the old mechanical systems can stop a train automatically too. They have levers that move out onto the tracks and hit brake activator panels on the trains themselves, which force the air brakes on if the train tries to go through the red signal.
The old mechanical systems aren't really unsafe. They're just inefficient, full of moving parts so they break all the time, and require the transit agencies to run their own workshops to maintain them because they're so obsolete.
The newer computerised systems tend to have no moving parts and can pack more trains onto the tracks.
The old mechanical systems aren't really unsafe. They're just inefficient, full of moving parts so they break all the time, and require the transit agencies to run their own workshops to maintain them because they're so obsolete.
The newer computerised systems tend to have no moving parts and can pack more trains onto the tracks.
I agree that that is a puzzling statement.
Perhaps the intent was to say something like "you can't reap marginal benefits from the replacement of a single signal since the whole system must be modernized to allow the advanced signalling afforded by new technology," or something.
Perhaps the intent was to say something like "you can't reap marginal benefits from the replacement of a single signal since the whole system must be modernized to allow the advanced signalling afforded by new technology," or something.
Its actually worse, in that replacing 1% of switches means the switch repair shop and field techs now have to learn 100% more and stock 100% varieties of spare parts etc. Eventually a breakeven point is found where the reduction in labor on the old switches exceeds the cost of being dual capable.
Its possible to imagine numbers where the cost of being dual capable is so high, the cheapest solution is some kind of forklift upgrade of the whole system. Which is kinda what they're doing.
A good hardware analogy is you have a company thats a windoze PC shop. Maybe even all the hardware comes from the same Dell contract, same OS image on every box, its pretty cheap. Surely OSX would improve any end user experience, but imagine the cost of instantly doubling the helpdesk workload to now handle two kinds of hardware and two OS images and two licensing agreements and generally two of everything.
Or a good software analogy is given a somewhat reliable mysql cluster, dropping postgresql on it and trying to use both at the same time is likely to result in some heartache.
Its possible to imagine numbers where the cost of being dual capable is so high, the cheapest solution is some kind of forklift upgrade of the whole system. Which is kinda what they're doing.
A good hardware analogy is you have a company thats a windoze PC shop. Maybe even all the hardware comes from the same Dell contract, same OS image on every box, its pretty cheap. Surely OSX would improve any end user experience, but imagine the cost of instantly doubling the helpdesk workload to now handle two kinds of hardware and two OS images and two licensing agreements and generally two of everything.
Or a good software analogy is given a somewhat reliable mysql cluster, dropping postgresql on it and trying to use both at the same time is likely to result in some heartache.
> What prevents new signals from being backwards compatible with the old system?
Nothing, of course, despite the naysayers. You wouldn't, initially, try to replace - just augment; record and report the inputs and be a test-bed for the eventual digital replacement. In this case, because the originals are physical switches, it'd be more like a modern-machined duplicate of the original with a micro-controller stuck on the side.
The data it would send be immediately helpful for arrival timers and such.
Nothing, of course, despite the naysayers. You wouldn't, initially, try to replace - just augment; record and report the inputs and be a test-bed for the eventual digital replacement. In this case, because the originals are physical switches, it'd be more like a modern-machined duplicate of the original with a micro-controller stuck on the side.
The data it would send be immediately helpful for arrival timers and such.
> Does anyone have any idea why this is? What prevents new signals from being backwards compatible with the old system?
Presumably you can't use anything currently manufactured.
So, you'd have to commission a unique, backward compatible design. And test it. And qualify it.
And no company is going to do that without you basically paying them the same up-front amount as if you just contracted to replace all the signals.
Presumably you can't use anything currently manufactured.
So, you'd have to commission a unique, backward compatible design. And test it. And qualify it.
And no company is going to do that without you basically paying them the same up-front amount as if you just contracted to replace all the signals.
I work in a 50 year old hospital. There was some work on the nurse call bell system and the maintenance staff had to scour eBay for old parts to fix the outdated system. I think procurement had a fit when they saw eBay receipts show up.
I worked on a radar project with a similar problem. The target system was a computer called HP superdomes. HP discontinued those (the PA-RISC version anyway) . I think they bought a couple spares. Ebay was mentioned as a source for parts, which doesn't always give you a good feeling.
In a previous job at a University, we had a couple SGI big irons from the 1999-2002 era. A 64-way Origin 3800 and a 16-way Onyx 3800. Along with a similar vintage SAN and some misc bits.
When I left in 2010, eBay was the source of something like one replacement HDD per 10-14 days for the SAN (73GB dual-port fibre channel, full-height...) and the two machines (made of modular "bricks") had been whittled down to a single franken-SGI of something like 20 CPUs. Some of the older processing workload still ran on there (i.e.: the student graduated ages ago...)
A few dual 5300-series (8 cores) Xeons had replaced the SGIs in most workloads and left the old 500Mhz R14000 far in the dust. Lots more RAM too. Let's not even get started on the advantage as far as HVAC and power went.
When I left in 2010, eBay was the source of something like one replacement HDD per 10-14 days for the SAN (73GB dual-port fibre channel, full-height...) and the two machines (made of modular "bricks") had been whittled down to a single franken-SGI of something like 20 CPUs. Some of the older processing workload still ran on there (i.e.: the student graduated ages ago...)
A few dual 5300-series (8 cores) Xeons had replaced the SGIs in most workloads and left the old 500Mhz R14000 far in the dust. Lots more RAM too. Let's not even get started on the advantage as far as HVAC and power went.
FWIW, the same thing goes on in Miami. I've been in the electronics shop for Miami-Dade Transit and there are always a bunch of electronic signs and signals of different vintages under repair.
A lesson in bad UX, or perhaps just bad editing: Nowhere in the article is the abbreviation MBTA expanded. I shouldn't have to guess at or Google for your article's main actor.
WBUR is Boston's NPR radio station. It's the 21st Century equivalent of a local news piece, while they could work harder to adapt it for a global audience it isn't really in their remit to do so.
For a century, replacements were not much of a problem. Most of the components were either from Union Switch and Signal, or General Railway Signal, both in the US, and they still made them. Then US&S (which used to have www.switch.com [1]) was sold to Ansaldo and then to Hitachi, and General Railway Signal was acquired by Alstom. Alstom continues to make some GRS relays.[2] But not the whole line of them. Ansaldo offers a remanufacturing service for the older components.
The New York City system uses mostly General Railway Signal components, while Boston uses US&S. NYC had to rebuild old relays after the big flood a few years ago, because all available spare stocks, both theirs and the manufacturers', had been exhausted. But they were able to get new ones from Alstom over time. Boston seems to be having more of a parts problem.
These relay-based systems are designed to fail safe, in a very literal way. "Vital" relays (the ones in safety-critical circuits) open by gravity, and the open direction is always towards a red signal. Broken wires always force things towards a safer condition. This is not a property of computer-based systems. There are modern replacements, such as Ansaldo MicroLok II, which are programmable logic controllers which emulate relay systems.[5] These, unfortunately, can be Internet-connected.[6]
Here's a simulator for a GRS relay-based signaling system, down to the relay level.[7]
[1] https://web.archive.org/web/20080103132002/http://www.switch... [2] http://alstomsignalingsolutions.com/OurProducts/WaysideProdu... [3] http://www.rrsignalpix.com/grs_relays.html [4] http://www.ansaldo-sts.com/en/about-us/ansaldo-around-world/... [5] http://www.ansaldo-sts.com/sites/ansaldosts.message-asp.com/... [6] http://www.westermo.com/web/web_en_idc_com.nsf/AllDocuments/... [7] http://www.nycsubway.org/wiki/NXSYS,_Signalling_and_Interloc...
The New York City system uses mostly General Railway Signal components, while Boston uses US&S. NYC had to rebuild old relays after the big flood a few years ago, because all available spare stocks, both theirs and the manufacturers', had been exhausted. But they were able to get new ones from Alstom over time. Boston seems to be having more of a parts problem.
These relay-based systems are designed to fail safe, in a very literal way. "Vital" relays (the ones in safety-critical circuits) open by gravity, and the open direction is always towards a red signal. Broken wires always force things towards a safer condition. This is not a property of computer-based systems. There are modern replacements, such as Ansaldo MicroLok II, which are programmable logic controllers which emulate relay systems.[5] These, unfortunately, can be Internet-connected.[6]
Here's a simulator for a GRS relay-based signaling system, down to the relay level.[7]
[1] https://web.archive.org/web/20080103132002/http://www.switch... [2] http://alstomsignalingsolutions.com/OurProducts/WaysideProdu... [3] http://www.rrsignalpix.com/grs_relays.html [4] http://www.ansaldo-sts.com/en/about-us/ansaldo-around-world/... [5] http://www.ansaldo-sts.com/sites/ansaldosts.message-asp.com/... [6] http://www.westermo.com/web/web_en_idc_com.nsf/AllDocuments/... [7] http://www.nycsubway.org/wiki/NXSYS,_Signalling_and_Interloc...
They go into a lot of detail on how the past-useful-live infrastructure works, modern replacement technologies, and even a race between infrastructure-giant Siemans and a bunch of hackers kludging together a countdown clock:
http://www.theatlantic.com/technology/archive/2015/11/why-do...
A more interesting read than this blog post.
----------
On a different note, these "they don't make these anymore" problems make me wonder if maybe we should be pushing government to look for open-source hardware for these types of infrastructure work.
When you have equipment lifetimes of (officially) 35 years (realistically up to 100 years), can you really rely on the same Sieman's division to still be in business supporting it? In this Boston story, I wonder how much of their repair workshop is reverse-engineering some hardware whos blueprints long ago decayed in some rusting factory.
Given the right open-source hardware, could you make "they don't make these anymore" obsolete because the technology has become accessible enough that you could build it yourself? (And in the process employing "New Economy" type high-tech jobs). More importantly, should a public entity push for making it's infrastructure publicly accessible? (Pragmatically, could you find someone to build it under those terms?)