Medical devices: A ticking time-bomb(economist.com)
economist.com
Medical devices: A ticking time-bomb
http://www.economist.com/blogs/babbage/2012/05/medical-devices
8 comments
The research tested 1700 devices. A single device that is off by 42 years (reset to Unix epoch?) would have skewed the average by over 9 days. So unless a number of other devices had similarly extreme errors in the opposite direction (and even if they did, I'm pretty sure the researcher would have only looked at the absolute magnitude of the error, not their direction), I think it would be safe to assume that extreme cases were excluded from the average.
Bnd you're right, 80% of them were within 30 minutes of the right time.
> "One in five were off by more than 30 minutes"
In fact, given that the article mentions daylight saving time as a major problem, I'm not surprised that the average error approaches half an hour. At any given time, a lot of machines would be running in the wrong timezone.
Bnd you're right, 80% of them were within 30 minutes of the right time.
> "One in five were off by more than 30 minutes"
In fact, given that the article mentions daylight saving time as a major problem, I'm not surprised that the average error approaches half an hour. At any given time, a lot of machines would be running in the wrong timezone.
Good point about the stats.
doctors are supposed to be able to reconcile conflicting information
They're supposed to wash their hands, too. (http://boingboing.net/2010/03/09/study-says-us-doctor.html) As a guy who makes dozens of mistakes a day, the ones I know about because a computer catches them, I'm really not reassured by the idea that human judgement makes up for machine error.
doctors are supposed to be able to reconcile conflicting information
They're supposed to wash their hands, too. (http://boingboing.net/2010/03/09/study-says-us-doctor.html) As a guy who makes dozens of mistakes a day, the ones I know about because a computer catches them, I'm really not reassured by the idea that human judgement makes up for machine error.
Average is a fine measure. You've inferred that they used a mean average. The article doesn't say what kind of average they took.
Let's assume they did take a mean. If the mean error for 1700 devices was 24 minutes, the total error is 40,800 minutes. That's 28 days.
Therefore, they didn't include the outlier of 42 years in their average. It's most likely a median (or mode) average.
Let's assume they did take a mean. If the mean error for 1700 devices was 24 minutes, the total error is 40,800 minutes. That's 28 days.
Therefore, they didn't include the outlier of 42 years in their average. It's most likely a median (or mode) average.
It's pretty unlikely to be a median; if 20% of machines are 30 minutes off and 24 minutes is a median, then 30% of machines are between 24-30 minutes off, which would seem to be a strangely narrow band.
More likely, the 24 minutes is a mean average after excluding outliers. One of the standard ways to compute outliers is to measure the inter-quartile range---the "distance" between the 25th percentile value and the 75th percentile value---and call everything that is more than 1.5*IQR past the 75th percentile an outlier. You can do this without computing a mean first, and you can use it to exclude values before computing a modified mean.
More likely, the 24 minutes is a mean average after excluding outliers. One of the standard ways to compute outliers is to measure the inter-quartile range---the "distance" between the 25th percentile value and the 75th percentile value---and call everything that is more than 1.5*IQR past the 75th percentile an outlier. You can do this without computing a mean first, and you can use it to exclude values before computing a modified mean.
[deleted]
Time errors between ten minutes and an hour are probably the most dangerous, so most devices being less than half an hour off is barely a comfort.
NTP is great and all, but...
When you're timing something that needs accuracy, you use a timer not a clock. If you need to use a clock, you use one clock not all the clocks. Yes, I work in the medical industry. Yes, we generally use timers not clocks. Yes, our clocks are all wrong too.
If you really do need a solution to the "time" problem (accurate time all the time), you need to issue everyone a smartphone.
When you're timing something that needs accuracy, you use a timer not a clock. If you need to use a clock, you use one clock not all the clocks. Yes, I work in the medical industry. Yes, we generally use timers not clocks. Yes, our clocks are all wrong too.
If you really do need a solution to the "time" problem (accurate time all the time), you need to issue everyone a smartphone.
im a doctor also and the need for highly accurate timing is exceedingly rare.
A lot of medical timing is guess work - why are antibiotics commonly prescribed for 5, 7, 10 or 14 days? Whynot 6?
Automatic filing of EMR information would be important and the legal aspect would unfortunately demanc the greatest amount of accuracy.
Surely connected devices could use one centrally calculated timesource?
A lot of medical timing is guess work - why are antibiotics commonly prescribed for 5, 7, 10 or 14 days? Whynot 6?
Automatic filing of EMR information would be important and the legal aspect would unfortunately demanc the greatest amount of accuracy.
Surely connected devices could use one centrally calculated timesource?
> "Surely connected devices could use one centrally calculated timesource?"
... like NTP?
Sorry, but I'm not sure what point you are trying to make here. Anything with a computer, even a microcontroller, and basic network connectivity can use NTP, it is an extremely simple protocol.
Anything that doesn't connect to a network could be kept in sync via a wireless pulse/tick, though then you have security issues unless you cryptographically sign the clock signal. Though I'm not sure why anybody would try and screw up a clock signal like that other than general asshattedness.
That aside, I'm unsure why they are using local time for logging data anyhow. If the folks managing the hospital were rational, shouldn't there be a regulation to use GMT and 24-hour clock? My home town is right on the Eastern/Central timezone split, and I know it causes all kinds of problems in day-to-day life. I just hate that kind of ambiguity.
... like NTP?
Sorry, but I'm not sure what point you are trying to make here. Anything with a computer, even a microcontroller, and basic network connectivity can use NTP, it is an extremely simple protocol.
Anything that doesn't connect to a network could be kept in sync via a wireless pulse/tick, though then you have security issues unless you cryptographically sign the clock signal. Though I'm not sure why anybody would try and screw up a clock signal like that other than general asshattedness.
That aside, I'm unsure why they are using local time for logging data anyhow. If the folks managing the hospital were rational, shouldn't there be a regulation to use GMT and 24-hour clock? My home town is right on the Eastern/Central timezone split, and I know it causes all kinds of problems in day-to-day life. I just hate that kind of ambiguity.
> If the folks managing the hospital were rational, shouldn't there be a regulation to use GMT and 24-hour clock?
People are not rational, that's the problem! Just look at any web app that can't get timezones right.
In addition, most Americans swear by EST, CST, PST, etc. and have no idea how many hours behind GMT they are. Record everything in GMT and soon there will be a similar article documenting how medical professionals routinely misinterpret GMT timestamps. Especially in March and November.
People are not rational, that's the problem! Just look at any web app that can't get timezones right.
In addition, most Americans swear by EST, CST, PST, etc. and have no idea how many hours behind GMT they are. Record everything in GMT and soon there will be a similar article documenting how medical professionals routinely misinterpret GMT timestamps. Especially in March and November.
When lives are at stake, people tend to be more amenable to changes like GMT and 24-hour clock. If the military can do it, a civilian organization with sufficient motivation (insurance terms, regulation, etc) should be able to make the same shift.
However, I was referring more toward the recording system than the staff. If the recording systems were all using unix time stamp with a GMT system clock synchronized over NTP, there would be no ambiguity as to when a record was added to the system. End user systems could be responsible for keeping track of what time zone the user is in, and could convert as necessary. The important thing is that the "One True Datum" have the appropriate stamp.
However, I was referring more toward the recording system than the staff. If the recording systems were all using unix time stamp with a GMT system clock synchronized over NTP, there would be no ambiguity as to when a record was added to the system. End user systems could be responsible for keeping track of what time zone the user is in, and could convert as necessary. The important thing is that the "One True Datum" have the appropriate stamp.
> End user systems could be responsible for keeping track of what time zone the user is in, and could convert as necessary.
That would require routine updates to the timezone database, which is a bit more complicated than NTP. (Yes, DST rules change all the time. The U.S. changed its rules just a few years ago.)
Hopefully, the rise of cheap and powerful ARM SoCs will make it impossible for device manufacturers to make excuses about not running a proper OS on their machines anymore. I would love to see more Linux and BSD on medical devices.
That would require routine updates to the timezone database, which is a bit more complicated than NTP. (Yes, DST rules change all the time. The U.S. changed its rules just a few years ago.)
Hopefully, the rise of cheap and powerful ARM SoCs will make it impossible for device manufacturers to make excuses about not running a proper OS on their machines anymore. I would love to see more Linux and BSD on medical devices.
Except that my iphone is usually off by a few minutes. I don't think Apple got their NTP implementation or usage quite right.
Most mobile phones usually get their time from the phone towers. Of course, this relies on them having the right time.
Interestingly the most effective way to get syncronised clocks is GPS.
Interestingly the most effective way to get syncronised clocks is GPS.
I'm pretty sure cell towers all use gps for time synchronization, which is necessary for the network to work.
I always trust my cell phone time.
I always trust my cell phone time.
Cell towers need synchronized clocks? For what?
I'm pretty certain that CDMA towers, at least, had exceptionally accurate clocks for something related to the technology (synchronizing the chip code maybe? it's a guess).
Not sure if this applies to GSM as well, but CDMA used to be an acceptable clock source for a Statum 0 clock source.
Not sure if this applies to GSM as well, but CDMA used to be an acceptable clock source for a Statum 0 clock source.
You technically don't even need accurate time, you need consistent time. If all your clocks are consistently slow by 10 minutes, there's less of an issue then if some of the clocks are slow by 10 minutes.
I have two questions:
Since the EMR is updated constantly, data from devices
whose clocks are way off would simply never be recorded.
Others might bury current information in older files.
Worse, they may insert the data into the EMR when the
patient they concern has left the given clinical
environment and another has come in.
Daylight saving time corrections twice a year require
tedious manual tweaks that the MD PnP Program, an
initiative supporting networking standards for medical
devices, estimates cost American hospitals over $17m
annually. At Massachusetts General, a patient-monitoring
system deletes an hours' worth of data when rolling back
from 2am to 1am every autumn, while drug pumps are kept
permanently on standard time, so they are (at least) one
hour off for half the year.
Hi. Written EMRs and built medical devices (10-12 years exp between them). In many cases messages from devices and external systems are processed asynchronously, so there isn't much choice but to trust remote timestamps.. I may be getting data minutes or hours later. Just a distributed system challenge.
The time zone issue is a human one. Historically clinicians schedule doses at "Every 3 hours" or "Once per day, 8am". Humans figure out what that means, but a dose scheduling system that shifts 8am an hour after daylight savings (because it interpreted the schedule as "every 24 hours" is going to run aground with nurses when it tells them they're late). However, every 3 hours is pretty obviously an interval and likely shouldn't shift by 30% of the interval!
This isn't a distinction that clinicians are used to making, but as someone who had to write some of the first dose-scheduling systems focused on clinical accuracy, we ran into this challenge and did a lot of thinking about which schedules really meant 'intervals' of specific units of time and which ones meant 'due at these calendar times every day'.
It's an interesting example of a problem that introducing technology introduces/exposes while showing up to fix other things.
The time zone issue is a human one. Historically clinicians schedule doses at "Every 3 hours" or "Once per day, 8am". Humans figure out what that means, but a dose scheduling system that shifts 8am an hour after daylight savings (because it interpreted the schedule as "every 24 hours" is going to run aground with nurses when it tells them they're late). However, every 3 hours is pretty obviously an interval and likely shouldn't shift by 30% of the interval!
This isn't a distinction that clinicians are used to making, but as someone who had to write some of the first dose-scheduling systems focused on clinical accuracy, we ran into this challenge and did a lot of thinking about which schedules really meant 'intervals' of specific units of time and which ones meant 'due at these calendar times every day'.
It's an interesting example of a problem that introducing technology introduces/exposes while showing up to fix other things.
Hi. Written EMRs and built medical devices (10-12 years exp between them). In many cases messages from devices and external systems are processed asynchronously, so there isn't much choice but to trust remote timestamps.. I may be getting data minutes or hours later. Just a distributed system challenge.
So I'm guessing you might solve that by asking the device what its internal clock is at the time of synchronization, and adjust the logs by the offset between device time and EMR time (with manual device time changes included in the log)?
So I'm guessing you might solve that by asking the device what its internal clock is at the time of synchronization, and adjust the logs by the offset between device time and EMR time (with manual device time changes included in the log)?
Sure. If there was a 'time of synchronization'. There's generally not. The device often doesn't know when/if/who is consuming it's data. Likewise the EMR systems don't generally have any access to the device other than receiving a stream of messages some non-deterministic after they were produced.
As someone who works with devices which require good timekeeping, I'll add another question: if the devices are communicating with a central server, why doesn't that protocol include a time update from the server?
I wrote software for medical devices for 3 years. Saying 'just use NTP' is all well and good, except that getting Internet access inside of a hospital is usually impossible. So of course we offered a configurable setting so that they could use an internal or firewall-allowed server. I think one hospital took advantage of that setting. Everybody else was just wrong.
NTP settings can be distributed by DHCP, which in a managed networked environment should be reasonably trustable.
Unfortunately Windows ignores this, and also sets the system clock to a timezone-specific time, not UTC as nearly all other OS's do.
Unfortunately Windows ignores this, and also sets the system clock to a timezone-specific time, not UTC as nearly all other OS's do.
Configuring every single device to use NTP is a hassle. Do medical devices on an intranet use DHCP? DHCP is capable of handing out NTP server addresses, although I don't think it's very common. Another hacky option would be to override "pool.ntp.org" on the intranet's DNS server so it points to an internal server.
Well, if it could be regulated, you could require that the hospital put the NTP server or equivalent on IP over power line.
Not sure how old your experience is, but there's increasingly less friction here.
Previous submission, with a few comments:
http://news.ycombinator.com/item?id=4013021
I'm glad that there is some more discussion of this issue, as medical devices are some of the main high technology products in my part of the United States, and my son (also a Hacker News participant) worked as an intern at one of the medical device companies here last year. Medical device businesses are much more regulated as an industry than are most Web-based SaaS businesses.
http://news.ycombinator.com/item?id=4013021
I'm glad that there is some more discussion of this issue, as medical devices are some of the main high technology products in my part of the United States, and my son (also a Hacker News participant) worked as an intern at one of the medical device companies here last year. Medical device businesses are much more regulated as an industry than are most Web-based SaaS businesses.
I just hope this doesn't result in poorer security between outside networks (including the internet) and medical devices. I'd argue there should be an "air gap" between the two, but I'm probably more paranoid than most.
Certainly it's fine to use NTP, but I'd want to use some kind of GPS-receiver NTP appliances at each site (possibly redundantly).
Certainly it's fine to use NTP, but I'd want to use some kind of GPS-receiver NTP appliances at each site (possibly redundantly).
[deleted]
I'll bet that out of the clocks that doctors actually use to decide on treatment (tallying every device is a bit unfair; if a blood pressure monitor on some cart in a closet has the wrong time of day who cares) 80% of them are within a few minutes of the right time. And at the end of the day, doctors are supposed to be able to reconcile conflicting information, it's part of their training. This is a little bit like tallying up every textbook lying around a high school and getting concerned that the average book is three editions out of date and has 142.8 errors--calm down, most of those are not in active use and have some faith in teachers to check their material.