UK legal system assumes that computers don't have bugs(postofficetrial.com)
postofficetrial.com
UK legal system assumes that computers don't have bugs
https://www.postofficetrial.com/2020/12/horizon-trial-judgment-could-force.html
3 comments
EDIT. Here's a useful radio programme that goes through what happened: https://www.bbc.co.uk/programmes/m000jf7j
Bob works as a postmaster running a small local post office. Bob must use software provided to him by The Post Office. That software is faulty. One of the errors is that it reports more money present in Bob's office than there actually is. Bob contacts his superiors who tell him that no-one else is having a problem (this is a lie, other people are having the same problem and they know this) and that the software is correct. They give Bob a choice: supply the missing funds or deal with the police. Bob is acting in good faith and he assumes this is a simple mistake that'll get sorted out, so he supplies the missing money from his own funds, and keeps telling his superiors to fix it. Eventually, Bob runs out of money (or, maybe, he says "get the police involved".) So, the police arrest Bob (because we have a low threshold for arrest over here) and interview Bob under caution. Police are told by Post Office that the software is reliable, and that Bob is lying. Police aggressively question Bob. It goes to court. Bob is convicted.
Because this is what happened to hundreds of people.
What's Bob supposed to do? Ask the Post Office to supply the source code so he can have a look for bugs?
Bob works as a postmaster running a small local post office. Bob must use software provided to him by The Post Office. That software is faulty. One of the errors is that it reports more money present in Bob's office than there actually is. Bob contacts his superiors who tell him that no-one else is having a problem (this is a lie, other people are having the same problem and they know this) and that the software is correct. They give Bob a choice: supply the missing funds or deal with the police. Bob is acting in good faith and he assumes this is a simple mistake that'll get sorted out, so he supplies the missing money from his own funds, and keeps telling his superiors to fix it. Eventually, Bob runs out of money (or, maybe, he says "get the police involved".) So, the police arrest Bob (because we have a low threshold for arrest over here) and interview Bob under caution. Police are told by Post Office that the software is reliable, and that Bob is lying. Police aggressively question Bob. It goes to court. Bob is convicted.
Because this is what happened to hundreds of people.
What's Bob supposed to do? Ask the Post Office to supply the source code so he can have a look for bugs?
I don't see the evidence against Bob. If I were Bob, my defence would be along the lines of 'The software says there was $x in the office. Then surely all transactions must add up to this number. Your allegations are baseless without the transaction data.'.
Surely someone made this argument. Was the response 'This aggregate value proves you wrong. We may not know where it came from, but it is all the evidence we need.'?
Surely someone made this argument. Was the response 'This aggregate value proves you wrong. We may not know where it came from, but it is all the evidence we need.'?
> I don't see the evidence against Bob. If I were Bob, my defence would be along the lines of 'The software says there was $x in the office. Then surely all transactions must add up to this number. Your allegations are baseless without the transaction data.'.
Presumably, they have the transaction data from the software, and the transaction data is saying he's lying. Software bugs are perfectly capable of creating bad transactions (i.e. by converting currencies incorrectly, double-submitting transactions, failing to actual cancel orders if the user cancels, etc).
In those cases, the presumption is that the software is working correctly. Those transactions are deemed legitimate because the software is presumed to be working correctly. That is the evidence against Bob; an "infallible" transaction registry that says he's embezzling.
If you are Bob, you have to prove that those transactions aren't legitimate, and never actually occurred. Or that they did occur, but the actual amount transacted was less than what the computer recorded. In essence, Bob is fucked, because he has to prove that something didn't happen. Probably something that he wasn't even there for (since presumably other employees would have handled the actual transaction).
Presumably, they have the transaction data from the software, and the transaction data is saying he's lying. Software bugs are perfectly capable of creating bad transactions (i.e. by converting currencies incorrectly, double-submitting transactions, failing to actual cancel orders if the user cancels, etc).
In those cases, the presumption is that the software is working correctly. Those transactions are deemed legitimate because the software is presumed to be working correctly. That is the evidence against Bob; an "infallible" transaction registry that says he's embezzling.
If you are Bob, you have to prove that those transactions aren't legitimate, and never actually occurred. Or that they did occur, but the actual amount transacted was less than what the computer recorded. In essence, Bob is fucked, because he has to prove that something didn't happen. Probably something that he wasn't even there for (since presumably other employees would have handled the actual transaction).
> Presumably, they have the transaction data from the software, and the transaction data is saying he's lying.
No, the error seems to be just in the aggregates:
> We went through the system together, and found nothing. The other weird thing is that her stock unit basically almost never had that amount of cash in it - for it to be that short, there had to be a random high value transaction put through by mistake. We searched, and found nothing. - https://www.postofficetrial.com/2019/05/the-post-office-clai...
No, the error seems to be just in the aggregates:
> We went through the system together, and found nothing. The other weird thing is that her stock unit basically almost never had that amount of cash in it - for it to be that short, there had to be a random high value transaction put through by mistake. We searched, and found nothing. - https://www.postofficetrial.com/2019/05/the-post-office-clai...
Either this is a massive failure on the government's part, or something else is going on. I can't believe that the ledger doesn't match the balance, and that that still doesn't count as proof that there's a bug.
They had recorded transactions happening at times when the branch was closed and noone could possibly be at the terminal, yet the guy was still sent to prison
Folks from Fujitsu, creators of the system, claimed i court that the system was flawless, and, apparently, were taken at their word.
Folks from Fujitsu, creators of the system, claimed i court that the system was flawless, and, apparently, were taken at their word.
"(this is a lie, other people are having the same problem and they know this)" So this is fraud, and makes the validate nature of a program immaterial. If they are willing to lie about this, they can produce false logs that show their system is working.
But if Bob, kept records, of his transaction, and could show an irregularity between the his records and theirs this could be used as evidence of a faulty system an warrant further investigation.
But you listed another issue that the law wouldn't help, the police don't believe Bob, they don't investigate, The courts don't believe bob.
I don't see how you think this different finding would change that.
In almost any case, if the cops don't believe you, and the court doesn't believe you, you are going to get screwed.
But if Bob, kept records, of his transaction, and could show an irregularity between the his records and theirs this could be used as evidence of a faulty system an warrant further investigation.
But you listed another issue that the law wouldn't help, the police don't believe Bob, they don't investigate, The courts don't believe bob.
I don't see how you think this different finding would change that.
In almost any case, if the cops don't believe you, and the court doesn't believe you, you are going to get screwed.
If Bob was a fraudster, any records kept by Bob would also be fraudulets, so he can't use them as evidence.
He would have to produce material proof of a kind he could not easilly forge - maybe hours of CCTV footage poiting at the till?
He would have to produce material proof of a kind he could not easilly forge - maybe hours of CCTV footage poiting at the till?
> If Bob was a fraudster, any records kept by Bob would also be fraudulets, so he can't use them as evidence.
Would that really be true? In a trial, I would expect part of the proceedings to include arguing to the jury that these records or those records are more reliable. Just because Bob is accused of fraud shouldn't automatically make his records inadmissable...
Would that really be true? In a trial, I would expect part of the proceedings to include arguing to the jury that these records or those records are more reliable. Just because Bob is accused of fraud shouldn't automatically make his records inadmissable...
Well they could likely be admitted, depending on the context that they were admitted, this is where shit gets complex in legal matters, if there is a legal expert they maybe able to field this better
I probably shouldn't have said the word evidence, but I didn't mean as evidence as a part of his defense, but to justify further investigation.
They don't even need to produce false records/logs if the courts believe that any computer system is working as intended without evidence that it isn't then they don't need to produce any records as the court will already accept their computers result as correct.
previously they would have, which was the point I was going after, its moot.
additionally, if bob kept a personal leisure that didn't not reconcile with their system, or if bob reached out to other branches and had others make the same complaint, both of those could be used to cause the company to defend their system.
Again, if bob is believed by the courts.
additionally, if bob kept a personal leisure that didn't not reconcile with their system, or if bob reached out to other branches and had others make the same complaint, both of those could be used to cause the company to defend their system.
Again, if bob is believed by the courts.
Er, have you ever used a computer system, or written a computer program?
Their position was that you don’t need to check the calculations, don’t need to show why it produced the results it did, you just assume it’s always correct. I work in the finance industry myself and have supported financial applications in banks and exchanges and the way these cases were handled is unbelievable. We’d never make assumptions about our systems like this, it was grossly negligent and incompetent and the courts need very clear guidance on how to do better.
Their position was that you don’t need to check the calculations, don’t need to show why it produced the results it did, you just assume it’s always correct. I work in the finance industry myself and have supported financial applications in banks and exchanges and the way these cases were handled is unbelievable. We’d never make assumptions about our systems like this, it was grossly negligent and incompetent and the courts need very clear guidance on how to do better.
The Previously the legal position was
"In any proceedings, a statement in a document produced by a computer shall not be admissible as evidence of any fact stated therein unless it is shown. That at all material times the computer was operating properly, or if not, that any respect in which it was not operating properly or was out of operation was not such as to affect the production of the document or the accuracy of its contents."
they have now changed the position to
"in the absence of evidence to the contrary, the courts will presume that mechanical instruments were in order at the material time... The principle has been applied to such devices as speedometers and traffic lights and in the consultation paper we saw no reason why it should not apply to computers."
I have no earthly idea, where the assumption about not checking calculations, or "don’t need to show why it produced the results it did." comes from or how you arrived at that conclusion.
your logical seems as daft as asking if I've ever used a computer system, on a computer system.
"In any proceedings, a statement in a document produced by a computer shall not be admissible as evidence of any fact stated therein unless it is shown. That at all material times the computer was operating properly, or if not, that any respect in which it was not operating properly or was out of operation was not such as to affect the production of the document or the accuracy of its contents."
they have now changed the position to
"in the absence of evidence to the contrary, the courts will presume that mechanical instruments were in order at the material time... The principle has been applied to such devices as speedometers and traffic lights and in the consultation paper we saw no reason why it should not apply to computers."
I have no earthly idea, where the assumption about not checking calculations, or "don’t need to show why it produced the results it did." comes from or how you arrived at that conclusion.
your logical seems as daft as asking if I've ever used a computer system, on a computer system.
phaemon(1)
It's perfectly reasonable to assume software generally works, because it generally does, and if it didn't we couldn't run modern society on software like we do right now. It's the proper default presumption to make.
But the evidence to the contrary could have been something as simple as showing patch notes addressing a bug, or online complaints about that bug, which could have affected the software in a way that gave rise to the circumstances of the lawsuit.
But the evidence to the contrary could have been something as simple as showing patch notes addressing a bug, or online complaints about that bug, which could have affected the software in a way that gave rise to the circumstances of the lawsuit.
In one case? Nothing.
But if you get 100s of cases and it appears that ever user of this software without failure is stealing despite no other evidence and all of them protesting their innocence, then you might wanna reconsider.
But if you get 100s of cases and it appears that ever user of this software without failure is stealing despite no other evidence and all of them protesting their innocence, then you might wanna reconsider.
Babbage supposedly said something like "I cannot rightly understand how this question came to be, a politician asked me once: "If one enters the wrong information into the engine, will it still provide a correct answer?". "
Why not make a compromise?
Assume that software has no bugs if there is a publically available machine-checkable proof of the program correctness.
Assume that software has no bugs if there is a publically available machine-checkable proof of the program correctness.
Because they would put an insane unreasonable burden on everyone who has a computer. and basically make everyone but multi-billion companies have to assume their systems are at fault.
Do you see the irony in your reply or would you like me to explain it?
"in the absence of evidence to the contrary, the courts will presume that mechanical instruments were in order at the material time... The principle has been applied to such devices as speedometers and traffic lights and in the consultation paper we saw no reason why it should not apply to computers."