UK judge says ‘freedom of information’ means choice of digital file format(washingtonpost.com)
washingtonpost.com
UK judge says ‘freedom of information’ means choice of digital file format
http://www.washingtonpost.com/blogs/the-switch/wp/2014/08/05/uk-judge-says-freedom-of-information-means-choice-of-digital-file-format/
6 comments
I can't think of anything more obtuse than releasing a database as a hundred plus page PDF. It seems like the bureaucrats responding to these requests are specifically trained to not release information.
>'It seems like the bureaucrats responding to these requests are specifically trained to not release information.'
They are.
I have witnessed someone reprimanded specifically for 'making it too easy' by providing an Excel sheet instead of printing the same to a PDF.
They are.
I have witnessed someone reprimanded specifically for 'making it too easy' by providing an Excel sheet instead of printing the same to a PDF.
What can be done about this? Can we call our congressmen or something?
This is about the UK, so you should probably call your MP instead.
They want to minimize the amount of time they spend on the FOIA requests, because it comes out of their operating budget. So they select the releasable records and then print them to pdf, where they can do any necessary redaction before release.
Exporting them into another database where they can be manipulated freely is beyond the skills of most database users, and requires an expensive database administrator, who will have to have meetings with the administrator of the original database, which might result in travel requests, expenses...at which point most organizations will invoke the clause in FOIA rules (at least in America) of charging the requestor the reasonable costs of the data retrieval, which in a case like this will be substantial - at least a few thousand. Bear in mind that that FOIA requests are rarely so broad in scope as to justify the release of an entire database; most requests are for a subset of information about particular individuals, timeframes, political administrations or suchlike.
I often dig through stuff like this in PDF format (because I enjoy reading court cases), and while I can't say I like it, if I really need to get the data out I set up some sort of scraping macro. Most people want something in human readable form and I suspect that if they were given an optical disc with an SQL dtabase on it, or even an XML file, they'd complain about the impenetrability of it.
Perhaps you could suggest some good candidates for release formats that met the criteria of being easily accessible/convertible as well as being manipulable and easy to produce.
Exporting them into another database where they can be manipulated freely is beyond the skills of most database users, and requires an expensive database administrator, who will have to have meetings with the administrator of the original database, which might result in travel requests, expenses...at which point most organizations will invoke the clause in FOIA rules (at least in America) of charging the requestor the reasonable costs of the data retrieval, which in a case like this will be substantial - at least a few thousand. Bear in mind that that FOIA requests are rarely so broad in scope as to justify the release of an entire database; most requests are for a subset of information about particular individuals, timeframes, political administrations or suchlike.
I often dig through stuff like this in PDF format (because I enjoy reading court cases), and while I can't say I like it, if I really need to get the data out I set up some sort of scraping macro. Most people want something in human readable form and I suspect that if they were given an optical disc with an SQL dtabase on it, or even an XML file, they'd complain about the impenetrability of it.
Perhaps you could suggest some good candidates for release formats that met the criteria of being easily accessible/convertible as well as being manipulable and easy to produce.
> Perhaps you could suggest some good candidates for release formats that met the criteria of being easily accessible/convertible as well as being manipulable and easy to produce.
The entire case here was around the requestor specifying he'd like the data in Excel format (although presumably CSV would also be acceptable). I can't think of many databases which can't export query results to CSV, or indeed many civil servants who are so completely inept with a computer that they couldn't redact information as (or more) easily in Excel than they could with a PDF.
The entire case here was around the requestor specifying he'd like the data in Excel format (although presumably CSV would also be acceptable). I can't think of many databases which can't export query results to CSV, or indeed many civil servants who are so completely inept with a computer that they couldn't redact information as (or more) easily in Excel than they could with a PDF.
>'I can't think of many databases which can't export query results to CSV...'
This is pretty much irrelevant as these requests are not going to end up in the hands of a DBA, or anyone with anything approaching direct access to the databases in question.
It's going to end up with someone who has an adversarial relationship with their own IT department as a 'maker of work'. Bureaucracy isn't something these agencies foist on the public, it just as bad if not worse inside.
The request will go up and down through a series of managers, directors and deputies before being put on the schedule for a change control board meeting.
In that meeting, a representative of the database administrators will explain that this request is not only complex, but potentially quite dangerous. He will point out that required data is actually completely accessible via a clunky mainframe interface wrapped in Java.
Some plebe in the receiving office of the request will end up clicking through that Java interface and printing/exporting the requested data one page at a time.
This process will repeat itself until such requests are a weekly if not daily occurrence at which point the plebe might be given a dedicated interface for exporting such requests.
>'...or indeed many civil servants who are so completely inept with a computer that they couldn't redact information as (or more) easily in Excel than they could with a PDF.'
I take it you've not met many civil servants then?
In any case, the mechanics aren't really the issue. It's the game of hot potato behind the scenes and the eschewing of responsibility that takes place at all levels.
I can virtually guarantee that even if all the stars aligned such that someone did obtain and redact a fully electronic copy of such data it would be rejected for release due to the 'unfamiliar' format. Someone in the chain would fear that they couldn't be sure of what they were releasing because it's not in the specific format they're used to.
This is pretty much irrelevant as these requests are not going to end up in the hands of a DBA, or anyone with anything approaching direct access to the databases in question.
It's going to end up with someone who has an adversarial relationship with their own IT department as a 'maker of work'. Bureaucracy isn't something these agencies foist on the public, it just as bad if not worse inside.
The request will go up and down through a series of managers, directors and deputies before being put on the schedule for a change control board meeting.
In that meeting, a representative of the database administrators will explain that this request is not only complex, but potentially quite dangerous. He will point out that required data is actually completely accessible via a clunky mainframe interface wrapped in Java.
Some plebe in the receiving office of the request will end up clicking through that Java interface and printing/exporting the requested data one page at a time.
This process will repeat itself until such requests are a weekly if not daily occurrence at which point the plebe might be given a dedicated interface for exporting such requests.
>'...or indeed many civil servants who are so completely inept with a computer that they couldn't redact information as (or more) easily in Excel than they could with a PDF.'
I take it you've not met many civil servants then?
In any case, the mechanics aren't really the issue. It's the game of hot potato behind the scenes and the eschewing of responsibility that takes place at all levels.
I can virtually guarantee that even if all the stars aligned such that someone did obtain and redact a fully electronic copy of such data it would be rejected for release due to the 'unfamiliar' format. Someone in the chain would fear that they couldn't be sure of what they were releasing because it's not in the specific format they're used to.
I know many civil servants. The ones with access to run queries on a database know how to export to Excel. If anything, there's an over reliance on Excel.
[deleted]
You nailed it. This is exactly the sort of crap I had to deal with when I was doing public sector contract work.
If you consider perhaps maybe 3-5% of staff are responsible and competent as well and they are bashed down the ladder by the incompetents as they are seen as a threat then you will see how this sort of stuff persists. The incompetents also persist through incestuous promotions (check on your council register how many people in the same family work in the same chain of command) and arse licking.
Most of these departments need the chain pulling like a toilet.
I've never encountered anything quite like it in the private sector apart from one large defence contractor who employed most of the surrounding area for ten miles.
If you consider perhaps maybe 3-5% of staff are responsible and competent as well and they are bashed down the ladder by the incompetents as they are seen as a threat then you will see how this sort of stuff persists. The incompetents also persist through incestuous promotions (check on your council register how many people in the same family work in the same chain of command) and arse licking.
Most of these departments need the chain pulling like a toilet.
I've never encountered anything quite like it in the private sector apart from one large defence contractor who employed most of the surrounding area for ten miles.
I don't think anyone is complaining about being charged reasonable rates for converting data to the requested format.
But I really don't think "run a query and export the result" requires the skills and expenses you describe. It wouldn't be in a database in the first place if the government didn't at least occasionally have to run their own queries against it.
But I really don't think "run a query and export the result" requires the skills and expenses you describe. It wouldn't be in a database in the first place if the government didn't at least occasionally have to run their own queries against it.
I can imagine something even more obtuse: have the pdfs contain bitmap renderings of the pages, so you would need not to parse them, but OCR them...
Unfortunately, this is not my wild imagination, but this happened in this reality. Sad, I know.
Unfortunately, this is not my wild imagination, but this happened in this reality. Sad, I know.
A classic practice is to archive pdf's of printed and scanned documents, instead the source digital documents - based on the concept that you need to have any signatures, approval stamps or notes/amendments as they might be on the actual document.
So instead of an excel worksheet you get a scanned picture that's possibly even unusable for OCR if someone has a wide signature and some line(s) of it go inside the actual table.
So instead of an excel worksheet you get a scanned picture that's possibly even unusable for OCR if someone has a wide signature and some line(s) of it go inside the actual table.
I requested all my data from a department of my local council.
Someone print-screened all the screens of their database with my information. I had several hundred pages of badly printed black and white pages of generic enterprise database screens.
> It seems like the bureaucrats responding to these requests are specifically trained to not release information.
I kind of wish that were true, because maliciousness is somehow more comforting than incompetence.
Someone print-screened all the screens of their database with my information. I had several hundred pages of badly printed black and white pages of generic enterprise database screens.
> It seems like the bureaucrats responding to these requests are specifically trained to not release information.
I kind of wish that were true, because maliciousness is somehow more comforting than incompetence.
FWIW, in some US states you can request documents be provided in electronic formats. The Federal FOIA law is fairly weak.
In the 1980s they did source escrow on paper. This meant printing out several boxes of fanfold paper for a medium sized project and shipping them to the escrow company. Imagine typing all that back in...
I'm sure there are political reasons, but it is very hard for me to understand why agencies purposefully keep the data obscure.
Opening data stimulates innovation around that data. It benefits society by allowing analysis to provide value, and improves the economy through the sale of related products.
Every time I see things like this, I cannot help but hope that if not my children's generation, at least my grandchildren's will be one that takes for granted accessible free information from public agencies. So many good things would come from old world beaurocrats relinquishing their stranglehold on data that is payed for by the people who need it the most.
Opening data stimulates innovation around that data. It benefits society by allowing analysis to provide value, and improves the economy through the sale of related products.
Every time I see things like this, I cannot help but hope that if not my children's generation, at least my grandchildren's will be one that takes for granted accessible free information from public agencies. So many good things would come from old world beaurocrats relinquishing their stranglehold on data that is payed for by the people who need it the most.
>'I'm sure there are political reasons, but it is very hard for me to understand why agencies purposefully keep the data obscure.'
I can understand / attest to both of the following stances by experience:
Thinking idealistically, it's a basic deterrent against lazy, disingenuous, 'data out of context' arguments. By keeping that data obscure you're building in a selection process. Requiring potential detractors to put forth even the small effort of cutting, pasting and sorting will deter many before they begin.
Thinking cynically, it's a basic deterrent against uncovering widespread lies and corruption. Keeping the data obscure will encourage the people who might uncover your schemes to look for easier, juicier targets.
>'Opening data stimulates innovation around that data.'
It's an enabler sure, but I'm not so sure about a stimulator.
As best I can tell, most FOIA requests primarily serve the lawyers hired by various agencies to sue each other or the Government in the process of contract grievances.
>'It benefits society by allowing analysis to provide value, and improves the economy through the sale of related products.'
It certainly could, but it's easy to forget that many people working in Government are just as self-interested as anyone else. They're no more interested in the big picture than the next man. More importantly, no one below the level of perhaps a local Mayor and his/her lieutenants have their jobs tied to economic performance or general well-being of their constituents.
I can understand / attest to both of the following stances by experience:
Thinking idealistically, it's a basic deterrent against lazy, disingenuous, 'data out of context' arguments. By keeping that data obscure you're building in a selection process. Requiring potential detractors to put forth even the small effort of cutting, pasting and sorting will deter many before they begin.
Thinking cynically, it's a basic deterrent against uncovering widespread lies and corruption. Keeping the data obscure will encourage the people who might uncover your schemes to look for easier, juicier targets.
>'Opening data stimulates innovation around that data.'
It's an enabler sure, but I'm not so sure about a stimulator.
As best I can tell, most FOIA requests primarily serve the lawyers hired by various agencies to sue each other or the Government in the process of contract grievances.
>'It benefits society by allowing analysis to provide value, and improves the economy through the sale of related products.'
It certainly could, but it's easy to forget that many people working in Government are just as self-interested as anyone else. They're no more interested in the big picture than the next man. More importantly, no one below the level of perhaps a local Mayor and his/her lieutenants have their jobs tied to economic performance or general well-being of their constituents.
>I'm sure there are political reasons, but it is very hard for me to understand why agencies purposefully keep the data obscure.
It's simple, really. There is no value to that individual or agency in handing out nicely formatted data to someone who might (probably will, if they're using FOIA) try to use it against them.
Very, very few people truly, through their actions, care about benefiting society. Most care only about benefiting themselves (which, I mean, is relatively understandable, in an evolutionary sense). And those individuals care so much about themselves that they often do things that they know are "wrong," or even blatantly illegal, and they obviously don't want to just open those records up to the analysts for the greater good.
It's simple, really. There is no value to that individual or agency in handing out nicely formatted data to someone who might (probably will, if they're using FOIA) try to use it against them.
Very, very few people truly, through their actions, care about benefiting society. Most care only about benefiting themselves (which, I mean, is relatively understandable, in an evolutionary sense). And those individuals care so much about themselves that they often do things that they know are "wrong," or even blatantly illegal, and they obviously don't want to just open those records up to the analysts for the greater good.
"Open" is still a very new concept in government. While words like "oversight" and "transparency" have floated around parliament, there was still very little precedence previously and old, ingrained, attitudes and methodologies take a while to change. Systems and minds need quite a bit of rewiring for this to work.
The old adage "Knowledge is power" would appear to be relevant.
Covering their asses.
My general impression, having seen the 'inside' of such processes, is that everyone would be better served by mandates around open data than continued interpretations and amendments around FOIA.
1.) The most common sort of requests (salary schedules, the outcomes of past competitive bid processes) are for information that could/should simply be public in the first place.
All this stuff can simply be published as part of a normal workflow rather than manually compiled on a per request basis.
2.) Compliance could be checked trivially via API rather than having to wait for failures to bubble up over time and potentially raise scandal.
3.) Proper funding could be allocated and structure put in place to build and maintain these services. As it is, there's a good chance any given request lands on the desk of some plebe who lacks the knowledge, tools and authority to handle the request correctly or efficiently.
1.) The most common sort of requests (salary schedules, the outcomes of past competitive bid processes) are for information that could/should simply be public in the first place.
All this stuff can simply be published as part of a normal workflow rather than manually compiled on a per request basis.
2.) Compliance could be checked trivially via API rather than having to wait for failures to bubble up over time and potentially raise scandal.
3.) Proper funding could be allocated and structure put in place to build and maintain these services. As it is, there's a good chance any given request lands on the desk of some plebe who lacks the knowledge, tools and authority to handle the request correctly or efficiently.
Well that will just result in someone manually cutting and pasting stuff into excel and exporting a CSV. We are doomed to live the following by specifying it:
http://wyorock.com/excelasadatabase.htm
This is due to government assignment policy which is delegate it until there is no one left to delegate it further resulting in the lowest paid and skilled workers performing the above.
http://wyorock.com/excelasadatabase.htm
This is due to government assignment policy which is delegate it until there is no one left to delegate it further resulting in the lowest paid and skilled workers performing the above.
I don't think that's peculiar to government - sounds like any other large organization with a hierarchical structure.
I've worked in various capacities in government and non-government organisations. It is much worse in government than not from experience because they are cesspools for the incompetent at best. In a private sector company people are considerably more aware of their standing and not heavily unionised into artificially secure positions.
I can see this backfiring if someone requests data in some ancient and obscure format just to be annoying.
Data needs to be provided in a reasonable and usable format; that imposes some responsibility on both the provider and the requester.
Data needs to be provided in a reasonable and usable format; that imposes some responsibility on both the provider and the requester.
The FOI laws in the UK contain a reasonableness hurdle:
http://ico.org.uk/for_organisations/freedom_of_information/guide/refusing_a_request
Would there be anything to prevent someone from creating a one off file format that requires a license fee and just start making FOIA requests in that format. If not, this seems like something that could be exploited for profit.
Don't FOIA requests usually require you to pay reasonable fees? I know they're not free....
Recently I was trying to get hold of local plan housing allocation which detail where new building is likely to be. The local authority would only give me PDFs of maps with hundreds of insets which obfuscate everything. With a proper digital format national mapping would be possible. The public could easily find out where new developments are going, something that is very hard to do now.