Archive for the ‘open data’ Category
Today, I submitted my response to the UK Cabinet Office’s Open Data Consultation,”Making Open Data Real”, and this is it.
I have been dealing at the sharp end of open data for a couple of years now, co-founding OpenCorporates and founding OpenlyLocal, both of which have massively increased the availability of company and UK local data respectively, and, I hope, in some tiny way have helped give the UK its worldwide reputation of leading the way in open data.
Through sitting on the Local Public Data Panel and countless other government programmes and meetings, I’ve also encountered local and central government bureaucracy in the raw. I’ve seen in detail how too often the bureaucracy subverts complex rules drawn up with the best of intentions to stifle innovation, exclude the most important ‘stakeholders’ of all (the people), and reward those behind big, multimillion-pound projects with promotion and further contracts.
All this experience has, I think, led me to a fairly comprehensive understanding of the issues, the blockages, the hype and the potential of open data. And it is with this understanding that I am responding to the consultation.
The truth is, like it or not, we now live in a ‘Big Data’ world, where our lives are not just governed by data but are data, from bank accounts to loyalty cards, smart phones to smart meters, televisions to travel cards. Even those who have never been on the internet are producing bucketfuls of data as they shop, watch, or catch the bus using free travel cards for the elderly and disabled.
Yet their access to data, both the data they produce and that is produced on their behalf by government and the public sector, is fundamentally restricted. Not only do they have no access to many of the datasets that affects their lives, those who are innovating to help them make sense of it are fatally hobbled by open access to the core public datasets which underly our modern world – for example, geographic data, company data, health data, and democratic & electoral data.
Public sector data is still being treated as an asset to be sold, rather than an underlying infrastructure of a modern democratic society, and with this approach people and the innovators who seek to empower them are marginalised and disenfranchised.
That is why the risk here is not of making changes, but of making no changes, and why what is needed is not a set of rules to be gamed and worked around by the existing ‘stakeholders’ (who after all have a stake in preserving their existing, out-of-date business models), but a core set of principles.
Open data is no silver bullet, and won’t on its own solve these problems, but it is an essential requirement for a ‘more open, more fair and more prosperous‘ society.
Fortunately the consultation provides such a set in Annex 2 of the consultation (The Public Sector Data Principles). These should be issued to every government department, quango, health authority and public sector body (including the PDC), with the order to follow them in letter and spirit. Backing these up, we also need an independent body needs to be appointed with the power and resources to enforce them. With these two things – good public principles, and an effective enforcer – we have a chance to achieve the innovation and fairer society we need.
Chris TaggartCEO & Co-Founder OpenCorporatesFounder OpenlyLocalMember of Local Public Data PanelMember of Mayor of London’s Digital Advisor
This is my presentation to the superb OKCON2011 conference in Berlin last week. It’s obviously openly licensed (CC-BY), so feel free to distribute widely. Comments also welcome.
When the amazing Emer Coleman first approached me a year and a half to get feedback on the plans for the London datastore, I told her that the gold standard for such datastores was that run by the District of Columbia, in the US. It wasn’t just the breadth of the data; it was that DC seemed to have integrated the principles of open data right into its very DNA.
And we had this commitment in mind that when we were thinking which were the US jurisdictions we’d scrape first for OpenCorporates, whose simple (but huge) goal is to build an open global database of every registered company in the world.
While there were no doubt many things that the DC company registry could be criticised for (perhaps it was difficult for the IT department to manage, or problematic for the company registry staff), for the visitors who wanted to get the information it worked pretty well.
What do I mean by worked well? Despite or perhaps because it was quite basic, it meant you could use any browser (or screenreader, for those with accessibility issues) to search for a company and to get the information about it.
It also had a simple, plain structure, with permanent URLs for each company, meaning search engines could easily find the data, so that if you search for a company name on Google there’s a pretty good chance you’ll get a link to the right page. This also means other websites can ‘deep-link’ to the specific company, and that links could be shared by people, in social networking, emails, whatever.
Finally, it meant that it was easy to get the information out of the register, by browsing or by scraping (we even used the scraper we wrote on ScraperWiki as an example of how to scrape a straightforward company register as part of our innovative bounty program).
It was, for the most part, what a public register should be, with the exception of providing a daily dump of the data under an open licence.
So it was a surprise a couple of weeks ago to find that they had redone the website, and taken a massive step back, essentially closing the site down to half the users of the web, and to those search engines and scrapers that wanted to get the information in order to make it more widely available.
In short it went from being pretty much open, to downright closed. How did they do this? First they introduced a registration system. Now, admittedly, it’s a pretty simple registration process, and doesn’t require you to submit any personal details. I registered as ‘Bob’ with a password of ‘password’ just fine. But as well as adding friction to the user experience, it also puts everything behind the signup out of the reach of search engines. Really dumb. Here’s the Google search you get now (a few weeks ago there were hundreds of thousands of results):
The other key point about adding a registration system is that the sole reason is to be able to restrict access to certain users. Let me repeat that, because it goes to the heart of the issue about openness and transparency, and why this is a step back from both by the District of Columbia: it allows them to say who can and can’t see the information.
If open data and transparency is about anything, it’s about giving equal access to information no matter who you are.
The second thing they did was build a site that doesn’t work for those who don’t use Internet Explorer 7 and above, including those who used screenreaders. That’s right. In the year 2011, when even Microsoft are embracing web standards, they decided to ditch them, and with them nearly half the web’s users, and all those who used screenreaders (Is this even allowed? Not covered by Americans With Disabilities Act?).
In the past couple of weeks, I’ve been in an email dialogue with the people in the District of Columbia behind the site, to try to get to the bottom of this, and the bottom seems to be, that the accessibility of the site, the ability for search engines to index it, and for people to reuse the data isn’t a priority.
In particular it isn’t a priority compared with satisfying the needs of their ‘customers’, meaning those companies that file their information (and perhaps more subtly those companies whose business models depend on the data being closed). Apparently some of the companies had complained that they were being listed, contacted and or solicited without their approval.
That’s right, the companies on the public register were complaining that their details were public. Presumably they’d really rather nobody had this information. We’re talking about companies here, remember, who are supposed to thrive or fail in the brutal world of the free market, not vulnerable individuals.
It’s worth mentioning here that this tendency to think that the stakeholders (hate that word) are those you deal with day-to-day is a pervasive problem in government in all countries, and is one of the reasons why they are failing to benefit from open data the way they should and failing too to retool and restructure for the modern world.
Sure, we can work around these restrictions and probably figure out a way to scrape the data, but it’s a sad day to see one of the pioneers of openness and transparency take such a regressive step. What’s next? Will the DC datastore take down its list of business licence holders, or maybe the DC purchase order data, all of which could be used for making unsoliticited requests to these oversensitive and easily upset businesses?
p.s. Apparently this change was in response to an audit report, which I’ve asked for a copy of but which hasn’t yet been sent to me. Any sleuthing or FOI requests gratefully received.
p.p.s. I also understand there’s also new DC legislation that’s been recently been passed that require further changes to the website, although again the details weren’t given to me, and I haven’t had time to search the DC website for them
As a bit of an outsider, reading the government’s pronouncements on open data feels rather like reading official Kremlin statements during the Cold War. Sometimes it’s not what they’re saying, it’s who’s saying it that’s important.
And so it is, I think, with George Osborne’s speech yesterday morning at Google Zeitgeist, at which he stated, “Our ambition is to become the world leader in open data, and accelerate the accountability revolution that the internet age has unleashed“, and “The benefits are immense. Not just in terms of spotting waste and driving down costs, although that consequence of spending transparency is already being felt across the public sector. No, if anything, the social and economic benefits of open data are even greater.”
This is strong, and good stuff, and that it comes from Osborne, who’s not previously taken a high profile position on open data and open government, leaving that variously to the Cabinet Office Minister, Francis Maude, Nick Clegg & even David Cameron himself.
It’s also intriguing that it comes in the apparent burying of the Public Data Corporation, which got just a holding statement in the budget, and no mention at all in Osborne’s speech.
But more than that it shows the Treasury taking a serious interest for the first time, and that’s both to be welcomed, and feared. Welcomed, because with open data you’re talking about sacrificing the narrow interests of small short-term fiefdoms (e.g. some of the Trading Funds in the Shareholder Executive) for the wider interest; you’re also talking about building the essential foundations for the 21st century. And both of these require muscle and money.
It also overseas a number of datasets which have hitherto been very much closed data, particularly the financial data overseen by the Financial Services Authority, the Bank of England and even perhaps some HMRC data, and I’ve started the ball rolling by scraping the FSA’s Register of Mutuals, which we’ve just imported into OpenCorporates, and tying these to the associated entries in the UK Register of Companies.
Feared, because the Treasury is not known for taking prisoners, still less working with the community. And the fear is that rather than leverage the potential that open data allows for a multitude of small distributed projects (many of which will necessarily and desirably fail), rather than use the wealth of expertise the UK has built up in open data, they will go for big, highly centralised projects.
I have no doubt, the good intentions are there, but let’s hope they don’t do a Team America here (and this isn’t meant as a back-handed reference to Beth Noveck, who I have a huge amount of respect for, and who’s been recruited by Osborne), and destroy the very thing they’re trying to save.
Like buses, you wait ages for local councils to publish their spending data, then a whole load come at once… and consequently OpenlyLocal has been importing the data pretty much non-stop for the past month or so.
We’ve now imported spending data for over 140 councils with more being added each day, and now have over a million and a half payments to suppliers, totalling over £10 billion. I think it’s worth repeating that figure: Ten Billion Pounds, as it’s a decent chunk of change, by anybody’s measure (although it’s still only a fraction of all spending by councils in the country).
Along with that we’ve also made loads of improvements to the analysis and data, some visible, other not so much (we’ve made loads of much-needed back-end improvements now that we’ve got so much more data), and to mark breaking the £10bn figure I thought it was worth starting a series of posts looking at the spending dataset.
Let’s start by having a look at those headline figures (we’ll be delving deeper into the data for some more heavyweight data-driven journalism over the next few weeks):
144 councils. That’s about 40% of the 354 councils in England (including the GLA). Some of the others we just haven’t yet imported (we’re adding them at about 2 a day); others have problems with the CSV files they are publishing (corrupted or invalid files, or where there’s some query about the data itself), and where there’s a contact email we’ve notified them of this.
The rest are refusing to publish the CSV files specified in the guidelines, deciding to make it difficult to automatically import by publishing an Excel file or, worse, a PDF (and here I’d like to single out Birmingham council, the biggest in the UK, which shamefully is publishing it’s spending only as a PDF, and even then with almost no detail at all. One wonders what they are hiding).
£10,184,169,404 in 1,512,691 transactions. That’s an average transaction value of £6,732 per payment. However this is not uniform across councils, varying from an average transaction value of £669 for Poole to £46,466 for Barnsley. (In future posts, I’ll perhaps have a look at using the R statistical language to do some histograms on the data, although I’d be more than happy if someone beat me to that).
194,128 suppliers. What does this mean? To be accurate, this is the total number of supplying relationships between the councils and the companies/people/things they are paying.
Sometimes a council may have (or appear to have) several supplier relationships with the same company (charity/council/police authority), using different names or supplier IDs. This is sometimes down to a mistake in keying in the data, or for internal reasons, but either way it means several supplier records are created. It’s also worth noting that redacted payments are often grouped together as a single ‘supplier’, as the council may not have given any identifier to show that a redacted payment of £50,000 to a company (and in general there’s little reason to redact such payments) is to a different recipient than a redacted payment of £800 to a foster parent, for example.
However, using some clever matching and with the help of the increasing number of users who are matching suppliers to companies/charities and other entities on OpenlyLocal (just click on ‘add info’ when you’re looking at a supplier you think you can match to a company or charity)., we’ve matched about 40% of these to real-world organisations such as companies and charities.
While that might not seem very high, a good proportion of the rest will be sole-traders, individuals, or organisations we’ve not yet got a complete list of (Parish and Town councils, for example). And what it does mean is we can start to get a first draft of who supplies local government. And this is what we’ve got:
66,165 companies, with total payments of £3,884,271,203 (£3.88 billion), 38.1% of the total £10bn, in 579,518 transactions, making an average payment of £6,702.
8,236 charities, with total payments of £415,878,177, 4.1% of the total, in 55,370 transactions, making an average payment of £7,511.
Next time, we’ll look at the company suppliers in a little more detail, and later on the charities too, but for the moment, as you can see we’re listing the top 20 matched indivudual companies and charities that supply local government. Bear in mind a company like Capita does business with councils through a variety of different companies, and there’s no public dataset of the relationships between the companies, but that’s another story.
Finally, the whole dataset is available to download as open data under the same share-alike attribution licence as the rest of OpenlyLocal, including the matches to companies/charities that are receiving the money (the link is at the bottom of the Council Spending Data Dashboard). Be warned, however, it’s a very big file (there’s a row for every transaction), and so is too big for Excel (or even Google Fusion tables for that matter), so it’s most use to those using a database, or doing academic research.
* Note: there are inevitably loads of caveats to this data, including that councils are (despite the guidance) publishing the data in different ways, including, occasionally, aggregating payments, and using over-aggressive redaction. It’s also, obviously, only 40% of the councils in England., although that’s a pretty big sample size. Finally there may be errors both in the data as published, and in the importing of it. Please do let us know at firstname.lastname@example.org if you see any errors, or figures that just look wrong.
A couple of days ago, there was a brief announcement from the UK Government of plans for a new Public Data Corporation, which would “bring together Government bodies and data into one organisation”.
A good thing, no? Well, up to a point, Lord Copper.
I tweeted after the announcement: “Is it just me, or does the tone of the Public Data Corp make any other #opendata types uneasy?” From the responses, I clearly wasn’t the only one, and in my discussions since then it’s clear there’s a lot of nervousness out there.
So, what is it, and should we be afraid? The answers are ‘Nobody knows’, and ‘Yes’.
To flesh that out a bit, none of the open data activists and developers that I’ve spoken to knows what it is, or what the real motivation is, and remember these are the people who did much to get us into a place where the UK government has declared that the public has a ‘Right To Data’ and that the excellent ‘Open Government Licence‘ should be the default licence.
In that context, the announcement of a ‘Public Data Corporation’ should be be treated with some wariness.
However, this wariness turns into suspicion, when you read the press release.
First the announcement is a joint one from the Cabinet Office minister Francis Maude (who seems to very much get the need for open public data in the changed world in which we live) and from Business Minister Edward Davey, who I know nothing about, but his department BIS (Dept of Business, Innovation & Skills) has very much not been pushing for open data, and in fact has in the past refused to make data it oversees openly available.
(My sources tell me the proposal in fact originated from BIS, and thus could be seen as an attempt by the incumbents to co-opt the open data agenda, as a way of shutting it down, smothering it if you like.)
Second, despite the upbeat headline “Public Data Corporation to free up public data and drive innovation” (Shock horror: org states its aim is to innovate & be successful), the text contains a number of worrying statements:
- “By bringing valuable Government data together, governed by a consistent set of principles around data collection, maintenance, production and charging[my emphasis], the Government can share best practice, drive efficiencies and create innovative public services for citizens and businesses. The Public Data Corporation will also provide real value for the taxpayer.”
The idea of ‘value for the taxpayer’ is the same old stuff that got us into the unholy mess of trading funds, and the gordian knot of the Ordnance Survey licence wich is still being unpicked. This nearly always translates as value we can measure in £s, which in turn means what income we’ve got coming in (even if it’s from other public sector bodies).
- “It will provide stability and certainty for businesses and entrepreneurs, attracting the investment these operations need to maintain their capabilities and drive growth in the economy” – quote from Edward Davey.
If I were a cynic I’d say stability and certainty translates to stagnation and rent-seeking businesses, which may be music to civil servants’ ears but does nothing to help innovation. We’re in a rapidly changing world. Get over it.
- “bringing valuable Government data together, governed by a consistent set of principles around data collection, maintenance, production and charging”.
If this is the PDC’s mandate I think it could end up focused on the last of these, short-sighted though that would be.
- “It will also provide opportunities for private investment in the corporation.”
Great. A conflicting priority, to delight the bureaucrats and muddy the focus. Keep it small, keep it simple, keep it agile.
Finally, there’s no mention of open data, no mention of the Open Government Licence, the Transparency Board and only one mention of transparency, and that’s in Francis Maude’s quote.
If you’re a natural cynic, you’ll just say the government has already decided to flog everything off to the highest bidder. If you adopt that position, and give up without a fight, the people in Whitehall and the trading funds who want to do that will almost certainly win.
However, if you believe me when I say things are finely balanced, that either side could win, and enough well-organised external pressure could really make a difference over the next year, then you won’t just bitch, you’ll get stuck in.
He’s not wrong there. We’ve got perhaps 6 months to make this story turn out good for open data, and good for the wider community, and I suspect that means some messy battles along the way, forcing government to take the right path rather than slide into its bad old habits, perhaps with some key datasets, which should undoubtedly be public open data, but are currently under a restrictive licence.
I’ve got a couple in my sights. Watch this space.
I took a very frustrating phone call earlier today from NESTA, an organisation I’ve not had any dealings with it before, and don’t actually have a view about it, or at least didn’t.
It followed from an email I’d received a couple of days earlier, which read:
I am contacting you about a project NESTA are currently working on in partnership with the Big Society Network called Your Local Budget.
Working with 10 pioneer local authorities, we are looking at how you can use participatory budgeting to develop new ways to give people a say in how mainstream local budgets are spent. Alongside this we will also be developing an online platform that enables members of the public to understand and scrutinise their local authority’s spending, and connect with each other to generate ideas for delivering better value for money in public spending.
We would like to share our thinking and get your thoughts on the online tool to get a sense of what is needed and where we can add value. You are invited to a round table discussion on Friday 19 November, 11am – 12.30pm at NESTA that will be chaired by Philip Colligan, Executive Director of the Public Services Lab. Following the meeting we intend to issue an invitation to tender for the online tool.
Apart from the short notice & terrible timing (it clashes with the Open Government Data Camp, to which you’d hope most of the people involved would be going), the main question I had was this:
I got the phone call because I couldn’t make the round table, and for some feedback, and this was the feedback I gave: I don’t understand why this is being done. At all.
Putting aside the participatory budgeting part (although this problem seems to be getting dealt with by Redbridge council and YouGov, whose solution is apparently being offered to all councils), there’s the question of the “online platform that enables members of the public to understand and scrutinise their local authority’s spending, and connect with each other to generate ideas for delivering better value for money in public spending.”
Excuse me? Most of the data hasn’t been published yet, there are several known organisations and groups (including OpenlyLocal) that have publicly stated they going to to be importing this data and doing things with it – visualising it, and allowing different views and analysis. Additionally, OpenlyLocal is already talking with several newspaper groups to help them re-use the data, and we are constantly evolving how we match and present the data.
Despite this, Nesta seems to have decided that it’s going to spend public money on coming up with a tendered solution to solve a problem that may be solved for zero cost by the private sector. Now I’m no roll-back-the-government red-in-tooth-and-claw free marketeer, but this is crazy, and I said as much to the person from Nesta.
Is the roundtable to decide whether the project should be done, or what should be done? I asked. The latter I was told. So, they’ve got some money and have decided they’re going to spend it, even though the need may not be there. At a time when welfare payments are being cut, essential services are being slashed, for this sort of thing to happen is frankly outrageous.
There are other concerns here too – I personally think websites such as this are not suitable for a tender process, as that doesn’t encourage or often even allow the sort of agile, feedback-led process that produces the best websites. They also favour those who make their living by tendering.
So, Nesta, here’s a suggestion. Park this idea for 12 months, and in the meantime give the money back to the government. If you want to act as an angel funding then act as such (and the ones I’ve come across don’t do tendering). A reminder, your slogan is ‘making innovation flourish’, but sometimes that means stepping back and seeing what happens. This is not the way to building a Big Society