Open data and all that

The open spending data that isn’t… this is not good

with 28 comments

When the coalition announced that councils would have to publish all spending over £500 by January next year, there’s been a palpable excitement in the open data and transparency community at the thought of what could be done with it (not least understanding and improving the balance of councils’ relationships with suppliers).

Secretary of State for Communities & Local Government Eric Pickles followed this up with a letter to councils saying, “I don’t expect everyone to do it right first time, but I do expect everyone to do it.” Great. Raw Data Now, in the words of Tim-Berners Lee.

Now, however, with barely the ink dry, the reality is looking not just a bit messy, a bit of a first attempt (which would be fine and understandable given the timescale), but Not Open At All.

As a member of the Local Public Data Panel, I’ve worked with other members and councils to draw up some clear and pragmatic draft guidelines for publishing the local spending data. We’ve had a great response in the comments and in conversations, and together with some lessons I did on importing the existing data, I think these will allow us to do a second draft soon.

One thing we weren’t explicit in that first draft – because we took it for granted – was that the data had to be open, and free for reuse by all. Equality of access by all is essential.

So I’ve been watching the activities of Spikes Cavell’s SpotlightOnSpend with some wariness and now those fears seem to have been borne out, as the company seems to set out not to consume the open data that councils are publishing, but to control this data.

The idea seems to be that councils should give Spikes Cavell privileged access to their detailed invoice information, which the company then adds to their proprietry and definitely non-open database, and then publishes an extract of this information on the SpotlightOnSpend website. Exactly what information they get, and under what terms isn’t disclosed anywhere.

The website’s got most of the buzzwords: transparency, accessible, efficiency. It’s even got a friendly domain. If that’s not enough to convince councils, liberally sprinkled around the site is an apparent endorsement from the Secretary of State himself:

I’m really excited about the opportunities of transparency and it’s something this government is utterly committed to. spotlightonspend demonstrates that, when innovative businesses work with far-sighted public bodies, we can inform the public, reduce costs and improve democracy both locally and nationally.

Eric Pickles
Secretary of State
Communities and Local Government

However, when you go to the data and click on the download link this is what you get:

Note the “This data is for your personal use only”  (not to mention the fact that the use of a captcha’ to screen out machines downloading the data means, er, you can’t use machines to automatically download the data, which is sort of the point of publishing the data in a machine-readable way).

Never mind, surely you can just head over to the council’s website and download the data from there? No chance. This is what you get on the Guildford website:

You can search and view this financial data using a new Spotlight on Spend national website. Just follow the link found in the offsite links section of this page.

What about Mole Valley Council:

This data is now available on the spotlight on spend website. You can look at categories and individual suppliers to see how much has been spent in each area or you can download all the data to see individual transactions.

But what about Windsor & Maidenhead, who are closely affiliated with the project, and who are publishing data on their website? Well, download the data from SpotlightOnSpend and it’s rather different from the published data. Different in that it is missing core data that is in W&M published data (e.g. categories), and that includes data that isn’t in the published data (e.g. data from 2008).

So the upshot seems to be this, councils hand over all their valuable financial data to a company which aggregates for its own purposes, and, er, doesn’t open up the data, shooting down all those goals of mashing up the data, using the community to analyse and undermining much of the good work that’s been done.

It’s worth linking here to the Open Knowledge Foundation’s draft guidelines on reporting of Government Finances (disclosure: I helped draw them up), of which the first point is ‘Make data openly available using an explicit license’. And let me be absolutely clear here: this is not open data, not a desirable approach, will not achieve the results of transparency or of equality of access, and is not good for the public sector.

I’m hoping this is a matter of councils and the Secretary of State not understanding the process and implications of giving this data to Spikes Cavell on a privileged basis. If not, perhaps it could be the first test case for the newly setup of Public Sector Transparency Board to rule on.

Update: With lightning fast speed, the Transparency Board has issued a statement about this issue reiterating the open data principles, and saying that measures are taking place to rectify the problem.

There are many questions remaining, not least the nature of the relationship with Spikes Cavell, and the undesirability about their privileged access to the information, but the Board should be congratulated for their quick reaction to the situation, and bodes well for the future issues that will undoubtedly come up.

In the meantime, I’ll keep on the case, and update with blog/tweet as I get more information

Written by countculture

July 2, 2010 at 9:57 am

28 Responses

Subscribe to comments with RSS.

  1. Well quite.

    More worrying is that the Windsor and Maidenhead data is actually available under an implicit open licence as “free for reuse” (not perfect, I agree) and yet Spikes Cavell claim copyright on their pages showing W&M’s data and don’t even let you print it except to make a single copy for personal use only. In my understanding the licence should be viral and not impose restrictions that the owner doesn’t make.

    Adrian Short

    July 2, 2010 at 10:14 am

  2. […] Knowledge Foundation’s Working Group on Open Government Data. This is a cross-post — Chris’ original post here […]

  3. […] The open spending data that isn’t… this is not good « countculture Kick up a stink about this: RT @delineator: council #opendata access restricted by proprietary vendor: (tags: opendata) […]

  4. I think we are going to see more of this with data. The pressure for local government to change has created an opportunity for commercial organisations to offer ‘easy’ and ‘pain free’ pathways for LAs to release this information.

    I have sat in many meetings where LAs are looking for ways of controlling the data that they release and Spikes Cavell has come up with the solution. It will be the preferred route for many unless it is legislated against.


    July 2, 2010 at 3:02 pm

  5. Doesn’t the Re-use of Public Sector Information regulations prevent exclusivity arrangements?


    July 2, 2010 at 6:49 pm

  6. Something to investigate on Help Me Investigate?

    Paul Bradshaw

    July 2, 2010 at 7:41 pm

  7. “this is not open data, not a desirable approach, will not achieve the results of transparency or of equality of access, and is not good for the public sector.”

    However, it is, in this instance, good for a particular business in the creative industries. Re-use of data was supposed to facilitate innovation in creative SMEs and this is a company now creating value and employment and contributing to the Creative Industries GVA.

    Problem is it seems to have a monopoly, given it has this data handed over to it by the councils it works with to the exclusion of all the other creative companies who might compete in this arena. But then that is how the creative industries work; creating value from restriction of access to content, not by giving it away for nothing.

    Dave Harte

    July 2, 2010 at 7:47 pm

  8. what is interesting is their t&c where they claim copyright for all content then deny any responsibility for its accuracy etc.
    one question, how much money are spike cavell making from this?

    david willmot

    July 3, 2010 at 2:37 am

  9. I’ve now got a council spending data scoreboard up on Armchair Auditor so we can clearly see which councils are publishing this data and whether it’s truly open:

    An important point to note is that all eight of these councils are ahead of the others in that they’ve taken some steps to be more transparent about their spending, even though in most cases the implementation leaves much to be desired.

    Adrian Short

    July 3, 2010 at 7:21 am

    • Adrian
      Great stuff re scoreboard. I’ll link to it from the overall Council open data scoreboard.

      You’ve probably got a closer insight into the councils that have entered into this arrangement. I’m not sure I’d be quite so sanguine about it, however, given the whole thing seemed to have been wrapped up by the South East Regional Partnership quango, and the councils they didn’t have to do anything, other than say to Spikes Cavell, please have access to all our accounts systems.

      And even if there is a genuine desire for transparency (I’m sure there is with Windsor & Maidenhead, who’ve have genuinely been pushing this agenda — the others I’m not aware of other pro-transparency/open data actions), there are so many examples of councils and other public sector bodies handing over their data wholesale to a single third party with disastrous consequences that this looks misguided at best.


      July 3, 2010 at 9:22 am

  10. […] The open spending data that isn’t… this is notgood (EN) […]

  11. Spikes Cavell has responded to this post on Information Age. I’ve left a detailed comment below the main article.

    Adrian Short

    July 5, 2010 at 5:27 pm

  12. […] A very disturbing discovery by Chris Taggart last week: a number of councils in the UK are handing over their ‘open’ data to a company which only allows it to be downloaded for “personal” use. […]

  13. […] a comment » My blog post on Friday about the local spending information, the open data that isn’t, and the ag… raised a flurry of tweets, emails, and a reassuringly fast response from the government’s […]

  14. […] via The open spending data that isn’t… this is not good « countculture. […]

  15. Where is the link to download the data? I don’t see it.

    Has it been removed since you pointed this out?


    July 6, 2010 at 5:42 pm

    • It’s at the bottom right of the page of the stats for a council


      July 6, 2010 at 6:10 pm

      • Thanks. Now got it.


        July 6, 2010 at 9:20 pm

  16. […] getting it wrong might be the default position for some local authorities. CountCulture’s Chris Taggart is concerned about data company Spikes Cavell’s SpotlightOnSpend muscling in on local […]

  17. […] up their financial data, Chris Taggart/@countculture, developer of OpenlyLocal posted a piece on The open spending data that isn’t… this is not good in which he described how apparently privileged access to financial data from several councils was […]

  18. […] I was interested to see how they’d tackle the controversy over Chris Taggart’s criticism of the way in which Spikes Cavell are publishing council spending data on their SpotlightOnSpend […]

  19. […] were keen to look up Chris’ blog post and the open data standards and said they would make sure they published their data following those […]

  20. […] key players in the local spending/Spikes Cavell issue that I’ve written about previous (see The open data that isn’t and Update on the local spending data scandal… the empire strikes […]

  21. Greetings,
    I am doing a research on Open Data in EU public administrations, please see for all details

    Any info on stories like this one and especially on the economic value for local businesses of local open data is very welcome (see the “Call for help” section of the link above)

    Thank you in advance for any feedback,
    M. Fioretti

    M. Fioretti

    July 21, 2010 at 4:52 pm

  22. All SpotlightOnSpend download data has been free to reuse since early August.


    August 21, 2010 at 11:28 pm

  23. […] And let’s face it, this is one of the bug bears – a great example of which is Chris Taggart’s work on Spikes Cavell – so that’s probably a damn good […]

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: