Open data and all that

Some progress on the Local Spending/Spikes Cavell issue

with 5 comments

Yesterday I was invited to a meeting at the Department for Communities and Local Government with the key players in the local spending/Spikes Cavell issue that I’ve written about previous (see The open data that isn’t and Update on the local spending data scandal… the empire strikes back).

The meeting included Luke Spikes from Spikes Cavell as well as Andrew Larner from IESE (the Regional Improvement and Efficiency Partnership for the South East), which helped set up the deal, as well as myself and Nigel Shadbolt, who chairs the Local Public Data Panel and sits on the government’s Transparency Board. I won’t go into all the details, but the meeting was cordial and constructive, produced a lot of information about how the deal works and also potentially made progress in terms of solving some of the key issues.

We can now, for example, start to understand the deal – it’s called the Transform project – which as I understand it is a package deal to take raw information from the councils accounts and other systems (e.g. purchase & procurement systems) to SC’s specification, clean up and depersonalise the data, then analyse to show the councils potential savings/improvements, and finally to publish a cut of this information on the Spotlight on Spend website. Essentially we have this:

There are still some details missing from this picture – we haven’t yet seen the Memorandum of Understanding which frames the deal, nor the specification of the raw information that is provided to Spikes Cavell, but we have been promised both of these imminently. This last one in particular will be very useful as it will allow us to refine the advice we are giving councils about the data they should be publishing in order to make the spending information useful and comparable (it’s not been suggested previously, for example, that it would be useful to include details from the council’s procurement systems, though in hindsight this makes a lot of sense).

Crucially, it was also agreed that all the input data into Spikes Cavell’s proprietary systems (the ‘Cleaned-up but non-proprietary data’ in the diagram above) would be published, so the wider community would be on the same footing as Spikes Cavell as far as access to the raw data goes. This is crucial and worth repeating: it means that anyone else will have access to the same base data as Spikes Cavell, and the playing field is therefore pretty much level.

There are still issues to be sorted out, the chief of which is that while Spikes Cavell is happy to publish the raw data under a completely open licence, they will require the OK of the council to do so. (However, armed with this knowledge it will be easy to identify those councils that refuse, and then possible to tackle them either through persuasion or ultimately legislation.)

The other issues are, briefly: liability for depersonalising the data; where the data is published (I think it should be on the council’s own website or a, or for London councils the London Datastore, not on the Spotlight On Spend website); whether the Spotlight On Spend website itself is necessary and cost-effective (it’s impossible to know how much it costs as it’s bundled in with the whole deal); and whether the data-cleansing should be stripped out from the rest of the deal.

However, it’s worth saying that this agreement goes beyond just the member councils of the IESE, but to all councils that in the future use a similar agreement (obviously it’s ultimately up to them, but certainly this was the wish of everyone at the meeting).

Finally, I’d like to thank Andrew Larner at IESE for his open approach, and for Spikes Cavell for their willingness to engage. What we have here isn’t perfect (and I still fundamentally believe that councils should be doing the cleansing and publishing of the data themselves, and exchanging that knowledge with other councils and using it to improve their own data processes), but it’s a big step forward in genuinely opening up raw council data.

Update: The official notes of the meeting have now been published on the Local Public Data panel blog:


Written by countculture

July 20, 2010 at 10:57 pm

5 Responses

Subscribe to comments with RSS.

  1. “…and I still fundamentally believe that councils should be doing the cleansing and publishing of the data themselves”

    I agree strongly.

    Essentially this means the suppliers of “Account Systems” and “Procurement and Other Systems” on the left of your diagram should create the widgets/addins/extensions to their own software layers to enable councils to do this themselves.

    There ought to be no need to create yet another cost to for them to be able to get to their own data and get it to a public endpoint.

    This issue should be in the forefront of council IT purchases from now on, and in one voice they should be demanding it by end Jan ’11.


    July 21, 2010 at 8:48 am

  2. This is really helpful Chris – I’ll add a post to the Open Data Community refereing to this, if you haven’t already, and circulate to councils attending the Socitm >£500 workshop on 2 August.

    Vicky Sargent

    July 21, 2010 at 8:53 am

  3. “…and I still fundamentally believe that councils should be doing the cleansing and publishing of the data themselves”

    Sorry, but long and painful experience tells me that it is far quicker, easier and cheaper to outsource this. Once they see the benefits of better data quality and interoperability, there is the vague chance that they might do something internally but otherwise you are talking about a culture change for the whole of a slow moving organisation – not going to happen.

    Christopher Osborne

    July 21, 2010 at 9:28 am

  4. so was this part of the council entire transparency plan or just something mostly internal for the council which cavell process and showed some council savings publically.

    i think cavell may have had a point a miffed at being accused of withholding information in anyway as its the councils responsibility but this does foreshadow they way one suspects a lot of the councils will deal with it by allowing themselves to consultants to misinterpret open data for a big contract.

    steve white

    July 21, 2010 at 3:50 pm

  5. Chris,

    All sounds interesting, and provides more background regarding the ‘why’ and the ‘how’.

    I’d agree regarding data cleansing, what the exercise probably does show (if SC feedback), is where the councils can improve (vendor master duplication and the like).

    Often in scenarios like this, it is better to bring in a consultant/expert in the field to give a 3rd party viewpoint as how to remedy and remediate.

    Either way, it sounds rather good – especially if data is obtained (pre or post cleanse, I’m guessing it would be Pre – dependant on contract). Obviously SC wouldn’t want to host this either (due to bandwidth, and not part of contract).

    Looking forward to an open future,


    July 21, 2010 at 10:08 pm

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

%d bloggers like this: