One of the things I’ve had on my ToDo list for OpenlyLocal for a while was providing a a list of links to online services provided by each Local Authority.

Seemed like something that should be on the site, and available as structured data; it also looked like it should be fairly easy to do, as it’s a service that’s sort of provided by central government (LocalDirectGov), though with some shortcomings.

The problem is that from a usability point of view the Local DirectGov interface is a bit clunky. First you choose the service you want the link for, which means using an A-Z (always a bit of a problem). This is the landing page, and as you can see you’re on the A’s.

So let’s say you want Hazardous Waste. Is that under H or W? Actually it’s under W, so click on W, and then on “Waste – Hazardous” and a new window opens (why?). You then need to enter your postcode, town or council in a form and you’ll then be (usually) given a link to click through to get to the council page.

However, depending on what you put in there and what category you want you may be asked to choose a particular council or be told that you council does not provide the service online:

Frustrating.

Now there is a limited way for external websites to interact with this service, using the ‘white-label’ Local DirectGov application. There’s even a case study. Basically, you download a list of services provided by each type of council, and then build a LocalDirectGov URL, which redirects to the council service.

Terrific. Not hard to do, even for a coder as slow as me. The only problem is that it doesn’t work. For the end user that is.

The thing is, there’s no way of knowing whether the local authority actually provides a given service online, and there’s a fair chance that the URL you’ve just built up will resolve to a bog-standard contact page, or even worse non-existent page resulting in a 404 error. Not great for users, and there appears no way of programmatically finding out if link will work, even though it’s there in Local DirectGov’s database (which is how it says that the service isn’t provided).

So, we’ve tried to fix on OpenlyLocal this and provide a better version. First we’ve collected up the useful data for each authority (i.e. where there’s a specific page to that subject, and not a 404 or generic “contact us” page). Then we’ve put it all on one page, and made it searchable too. It’s clean, simple, and works:

You can also search it from the main council page if you want to in an Ajaxy live-search way (obviously the search also works without javascript, for screenreaders and other text browsers):

Finally, you can access the data through the API as XML or JSON. So far, we’ve done a little over half the local authorities, and should have all the rest done by sometime next week (it’s just a matter of tying the remaining local authorities to their LocalDirectGov IDs, which has to be done manually).

As ever, comments, bug reports and feature requests welcome.

1. As I promised at the excellent TalkAboutLocal unconference in Stoke, I’ve been working on ways of helping non-techies use the local council data opened up by OpenlyLocal.com . The first of these is a Google Gadget, which can be added to your iGoogle page or your hyperlocal Blogger blog.

Though writing a Google Gadget can be a frustrating business, adding and using it is a doddle. Basically, if OpenlyLocal is extracting the data from your local authority (and we’ve got over 70 councils so far, with more being added every week — see the parsed council list for details), the gadget will show the key info at a glance — basic contact details, updated info, members, committees and forthcoming meetings.

For iGoogle users

1. Go to iGoogle and click on “Add stuff” in the top right-hand corner of the page.
   
2. Click on “Add feed or gadget” at the bottom of the left column
   
3. In the box that pops up enter (or paste) http://openlylocal.com/tools/gadget.xml and click “Add”. Click OK to the box asking you if you want to add the gadget.
   
4. Go back to the iGoogle home page, and the gadget will have appeared and will need you to choose your council. Choose the council, click “save”, and you’re done.
   

Note you can drag the gadget anywhere you want on the page, and even add more than one copy of the gadget, if you want to have ones for different councils.

You can also share the gadget with friends, neighbours, colleagues etc. Just click on the triangle in the top right of the gadget. This is also where you go if you want to delete the gadget.

For Blogger users

1. Make sure you’re logged in and go to the Customize area (the link is in the top right hand corner of the screen) where should choose “Layout” and “Page Elements”
   
2. On the template that you’re presented with there’s a block of boxes representing widgets/sidebars, including “Add a gadget”. Click on this.
   
3. You’ll then be given a selection of gadgets, with a menu on the left-hand side. Click the link that says “Add your own”
   
4. A form will appear with a space for the gadget’s URL. Enter (or paste) http://openlylocal.com/tools/gadget.xml and click “Add”.
   
5. You will be then asked to configure the gadget. Select the council and click the “Save” button.
   

We’re planning on introducing more features in the future, but even as it stands, I think it’s a useful tool for your iGoogle page, or if you’re a hyperlocal blogger, a great way to add up-to-date and relevant info to your blog.

By the way, you can see the code behind the gadget at http://openlylocal.com/tools/gadget.xml (it basically makes a single call to an OpenlyLocal API url — http://openlylocal.com/councils/[council_id].json — and then builds the gadget using javascript) and I’ll be creating a github project for it so you can help improve it/report bugs/request features.

The next step in making info more available is a Ning app, as quite a few hyperlocal sites seem to be using Ning as their platform of choice, and there’s someone who’s promised to write a WordPress plugin to provide the same or similar functionality to the gadget.

Mentioned David Eaves’ Three Laws of Open Government Data at yesterday’s excellent Talk About Local unconference, and had a few people asking me what they were and where to find them. So here they are (from http://eaves.ca/2009/09/30/three-law-of-open-government-data/):

The Three Laws of Open Government Data:

1. If it can’t be spidered or indexed, it doesn’t exist
2. If it isn’t available in open and machine readable format, it can’t engage
3. If a legal framework doesn’t allow it to be repurposed, it doesn’t empower

There are also a few other useful links in the comments.

p.s. I don’t know David, but I really like the conciseness of these

This post is something of a brain dump about some possible common principles for open data for Local Authorities. It’s pretty much the text of a post I made to the tyoc google group, which is helping to organise a TheyWorkForYou-type website for Manchetser County Council.

It’s probably not the first post on the subject (link to other ones welcome), and certainly won’t be the last, but hopefully will provide some useful thoughts for those councils or groups working on exposing their data. (Maybe if there’s anyone else interested we can get this a bit more formalized.)

==========

My thoughts have been influenced by exposing the data from OpenlyLocal.com and also from consuming XML data from other authorities, but obviously these are only my initial ideas, and I’m using as an example OpenlyLocal urls and also those for Lichfield District Council, who’s got a great webmaster who kindly exposed all the council democratic data they could as XML.

1. The api should expose the authority’s internal UIDs (as well as the id of the record in the application if it’s not the council exposing the data). The idea is to open up the data, not creating another walled garden. See http://openlylocal.com/members/3114.xml and http://www.lichfielddc.gov.uk/site/custom_scripts/meetings_committees…
2. UIDs should be absolutely unique to the object being exposed and should not change if the name changes (so no strings for councillor IDs, as these can change if the councillors name changes, e.g. through marriage, titles or simply what they prefer to be known as).
3. The api should use and expose common identifiers when possible to allow definitive identification, e.g. Wards should expose the ONS Snac Ids. E.g. http://openlylocal.com/wards/982.xml
4. The api should given information about when the object was last updated. At the moment on OpenlyLocal, all objects have created_at and updated_at fields exposed. However, given OpenlyLocal.com and the ‘Your MCC’ projects are basically proxies we should possibly also expose a “last_checked_at” field, so the timeliness can be worked out.

Just got back from doing a presentation to the Advisory Panel on Public Sector Information

Though there’s a bit about OpenlyLocal.com (the site I run that opens up and makes accessible local government data), most of it is about the present and future of local government data, and the obstacles that need to be overcome.

The presentation seemed to go down reasonably well, despite Powerpoint messing up the formatting (it was created on Keynote), and I’m hoping we can start to get some traction on opening up local government data.

I’m embedding it and making it available under Creative Commons non-commercial share-alike licence. Comments welcome:

Sometimes you just have to get on with it. After adding a few more UK Local authorities to OpenlyLocal this weekend (we’ve now opened up information from over 70 councils), I bit the bullet and added basic entries for all the remaining UK local authorities.

[Props should go to Dane at the excellent eGovernment Register who gave me permission to use the basic info I got from there for open distribution, though other restrictions may apply -- the ONS info is Crown Copyright, for example.]

What use is that, if we’re not yet extracting the councillors, committees and meetings information from them?

Plenty, because it now means you can get programmatic access to the Full List of UK councils in one place, at one url: http://OpenlyLocal.com/councils/all. You can also get the data as XML or Json just by adding .xml or .json to the url.

Plus the XML and Json feeds also include extra information (all which on the HTML version is on the info page for each council):

• Basic info for the council — address, telephone number, website
• The ONS SNAC ids used by central govt to refer to Local Authorities
• The WhatDoTheyKnow id, so you can tie into Freedom of Information requests for that council
• Plus — for those councils that we’ve opened up democratic data for — councillors, committees and wards

If there are any errors (some of this info has been added by hand, after all, do let me know in the comments), and I’ll aim to get them sorted straight away .

Till then, I’ll leave you with an example of how to use this data — I’ve created a Google Spreadsheet of All UK Local Authorities from it (using the ImportXML function): http://spreadsheets.google.com/ccc?key=0AhOqra7su40fdFNybEV6TlFGR1hQNHY5T0hQUDduUnc&hl=en_GB

I’ve made it accessible to all, so you could just go ahead and download it to you computer, or create a copy of it on Google Spreadsheets, and use it as the basis for your hack/analysis, FoI investigation or whatever. (For those who want to understand how it works, have a look at the the functions — the main one is ImportXML, which is not well documented, but you can get by with trial-and-error and a bit of Googling).

Alternatively, have a play with the XML feed in Yahoo Pipes. Have fun.

Time for a quick update on OpenlyLocal.com — the website that opens up local government in an accessible and consistent way (a sort of local government TheyWorkForYou). Since the last post announcing the site I’ve added quite a few new features, as well as a fair bit of tweaking to the back-end .

The main ones are:

• Loads more councils (we have now opened up over 65 councils, with info on more than 4000 councillors and 2300 committees). If you council isn’t there give me a nudge and I’ll see what we can do
• Keyword filtering of committee minutes and agendas. So if you want to find a council’s minutes and agendas featuring Tesco, for example you can now do that.

  

• RSS feeds for minutes and agendas, including ones restricted to keywords. Just click on the RSS feed link at the bottom of the page.
• Even more Calendar feeds. You can now subscribe to a calendar for a whole council, an individual councillor, or a particular council committee.
• Added Wards to the information we’re extracting from websites.
• Added RDFa markup, to allow integration with other datasets and become part of the semantic web.
• Plus tons of backend improvements making it easier for me to add new councils, and loads of bugfixes

There’s still a long way to go, but OpenlyLocal does feel as if it’s moved a little beyond the proof-of-concept stage and is now genuinely useful for local democracy.

While all the hullaballo about MPs expenses has been raging, I decided to leave the parliamentary stuff for a while to the big boys, and have a look at local government instead. Wasn’t looking for any dirt; just trying to get a feel for how it worked, possibly run a few comparisons.

Unfortunately, as anyone who has been down this road knows, pulling together information on local authorities is fraught with difficulties, of which the most serious is the fact that for most things there’s no single source of data

Now, perhaps things may change now that Tim Berners-Lee is on the case, or perhaps David Cameron will really make a difference to local authority data (though he seems to misunderstand the problem — it’s not that council minutes aren’t published, it’s that they’re in a form that can’t easily be monitored, analysed or merged with other data).

So decided to scratch my own itch, as they say. The result is OpenlyLocal.com, a first draft of getting hold of and making accessible local government data in a consistent and useful form – a TheyWorkForYou Local, if you like.

To start off, I’ve concentrated on getting the basic information — current councillors, committees, membership of those committees, and minutes for those committee meetings. I started off with a little over a dozen councils (though that’s since increased to over 40 of them). Since starting the project I’ve had some great feedback and, increasingly, cooperation from many of the people working in this field, including in councils.

Don’t expect fancy styling(as you can see from the screenshot above)  – the key is to get the data, and make it accessible. That means making the data available through an API (either as XML or json) from day one, and supporting open standards (I’m looking at RDFa at the moment).

Want to help? If you can code in Ruby, then grab the code at github and start contributing patches (with tests). Or if your council’s already listed, see if you can find any errors, and let us know what other data you’d like on OpenlyLocal. Or build a mashup based on the data exposed by the API.

[Note: Voting attendance is an imperfect proxy for actual attendance, as the figure may be depressed by silent abstentions (i.e. not voting in a division, rather than voting both ‘aye’ and ‘no’) and by just turning up to vote, but failing to attend the debate. However, until Parliament provides a better measure for attendance, or more transparency of MPs actions, this is the only one we have.]

It’s recess time again, and time for MPs’ end-of-term report. I’ll leave it to others to comment on how they’ve dealt with some of the genuinely momentous events since the summer recess. This post deals solely with their voting attendance record.

First off, let’s have an overall look at the overall figures for the period:

The figures above are pretty self-explanatory. All parties have improved their attendance of votes, by 5 to 8 percentage points. Perhaps not surprising given the financial crisis.

Now let’s have a look at the main parties in detail, using the same histograms used before to show the distribution of the parties attendance figures. Interestingly (well, in a wonkish sort of way), the distributions are a bit more spread out than the long-term average. In part this is probably down to the shorter time period showing up variations that are hidden in longer period, but it’s interesting nevertheless to note that though all parties have improved their overall attendance figures, the number and proportion of Labour MPs who’ve voted in fewer than half the divisions has nearly tripled, from 11 MPs to 30 of them.

[Note: there's no significance to the width of the columns -- the recent ones are narrower so that both can be seen on the same graph]

Finally, let’s have a look at those outliers, first, the MPs who attended divisions less than than 50% of the time:

Attended fewer than 50% of divisions Oct–Dec 08

No surprise that GB’s in this list, voting in only 3 divisions (see here for the divisions he did take time out to vote on). Ditto Alistair Darling and David Cameron.

However, the rest of the list is more interesting. Some of those on the front bench, for example (e.g. Jacqui Smith, Jack Straw), surprisingly don’t make the list, i.e. they voted in at least 50% of the divisions. Ditto some of the opposition spokespeople.

But what about the backbenchers who are on the list. Possibly there’s a good reason for Margaret Hodge and Jessica Morden for failing to attend a single division — illness perhaps (though there’s nothing on either of their websites to indicate such a factor)? And what about Kali Mountford (14.3%) and Khalid Mahmood (21.4%).

If I was in their constituency, I’d like to know, particularly since they took little part in debates, either. Similarly for the low-raters for the Conservatives — Michael Mates and Tim Yeo (6 directorships!) at 35.7% each.

Now the MPs who voted more than 90% of the time:

Attended more than 90% of divisions

Some amazing figures in there. In fact, given his 100% record you wonder if the Beast of Bolsover has got a home to go to.

C.

Notes on calculations

• The above calculations were derived from the voting record freely available from the Public Whip project, and cover the period from Oct 2008 to Dec 2008. The data can be downloaded in the form of a MySQL database, and this was used together with custom MySQL queries to generate the figures.

[Note: Voting attendance is an imperfect proxy for actual attendance, as the figure may be depressed by silent abstentions (i.e. not voting in a division, rather than voting both ‘aye’ and ‘no’) and by just turning up to vote, but failing to attend the debate. However, until Parliament provides a better measure for attendance, or more transparency of MPs actions, this is the only one we have.]

A frequent arguments for low attendance of voting divisions by MPs is that the figure is depressed by ministers (and shadow spokespersons), whose other responsibilities prevent them from attending as many votes (as they’d like to), thus bringing down the overall average.

Seems reasonable, so let’s have a look at just how much of an influence this ‘ministerial effect’ has on the overall figures. First, let’s look at the average voting attendance for ministers and non-ministers (calculation details below):

Er, wait a minute, so the average voting attendance rate for ministers is higher than non-ministers? That’s not what we expected. However, basic averages (i.e. the mean) can hide a multitude of sins, so let’s have a look at the distribution of those attendance figures.

As you can see, while the peak of the ministerial attendance is around the 65% mark (less than that for the non-ministerial one), there were far more divisions in which 90%+ of ministers voted than there were for which 90%+ of non-ministers voted.

This makes sense, in a way, as ministers are far more likely than backbenchers to turn up en masse for votes their party sees as important. It’s this that largely accounts for the figures we saw in the table above. However, what the graph also shows is that when you take the ministers out of the equation, attendance definitely does not shoot up. There is, in short, no ‘ministerial effect’ to account for the low attendance of MPs.

[It's worth mentioning that the ministerial office records are slightly incomplete -- the record of Parliamentary Private Secretaries is missing during some periods -- so I've run the figures for ministers both including and excluding PPSs. As you can see, it doesn't make a lot of difference.]

The party lines

Having looked at the big picture, it’s time to look at the ministerial vs non-ministerial attendance by party, specifically the three main parties in Parliament.

As you can see, the relationship between ministerial and non-ministerial attendance is noticeably different for each of the parties. Labour ministers do indeed have noticeably lower attendance rates than their backbenchers, though not as much as I’d expected and not enough to alter the distribution massively.

However, for the Tories and LibDems, the surprising thing — for me, at least — was the attendance rates for their spokespersons are actually noticeably better than their backbenchers, raising rather than lowering the overall figures. What, I wonder, is the reason for this?

Finally, a couple of quick graphs to wrap this post up. One shows, perhaps not surprisingly, that Labour ministerial attendance rates are less than for the shadow spokespersons — presumably the time commitment for a governmental position is greater than that for the equivalent shadow position.

The other shows the distribution of backbenchers attendance figures, by party. I’ll leave that one without making any further comment.

C.

Notes on calculations

• The Ministerial/non-ministerial attendance rates were calculated by looking at every Commons division between May 1997 and July 2008, and working out the number of ministers/non-ministers who could have voted in that division, and the number who actually did vote. The average attendance figures in the table were calculated by dividing the aggregate number of votes by the aggregate number of possible votes.
  To calculate the distribution of attendance rates I calculated the ministerial/non-ministerial attendance rate for each division, and plotted these on a graph to show how those attendance rates are distributed (as usual, I’ve made the underlying figures are available as a spreadsheet here and here if you want to examine them further).
• Ministers are those holding any sort of ministerial office as per the PublicWhip database, including whips, but excluding select committee members (although it wouldn’t be hard to run the figures to include select committee members). The Parliamentary Private Secretaries record at the Public Whip is incomplete for several periods, and unfortunately (and ridiculously) there is no historical record of ministers available from Parliament’s own website.
• The above calculations were derived from the voting record freely available from the Public Whip project, and cover the period from May 1997 to July 22, 2008 (when the house rose for the summer recess). The data can be downloaded in the form of a MySQL database, and this was used together with custom MySQL queries to generate the figures.
• The graphs are visual representations of the density of the distribution, and were plotted using R using the kernel densityplot function.