A bit overdue (I’ve been talking about doing this for a couple of months), but at last there’s now a Ning app for OpenlyLocal local data so anyone who has a UK Ning hyperlocal site (well, anyone in the 90+ councils we’ve opened up data for) can now have information about their council right there in their site.

Like the OpenlyLocal Google gadget (see OpenlyLocal info on your website, Part 1: Google Gadgets), it’s fairly straightforward to use. You need to be the owner of the Ning community to add it, and then it’s automatically available to users as just another tab (like Forums, Videos, Photos, etc). Once you’ve done it you (and the other members of the community) can see the council’s key data, upcoming meetings, members and committees. More features and functionality will be added, but it’s already a very useful addition to any hyperlocal site.

This is what it looks like in action:

So how do you add the OpenlyLocal application to your Ning community (NB Ning Apps need to be added by the network creator). It’s a breeze, and should take no more than a couple of minutes (probably a lot less):

1. Go to the ‘Manage’ Tab on your network:
2. Click on Ning Apps and you’ll be shown the Ning App directory. Quickest thing is just to search for the OpenlyLocal app:
3. Choose the OpenlyLocal application:
4. You’ll then be redirected to the Tab management screeen, where you can change the name of the Tab for the app . By default it’s “Council info :: OpenlyLocal”, but might be better to be just “Our Council” or “Council watch” if space is tight. *Important*: If you do change the name, you must click the Save Tab Settings button, otherwise just click on the link:
   
5. You should now be shown the OpenlyLocal page (if not, just click on the tab), and you should click on the ‘edit settings’ link (in the top RIGHT of the info area, not the ’settings’ link just above it and to the left).
6. Select your local authority, and then “Save Changes”
   
7. The app will then get the data from OpenlyLocal (but some may be hidden – if so so just reload the page).
8. That’s it.

There’s more features coming, but I hope you’ll agree it’s an essential addition to any Ning hyperlocal community. Comments as ever welcome, and the code behind the application will be shortly uploaded to the OpenlyLocal github tools page.

p.s. To remove the app, just go back to the tab management page and click on the ‘x’ beside the tab.

One of the things I’ve had on my ToDo list for OpenlyLocal for a while was providing a a list of links to online services provided by each Local Authority.

Seemed like something that should be on the site, and available as structured data; it also looked like it should be fairly easy to do, as it’s a service that’s sort of provided by central government (LocalDirectGov), though with some shortcomings.

The problem is that from a usability point of view the Local DirectGov interface is a bit clunky. First you choose the service you want the link for, which means using an A-Z (always a bit of a problem). This is the landing page, and as you can see you’re on the A’s.

So let’s say you want Hazardous Waste. Is that under H or W? Actually it’s under W, so click on W, and then on “Waste – Hazardous” and a new window opens (why?). You then need to enter your postcode, town or council in a form and you’ll then be (usually) given a link to click through to get to the council page.

However, depending on what you put in there and what category you want you may be asked to choose a particular council or be told that you council does not provide the service online:

Frustrating.

Now there is a limited way for external websites to interact with this service, using the ‘white-label’ Local DirectGov application. There’s even a case study. Basically, you download a list of services provided by each type of council, and then build a LocalDirectGov URL, which redirects to the council service.

Terrific. Not hard to do, even for a coder as slow as me. The only problem is that it doesn’t work. For the end user that is.

The thing is, there’s no way of knowing whether the local authority actually provides a given service online, and there’s a fair chance that the URL you’ve just built up will resolve to a bog-standard contact page, or even worse non-existent page resulting in a 404 error. Not great for users, and there appears no way of programmatically finding out if link will work, even though it’s there in Local DirectGov’s database (which is how it says that the service isn’t provided).

So, we’ve tried to fix on OpenlyLocal this and provide a better version. First we’ve collected up the useful data for each authority (i.e. where there’s a specific page to that subject, and not a 404 or generic “contact us” page). Then we’ve put it all on one page, and made it searchable too. It’s clean, simple, and works:

You can also search it from the main council page if you want to in an Ajaxy live-search way (obviously the search also works without javascript, for screenreaders and other text browsers):

Finally, you can access the data through the API as XML or JSON. So far, we’ve done a little over half the local authorities, and should have all the rest done by sometime next week (it’s just a matter of tying the remaining local authorities to their LocalDirectGov IDs, which has to be done manually).

As ever, comments, bug reports and feature requests welcome.

1. As I promised at the excellent TalkAboutLocal unconference in Stoke, I’ve been working on ways of helping non-techies use the local council data opened up by OpenlyLocal.com . The first of these is a Google Gadget, which can be added to your iGoogle page or your hyperlocal Blogger blog.

Though writing a Google Gadget can be a frustrating business, adding and using it is a doddle. Basically, if OpenlyLocal is extracting the data from your local authority (and we’ve got over 70 councils so far, with more being added every week — see the parsed council list for details), the gadget will show the key info at a glance — basic contact details, updated info, members, committees and forthcoming meetings.

For iGoogle users

1. Go to iGoogle and click on “Add stuff” in the top right-hand corner of the page.
   
2. Click on “Add feed or gadget” at the bottom of the left column
   
3. In the box that pops up enter (or paste) http://openlylocal.com/tools/gadget.xml and click “Add”. Click OK to the box asking you if you want to add the gadget.
   
4. Go back to the iGoogle home page, and the gadget will have appeared and will need you to choose your council. Choose the council, click “save”, and you’re done.
   

Note you can drag the gadget anywhere you want on the page, and even add more than one copy of the gadget, if you want to have ones for different councils.

You can also share the gadget with friends, neighbours, colleagues etc. Just click on the triangle in the top right of the gadget. This is also where you go if you want to delete the gadget.

For Blogger users

1. Make sure you’re logged in and go to the Customize area (the link is in the top right hand corner of the screen) where should choose “Layout” and “Page Elements”
   
2. On the template that you’re presented with there’s a block of boxes representing widgets/sidebars, including “Add a gadget”. Click on this.
   
3. You’ll then be given a selection of gadgets, with a menu on the left-hand side. Click the link that says “Add your own”
   
4. A form will appear with a space for the gadget’s URL. Enter (or paste) http://openlylocal.com/tools/gadget.xml and click “Add”.
   
5. You will be then asked to configure the gadget. Select the council and click the “Save” button.
   

We’re planning on introducing more features in the future, but even as it stands, I think it’s a useful tool for your iGoogle page, or if you’re a hyperlocal blogger, a great way to add up-to-date and relevant info to your blog.

By the way, you can see the code behind the gadget at http://openlylocal.com/tools/gadget.xml (it basically makes a single call to an OpenlyLocal API url — http://openlylocal.com/councils/[council_id].json — and then builds the gadget using javascript) and I’ll be creating a github project for it so you can help improve it/report bugs/request features.

The next step in making info more available is a Ning app, as quite a few hyperlocal sites seem to be using Ning as their platform of choice, and there’s someone who’s promised to write a WordPress plugin to provide the same or similar functionality to the gadget.

UPDATE:

As Helen reported in the comments, the gadget wasn’t working in Internet Explorer. Debugging it was not a pleasant process, but I’ve now found and fixed the bug (I believe). Let me know in the comments if the are any probs. It might take an hour or so before Google updates its cache with the latest code, but then should be fine.

Mentioned David Eaves’ Three Laws of Open Government Data at yesterday’s excellent Talk About Local unconference, and had a few people asking me what they were and where to find them. So here they are (from http://eaves.ca/2009/09/30/three-law-of-open-government-data/):

The Three Laws of Open Government Data:

1. If it can’t be spidered or indexed, it doesn’t exist
2. If it isn’t available in open and machine readable format, it can’t engage
3. If a legal framework doesn’t allow it to be repurposed, it doesn’t empower

There are also a few other useful links in the comments.

p.s. I don’t know David, but I really like the conciseness of these

This post is something of a brain dump about some possible common principles for open data for Local Authorities. It’s pretty much the text of a post I made to the tyoc google group, which is helping to organise a TheyWorkForYou-type website for Manchetser County Council.

It’s probably not the first post on the subject (link to other ones welcome), and certainly won’t be the last, but hopefully will provide some useful thoughts for those councils or groups working on exposing their data. (Maybe if there’s anyone else interested we can get this a bit more formalized.)

==========

My thoughts have been influenced by exposing the data from OpenlyLocal.com and also from consuming XML data from other authorities, but obviously these are only my initial ideas, and I’m using as an example OpenlyLocal urls and also those for Lichfield District Council, who’s got a great webmaster who kindly exposed all the council democratic data they could as XML.

1. The api should expose the authority’s internal UIDs (as well as the id of the record in the application if it’s not the council exposing the data). The idea is to open up the data, not creating another walled garden. See http://openlylocal.com/members/3114.xml and http://www.lichfielddc.gov.uk/site/custom_scripts/meetings_committees…
2. UIDs should be absolutely unique to the object being exposed and should not change if the name changes (so no strings for councillor IDs, as these can change if the councillors name changes, e.g. through marriage, titles or simply what they prefer to be known as).
3. The api should use and expose common identifiers when possible to allow definitive identification, e.g. Wards should expose the ONS Snac Ids. E.g. http://openlylocal.com/wards/982.xml
4. The api should given information about when the object was last updated. At the moment on OpenlyLocal, all objects have created_at and updated_at fields exposed. However, given OpenlyLocal.com and the ‘Your MCC’ projects are basically proxies we should possibly also expose a “last_checked_at” field, so the timeliness can be worked out.

Just got back from doing a presentation to the Advisory Panel on Public Sector Information

Though there’s a bit about OpenlyLocal.com (the site I run that opens up and makes accessible local government data), most of it is about the present and future of local government data, and the obstacles that need to be overcome.

The presentation seemed to go down reasonably well, despite Powerpoint messing up the formatting (it was created on Keynote), and I’m hoping we can start to get some traction on opening up local government data.

I’m embedding it and making it available under Creative Commons non-commercial share-alike licence. Comments welcome:

Sometimes you just have to get on with it. After adding a few more UK Local authorities to OpenlyLocal this weekend (we’ve now opened up information from over 70 councils), I bit the bullet and added basic entries for all the remaining UK local authorities.

[Props should go to Dane at the excellent eGovernment Register who gave me permission to use the basic info I got from there for open distribution, though other restrictions may apply -- the ONS info is Crown Copyright, for example.]

What use is that, if we’re not yet extracting the councillors, committees and meetings information from them?

Plenty, because it now means you can get programmatic access to the Full List of UK councils in one place, at one url: http://OpenlyLocal.com/councils/all. You can also get the data as XML or Json just by adding .xml or .json to the url.

Plus the XML and Json feeds also include extra information (all which on the HTML version is on the info page for each council):

• Basic info for the council — address, telephone number, website
• The ONS SNAC ids used by central govt to refer to Local Authorities
• The WhatDoTheyKnow id, so you can tie into Freedom of Information requests for that council
• Plus — for those councils that we’ve opened up democratic data for — councillors, committees and wards

If there are any errors (some of this info has been added by hand, after all, do let me know in the comments), and I’ll aim to get them sorted straight away .

Till then, I’ll leave you with an example of how to use this data — I’ve created a Google Spreadsheet of All UK Local Authorities from it (using the ImportXML function): http://spreadsheets.google.com/ccc?key=0AhOqra7su40fdFNybEV6TlFGR1hQNHY5T0hQUDduUnc&hl=en_GB

I’ve made it accessible to all, so you could just go ahead and download it to you computer, or create a copy of it on Google Spreadsheets, and use it as the basis for your hack/analysis, FoI investigation or whatever. (For those who want to understand how it works, have a look at the the functions — the main one is ImportXML, which is not well documented, but you can get by with trial-and-error and a bit of Googling).

Alternatively, have a play with the XML feed in Yahoo Pipes. Have fun.

Time for a quick update on OpenlyLocal.com — the website that opens up local government in an accessible and consistent way (a sort of local government TheyWorkForYou). Since the last post announcing the site I’ve added quite a few new features, as well as a fair bit of tweaking to the back-end .

The main ones are:

• Loads more councils (we have now opened up over 65 councils, with info on more than 4000 councillors and 2300 committees). If you council isn’t there give me a nudge and I’ll see what we can do
• Keyword filtering of committee minutes and agendas. So if you want to find a council’s minutes and agendas featuring Tesco, for example you can now do that.

  

• RSS feeds for minutes and agendas, including ones restricted to keywords. Just click on the RSS feed link at the bottom of the page.
• Even more Calendar feeds. You can now subscribe to a calendar for a whole council, an individual councillor, or a particular council committee.
• Added Wards to the information we’re extracting from websites.
• Added RDFa markup, to allow integration with other datasets and become part of the semantic web.
• Plus tons of backend improvements making it easier for me to add new councils, and loads of bugfixes

There’s still a long way to go, but OpenlyLocal does feel as if it’s moved a little beyond the proof-of-concept stage and is now genuinely useful for local democracy.

While all the hullaballo about MPs expenses has been raging, I decided to leave the parliamentary stuff for a while to the big boys, and have a look at local government instead. Wasn’t looking for any dirt; just trying to get a feel for how it worked, possibly run a few comparisons.

Unfortunately, as anyone who has been down this road knows, pulling together information on local authorities is fraught with difficulties, of which the most serious is the fact that for most things there’s no single source of data

Now, perhaps things may change now that Tim Berners-Lee is on the case, or perhaps David Cameron will really make a difference to local authority data (though he seems to misunderstand the problem — it’s not that council minutes aren’t published, it’s that they’re in a form that can’t easily be monitored, analysed or merged with other data).

So decided to scratch my own itch, as they say. The result is OpenlyLocal.com, a first draft of getting hold of and making accessible local government data in a consistent and useful form – a TheyWorkForYou Local, if you like.

To start off, I’ve concentrated on getting the basic information — current councillors, committees, membership of those committees, and minutes for those committee meetings. I started off with a little over a dozen councils (though that’s since increased to over 40 of them). Since starting the project I’ve had some great feedback and, increasingly, cooperation from many of the people working in this field, including in councils.

Don’t expect fancy styling(as you can see from the screenshot above)  – the key is to get the data, and make it accessible. That means making the data available through an API (either as XML or json) from day one, and supporting open standards (I’m looking at RDFa at the moment).

Want to help? If you can code in Ruby, then grab the code at github and start contributing patches (with tests). Or if your council’s already listed, see if you can find any errors, and let us know what other data you’d like on OpenlyLocal. Or build a mashup based on the data exposed by the API.

[Note: Voting attendance is an imperfect proxy for actual attendance, as the figure may be depressed by silent abstentions (i.e. not voting in a division, rather than voting both ‘aye’ and ‘no’) and by just turning up to vote, but failing to attend the debate. However, until Parliament provides a better measure for attendance, or more transparency of MPs actions, this is the only one we have.]

It’s recess time again, and time for MPs’ end-of-term report. I’ll leave it to others to comment on how they’ve dealt with some of the genuinely momentous events since the summer recess. This post deals solely with their voting attendance record.

First off, let’s have an overall look at the overall figures for the period:

The figures above are pretty self-explanatory. All parties have improved their attendance of votes, by 5 to 8 percentage points. Perhaps not surprising given the financial crisis.

Now let’s have a look at the main parties in detail, using the same histograms used before to show the distribution of the parties attendance figures. Interestingly (well, in a wonkish sort of way), the distributions are a bit more spread out than the long-term average. In part this is probably down to the shorter time period showing up variations that are hidden in longer period, but it’s interesting nevertheless to note that though all parties have improved their overall attendance figures, the number and proportion of Labour MPs who’ve voted in fewer than half the divisions has nearly tripled, from 11 MPs to 30 of them.

[Note: there's no significance to the width of the columns -- the recent ones are narrower so that both can be seen on the same graph]

Finally, let’s have a look at those outliers, first, the MPs who attended divisions less than than 50% of the time:

Attended fewer than 50% of divisions Oct–Dec 08

No surprise that GB’s in this list, voting in only 3 divisions (see here for the divisions he did take time out to vote on). Ditto Alistair Darling and David Cameron.

However, the rest of the list is more interesting. Some of those on the front bench, for example (e.g. Jacqui Smith, Jack Straw), surprisingly don’t make the list, i.e. they voted in at least 50% of the divisions. Ditto some of the opposition spokespeople.

But what about the backbenchers who are on the list. Possibly there’s a good reason for Margaret Hodge and Jessica Morden for failing to attend a single division — illness perhaps (though there’s nothing on either of their websites to indicate such a factor)? And what about Kali Mountford (14.3%) and Khalid Mahmood (21.4%).

If I was in their constituency, I’d like to know, particularly since they took little part in debates, either. Similarly for the low-raters for the Conservatives — Michael Mates and Tim Yeo (6 directorships!) at 35.7% each.

Now the MPs who voted more than 90% of the time:

Attended more than 90% of divisions

Some amazing figures in there. In fact, given his 100% record you wonder if the Beast of Bolsover has got a home to go to.

C.

Notes on calculations

• The above calculations were derived from the voting record freely available from the Public Whip project, and cover the period from Oct 2008 to Dec 2008. The data can be downloaded in the form of a MySQL database, and this was used together with custom MySQL queries to generate the figures.