Open data and all that

Archive for the ‘api’ Category

Tweeting councillors, and why open, connected data matters

leave a comment »

Cllr Tweeps twitter directory of UK councillors closes

A couple of days ago I heard that the rather excellent CllrTweeps website was closing down. At its heart, CllrTweeps was a directory of councillors on Twitter, matching them up to council and party. My first thought was, wow that’s a shame to let that all that accumulated data go to waste.

The second was, wouldn’t it be great to put it on OpenlyLocal as open data, then not only would it be available to everyone via the API but it would also link the twitter accounts not just to the council, but also the ward, committees and so on.

So I dropped CllrTweeps a quick note, and Dafydd and James, the guys behind CllrTweeps, were well up for it. Within less than 48 hours, they’d sent me the data, agreed to make it open data, and I’d matched the first batch against the councillor records already on OpenlyLocal. What’s more, as a bonus, they’d also been collating info on councillor blogs, and so we could add that too.

Why is all this important — after all there are other pretty good sites listing councilors on twitter (although I’m not sure they’re as extensive as the CllrTweeps list)? It matters for the same reason as it was worth doing the open data Hyperlocal Directory (which is going gangbusters).

The  point is not who is maintaining the list — whether it’s twitter accounts or hyperlocal sites. What matter is whether the information is open for reuse by hyperlocal sites, bloggers, mashups, or anybody else and whether that information is it able to be connected to other bits of information, or is it — like the government data we often criticise — locked up in its own silo, not able to be matched to or combined with other information.

There’s a few tweeks we’re going to adding over the next couple of weeks, but for now if you’re a tweeting councillor (county/district/borough for the moment; parish and town councillors soon), let us know by tweeting to @OpenlyLocal with the hashtag #ukcouncillors (e.g. like this) and either the URL address of your OpenlyLocal page or your council.

Even better, you’ll automatically be added to the twitter list of UK local councillors we’ve started (see below). Finally if you have a blog and you include the URL address of that in the tweet we can add that to the info on your OpenlyLocal page.

List of UK local councillors who tweet

p.s Because the twitter accounts on OpenlyLocal are open data, there’s obviously no reason why they can’t be combined with other such listings. Hopefully we can get this arrangement to be reciprocal 😉

Written by countculture

February 13, 2010 at 12:37 pm

Yet another UK Hyperlocal Directory… but this time it’s open data

with 13 comments

At OpenlyLocal we’ve long been fans of hyperlocal sites, seeing them as a crucial part of the media future as the traditional local media dies or is cut back to a shadow of its former self.

And for a while I’ve been looking for a good directory of such sites, whether pure community ones such as HaringayOnline, ones with serious journalistic depth such as Pits’N’Pots, The Lichfield Blog, or all-rounders such as VentnorBlog (who do so many things well). Mainly I wanted it for selfish reasons, so I could make OpenlyLocal a better site, by linking to relevant hyperlocal sites on council pages.

Seems to me the community could do with such a thing too, as a way of new sites alerting the community to them and of course help with their google juice. Sure, there are a few — recently the one over at has been getting stronger, and is now pretty good — but there are problems, at least from my perspective.

So what are these, and why have I spent the past couple of days doing a UK hyperlocal directory as part of OpenlyLocal. Three reasons:

  1. Most importantly, I thought the directory should be open data which could be reused by anyone and not just by the person or company running the directory. The one at isn’t (as far as I can tell), and so if you wanted to to put the information on your website, say to allow people to see the closest hyperlocal sites to them, you couldn’t.
  2. I thought such a directory should be run by someone who wasn’t publishing a hyperlocal site or several hyperlocal sites. Perception is important in these matters, and conflicts of interests have a way of raising their head despite the best intentions.
  3. There lots of useful things we can do when we know the location of a hyperlocal site, not just put it on a map. We can use the info in mashups, we can use it in tweets, and we can find the nearest sites to a given address — if the info is made available as open data.

So after a couple of days of coding we have the first draft of the OpenlyLocal UK Hyperlocal Directory.

Here’s how it’s different:

  1. The information on the OpenlyLocal UK Hyperlocal Directory is licensed under the CC SA licence, and can be reused by anyone.
  2. You can enter you own data. Just go to, click on “Add your hyperlocal site” and fill in the form. Even specifying the area covered should be a breeze — you just drag the pointer on the map to where the blog is about, and you can also chose the radius of the circle covered by the site. We aim to approve all sites within 24 hours, and you’ll be tweeted automatically on approval from the OpenlyLocal twitter account.
  3. We allow non-commercial and commercial sites. The only sites we won’t allow are those behind a paywall or those that are pure listings sites (and don’t have a significant news or community aspect). So even local newspaper sites can be included as long as there’s free access to them.
  4. People can search for the sites closest to to them — just put an address or postcode in the search form and it’ll give you the nearest ones with distance.
  5. The list can be output as XML or JSON data for mashups or anything else, as can the results of searches for closest sites.
  6. All approved sites also appear on the correct council’s page (just choose a council when you fill in your entry).

There’s more we could do with this, but really it’s about generating a community resource, and one that’s open data. So if you want to help build the first open directory of UK hyperlocal sites first open directory of UK hyperlocal sites , get over to and click on “Add your hyperlocal site“.

And if you’ve got any suggestions, leave them in the comments or contact me on twitter.

Written by countculture

January 13, 2010 at 12:04 pm

Posted in api, hyperlocal, open data

Online services provided by your council: rewiring LocalDirectGov

with 8 comments

One of the things I’ve had on my ToDo list for OpenlyLocal for a while was providing a a list of links to online services provided by each Local Authority.

Seemed like something that should be on the site, and available as structured data; it also looked like it should be fairly easy to do, as it’s a service that’s sort of provided by central government (LocalDirectGov), though with some shortcomings.

The problem is that from a usability point of view the Local DirectGov interface is a bit clunky. First you choose the service you want the link for, which means using an A-Z (always a bit of a problem). This is the landing page, and as you can see you’re on the A’s.

LocalDirectGov landing page

So let’s say you want Hazardous Waste. Is that under H or W? Actually it’s under W, so click on W, and then on “Waste – Hazardous” and a new window opens (why?). You then need to enter your postcode, town or council in a form and you’ll then be (usually) given a link to click through to get to the council page.

However, depending on what you put in there and what category you want you may be asked to choose a particular council or be told that you council does not provide the service online:


LocalDirectGov no service


Now there is a limited way for external websites to interact with this service, using the ‘white-label’ Local DirectGov application. There’s even a case study. Basically, you download a list of services provided by each type of council, and then build a LocalDirectGov URL, which redirects to the council service.

Terrific. Not hard to do, even for a coder as slow as me. The only problem is that it doesn’t work. For the end user that is.

The thing is, there’s no way of knowing whether the local authority actually provides a given service online, and there’s a fair chance that the URL you’ve just built up will resolve to a bog-standard contact page, or even worse non-existent page resulting in a 404 error. Not great for users, and there appears no way of programmatically finding out if link will work, even though it’s there in Local DirectGov’s database (which is how it says that the service isn’t provided).

So, we’ve tried to fix on OpenlyLocal this and provide a better version. First we’ve collected up the useful data for each authority (i.e. where there’s a specific page to that subject, and not a 404 or generic “contact us” page). Then we’ve put it all on one page, and made it searchable too. It’s clean, simple, and works:

Council Services list

You can also search it from the main council page if you want to in an Ajaxy live-search way (obviously the search also works without javascript, for screenreaders and other text browsers):

Council page with services search


Finally, you can access the data through the API as XML or JSON. So far, we’ve done a little over half the local authorities, and should have all the rest done by sometime next week (it’s just a matter of tying the remaining local authorities to their LocalDirectGov IDs, which has to be done manually).

As ever, comments, bug reports and feature requests welcome.

Written by countculture

October 27, 2009 at 4:49 pm

OpenlyLocal info on your website, Part 1: Google Gadgets

with 18 comments

  1. As I promised at the excellent TalkAboutLocal unconference in Stoke, I’ve been working on ways of helping non-techies use the local council data opened up by . The first of these is a Google Gadget, which can be added to your iGoogle page or your hyperlocal Blogger blog.

UK Councils Google Gadget

Though writing a Google Gadget can be a frustrating business, adding and using it is a doddle. Basically, if OpenlyLocal is extracting the data from your local authority (and we’ve got over 70 councils so far, with more being added every week — see the parsed council list for details), the gadget will show the key info at a glance — basic contact details, updated info, members, committees and forthcoming meetings.

For iGoogle users

  1. Go to iGoogle and click on “Add stuff” in the top right-hand corner of the page.
  2. Click on “Add feed or gadget” at the bottom of the left column
  3. In the box that pops up enter (or paste) and click “Add”. Click OK to the box asking you if you want to add the gadget.
  4. Go back to the iGoogle home page, and the gadget will have appeared and will need you to choose your council. Choose the council, click “save”, and you’re done.

Note you can drag the gadget anywhere you want on the page, and even add more than one copy of the gadget, if you want to have ones for different councils.

You can also share the gadget with friends, neighbours, colleagues etc. Just click on the triangle in the top right of the gadget. This is also where you go if you want to delete the gadget.


For Blogger users

  1. Make sure you’re logged in and go to the Customize area (the link is in the top right hand corner of the screen) where should choose “Layout” and “Page Elements”
  2. On the template that you’re presented with there’s a block of boxes representing widgets/sidebars, including “Add a gadget”. Click on this.
  3. You’ll then be given a selection of gadgets, with a menu on the left-hand side. Click the link that says “Add your own”
  4. A form will appear with a space for the gadget’s URL. Enter (or paste) and click “Add”.
  5. You will be then asked to configure the gadget. Select the council and click the “Save” button.

We’re planning on introducing more features in the future, but even as it stands, I think it’s a useful tool for your iGoogle page, or if you’re a hyperlocal blogger, a great way to add up-to-date and relevant info to your blog.

By the way, you can see the code behind the gadget at (it basically makes a single call to an OpenlyLocal API url — — and then builds the gadget using javascript) and I’ll be creating a github project for it so you can help improve it/report bugs/request features.

The next step in making info more available is a Ning app, as quite a few hyperlocal sites seem to be using Ning as their platform of choice, and there’s someone who’s promised to write a WordPress plugin to provide the same or similar functionality to the gadget.


As Helen reported in the comments, the gadget wasn’t working in Internet Explorer. Debugging it was not a pleasant process, but I’ve now found and fixed the bug (I believe). Let me know in the comments if the are any probs. It might take an hour or so before Google updates its cache with the latest code, but then should be fine.

Written by countculture

October 9, 2009 at 10:56 am

Open Data Feeds from Councils: brain dump

with one comment

This post is something of a brain dump about some possible common principles for open data for Local Authorities. It’s pretty much the text of a post I made to the tyoc google group, which is helping to organise a TheyWorkForYou-type website for Manchetser County Council.

It’s probably not the first post on the subject (link to other ones welcome), and certainly won’t be the last, but hopefully will provide some useful thoughts for those councils or groups working on exposing their data. (Maybe if there’s anyone else interested we can get this a bit more formalized.)


My thoughts have been influenced by exposing the data from and also from consuming XML data from other authorities, but obviously these are only my initial ideas, and I’m using as an example OpenlyLocal urls and also those for Lichfield District Council, who’s got a great webmaster who kindly exposed all the council democratic data they could as XML.

  1. The api should expose the authority’s internal UIDs (as well as the id of the record in the application if it’s not the council exposing the data). The idea is to open up the data, not creating another walled garden. See and…
  2. UIDs should be absolutely unique to the object being exposed and should not change if the name changes (so no strings for councillor IDs, as these can change if the councillors name changes, e.g. through marriage, titles or simply what they prefer to be known as).
  3. The api should use and expose common identifiers when possible to allow definitive identification, e.g. Wards should expose the ONS Snac Ids. E.g.
  4. The api should given information about when the object was last updated. At the moment on OpenlyLocal, all objects have created_at and updated_at fields exposed. However, given and the ‘Your MCC’ projects are basically proxies we should possibly also expose a “last_checked_at” field, so the timeliness can be worked out.

Written by countculture

September 21, 2009 at 1:24 pm

Full List of UK Councils now online (and accessible via API)

with 29 comments

Sometimes you just have to get on with it. After adding a few more UK Local authorities to OpenlyLocal this weekend (we’ve now opened up information from over 70 councils), I bit the bullet and added basic entries for all the remaining UK local authorities.

[Props should go to Dane at the excellent eGovernment Register who gave me permission to use the basic info I got from there for open distribution, though other restrictions may apply — the ONS info is Crown Copyright, for example.]

What use is that, if we’re not yet extracting the councillors, committees and meetings information from them?

Plenty, because it now means you can get programmatic access to the Full List of UK councils in one place, at one url: You can also get the data as XML or Json just by adding .xml or .json to the url.

Plus the XML and Json feeds also include extra information (all which on the HTML version is on the info page for each council):

  • Basic info for the council — address, telephone number, website
  • The ONS SNAC ids used by central govt to refer to Local Authorities
  • The WhatDoTheyKnow id, so you can tie into Freedom of Information requests for that council
  • Plus — for those councils that we’ve opened up democratic data for — councillors, committees and wards

If there are any errors (some of this info has been added by hand, after all, do let me know in the comments), and I’ll aim to get them sorted straight away .

Till then, I’ll leave you with an example of how to use this data — I’ve created a Google Spreadsheet of All UK Local Authorities from it (using the ImportXML function):

Picture 3

I’ve made it accessible to all, so you could just go ahead and download it to you computer, or create a copy of it on Google Spreadsheets, and use it as the basis for your hack/analysis, FoI investigation or whatever. (For those who want to understand how it works, have a look at the the functions — the main one is ImportXML, which is not well documented, but you can get by with trial-and-error and a bit of Googling).

Alternatively, have a play with the XML feed in Yahoo Pipes. Have fun.

Written by countculture

September 9, 2009 at 10:56 am