Yet another UK Hyperlocal Directory… but this time it’s open data

At OpenlyLocal we’ve long been fans of hyperlocal sites, seeing them as a crucial part of the media future as the traditional local media dies or is cut back to a shadow of its former self.

And for a while I’ve been looking for a good directory of such sites, whether pure community ones such as HaringayOnline, ones with serious journalistic depth such as Pits’N'Pots, The Lichfield Blog, or all-rounders such as VentnorBlog (who do so many things well). Mainly I wanted it for selfish reasons, so I could make OpenlyLocal a better site, by linking to relevant hyperlocal sites on council pages.

Seems to me the community could do with such a thing too, as a way of new sites alerting the community to them and of course help with their google juice. Sure, there are a few — recently the one over at hyperlocal.co.uk has been getting stronger, and is now pretty good — but there are problems, at least from my perspective.

So what are these, and why have I spent the past couple of days doing a UK hyperlocal directory as part of OpenlyLocal. Three reasons:

1. Most importantly, I thought the directory should be open data which could be reused by anyone and not just by the person or company running the directory. The one at hyperlocal.co.uk isn’t (as far as I can tell), and so if you wanted to to put the information on your website, say to allow people to see the closest hyperlocal sites to them, you couldn’t.
2. I thought such a directory should be run by someone who wasn’t publishing a hyperlocal site or several hyperlocal sites. Perception is important in these matters, and conflicts of interests have a way of raising their head despite the best intentions.
3. There lots of useful things we can do when we know the location of a hyperlocal site, not just put it on a map. We can use the info in mashups, we can use it in tweets, and we can find the nearest sites to a given address — if the info is made available as open data.

So after a couple of days of coding we have the first draft of the OpenlyLocal UK Hyperlocal Directory.

Here’s how it’s different:

1. The information on the OpenlyLocal UK Hyperlocal Directory is licensed under the CC SA licence, and can be reused by anyone.
2. You can enter you own data. Just go to http://openlylocal.com/hyperlocal_sites, click on “Add your hyperlocal site“ and fill in the form. Even specifying the area covered should be a breeze — you just drag the pointer on the map to where the blog is about, and you can also chose the radius of the circle covered by the site. We aim to approve all sites within 24 hours, and you’ll be tweeted automatically on approval from the OpenlyLocal twitter account.
3. We allow non-commercial and commercial sites. The only sites we won’t allow are those behind a paywall or those that are pure listings sites (and don’t have a significant news or community aspect). So even local newspaper sites can be included as long as there’s free access to them.
4. People can search for the sites closest to to them — just put an address or postcode in the search form and it’ll give you the nearest ones with distance.
5. The list can be output as XML or JSON data for mashups or anything else, as can the results of searches for closest sites.
6. All approved sites also appear on the correct council’s page (just choose a council when you fill in your entry).

There’s more we could do with this, but really it’s about generating a community resource, and one that’s open data. So if you want to help build the first open directory of UK hyperlocal sites first open directory of UK hyperlocal sites , get over to http://OpenlyLocal.com/hyperlocal_sites and click on “Add your hyperlocal site“.

And if you’ve got any suggestions, leave them in the comments or contact me on twitter.

The GLA and open data: did he really say that?

The launch on Friday of the Greater London Authority’s open data initiative (aka London Datastore) was a curious affair, and judging from some of the discussions in the pub after, I think that the strangeness – a joint teleconferenced event with CES Las Vegas – possibly overshadowed its significance and the boldness of the GLA’s action.

First off the technology let it down – if Skype wanted to give a demo of just how far short its video conferencing is from prime time they did a perfect job. Boris did a great impromptu stand-up routine, looking for the world like he was still up from the night before, but the people at CES in Las Vegas missed the performance and whose images and words occasionally stuttered in to life to interrupt the windows/skype error messages.

However in between the gags Boris came out with this nugget, “We will open up all the GLA’s data for free reuse”.

What does that mean, I wondered, all their data? All that’s easy to do? Does it include info from TransportForLondon (TfL), the Metropolitan Police? To be honest I sort of assumed it was Boris just paraphrasing. Nevertheless, I thought, it could be a good stick to enforce change later on.

However then it was Deputy Mayor Sir Simon Milton’s turn to give the more scripted, more plodding, more coherent version. This was the bit where we would find out what’s really going to happen. [What you need to realise that the GLA doesn’t actually have a lot of its own data – mostly it’s just some internal stuff, slices of central government data, and grouping of London council info. The good stuff is owned by those huge bodies, such as TfL and the Met, that it oversees.

So when Steve said: “I hope that our discussions with the GLA group will be fruitful and that in the short term we can encourage them to release that data which is not tied to commercial contracts and in the longer term encourage them when these contracts come up for renewal to apply different contractual principles that would allow for the release of all of their data into the public domain“, all I heard was yada yada yada.

The next bit, however, genuinely took me by surprise:

“I can confirm today, however, that as a result of our discussions around the Datastore, TfL are willing to make raw data available through the Datastore. Initially this will be data which is already available for re-use via the TfL website, including live feeds from traffic cameras, geo-coded information on the location of Tube, DLR and Overground stations, the data behind the Findaride service to locate licensed mini-cab and private hire operators and data on planned works affecting weekend Tube services.

“TfL will also be considering how best to make available detailed timetabling data for its services and welcomes examples of other data which could also be prioritised for inclusion in the Datastore such as the data on live departures and Tube incidents on TfL’s website”

So stunned was I in fact (and many others too) we that we didn’t ask any questions when he finished talking came to it , or for that matter congratulate Boris/Simon on the steps they were taking.

Yes, it’s nothing that hasn’t been done in Washington DC or San Francisco, and it isn’t as  big a deal as the Government’s open data announcement on December 7 (which got scandalously little press coverage, even in the broadsheets, yet may well turn out to be the most important act of this government).

However it is a huge step for local government in the UK and sets a benchmark for other local authorities to attain, and for the GLA to have achieved what it already has with Transport for London will only have come after a considerable trial of will, and one, significantly, that they won.

So, Simon & Boris, and all those who fought the battle with TfL, well done. Now let’s see some action with the other GLA bodies – the Met, London Development Agency, London Fire Brigade, he London Pensions Fund Authority in particular (I’m still trying to figure out its relationship to Visit London and the London Travel Watch).

Update: Video embedded below

Useful links:

• London Datastore
• List of packages that wil be available for full launch on Jan 29th
• Other blog posts on the event.
• Twitter channel

Opening up Local Spending Reports on OpenlyLocal

As I mentioned in the last post, I’ve recently added council- and ward-level statistics to OpenlyLocal, using the data from the Office of National Statistics Neighbourhood Statistics database. All very well and nice to have it in the same place as the democratic info.

However, what I was really interested in was getting and showing statistics about local areas that’s a bit more, well, meaty. So when I did that statistical backend of OpenlyLocal I wanted to make sure that I could use it for other datasets from other sources.

The first of those is now online, and it’s a good one, the 2006-07 Local Spending Report for England, published in April 2009. What is this? In a nutshell it lists the spending by category for every council in England at the time of the report (there have been a couple of new ones since then).

Now this report has been available to download online if you knew it existed, as a pretty nasty and unwieldy spreadsheet (in fact the recent report to Parliament, Making local public expenditure data public and the development of Local Spending Reports, even has several backhanded references to the inaccessibility of it).

However, unless you enjoy playing with spreadsheets (and at the very minimum know how to unhide hidden sheets and read complex formulae), it’s not much use to you. Much more helpful, I think, is an accessible table you can drill down for more details.

Let’s start with the overview:

Here you can see the total spending for each council over all categories (and also a list of the categories). Click on the magnifying glass at the right of each row and you’ll see a breakdown of spending by main category:

Local Spending breakdown for given council

Click again on the magnifying glass for any row now and you’ll see the breakdown of spending for the category of spending in that row:

Finally (for this part) if you click on the magnifying glass again you’ll get a comparison with councils of the same type (District, County, Unitary, etc) you can compare with other councils:

You can also compare between all councils. From the main page for the Local Spending Dataset, click on one of the categories and it will show you the totals for all councils. Click on one of the topics on that page and it will give you all councils for that topic. Well, hopefully you get the idea. Basically, have a play and give us some feedback.

[There'll also be a summary of the figures appearing on the front page for each council sometime in the next few hours.]

There’s no fancy javascript or visualizations yet (although we are talking with the guys at OKFN,  who do the excellent WhereDoesMyMoneyGo, about collaborating), but that may come. For the moment, we’ve kept it simple, understandable, and accessible.

Comments, mistakes found, questions all welcome in the usual locations (comments below, twitter or email at CountCulture at gmail dot com).

About your local area: ward-level statistics come to OpenlyLocal

Those who follow me on twitter will know that for the past couple of months I’ve been on-and-off looking at the Official for National Statistics Neighbourhood Statistics, and whether it would be possible and useful to show some of that information on OpenlyLocal.

Usually, when I’ve mentioned it on twitter it has usually been in the context of moaning about the less-than-friendly SOAP interface to the data (even by SOAP standards it’s unwieldy). There’s also the not insignificant issue of getting to grips with the huge amount of data, and how it’s stored on the ONS’s servers (at one stage I looked at downloading the raw data, but we’re talking about tens of thousands of files).

Still, like a person with a loose tooth, I’ve worried the problem on and off in quiet times with occasionally painful results (although the people at the ONS have been very helpful), and have now got to a level where (I think) it’s pretty useful.

Specifically, you can now see general demographic info for pretty much all the councils in England & Wales (unfortunately the ONS database doesn’t include Scotland or Northern Ireland, so if there’s anyone who can help me with those areas, I’d be pleased to hear from them).

More significantly, however, we’ve added a whole load of ward-level statistics:

Inevitably, much of the data comes from the 2001 Census (the next is due in 2011), and so it’s not bang up to date. However, it’s still useful and informative, particularly as you can compare the figures with the other wards in the council, or compare councils of similar type. Want to know which ward has the greatest proportion of people over the age of 90 years old. No prob, just click on the description (‘People aged 90 and over in this case) and you have it:

Doing the same on councils will bring up  a comparison with similar councils (e.g. District councils are compared with other district councils, London Authorities with other London Authorities):

As you can see from the list of ONS datasets, there’s huge amounts of data to be shown, and we’ve only imported a small section, in part while we’re working out the best way of making it manageable. As you can see from the religion graph, where it makes more sense for it to be graphed we’ve done it that way, and you can expect to see more of that in the futrue.

It’s also worth mentioning that there are some gaps in the ONS’s database — principally where ward boundaries have changed, or where new local authorities have been formed, and if there’s only a small amount of info for a ward or council, that’s why.

In the meantime, have a play, and if there’s a dataset you want us to expose sooner rather than later, let me know in the comments or via twitter (or email, of course).

C

p.s. In case you’re wondering the graphs and data are fully accessible so should be fine for screenreaders. The comparison tables are just plain ordinary HTML tables with a bit of CSS styling to make them look like graphs, and the pie charts have the underlying data accompanying them as tables on the page (and can be seen by anyone else just by clicking on the chart).

OpenlyLocal info on your hyperlocal website, Part 2: Ning app

A bit overdue (I’ve been talking about doing this for a couple of months), but at last there’s now a Ning app for OpenlyLocal local data so anyone who has a UK Ning hyperlocal site (well, anyone in the 90+ councils we’ve opened up data for) can now have information about their council right there in their site.

Like the OpenlyLocal Google gadget (see OpenlyLocal info on your website, Part 1: Google Gadgets), it’s fairly straightforward to use. You need to be the owner of the Ning community to add it, and then it’s automatically available to users as just another tab (like Forums, Videos, Photos, etc). Once you’ve done it you (and the other members of the community) can see the council’s key data, upcoming meetings, members and committees. More features and functionality will be added, but it’s already a very useful addition to any hyperlocal site.

This is what it looks like in action:

So how do you add the OpenlyLocal application to your Ning community (NB Ning Apps need to be added by the network creator). It’s a breeze, and should take no more than a couple of minutes (probably a lot less):

1. Go to the ‘Manage’ Tab on your network:
2. Click on Ning Apps and you’ll be shown the Ning App directory. Quickest thing is just to search for the OpenlyLocal app:
3. Choose the OpenlyLocal application:
4. You’ll then be redirected to the Tab management screeen, where you can change the name of the Tab for the app . By default it’s “Council info :: OpenlyLocal”, but might be better to be just “Our Council” or “Council watch” if space is tight. *Important*: If you do change the name, you must click the Save Tab Settings button, otherwise just click on the link:
   
5. You should now be shown the OpenlyLocal page (if not, just click on the tab), and you should click on the ‘edit settings’ link (in the top RIGHT of the info area, not the ’settings’ link just above it and to the left).
6. Select your local authority, and then “Save Changes”
   
7. The app will then get the data from OpenlyLocal (but some may be hidden – if so so just reload the page).
8. That’s it.

There’s more features coming, but I hope you’ll agree it’s an essential addition to any Ning hyperlocal community. Comments as ever welcome, and the code behind the application will be shortly uploaded to the OpenlyLocal github tools page.

p.s. To remove the app, just go back to the tab management page and click on the ‘x’ beside the tab.

Online services provided by your council: rewiring LocalDirectGov

One of the things I’ve had on my ToDo list for OpenlyLocal for a while was providing a a list of links to online services provided by each Local Authority.

Seemed like something that should be on the site, and available as structured data; it also looked like it should be fairly easy to do, as it’s a service that’s sort of provided by central government (LocalDirectGov), though with some shortcomings.

The problem is that from a usability point of view the Local DirectGov interface is a bit clunky. First you choose the service you want the link for, which means using an A-Z (always a bit of a problem). This is the landing page, and as you can see you’re on the A’s.

So let’s say you want Hazardous Waste. Is that under H or W? Actually it’s under W, so click on W, and then on “Waste – Hazardous” and a new window opens (why?). You then need to enter your postcode, town or council in a form and you’ll then be (usually) given a link to click through to get to the council page.

However, depending on what you put in there and what category you want you may be asked to choose a particular council or be told that you council does not provide the service online:

 

Frustrating.

Now there is a limited way for external websites to interact with this service, using the ‘white-label’ Local DirectGov application. There’s even a case study. Basically, you download a list of services provided by each type of council, and then build a LocalDirectGov URL, which redirects to the council service.

Terrific. Not hard to do, even for a coder as slow as me. The only problem is that it doesn’t work. For the end user that is.

The thing is, there’s no way of knowing whether the local authority actually provides a given service online, and there’s a fair chance that the URL you’ve just built up will resolve to a bog-standard contact page, or even worse non-existent page resulting in a 404 error. Not great for users, and there appears no way of programmatically finding out if link will work, even though it’s there in Local DirectGov’s database (which is how it says that the service isn’t provided).

So, we’ve tried to fix on OpenlyLocal this and provide a better version. First we’ve collected up the useful data for each authority (i.e. where there’s a specific page to that subject, and not a 404 or generic “contact us” page). Then we’ve put it all on one page, and made it searchable too. It’s clean, simple, and works:

You can also search it from the main council page if you want to in an Ajaxy live-search way (obviously the search also works without javascript, for screenreaders and other text browsers):

 

Finally, you can access the data through the API as XML or JSON. So far, we’ve done a little over half the local authorities, and should have all the rest done by sometime next week (it’s just a matter of tying the remaining local authorities to their LocalDirectGov IDs, which has to be done manually).

As ever, comments, bug reports and feature requests welcome.

OpenlyLocal info on your website, Part 1: Google Gadgets

1. As I promised at the excellent TalkAboutLocal unconference in Stoke, I’ve been working on ways of helping non-techies use the local council data opened up by OpenlyLocal.com . The first of these is a Google Gadget, which can be added to your iGoogle page or your hyperlocal Blogger blog.

Though writing a Google Gadget can be a frustrating business, adding and using it is a doddle. Basically, if OpenlyLocal is extracting the data from your local authority (and we’ve got over 70 councils so far, with more being added every week — see the parsed council list for details), the gadget will show the key info at a glance — basic contact details, updated info, members, committees and forthcoming meetings.

For iGoogle users

1. Go to iGoogle and click on “Add stuff” in the top right-hand corner of the page.
   
2. Click on “Add feed or gadget” at the bottom of the left column
   
3. In the box that pops up enter (or paste) http://openlylocal.com/tools/gadget.xml and click “Add”. Click OK to the box asking you if you want to add the gadget.
   
4. Go back to the iGoogle home page, and the gadget will have appeared and will need you to choose your council. Choose the council, click “save”, and you’re done.
   

Note you can drag the gadget anywhere you want on the page, and even add more than one copy of the gadget, if you want to have ones for different councils.

You can also share the gadget with friends, neighbours, colleagues etc. Just click on the triangle in the top right of the gadget. This is also where you go if you want to delete the gadget.

For Blogger users

1. Make sure you’re logged in and go to the Customize area (the link is in the top right hand corner of the screen) where should choose “Layout” and “Page Elements”
   
2. On the template that you’re presented with there’s a block of boxes representing widgets/sidebars, including “Add a gadget”. Click on this.
   
3. You’ll then be given a selection of gadgets, with a menu on the left-hand side. Click the link that says “Add your own”
   
4. A form will appear with a space for the gadget’s URL. Enter (or paste) http://openlylocal.com/tools/gadget.xml and click “Add”.
   
5. You will be then asked to configure the gadget. Select the council and click the “Save” button.
   

We’re planning on introducing more features in the future, but even as it stands, I think it’s a useful tool for your iGoogle page, or if you’re a hyperlocal blogger, a great way to add up-to-date and relevant info to your blog.

By the way, you can see the code behind the gadget at http://openlylocal.com/tools/gadget.xml (it basically makes a single call to an OpenlyLocal API url — http://openlylocal.com/councils/[council_id].json — and then builds the gadget using javascript) and I’ll be creating a github project for it so you can help improve it/report bugs/request features.

The next step in making info more available is a Ning app, as quite a few hyperlocal sites seem to be using Ning as their platform of choice, and there’s someone who’s promised to write a WordPress plugin to provide the same or similar functionality to the gadget.

UPDATE:

As Helen reported in the comments, the gadget wasn’t working in Internet Explorer. Debugging it was not a pleasant process, but I’ve now found and fixed the bug (I believe). Let me know in the comments if the are any probs. It might take an hour or so before Google updates its cache with the latest code, but then should be fine.

David Eaves’ Three Laws of Open Government Data

Mentioned David Eaves’ Three Laws of Open Government Data at yesterday’s excellent Talk About Local unconference, and had a few people asking me what they were and where to find them. So here they are (from http://eaves.ca/2009/09/30/three-law-of-open-government-data/):

The Three Laws of Open Government Data:

1. If it can’t be spidered or indexed, it doesn’t exist
2. If it isn’t available in open and machine readable format, it can’t engage
3. If a legal framework doesn’t allow it to be repurposed, it doesn’t empower

There are also a few other useful links in the comments.

p.s. I don’t know David, but I really like the conciseness of these

Open Data Feeds from Councils: brain dump

This post is something of a brain dump about some possible common principles for open data for Local Authorities. It’s pretty much the text of a post I made to the tyoc google group, which is helping to organise a TheyWorkForYou-type website for Manchetser County Council.

It’s probably not the first post on the subject (link to other ones welcome), and certainly won’t be the last, but hopefully will provide some useful thoughts for those councils or groups working on exposing their data. (Maybe if there’s anyone else interested we can get this a bit more formalized.)

==========

My thoughts have been influenced by exposing the data from OpenlyLocal.com and also from consuming XML data from other authorities, but obviously these are only my initial ideas, and I’m using as an example OpenlyLocal urls and also those for Lichfield District Council, who’s got a great webmaster who kindly exposed all the council democratic data they could as XML.

1. The api should expose the authority’s internal UIDs (as well as the id of the record in the application if it’s not the council exposing the data). The idea is to open up the data, not creating another walled garden. See http://openlylocal.com/members/3114.xml and http://www.lichfielddc.gov.uk/site/custom_scripts/meetings_committees…
2. UIDs should be absolutely unique to the object being exposed and should not change if the name changes (so no strings for councillor IDs, as these can change if the councillors name changes, e.g. through marriage, titles or simply what they prefer to be known as).
3. The api should use and expose common identifiers when possible to allow definitive identification, e.g. Wards should expose the ONS Snac Ids. E.g. http://openlylocal.com/wards/982.xml
4. The api should given information about when the object was last updated. At the moment on OpenlyLocal, all objects have created_at and updated_at fields exposed. However, given OpenlyLocal.com and the ‘Your MCC’ projects are basically proxies we should possibly also expose a “last_checked_at” field, so the timeliness can be worked out.

Opening Up Local Government Information: APPSI Presentation

Just got back from doing a presentation to the Advisory Panel on Public Sector Information

Though there’s a bit about OpenlyLocal.com (the site I run that opens up and makes accessible local government data), most of it is about the present and future of local government data, and the obstacles that need to be overcome.

The presentation seemed to go down reasonably well, despite Powerpoint messing up the formatting (it was created on Keynote), and I’m hoping we can start to get some traction on opening up local government data.

I’m embedding it and making it available under Creative Commons non-commercial share-alike licence. Comments welcome: