Open data and all that

Archive for the ‘hyperlocal’ Category

Ward maps on OpenlyLocal (& how I did it)

with 6 comments

Yesterday we added a feature to OpenlyLocal that I’ve been wanting to do since the beginning: ward maps. Why is this important? Simple, because hardly anyone knows what council ward they’re in, and nobody knows where the boundary lies, and the ward is the most basic unit of democratic accountability.

True, some councils have outlines of the wards (but without placing them on a map) and it is visible on the ONS’s Neighbourhood Statistics site, though it’s a pretty challenging user experience:

ONS Neighbourhood Statistics Ward Page

But to me, this should just be something that is there on the OpenlyLocal ward page, where the links to relevant councillors, stats and other data sit… and now it is. Here’s the page for Wembley Central ward in north London, for example, complete with zoomable maps with all the usual Google extras (satellite view etc):

Ward details and map for Wembley Central

Since tweeting about this, I’ve had a few people ask me how I did it, and so here are the details.

Much of the credit goes to the guys at #maptitude (writeup here), who put me in a room with the excellent Stuart Harrison from Lichfield District Council in a room and asked us to do something over the course of an afternoon, and to Stuart himself, who did most of the work.

Over that afternoon (on the suggestion of @danslee and with input from everyone else at the event) we did a ward comparison proof-of-concept using some boundaries Stuart had already imported into a database. The key thing, I think for both of us, was getting a few hours of focus just on the ward mapping problem, checking up on the various bits of code and interface with Google maps, and working out how to draw the outlines. Stuart used a Windows command line program and some PHP to get the boundaries; when I did it for the wards I used Ruby, the language OpenlyLocal is programmed in.

How I did it

The data comes from the newly opened-up OS BoundaryLine dataset (easiest to download from MySociety here), specifically the May 2010 data. The first problem is that this comes in the form of ESRI Shapefiles, which are standard for geo geeks, but not for data mashups, or mixing with online maps. The first stage was to import these into the database, and for this I used the GeoRuby library. Specifically I did this:

shpfile = File.join(RAILS_ROOT, "db/boundary_line/district_borough_unitary_ward_region") do |shp|
    geom = shape.geometry
    attribute_data =
    #do something with the data

This just reads the shapefile (and associated data files), goes through the shapes, extracts the geometries and associated attributes (e.g. IDs, area size etc), and lets you do something with them.

If you’re using the Ruby Spatial Adapter library you should then be able to store them in the database easily. Except…

Except, the geometries are polygons made up not of latitude/longitude points but of OS northings/eastings. More than that, they are using a different model for the shape of the Earth (OSGB36) rather than the more normal (although arguably US-centric WGS84). Now doing these two conversions is not trivial, and while there are quite a few libraries for other languages I didn’t find one for Ruby, and so I wrote one, basically converting a Javascript version line by line into Ruby (gist here).

Like all direct code conversions, it’s not pretty, it isn’t fast, but I have tested it and it seems to work well. After that it was a simple matter to add this to the loop, put it in a ruby command line script called a rake task and let it run (took about an hour or so). Specifically this is what I put in in place of the ‘do something with the data’ line:

wgs84_lat_long_groupings = geom.geometries.first.rings.collect do |ring|
   ring.points.collect{|pt| OsCoordsNewUtilities.convert_os_to_wgs84(pt.x,pt.y).reverse }
   # when creating from collections of coords supplied as x,y (where x is long, y lat)
wsg84_polygon = Polygon.from_coordinates(wgs84_lat_long_groupings)
# if you're using the Spatial Adapter you should be able to save this polygon in a 'polygon'
# type field in the database.

So far I’ve done all the district wards; I’ll do the Unitary and County Councils next, but showing them as polygons on a map you quickly start to add a lot of bulk to the page, which for people using screenreaders or mobile devices is not good. So I’m investigating other options, including Google Fusion tables (though I’d like to steer clear of Google-only solutions), and running my own tile server. Suggestions, comments welcome.

In the meantime, hope this helps, and if people will find it helpful, I’ll make the boundaries available via the API as a KML file.

Written by countculture

June 2, 2010 at 7:25 pm

Linking hyperlocal blog posts and councillors: a simple solution

with 10 comments

At last Saturday’s Talk About Local unconference, I chucked out an idea and problem I’d been playing with for a few weeks: how to show links to hyperlocal site blog posts about a given councillor, committee, meeting or what-have-you. We’ve got pages on over 8000 councillors over at OpenlyLocal and it would be good to show links to stories about them when they are written about by hyperlocal sites.

The solution I’d come up with was pretty simple, but seemed like it would have legs, and more to the point be easy to use: the pingbacks that WordPress and many other blogging systems send out (unknown to most users) when they link to another website. A pingback is a very short message to say: hey, I’ve linked to this page on your site from this blog post on my site.

Normally unless the thing you’re linking to is another blog (in which case it often appears in the comments), the pingback is ignored, but it doesn’t have to be. So my idea was to set up a pingback server on OpenlyLocal so that if you linked to a councillor, committee, meeting or poll (possibly soon candidates and documents if the interest is there), we could then link back to the piece (together with its title and few words of excerpt).

As with the OpenlyLocal widget, I got the excellent Philip John of the Lichfield Blog and JournalLocal to test it out once I’d built it. He updated three recent posts about Councillor David Smith, the departing leader of Lichfield council, with links to the OpenlyLocal page about him, and this is the result:

Related articles about councillors on OpenlyLocal

Thus your readers get links to more info about the councillor, meeting, committee etc, and you get more links and Google juice back to your site (it also acts as a loose webring for hyperlocal blogs covering the same area). One extra point, this service will only list pingbacks from approved entries in the Hyperlocal Directory. That way, we don’t have to worry about spamming from non-hyperlocal sites. As ever, comments welcome below.

Written by countculture

April 22, 2010 at 4:40 pm

New feature: search for information by postcode

leave a comment »

Why was it important that the UK government open up the geographic infrastructure? Because it makes so many location-based things that were tortuous, almost trivial.

Previously, getting open data about your local councillors, given just a postcode, was a tortuous business, requiring multiple calls to different sites. Now, it is easy. Just go to and, bingo, you’re done.

You can also just put your postcode in the search box on any OpenlyLocal page to do the same thing. And, obviously, you can also download the data as XML or JSON, and with an open data licence that allows reuse by anybody, even commercial reuse.

There’s still a little bit of tweaking to be done. I need to match up postcodes county electoral divisions, and I’m planning on adding RDF to the data types returned. Finally, it’d be great to show the ward boundaries on a map, but I think that may take a little more work.

Written by countculture

April 6, 2010 at 11:45 am

Yet another UK Hyperlocal Directory… but this time it’s open data

with 13 comments

At OpenlyLocal we’ve long been fans of hyperlocal sites, seeing them as a crucial part of the media future as the traditional local media dies or is cut back to a shadow of its former self.

And for a while I’ve been looking for a good directory of such sites, whether pure community ones such as HaringayOnline, ones with serious journalistic depth such as Pits’N’Pots, The Lichfield Blog, or all-rounders such as VentnorBlog (who do so many things well). Mainly I wanted it for selfish reasons, so I could make OpenlyLocal a better site, by linking to relevant hyperlocal sites on council pages.

Seems to me the community could do with such a thing too, as a way of new sites alerting the community to them and of course help with their google juice. Sure, there are a few — recently the one over at has been getting stronger, and is now pretty good — but there are problems, at least from my perspective.

So what are these, and why have I spent the past couple of days doing a UK hyperlocal directory as part of OpenlyLocal. Three reasons:

  1. Most importantly, I thought the directory should be open data which could be reused by anyone and not just by the person or company running the directory. The one at isn’t (as far as I can tell), and so if you wanted to to put the information on your website, say to allow people to see the closest hyperlocal sites to them, you couldn’t.
  2. I thought such a directory should be run by someone who wasn’t publishing a hyperlocal site or several hyperlocal sites. Perception is important in these matters, and conflicts of interests have a way of raising their head despite the best intentions.
  3. There lots of useful things we can do when we know the location of a hyperlocal site, not just put it on a map. We can use the info in mashups, we can use it in tweets, and we can find the nearest sites to a given address — if the info is made available as open data.

So after a couple of days of coding we have the first draft of the OpenlyLocal UK Hyperlocal Directory.

Here’s how it’s different:

  1. The information on the OpenlyLocal UK Hyperlocal Directory is licensed under the CC SA licence, and can be reused by anyone.
  2. You can enter you own data. Just go to, click on “Add your hyperlocal site” and fill in the form. Even specifying the area covered should be a breeze — you just drag the pointer on the map to where the blog is about, and you can also chose the radius of the circle covered by the site. We aim to approve all sites within 24 hours, and you’ll be tweeted automatically on approval from the OpenlyLocal twitter account.
  3. We allow non-commercial and commercial sites. The only sites we won’t allow are those behind a paywall or those that are pure listings sites (and don’t have a significant news or community aspect). So even local newspaper sites can be included as long as there’s free access to them.
  4. People can search for the sites closest to to them — just put an address or postcode in the search form and it’ll give you the nearest ones with distance.
  5. The list can be output as XML or JSON data for mashups or anything else, as can the results of searches for closest sites.
  6. All approved sites also appear on the correct council’s page (just choose a council when you fill in your entry).

There’s more we could do with this, but really it’s about generating a community resource, and one that’s open data. So if you want to help build the first open directory of UK hyperlocal sites first open directory of UK hyperlocal sites , get over to and click on “Add your hyperlocal site“.

And if you’ve got any suggestions, leave them in the comments or contact me on twitter.

Written by countculture

January 13, 2010 at 12:04 pm

Posted in api, hyperlocal, open data

OpenlyLocal info on your hyperlocal website, Part 2: Ning app

with 4 comments

A bit overdue (I’ve been talking about doing this for a couple of months), but at last there’s now a Ning app for OpenlyLocal local data so anyone who has a UK Ning hyperlocal site (well, anyone in the 90+ councils we’ve opened up data for) can now have information about their council right there in their site.

Like the OpenlyLocal Google gadget (see OpenlyLocal info on your website, Part 1: Google Gadgets), it’s fairly straightforward to use. You need to be the owner of the Ning community to add it, and then it’s automatically available to users as just another tab (like Forums, Videos, Photos, etc). Once you’ve done it you (and the other members of the community) can see the council’s key data, upcoming meetings, members and committees. More features and functionality will be added, but it’s already a very useful addition to any hyperlocal site.

This is what it looks like in action:

Screenshot of Hyperlocal Ning app from OpenlyLocal

So how do you add the OpenlyLocal application to your Ning community (NB Ning Apps need to be added by the network creator). It’s a breeze, and should take no more than a couple of minutes (probably a lot less):

  1. Go to the ‘Manage’ Tab on your network:
  2. Click on Ning Apps and you’ll be shown the Ning App directory. Quickest thing is just to search for the OpenlyLocal app:
  3. Choose the OpenlyLocal application:
  4. You’ll then be redirected to the Tab management screeen, where you can change the name of the Tab for the app . By default it’s “Council info :: OpenlyLocal”, but might be better to be just “Our Council” or “Council watch” if space is tight. *Important*: If you do change the name, you must click the Save Tab Settings button, otherwise just click on the link:
  5. You should now be shown the OpenlyLocal page (if not, just click on the tab), and you should click on the ‘edit settings’ link (in the top RIGHT of the info area, not the ‘settings’ link just above it and to the left).
  6. Select your local authority, and then “Save Changes”
  7. The app will then get the data from OpenlyLocal (but some may be hidden – if so so just reload the page).
  8. That’s it.

There’s more features coming, but I hope you’ll agree it’s an essential addition to any Ning hyperlocal community. Comments as ever welcome, and the code behind the application will be shortly uploaded to the OpenlyLocal github tools page.

p.s. To remove the app, just go back to the tab management page and click on the ‘x’ beside the tab.

Written by countculture

November 20, 2009 at 2:09 pm

OpenlyLocal info on your website, Part 1: Google Gadgets

with 18 comments

  1. As I promised at the excellent TalkAboutLocal unconference in Stoke, I’ve been working on ways of helping non-techies use the local council data opened up by . The first of these is a Google Gadget, which can be added to your iGoogle page or your hyperlocal Blogger blog.

UK Councils Google Gadget

Though writing a Google Gadget can be a frustrating business, adding and using it is a doddle. Basically, if OpenlyLocal is extracting the data from your local authority (and we’ve got over 70 councils so far, with more being added every week — see the parsed council list for details), the gadget will show the key info at a glance — basic contact details, updated info, members, committees and forthcoming meetings.

For iGoogle users

  1. Go to iGoogle and click on “Add stuff” in the top right-hand corner of the page.
  2. Click on “Add feed or gadget” at the bottom of the left column
  3. In the box that pops up enter (or paste) and click “Add”. Click OK to the box asking you if you want to add the gadget.
  4. Go back to the iGoogle home page, and the gadget will have appeared and will need you to choose your council. Choose the council, click “save”, and you’re done.

Note you can drag the gadget anywhere you want on the page, and even add more than one copy of the gadget, if you want to have ones for different councils.

You can also share the gadget with friends, neighbours, colleagues etc. Just click on the triangle in the top right of the gadget. This is also where you go if you want to delete the gadget.


For Blogger users

  1. Make sure you’re logged in and go to the Customize area (the link is in the top right hand corner of the screen) where should choose “Layout” and “Page Elements”
  2. On the template that you’re presented with there’s a block of boxes representing widgets/sidebars, including “Add a gadget”. Click on this.
  3. You’ll then be given a selection of gadgets, with a menu on the left-hand side. Click the link that says “Add your own”
  4. A form will appear with a space for the gadget’s URL. Enter (or paste) and click “Add”.
  5. You will be then asked to configure the gadget. Select the council and click the “Save” button.

We’re planning on introducing more features in the future, but even as it stands, I think it’s a useful tool for your iGoogle page, or if you’re a hyperlocal blogger, a great way to add up-to-date and relevant info to your blog.

By the way, you can see the code behind the gadget at (it basically makes a single call to an OpenlyLocal API url — — and then builds the gadget using javascript) and I’ll be creating a github project for it so you can help improve it/report bugs/request features.

The next step in making info more available is a Ning app, as quite a few hyperlocal sites seem to be using Ning as their platform of choice, and there’s someone who’s promised to write a WordPress plugin to provide the same or similar functionality to the gadget.


As Helen reported in the comments, the gadget wasn’t working in Internet Explorer. Debugging it was not a pleasant process, but I’ve now found and fixed the bug (I believe). Let me know in the comments if the are any probs. It might take an hour or so before Google updates its cache with the latest code, but then should be fine.

Written by countculture

October 9, 2009 at 10:56 am


Get every new post delivered to your Inbox.

Join 81 other followers