countculture

Open data and all that

Ward maps on OpenlyLocal (& how I did it)

with 6 comments

Yesterday we added a feature to OpenlyLocal that I’ve been wanting to do since the beginning: ward maps. Why is this important? Simple, because hardly anyone knows what council ward they’re in, and nobody knows where the boundary lies, and the ward is the most basic unit of democratic accountability.

True, some councils have outlines of the wards (but without placing them on a map) and it is visible on the ONS’s Neighbourhood Statistics site, though it’s a pretty challenging user experience:

ONS Neighbourhood Statistics Ward Page

But to me, this should just be something that is there on the OpenlyLocal ward page, where the links to relevant councillors, stats and other data sit… and now it is. Here’s the page for Wembley Central ward in north London, for example, complete with zoomable maps with all the usual Google extras (satellite view etc):

Ward details and map for Wembley Central

Since tweeting about this, I’ve had a few people ask me how I did it, and so here are the details.

Much of the credit goes to the guys at #maptitude (writeup here), who put me in a room with the excellent Stuart Harrison from Lichfield District Council in a room and asked us to do something over the course of an afternoon, and to Stuart himself, who did most of the work.

Over that afternoon (on the suggestion of @danslee and with input from everyone else at the event) we did a ward comparison proof-of-concept using some boundaries Stuart had already imported into a database. The key thing, I think for both of us, was getting a few hours of focus just on the ward mapping problem, checking up on the various bits of code and interface with Google maps, and working out how to draw the outlines. Stuart used a Windows command line program and some PHP to get the boundaries; when I did it for the wards I used Ruby, the language OpenlyLocal is programmed in.

How I did it

The data comes from the newly opened-up OS BoundaryLine dataset (easiest to download from MySociety here), specifically the May 2010 data. The first problem is that this comes in the form of ESRI Shapefiles, which are standard for geo geeks, but not for data mashups, or mixing with online maps. The first stage was to import these into the database, and for this I used the GeoRuby library. Specifically I did this:

shpfile = File.join(RAILS_ROOT, "db/boundary_line/district_borough_unitary_ward_region")
GeoRuby::Shp4r::ShpFile.open(shpfile) do |shp|
    geom = shape.geometry
    attribute_data = shape.data
    #do something with the data
end

This just reads the shapefile (and associated data files), goes through the shapes, extracts the geometries and associated attributes (e.g. IDs, area size etc), and lets you do something with them.

If you’re using the Ruby Spatial Adapter library you should then be able to store them in the database easily. Except…

Except, the geometries are polygons made up not of latitude/longitude points but of OS northings/eastings. More than that, they are using a different model for the shape of the Earth (OSGB36) rather than the more normal (although arguably US-centric WGS84). Now doing these two conversions is not trivial, and while there are quite a few libraries for other languages I didn’t find one for Ruby, and so I wrote one, basically converting a Javascript version line by line into Ruby (gist here).

Like all direct code conversions, it’s not pretty, it isn’t fast, but I have tested it and it seems to work well. After that it was a simple matter to add this to the loop, put it in a ruby command line script called a rake task and let it run (took about an hour or so). Specifically this is what I put in in place of the ‘do something with the data’ line:

wgs84_lat_long_groupings = geom.geometries.first.rings.collect do |ring|
   ring.points.collect{|pt| OsCoordsNewUtilities.convert_os_to_wgs84(pt.x,pt.y).reverse }
   # when creating from collections of coords supplied as x,y (where x is long, y lat)
end
wsg84_polygon = Polygon.from_coordinates(wgs84_lat_long_groupings)
# if you're using the Spatial Adapter you should be able to save this polygon in a 'polygon'
# type field in the database.

So far I’ve done all the district wards; I’ll do the Unitary and County Councils next, but showing them as polygons on a map you quickly start to add a lot of bulk to the page, which for people using screenreaders or mobile devices is not good. So I’m investigating other options, including Google Fusion tables (though I’d like to steer clear of Google-only solutions), and running my own tile server. Suggestions, comments welcome.

In the meantime, hope this helps, and if people will find it helpful, I’ll make the boundaries available via the API as a KML file.

Written by countculture

June 2, 2010 at 7:25 pm

6 Responses

Subscribe to comments with RSS.

  1. Yes, making .kml files available would be very much in the spirit of LOD & #opendata I would have thought – the kinda thing that tbl would have had wet dreams about a year ago.

    Bravo if you get time to do so.

    I suppose the next question should be where should they be kept and how would they be findable and query-able?

    What kind of datastore format?

    Could each ward have a url?

    ‘Twould be great boon, and thanks for sharing your trip to shape.hell not for the feint hearted!

    PaulG

    June 2, 2010 at 7:49 pm

  2. Another way of converting the eastings/northings to WGS84 is through ogr2ogr, something like this:

    ogr2ogr -s_srs EPSG:27700 -t_srs EPSG:4326 destination.shp source.shp

    Used this method for my election map experiment recently, seemed to work ok.

    You could possibly generalise the polygons which will help somewhat – there might be a Ruby implementation of the Douglas-Peucker algorithm somewhere which you could use to cut down on the number of points in each poly.

    I’m interested in the KmlLayer class as part of GMaps API v3, seems like it could be a good way of avoiding performance issues, though that would make your maps more tied to Google.

    Steve Woodward

    June 2, 2010 at 8:02 pm

  3. [the next day] I thought I had seen something similar: Unlock Places from Edina (“EDINA is a JISC National Data Centre based at the University of Edinburgh”)

    Wembley Central Ward.

    http://unlock.edina.ac.uk/ws/featureLookup?format=html&identifier=13299326

    … as kml, json, html, xml – but in this case the polygon seems to do it about distinct 32 points – is that accurate enough?

    PaulG

    June 3, 2010 at 2:10 pm

  4. The OS data is such a pain, it’s great it’s there though. It looks like it should be possible to get all the wards back by using the Edina API – I think they source their data from OS originally.

    Would be great if you could put up a single KML file though.

    James

    June 26, 2010 at 11:54 am

  5. Woot! KML ward boundary lines would be an awesome and welcomed addition. That was my prod of encouragement, now here’s a polite prod for action (please with datapoints on top)🙂

    Over a year ago I actually sent a FOI request to my local council namely Hull, for this data for a project I was about to start to help get me out of unemployment.

    Sadly the data was not forthcoming, instead I was forwarded to a URL containing some hideous non zoomable copyrighted maps. Pah!

    Having read this post my shelved project can hopefully be resurrected.

    naesk

    November 15, 2010 at 2:41 am

  6. naesk – http://mapit.mysociety.org/ has KML (and lots more) for all council, ward, Parliamentary, Super Output Area, etc. boundaries.

    Any database worth its salt nowadays can deal with co-ordinate transforms, which I strongly encourage people to use wherever possible. Mapit just imports Boundary-Line OpenData shapefiles into it directly with the database thankfully doing all the hard work of remembering the co-ordinate system used. It’s a far cry from what we used to do back in 2003-4 and very welcome too.

    Matthew Somerville

    December 9, 2010 at 1:08 am


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: