Today, we are using the phrase “Alaveteli upgrade” rather a lot – and not just because it’s such a great tongue-twister. It’s also a notable milestone for our open-source community.
Alaveteli is our software for running Freedom of Information websites. The code can be deployed by people in other countries who wish to set up a site like our original UK one, WhatDoTheyKnow. If you’re a developer who would like to use the platform in your own country, it makes several things easier for you.
Alaveteli will now be using the Rails 3 series – the series we were previously relying on, 2, has become obsolete. One benefit is that we’re fully supported by the core Rails team for security patches. But, more significant to our aim of sharing our software with organisations around the world, it makes Alaveteli easier to use and easier to contribute to. It’s more straightforward to install, dependencies are up-to-date, code is clearer, and there’s good test coverage – all things that will really help developers get their sites up and running without a problem.
Rails cognoscenti will be aware that series 4.0 is imminent – and that we’ve only upgraded to 3.1 when 3.2 is available. We will be upgrading further in due course – it seemed sensible to progress in smaller steps. But meanwhile, we’re happy with this upgrade! The bulk of the work was done by Henare Degan and Matthew Landauer of the Open Australia Foundation, as volunteers – and we are immensely grateful to them. Thanks, guys.
This entry is cross-posted from the main mySociety blog.
Image credit: Sashi Manek (cc)
If so, mySociety has some news of an offer that may interest you.
As of May 2013 we will be offering free technical time from mySociety’s developers to a limited number of people and organisations who want to get versions of Alaveteli (Freedom of Information requests) and FixMyStreet (street problem reports) working, anywhere in the world.
All you have to do to be considered is to send us a message expressing an interest in gaining our support, and telling us a bit about you and what you hope to achieve.
If you’re selected, we’ll help modify the software to make sense in relation to your own region’s laws or local authority’s systems, and we’ll even host the service if that is a problem.
More importantly, we’ll help to explain how the software works at a technical level, so you or a local developer can really understand how the open source code works, and how to make changes to it.
This service from mySociety is worth thousands of dollars a time. We are offering it because we think it is important to support people who have the enthusiasm, but perhaps not the means, to run a service like FixMyStreet or WhatDoTheyKnow.
In order to qualify, you must be a group or an individual who can show us that you have a desire to run online civic and democratic projects like FixMyStreet or WhatDoTheyKnow in the long term, and that you have access to some kind of web developer skills. You can be anywhere in the world.
What does commitment mean? Nothing impossible, but there are a couple of requirements.
You need long-lasting enthusiasm. We’ll be looking to make sure that you understand the ongoing time and energy commitments a project like this will involve. To put it frankly, we don’t want to invest in a project that may close down after a few months. So, we’ll want to have a chat to ensure that you really know what you’re getting into.
You need access to a web developer – at least sometimes. While these kinds of sites do, to some extent, run themselves, some work will always be necessary to keep them running smoothly*. And while our developers will help you get your site off the ground, you will need your own developer too, both at set-up, and as the site continues to run.
But don’t let that put you off – we also want to hear from you even if you haven’t yet got a group in place. The important thing is that you have the desire and the motivation to drive a project to completion.
Interested? Drop us a line now and let’s talk. Don’t forget to tell us what country, city or region you’re interested in covering, and what resources you can contribute to making your site into a success.
* See the following resources to understand what sort of work is involved in running a civic or democratic website:
Photo by Ken Hawkins (CC)
We’ve mentioned Components before on this blog – they’re modules which you can slot into your website, and which should save you a lot of time and effort. Today, we’re pleased to announce that a fundamental Component is ready for use – MapIt Global.
This Component will match geographical points to administrative areas anywhere in the world. So for example, you can use it on sites like FixMyStreet, where we ask the user for a zipcode/postcode, and then automatically knows which council to send their report to.
mySociety’s Director Tom has written an in-depth blog post about MapIt Global. You’ll want to read it if you’re thinking of building a site or app for reporting street faults, for contacting elected representatives, for parliamentary monitoring… or just maybe you have ideas for a type of website that we haven’t even thought of. We look forward to seeing how MapIt Global will be used.
Questions? Thoughts? We’d love to hear them, either on this post or on Tom’s.
Version 0.6 of Alaveteli, our Right to Know platform, has just been released. Developer Seb has blogged the major features of the new release. Some are for users, some for Admin:
- We’re all so used to the concept of getting updates on a ‘wall’ or a ‘stream’, thanks to Facebook and Twitter – and now users can do the same on Alaveteli sites.
- It’s difficult to moderate all unsuitable requests when you are running a high-traffic site – but now you can tap into the power of the crowd, with ‘report this request’ buttons (and a moderation list in Admin).
- The back-end looks extra smart now, thanks to some nifty code “built for and by nerds” by Twitter.
- And – one for developers – Alaveteli is now using Bundler wherever possible.
Seb’s also written a round-up of the most interesting changes and bug-fixes. If you’re running an Alaveteli site, or just thinking about it, you should head over there for a read.
Image by Caterpiya.
This is the first part of a two-part blog post about some of our work on making it easier to deploy FixMyStreet and MapIt in new countries. This part describes how to generate KML for a given boundary in OpenStreetMap. Update: the second part is now available.
As mentioned in a previous post, we’re looking at ways of making it smoother to deploy FixMyStreet for a new country, city or use-case. Essentially there are two fundamental bits of data that you need for this:
- a mapping between a latitude and longitude (or postcode) to all the adminstrative areas that cover that point.
- a mapping between problem type and adminstrative areas to an appropriate email address for reporting that problem.
The first of those is typically provided by a service called MapIt, an open source GeoDjango web application written by Matthew Somerville. In the UK we are fortunate that official boundary data and the postcode database have now been released under an open data license from the Ordnance Survey. However, in other many other countries similar data is unavailable, or not available under reasonable licensing conditions. In such cases, though, all is not lost thanks to the extraordinary work of contributors to the OpenStreetMap project. OpenStreetMap contains high-quality administrative boundary data for many countries of the world, and we know that data submitted to the project is available under the Creative Commons Attribution-ShareAlike license, so we can reuse it in a web service like MapIt.
The first step towards being able to build an instance of MapIt based on OpenStreetMap boundary data is to be able to generate a shapefile that represents a boundary in the project. (In OpenStreetMap’s data model, boundaries are represented as either ways or relations, and those that we are interested in are tagged with boundary=administrative.) Matthew had previously written code to generate KML files for boundaries in Norway in order to help to set up an instance of MapIt for Norway, which is used by FiksGataMi. However, that script was quite specific to the organization of boundaries in that country, and it did not deal with more complex boundary topologies (e.g. enclaves), or different representations of boundaries (e.g. multiply nested relations).
So, Matthew and I wrote a new version of the code to extract a boundary from OpenStreetMap and generate a KML representation of it. The new version uses the Overpass API instead of XAPI, since it allows us to specify multiple predicates in the query and recursively fetch the ways and nodes that are contained in a relation. Once all the ways that make up a relation have been fetched (ignoring those with roles like “defaults” or “subarea”), the script tries to join each unclosed way to any other with which it shares an endpoint. We should end up with a series of closed polygons – the script exits in error if there are any unclosed ways left. We can then directly create KML from these polygons, the only subtlety being that we need to mark certain boundaries as being an inner boundary (i.e., creating a hole in a boundary) if they had the role “enclave” or “inner” in an OpenStreetMap relation. For example, the South Cambridgeshire District Council boundary has a Cambridge City Council-shaped hole in it:
Similarly, the script has to cope with multiple distinct polygons, such as the boundary of Orkney.
$ bin/boundaries.py --help Usage: boundaries.py [options] Options: -h, --help show this help message and exit --test Run all doctests in this file --relation=<RELATION_ID> Output KML for the OSM relation <RELATION_ID> --way=<WAY_ID> Output KML for the OSM way <WAY_ID>
For example, to generate a KML boundary for the Hottingen area of Zürich, you can do:
$ bin/boundaries.py --relation=1701449 > hottingen.kml
In the next blog post in this series, we will discuss extracting such boundaries en masse and creating a service based on them.
At mySociety, we’re working really hard to create software tools that are attractive and easy to set up in diverse countries, cities and regions. Now we want to make sure everyone knows what we offer, and how it can be useful. This is a beginners’ guide to what mySociety can offer in the way of software tools.
First up, the basics:
- All our code is open source.
- Some of our code is available in simple-to-use packages.
- There are two types of package. We call them Platforms and Components. This post is about explaining the difference.
You can think of Platforms as flat-pack websites – like furniture that arrives in a cardboard box, with all the screws, instructions and tools included. Our Platforms provide everything you need to replicate a site like FixMyStreet.com or WhatDoTheyKnow.com in your own country, city or region, but you need to do a little work to get it up and running.
Platforms are great for people who don’t want to spend a long time reinventing the wheel, and who want to get a basic, functional site up and running as fast as possible.
We provide the software, and you just need to add:
- Data to populate it
For example, if you’re setting up a website using the FixMyStreet platform, you need the names and email addresses of every bit of government that you want to send reports to. (This isn’t as daunting as it might sound – it might just be one authority and one email address! And if not, well, we’ve had lots of success with crowd-sourcing this sort of information).
- A server to host it on
We can help you here, if it’s a problem for you. See step 1 on this page.
- Enthusiastic people to run it
Don’t forget this vital consideration! Computers are great, but they can’t do everything themselves. You will need people – volunteers or paid staff – to promote, improve, and interact with the users of your website.
The following platforms are available to download and install:
For reporting common street problems such as potholes or broken streetlights. Creates transparency about local government, at the same time as providing a practical service to users.
Our Freedom-of-Information Platform. Whether or not your country has a Right to Know law, this Platform lets people ask questions to public authorities, – and it publishes all the conversations online.
If Platforms are like a flat-pack piece of furniture, Components are more like the parts of a kitchen. When you have a kitchen built, you get to choose from a number of parts that fit together: cupboards, drawers, shelves, etc. You can ignore things you don’t want, and add in things you do – and you end up with a kitchen that suits your needs.
Components will save you a lot of time because you won’t need to create database structures, APIs, search mechanisms, admin interfaces, and so on. Just slot in a Component – like you might slot in a dishwasher – and it’s all done for you. We’ve done our best to make them easy to deploy, easy to customise, and easy to connect together.
You will definitely need technical skills, although we are working on lowering that barrier. Components cannot run on their own – they need a website to fit into. And just as with our Platforms, you’ll need data. But you don’t need a server – we host the Components ourselves.
Right now we just have one component which is fully documented and ready to use, but we’re working on followups right now. This component is called MapIt.
MapIt is a web service which you can use to work out which boundaries a point or postcode exists within. An essential foundation for geographic lookups of all kinds. You can play with the UK instance here. We use it on:
- Our parliamentary monitoring website TheyWorkForYou.com. Users are shown their own MP’s data even if they don’t know who that MP is – all they have to do is input their postcode.
- Our ‘contact your representative’ site WritetoThem.com. Users input their postcode and are shown everyone who represents them, from local to European level.
- Our street problem-reporting site FixMyStreet.com. It sends problem reports to the relevant local council, based on the co-ordinates of where the problem was reported.
We are also working on a new component for building Parliamentary Monitoring Websites on top of, called PopIt. It isn’t quite ready for prime time yet, but if you join the Components email list, you can follow progress.
Where can I get these Platforms and Components ?
They’re all on Github, as is all our code (including a lot that we haven’t made easy to re-install yet). As it’s open source code, you can take them for free.
If you want to use MapIt, or learn about our future components, please sign up for the Components mailing list at the same time – it can be an invaluable place to get support when you have questions. You can also improve the code – sharing your improvements with us is a great way to say thank you. Plus, if you have ideas for other Components that will work well with ours, we’d love to hear about them.
We don’t just build this stuff, we also help people install and run it. Keep in touch and let us know how you’re using our code, and what is or is not working. If you hit any problems, there is always someone who can help.