Today, we are using the phrase “Alaveteli upgrade” rather a lot – and not just because it’s such a great tongue-twister. It’s also a notable milestone for our open-source community.
Alaveteli is our software for running Freedom of Information websites. The code can be deployed by people in other countries who wish to set up a site like our original UK one, WhatDoTheyKnow. If you’re a developer who would like to use the platform in your own country, it makes several things easier for you.
Alaveteli will now be using the Rails 3 series – the series we were previously relying on, 2, has become obsolete. One benefit is that we’re fully supported by the core Rails team for security patches. But, more significant to our aim of sharing our software with organisations around the world, it makes Alaveteli easier to use and easier to contribute to. It’s more straightforward to install, dependencies are up-to-date, code is clearer, and there’s good test coverage – all things that will really help developers get their sites up and running without a problem.
Rails cognoscenti will be aware that series 4.0 is imminent – and that we’ve only upgraded to 3.1 when 3.2 is available. We will be upgrading further in due course – it seemed sensible to progress in smaller steps. But meanwhile, we’re happy with this upgrade! The bulk of the work was done by Henare Degan and Matthew Landauer of the Open Australia Foundation, as volunteers – and we are immensely grateful to them. Thanks, guys.
This entry is cross-posted from the main mySociety blog.
Image credit: Sashi Manek (cc)
If so, mySociety has some news of an offer that may interest you.
As of May 2013 we will be offering free technical time from mySociety’s developers to a limited number of people and organisations who want to get versions of Alaveteli (Freedom of Information requests) and FixMyStreet (street problem reports) working, anywhere in the world.
All you have to do to be considered is to send us a message expressing an interest in gaining our support, and telling us a bit about you and what you hope to achieve.
If you’re selected, we’ll help modify the software to make sense in relation to your own region’s laws or local authority’s systems, and we’ll even host the service if that is a problem.
More importantly, we’ll help to explain how the software works at a technical level, so you or a local developer can really understand how the open source code works, and how to make changes to it.
This service from mySociety is worth thousands of dollars a time. We are offering it because we think it is important to support people who have the enthusiasm, but perhaps not the means, to run a service like FixMyStreet or WhatDoTheyKnow.
In order to qualify, you must be a group or an individual who can show us that you have a desire to run online civic and democratic projects like FixMyStreet or WhatDoTheyKnow in the long term, and that you have access to some kind of web developer skills. You can be anywhere in the world.
What does commitment mean? Nothing impossible, but there are a couple of requirements.
You need long-lasting enthusiasm. We’ll be looking to make sure that you understand the ongoing time and energy commitments a project like this will involve. To put it frankly, we don’t want to invest in a project that may close down after a few months. So, we’ll want to have a chat to ensure that you really know what you’re getting into.
You need access to a web developer – at least sometimes. While these kinds of sites do, to some extent, run themselves, some work will always be necessary to keep them running smoothly*. And while our developers will help you get your site off the ground, you will need your own developer too, both at set-up, and as the site continues to run.
But don’t let that put you off – we also want to hear from you even if you haven’t yet got a group in place. The important thing is that you have the desire and the motivation to drive a project to completion.
Interested? Drop us a line now and let’s talk. Don’t forget to tell us what country, city or region you’re interested in covering, and what resources you can contribute to making your site into a success.
* See the following resources to understand what sort of work is involved in running a civic or democratic website:
Photo by Ken Hawkins (CC)
This is the second part of a two-part blog post about some of our work on making it easier to deploy FixMyStreet and MapIt in new countries. This part describes how to generate KML files for every useful administrative and political boundary in OpenStreetMap.
The previous post on this subject described how to take the ID for a particular relation or way that represents a boundary in OpenStreetMap, and generate a KML file for it. That’s much of the work that we needed to create MapIt Global, but there are a few more significant steps that were required:
Efficiently extracting boundaries en masse
The code I previously described for extracting a boundary from OpenStreetMap used a public Overpass API server. This is fine for occasional boundaries, but, given that there are hundreds of thousands of boundaries in OSM, ideally we don’t want to be hitting a public server that many times – it puts a large load on that server, and is extremely slow. As an alternative, I tried parsing the OSM planet file with a SAX-based parser, but this also turned out to be very slow – multiple passes of the file were required to pick out the required nodes, ways and elements, and keeping the memory requirements down to something reasonable was tricky. (Using the PBF format would have helped a bit, but presented the same algorithmic problems.) Eventually, I decided that a better approach was simply to set up a local Overpass API server, and to query that. This is a great improvement – it allows very fast lookups of the data we need, and the query language is flexible enough to be able to retrieve huge sets of relations and ways that match the tags we’re interested in. It also means we would no longer have problems if connectivity to the remote server went down.
Another question that arose when scaling up the boundary extraction was, “Which set of tags we should consider as boundaries of interest?” On the first import, we considered any relation or way with the tag boundary=”adminstrative” and where the admin_level tag was one of “2″, “3″, “4″, … “11″. At the time, there were about 225,000 such elements that represented closed boundaries. Afterwards, it was pointed out to me that we should also include elements with the tag boundary=”political”, which includes parliamentary constituencies in the UK, for example. For later import purposes, I gave each of these boundary types a 3-letter code in MapIt, which are as follows:
- O02 (boundary="administrative", admin_level="2")
- O03 (boundary="administrative", admin_level="3")
- O11 (boundary="administrative", admin_level="11")
- OLC (boundary="political", political_division="linguistic_community")
- OIC (boundary="political", political_division="insular_council")
- OEC (boundary="political", political_division="euro_const")
- OCA (boundary="political", political_division="canton")
- OCL (boundary="political", political_division="circonscription_législative")
- OPC (boundary="political", political_division="parl_const")
- OCD (boundary="political", political_division="county_division")
- OWA (boundary="political", political_division="ward")
Importing Boundaries into MapIt
The next step in building our service was to take the 236,000 KML files generated by the previous step and import them into MapIt.
The code that creates the KML file for an element includes all its OpenStreetMap tags in the <ExtendedData/> section. On importing the KML into MapIt, there are only a few of those tags that we’re interested in – chiefly those that describe the alternative names of the area. We have to pick a canonical name for the boundary, which is currently done by taking the first of name, name:en and place_name that exists. If none of those exist, the area is given a default name of the form “Unknown name for (way|relation) with ID (element ID)”. There are also tags for the name of a country in different languages, which we also import into the database so that localized versions of the name of the boundaries will be available through MapIt with their ISO 639 language code.
Another tricky consideration when importing the data is how to deal with boundaries that have changed or disappeared since the previous import. MapIt has a concept of generations, so we could perfectly preserve the boundaries from the previous import as an earlier generation. This would certainly be desirable in one respect, since if someone is depending on the service they should be able to pick a generation that they have tested their application against, and then not have to worry about a boundary disappearing on the next import. However, with quarterly imports the size of our database would grow quite dramatically: I found that approximately 50% of the boundaries in MapIt Global had changed over the 5 months since the initial import. Our proposed compromise solution is that we will only keep the polygons associated with areas in the two most recent generations, and notify any known users of the service when a new generation is available for them to test and subsequently migrate to.
For reference, you can see the script that extracts boundaries and generates the KML files and the Django admin command for importing these files into MapIt.
The end result: MapIt Global
The aim of all this work was to create our now-launched web service, MapIt Global. As far as we know, this is the only API that allows you to take the latitude and longitude of any point on the globe, and look up the administrative and political boundaries in OpenStreetMap that surround that point. We hope it’s of use to anyone trying to build services that need to look up administrative boundaries – please let us know!
Photo credit: Hadrian’s Wall by Joe Dunckley,
One of the projects we’ve been working on at mySociety recently is that of making it easier for people to set up new versions of our sites in other countries. Something we’ve heard again and again is that for many people, setting up new web applications is a frustrating process, and that they would appreciate anything that would make it easier.
You can use these AMIs to create a running version of one of these sites on an Amazon EC2 instance with just a couple of clicks. If you haven’t used Amazon Web Services before, then you can get a Micro instance free for a year to try this out. (We have previously published an AMI for Alaveteli, which helped many people to get started with setting up their own Freedom of Information sites.)
Each AMI is created from an install script designed to be used on a clean installation of Debian squeeze or Ubuntu precise, so if you have a new server hosted elsewhere, you can use that script to similarly create a default installation of the site without being dependent on Amazon:
In addition, we’ve launched new sites with documentation for FixMyStreet and MapIt, which will tell you how to customize those sites and import data if you’ve created a running instance using one of the above methods.
These documentation sites also have improved instructions for completely manual installations of either site, for people who are comfortable with setting up PostgreSQL, Apache / nginx, PostGIS, etc.
We hope that these changes make it easier than ever before to reuse our code, and set up new sites that help people fix things that matter to them.
Photo credit: duncan
One of the common elements you will find across mySociety’s sites is that they have features designed to reduce duplicate messages or reports being sent to politicians, governments or companies. We often do this in quite a subtle way, so it is worth spelling out here how we do this across several sites:
- If you start to report a broken street light or pothole on FixMyStreet, you’ll see local problems before you start to type in your own details. This means if the problem is already there, you can see before you waste any effort.
- If you use WhatDoTheyKnow to send a Freedom of Information request to a public body, we provide a facility which encourages users to search through other people’s requests before they type their new request in.
- If the 08:10 train you take to work is always late, when you go to report it on FixMyTransport, we show you all the other problems already reported on that route. If someone else has already set up a page, you can press the big green ’join’ button, and show your support.
- If more than a handful of people try to use WriteToThem to send an exact duplicate of the same message to a politician, it will prevent it. This is because we know that politicians listen much, much more to individual messages from constituents than bulk mails.
This pattern – trying to intervene before people write identical messages or reports – is a design decision that makes a big difference to the way these sites operate. As usual with mySociety sites, this little feature seems like the sort of thing that would be quite tempting to skip when building a copy. But it really matters to the long term success of the sites. There are three reasons why.
First, there is a simple public benefit that comes from saving time. There’s no point us wasting your time if a report or request has already been sent, especially around minor issues. Saving your users time makes them happier and more likely to enjoy their experience.
Second, if you can spot that someone is about to send a duplicate message, we may be able to encourage that user to support the existing report instead of making a new one. For example, on FixMyStreet you can add an update to an existing pothole report (“it’s getting worse!”).
This feature is most visible, and most mature, on FixMyTransport, where users are clearly encouraged to ‘support’ pre-existing reports, rather than making new copies. By discouraging duplicate reports, we let people with a shared problem work together, even if this only means adding themselves as a “supporter” and doing nothing else. We know that many people search for, and find, problem reports which have turned into these little campaigns, which they then join and help. So even if they are only reading them (not joining them) that exposure can have some value to the people affected. This would be diluted if we created lots of similar reports about the same problem.
Third, we discourage duplicates for the benefit of the governments and companies receiving messages. We don’t think FixMyStreet is effective because it lets people moan: we think it’s effective because it helps local government to be effective by giving them good quality reports about local problems, in formats that area easy to handle. This good quality reporting increases the chance that the government will understand the problem and act on it, which leads to our main goal – citizen empowerment. Recipients are unlikely to help users if many of the messages they get are confused, inaccurate or duplicates, so we work on all these fronts.
So if you haven’t thought about this before, notice how the “work flow” through our sites makes you see similar problems before you’ve finished reporting your own. This is the implicit way to prevent duplication. We don’t have “Stop! Warning! Check this is a new problem!” messages, because we never want to discourage genuine users. But the careful design of the interface gently discourages, successfully, duplicate reports, and encourages supporting of other items.
It’s never possible to entirely prevent duplication. But we try hard, because it’s always better to join people together around common causes, than it is to let them struggle alone.
When you report a problem on FixMyStreet.com, the site displays a map for you to click on to indicate its exact location. Of course, you can zoom in and out of that map, but when it is first displayed, FixMyStreet needs to use an initial ‘default’ zoom level. Ideally, this is a zoom level that reduces the number of clicks required before a user can pinpoint the location of their problem.
And here’s where we encounter a tricky problem. The world is a varied place – some towns are very dense with buildings and streets crammed close together. In these areas you need to default to a zoom that’s quite ‘close in’, otherwise it can be hard to locate your problem.
But out in the countryside, we have the opposite problem. You can have huge areas where there’s nothing but blank fields or moorlands. If the default map zoom is ‘close in’ here then users are likely to see a big map full of nothingness, or maybe just a single stretch of unidentifiable road.
So, what is to be done?
The answer is this – every time you search for a location in FixMyStreet the website does a check to see whether the location you typed is in an area where a lot of people live, or very few people live.
mySociety has been storing this population density data in a webservice which we call Gaze. If the area you searched for is in a densely populated area we assume that it must be an urban location, and the map starts with a helpfully zoomed-in map. But if you’re in a sparsely populated area then it’s probably rural, so FixMyStreet starts zoomed out, making it easier to get an overview of the whole area.
Where do we get the data from? Our late colleague Chris discusses this in a blog post from 2005 — the short answer is NASA SEDAC and LandScan. It’s an interesting example of how unexpected things can happen when data is made public — if population density wasn’t available to us, we wouldn’t have been able to put this small but clever detail into FixMyStreet’s interface.
If you’re building any civic or democratic website, we know that you’ll want to make its homepage beautiful and unique. After all, you want to make a good impression on your users, and first impressions are the ones that count.
Go ahead! Dive in and make it special… but also take our advice and don’t burn all your energy on building a perfect homepage. The reason why is that many of your users will come to your site without ever seeing your homepage.
The main reason for this is the influence of search engines, Twitter and Facebook. These have already changed users’ behaviour during the short history of the World Wide Web: there was a time when people could be expected to start on the front of your site and work inwards, clicking on the links you provided, step by step. But several things now work against this: search engine results link to pages deep within your website; links are easily and routinely shared (email, Twitter, Facebook, Pinterest, even QR codes); and people are becoming accustomed to simple URLs that they might compose by hand, such as mysociety.org/contact. Realistically you must expect a user’s first page to be any* page on your website.
This means that, on every page except the homepage, your users may have arrived without any context or history — and, importantly, without the benefit of all the fabulous design time you’ve put into that glorious homepage.
Spend some time thinking about how you are treating those people. Imagine they’ve been teleported into a strange land; they know, roughly, what planet they’re on, but otherwise it’s all new to them. That’s how bewildering this could be to a user who’s clicked on a link that drops them deep into your website. It’s a common experience and it’s easy to get right, but if you get it wrong, they will just as easily teleport out again. And they will think: “that is not somewhere I want to visit again.”
None of this is very radical. But the point is that it’s very easy to overlook — especially because, if you’re one of the developers working on a site, you already know too much about how it works to be objective when you look at what you’ve made.
Here are some things you should consider:
- Do your pages instantly say what they’re for? A new user landing on an unfamiliar page will grope for some explanation – “This is a page about a politician”, or “This is a page about problems in this street”. You need to make sure you spell it out somewhere easy to see.
- Can the user easily take an action from this page? If the user was looking for more than just a quick hit of information, the chances are that they want to get something done. Don’t fall into the trap of assuming that, just because they’ve landed on this page, this is the page where they want to be. For example, if you look at any one problem report on FixMyStreet and you will find a big “Report a problem” link in yellow at the top of the page.
So you should of course create a beautiful, functional homepage, but you should never forget that almost every page on your website will be a homepage for someone.
One of the democratic software tools mySociety has built is an online petitions tool for use by governments. We built the first version for the British Prime Minister’s office, and now it is used by a variety of British local governments.
When a user wants to make a new petition, they fill in a web form, containing the petition details.
During this process, a user must choose when they want their petition to end.
The question we are exploring today is this: what’s the best way to ask for that information?
From a developer’s point of view, the end of a petition is obviously a date. This is because the petition software really needs to test whether or not today’s date comes after the petition’s end-date. So an easy — and accessible — way to ask for this data would be as a
date input, which could even helpfully trigger a calendar pop-up menu or similar widget.
But we don’t do it like that, for a couple of reasons, both of which you might miss if you were to build this from scratch. Instead, we ask:
For how long would you like your petition to accept signatures? (e.g., “1 month”; maximum 1 year)
That’s not a date input. But why?
The first reason is that the user is thinking about running a petition as something they plan to do for a period of time. Put simply, if you’re asked about your next holiday, you’ll probably imagine what you’ll be doing (“I’m going to spend two weeks lying on the beach”) rather than the end of it (“I’m going to stop lying on the beach on the 25th of August”).
In terms of interface design, this means that users will tend to think about how long something lasts differently than from how a computer or a coder will tend to think about it. The general rule is: even for something as simple as this little web form, if it’s possible to create an interface that matches the way that a user thinks about the world, then it will be less confusing for them than one that does not. Usually this means the system has to be a bit more complicated to program, in order to convert what the user types to what the computer needs.
The second reason that we ask for a duration, not a date, is that in the case of the petition, the user doesn’t actually know when their petition is going to start. Although they’re entering it into the website right now, there will nearly always be a delay before the government approves it. So most of the time, by asking for a duration, we’re avoiding raising this issue at all, and the user simply says what they mean, and it works. A petition with a duration of “3 months” will run for three months, once it goes live. A date wouldn’t survive an unspecified delay like that.
So, we ask for a duration and parse the result to turn it into a date when it goes live… but there’s an extra twist. What if the user really does know the date their petition must end? Maybe it’s a petition about closing the main road for a New Year’s Party, and it’s got to be discussed at the council meeting that’s being held on the 5th of December. Well — OK! If the user puts a date into the duration field, even though we didn’t ask for a date… we accept it. We can do some validation to make sure it’s a sensible date, of course, but basically if someone does puts a date in there, then that’s fine too.
What we hope this has shown you is that even something as simple as asking a user for an end date can be done easily (just use a date input there) or thoughtfully. The system wouldn’t break if we did it the other way. But making things as convenient as this means that we are allowing users to focus on the thing we’re trying to let them do, instead of drawing attention to how we are making them do it.
Simple things are the most easily overlooked. Two examples: a magician taking a wand out of his pocket (see? so simple that maybe you’ve never thought about why it wasn’t on the table at the start), or the home page on www.fixmystreet.com.
The first thing FixMyStreet asks for is a location. That’s so simple most people don’t think about it; but it doesn’t need to be that way. In fact, a lot of services like this would begin with a login form (“who are you?”) or a problem form (“what’s the problem you want to report?”). Well, we do it this way because we’ve learned from years of experience, experiment and, yes, mistakes.
We start off by giving you, the user, an easy problem (“where are you?”) that doesn’t offer any barrier to entry. Obviously, we’re very generous as to how you can describe that location (although that’s a different topic for another blog post). The point is we’re not asking for accuracy, since as soon as we have the location we will show you a map, on which you can almost literally pinpoint the position of your problem (for example, a pothole). Pretty much everyone can get through that first stage — and this is important if we want people to use the service.
How important? Well, we know that when building a site like FixMyStreet, it’s easy to forget that nobody in the world really needs to report a pothole. They want to, certainly, but they don’t need to. If we make it hard for them, if we make it annoying, or difficult, or intrusive, then they’ll simply give up. Not only does that pothole not get reported, but those users probably won’t bother to try to use FixMyStreet ever again.
So, before you know it, by keeping it simple at the start, we’ve got your journey under way — you’re “in”, the site’s already helping you. It’s showing you a map (a pretty map, actually) of where your problem is. Of course we’ve made it as easy as possible for you to use that map (and, yes, we also let you skip it if for some reason a map doesn’t help you). You see other problems, already reported (maybe you’ll notice that your pothole is already there — and we haven’t wasted any of your time making you tell us about it, even though you probably didn’t realise we intended right from the start to show you other people’s problems before you reported your own). Meanwhile, behind the scenes, we now know which jurisdictions are responsible for the specific area, so the drop-down menu of categories you’re about to be invited to pick from will already be relevant for the council departments (for example) that your report will be going to.
And note that we still haven’t asked you who you are. We do need to know — we send your name and contact details to the council as part of your report — but you didn’t come to FixMyStreet to tell us who you are, you came first and foremost to report the problem. So we focus on the reporting, and when that is all done then, finally, we can do the identity checks.
Of course there’s a lot more to it than this, and it’s not just civic sites like ours that use such techniques (most modern commerce sites have realised the value of making it very easy to take your order before any other processing; many governmental websites have not). But we wanted to show you that if you want to build sites that people use, you should be as clever as a magician, and the secret to that is often keeping it simple — deceptively simple — on the outside.
We’ve mentioned Components before on this blog – they’re modules which you can slot into your website, and which should save you a lot of time and effort. Today, we’re pleased to announce that a fundamental Component is ready for use – MapIt Global.
This Component will match geographical points to administrative areas anywhere in the world. So for example, you can use it on sites like FixMyStreet, where we ask the user for a zipcode/postcode, and then automatically knows which council to send their report to.
mySociety’s Director Tom has written an in-depth blog post about MapIt Global. You’ll want to read it if you’re thinking of building a site or app for reporting street faults, for contacting elected representatives, for parliamentary monitoring… or just maybe you have ideas for a type of website that we haven’t even thought of. We look forward to seeing how MapIt Global will be used.
Questions? Thoughts? We’d love to hear them, either on this post or on Tom’s.