DIY.mySociety.org
"Boundary Stone 1" taken by Tim Green (CC-BY-licensed from Flickr)
This is the first part of a two-part blog post about some of our work on making it easier to deploy FixMyStreet and MapIt in new countries. This part describes how to generate KML for a given boundary in OpenStreetMap. Update: the second part is now available.
As mentioned in a previous post, we’re looking at ways of making it smoother to deploy FixMyStreet for a new country, city or use-case. Essentially there are two fundamental bits of data that you need for this:
- a mapping between a latitude and longitude (or postcode) to all the adminstrative areas that cover that point.
- a mapping between problem type and adminstrative areas to an appropriate email address for reporting that problem.
The first of those is typically provided by a service called MapIt, an open source GeoDjango web application written by Matthew Somerville. In the UK we are fortunate that official boundary data and the postcode database have now been released under an open data license from the Ordnance Survey. However, in other many other countries similar data is unavailable, or not available under reasonable licensing conditions. In such cases, though, all is not lost thanks to the extraordinary work of contributors to the OpenStreetMap project. OpenStreetMap contains high-quality administrative boundary data for many countries of the world, and we know that data submitted to the project is available under the Creative Commons Attribution-ShareAlike license, so we can reuse it in a web service like MapIt.
The first step towards being able to build an instance of MapIt based on OpenStreetMap boundary data is to be able to generate a shapefile that represents a boundary in the project. (In OpenStreetMap’s data model, boundaries are represented as either ways or relations, and those that we are interested in are tagged with boundary=administrative.) Matthew had previously written code to generate KML files for boundaries in Norway in order to help to set up an instance of MapIt for Norway, which is used by FiksGataMi. However, that script was quite specific to the organization of boundaries in that country, and it did not deal with more complex boundary topologies (e.g. enclaves), or different representations of boundaries (e.g. multiply nested relations).
So, Matthew and I wrote a new version of the code to extract a boundary from OpenStreetMap and generate a KML representation of it. The new version uses the Overpass API instead of XAPI, since it allows us to specify multiple predicates in the query and recursively fetch the ways and nodes that are contained in a relation. Once all the ways that make up a relation have been fetched (ignoring those with roles like “defaults” or “subarea”), the script tries to join each unclosed way to any other with which it shares an endpoint. We should end up with a series of closed polygons – the script exits in error if there are any unclosed ways left. We can then directly create KML from these polygons, the only subtlety being that we need to mark certain boundaries as being an inner boundary (i.e., creating a hole in a boundary) if they had the role “enclave” or “inner” in an OpenStreetMap relation. For example, the South Cambridgeshire District Council boundary has a Cambridge City Council-shaped hole in it:
Similarly, the script has to cope with multiple distinct polygons, such as the boundary of Orkney.
If you want to use this code to generate a KML representation of a closed way or boundary relation from OSM, just clone the MapIt repository and run bin/boundaries.py:
$ bin/boundaries.py --help Usage: boundaries.py [options] Options: -h, --help show this help message and exit --test Run all doctests in this file --relation=<RELATION_ID> Output KML for the OSM relation <RELATION_ID> --way=<WAY_ID> Output KML for the OSM way <WAY_ID>
For example, to generate a KML boundary for the Hottingen area of Zürich, you can do:
$ bin/boundaries.py --relation=1701449 > hottingen.kml
In the next blog post in this series, we will discuss extracting such boundaries en masse and creating a service based on them.
Romina Colman was one of the delegates at the Alaveteli conference. As well as making videos, tweeting at a good pace, and talking to everyone, Romina took the time to write up her experiences for Argentina’s national newspaper, La Nacion.
If Spanish is not your language, you can now read the English versions on the Alaveteli blog.
How to give a voice to the people
Eight steps to understanding and implementing Alaveteli
Just to finish off this collection of video clips from the Alaveteli conference, here are a couple featuring mySociety people. They were shot by Romina Colman.
First, mySociety Director Tom Steinberg, talking about what he hopes will happen as a result of the conference.
And below is Seb Bacon, Lead Developer of the Alaveteli Platform, explaining how the project began:
You can see all Romina’s videos from the Alaveteli Conference – some in English, some in Spanish – on YouTube. Romina also put together a Storify story of the conference.
Phew! Do you feel like you were there yet? If you’ve been inspired by the examples and advice from transparency hackers and activists around the world, you may be thinking about building your own Alaveteli site. Why not join our mailing list and introduce yourself? After all, if you’ve watched these videos, you’ll already be familiar with many of the people on the list!
Romina Colman is, in her own words, a Freedom of Information activist from Buenos Aires. She did a great job of recording events at AlaveteliCon, what with blogging for Argentina’s national newspaper La Nacion, copious tweeting, and videos.
Here, Romina speaks to Andrea Menapace from Italy, co-founder of Diritto di Sapere.
In this short clip (1:15), Andrea explains the current situation with Freedom of Information in Italy, and what his nascent organisation hopes to achieve.
Together with Guido Romeo (science editor at Wired Italy) I am the founder of Diritto di Sapere, a brand new organisation working on the Right to Information and Transparency in Italy. I am a lawyer by training and I have been working as a researcher and project manager in human rights and humanitarian organizations. I am currently working as a consultant for international NGOs on digital media and civil society capacity building projects.
This is a cross-post from the main mySociety blog.
Ah, summer: walks in the park, lazing in the long grass, and the sound of chirping crickets – all overlaid with the clatter of a thousand keyboards.
That may not be your idea of summer, but it’s certainly the ways ours is shaping up. We’re participating in Google’s Summer of Code, which aims to put bright young programmers in touch with Open Source organisations, for mutual benefit.
What do the students get from it? Apart from a small stipend, they have a mentored project to get their teeth into over the long summer hols, and hopefully learn a lot in the process. We, of course, see our code being used, improved and adapted – and a whole new perspective on our own work.
Candidates come from all over the world – they’re mentored remotely – so for an organisation like mySociety, this offers a great chance to get insight into the background, politics and technical landscape of another culture. Ideas for projects that may seem startlingly obvious in, say, Latin America or India would simply never have occurred to our UK-based team.
This year, mySociety were one of the 180 organisations participating. We had almost 100 enquiries, from countries including Lithuania, India, Peru, Georgia, and many other places. It’s a shame that we were only able to take on a couple of the many excellent applicants.
We made suggestions for several possible projects to whet the applicants’ appetite. Mobile apps were popular, in particular an app for FixMyTransport. Reworking WriteToThem, and creating components to complement MapIt and PopIt also ranked highly.
It was exciting to see so many ideas, and of course, hard to narrow them down.
In the end we chose two people who wanted to help improve our nascent PopIt service. PopIt will allow people to very quickly create a public database of politicians or other figures. No technical knowledge will be needed – where in the past our code has been “Just add developers”, this one is “Just add data”. We’ll host the sites for others to build on.
Our two successful applicants both had ideas for new websites that would use PopIt for their datastore, exactly the sort of advanced usage we hope to encourage. As well as making sure that PopIt actually works by using it they’ll both be creating transparency sites that will continue after their placements ends. They’ll also have the knowledge of how to set up such a site, and in our opinion that is a very good thing.
We hope to bring you more details as their projects progress, throughout the long, hot (or indeed short and wet) summer.
PS: There is a separate micro-blog where we’re currently noting some of the nitty gritty thoughts and decisions that go into building something like PopIt. If you want to see how the project goes please do subscribe! The Components mailing list is also a good way of staying in touch.
Top image by Elaine Millan, used with thanks under the Creative Commons licence.
DIY mySociety is all about making our code – and our experience – available to people who want to build similar websites in their own countries. We thought it would be helpful to list some examples of sites already using mySociety code, so you can see the variety of different possible outcomes.
It might seem like a simple task, but identifying sites in this way isn’t as straightforward as you might think – we don’t always know when people pick up our open source code! If we’ve missed any, please do comment below and we’ll add them.
There are also many sites around the world which were directly, or indirectly, “inspired by” ours. In these cases, the site’s owners have written their own code from scratch. That’s a subject – and a list – for another post. For now, here are all the international sites using mySociety’s code that we know about.
Alaveteli: our Right-to-Know Platform
WhatDoTheyKnow.com – our original Freedom of Information site
FYI.org.nz – New Zealand Freedom of Information site
Pravodaznam – Bosnia and Herzegovina Freedom of Information site
Queremossaber.br – Brazil Freedom of Information site
Informatazyrtare.org – Albania Freedom of Information site
Tuderechoasaber.es – Spain Freedom of Information site
AskTheEU – Europe Freedom of Information site
FixMyStreet: our fault-reporting Platform
FixMyStreet.com – our original fault-reporting site
Fiksgatami – Norway FixMyStreet
FixOurCity – Chennai FixMyStreet
FixMyStreet.br – Brazil FixMyStreet, based on both our code and FixMyStreet.ca from Canada
Parliamentary monitoring and access to elected representatives
TheyWorkForYou – our original parliamentary monitoring site
WriteToThem – our original ‘contact your representative’ site
Mzalendo – Kenya parliamentary monitoring site
Open Australia – Australia parliamentary monitoring site
Kildare Street – Ireland parliamentary monitoring site
Parlamany – Egypt parliamentary monitoring site
Mejlis – Tunisia parliamentary monitoring site
Find out more about the Components behind these sites, PopIt and MapIt, on the Components mailing list.
A community of people, waiting to help
Inspired by the examples above? If you’re thinking of going ahead and building your own site, we’re here to support you with our easy-to-understand guidebooks and our friendly mailing lists (see links to the right). In our online communities you’ll find many of the people who built the sites listed here. There’s no-one better to ask questions, because they’ve been through the process themselves, from early conception right up to completion.
If you are one of those people who has been through the whole process of building, launching and running a site like these (with or without our codebase), and lived to tell the tale, please shout in the comments below. And especially if you’re open to people approaching with questions. Perhaps add a note to say where you prefer to have those conversations – whether that’s via your favourite mailing lists, Twitter, email or simply in the comments to this post.
One last thought – it’s interesting to see that our code can be used for areas as small as a single city (FixMyStreet Chennai) or as large as a confederation of states (AskTheEU.org). In short, it’s scalable! How will you use it?
Image by Windell Oskay, used with thanks under the Creative Commons licence.
We asked mySociety’s Director, Tom Steinberg, a few questions. His answers help to explain DIY mySociety: what it is, why we created it, and who it’s for.
Can you briefly explain DIY mySociety?
DIY mySociety is the over-arching name for mySociety’s goal of making it really easy to set up versions of the websites we run, in countries, cities and regions around the world.
DIY mySociety consists of writings, software and face-to-face meetings that are all about helping people to get websites like WhatDoTheyKnow.com, FixMyStreet.com and TheyWorkForYou.com running wherever they are wanted, and customised to do the widely varying jobs that are required.
mySociety is a British institution and your sites deal with British politics. What is the motivation for this initiative?
mySociety started as a British NGO, a small group of staff and volunteers who built websites to help people in Britain become more powerful.
Over the last eight years we built a range of sites that worked in Britain, and that people around the world saw and wanted to copy. For a long time mySociety didn’t take many active steps to help other people, but in the last three years we’ve started working seriously to help people around the world.
Now we at mySociety think of ourselves as more of a global organisation, and we have friends with mySociety-inspired projects on every continent. But we’ve still not really done all we can to help people successfully run sites like those which we pioneered, and DIY mySociety is all about showing our intention to get really good at helping other people.
But there are already lots of mySociety-type sites in the world – do people really need your help?
Whilst there has been a huge explosion of digital democracy and transparency tools, there are still a huge number of countries, almost certainly a majority, where no such tools exist at all.
Even more serious than this is that we have seen people build copies of our tools without an understanding of the cultural or technical complexities that lie behind their surfaces. These sites normally struggle and frequently die as a consequence.
We believe that despite the massive variance between countries, almost everywhere probably has problems and needs that can be supported by some kinds of good quality democratic or transparency related web tools.
We want to help people to understand what they need to do to have the best shot to make something that will work where they are.
Surely, different countries have such different political systems that you can’t possibly offer ‘one size fits all’ codebases?
If you look at all the different websites out there that are like TheyWorkForYou.com (our parliamentary monitoring site) you will find that they are almost all built on different codebases – barely two projects share any code at all.
This is, in my view, an appalling waste of time, money and knowledge about what works.
Of course countries vary, and Parliaments most definitely do. But think how widely the companies vary that use Microsoft Office to carry out their work: almost every business in the rich world uses them, no matter what they do.
Good enough tools for monitoring parliaments will be customisable for widely varying parliaments, and they will save everyone involved precious time and money that can be spend on pushing for changes that matter.
What exactly can DIY mySociety offer?
We offer four kinds of service which we hope will be of use to people around the world.
- General knowledge – via this blog, Twitter and our project homepages
- Someone to ask questions to - in general, or on one of the specific projects
- Guides to read – currently on Alaveteli, FixMyStreet Platform
and TheyWorkForYou - Code to install and reuse
Who is it for?
We want to help anyone, anywhere who thinks that mySociety-style democracy or transparency websites (and apps) might make a positive difference where they are.
We’re setting things up so that we can be just as much help to a completely non-technical amateur as we are to a seasoned technical professional.
What should I do first if I’m interested in setting up a site like one of yours?
If you already know which project is right for you, join the appropriate mailing list and say hello [see links to the right of this page].
If you don’t know which project might be most appropriate for you, drop us an email and we can talk it through with you.
How can I help spread the word?
The most valuable thing you can do is tell us what you want to know, or what you think other people want to know. That way we can work more effectively to help people understand how we can help.
Image by Mark Hillary, used with thanks.
Two good reasons to use Alaveteli: it’s flexible, and there’s a supportive, worldwide community. So says Danko Nikolic from Serbia in this half-minute clip.
Danko is one of the founders of the Zajecar Initiative (ZI). ZI has grown into a leading civil society organization working outside the capital of Belgrade. On behalf of ZI, he has developed, co-managed and managed projects funded by various donors, such as National Endowment for Democracy (NED), USA Embassy Democracy Commission, USAID, Fund for an Open Society and others.
Zajecar Initiative is now working on the Serbian version of WriteToThem, aiming to enable the citizens of Serbia to communicate with their local representatives and MPs.
Image by Bixentro
There are quite a few people around the world who are interested in running websites like FixMyStreet.com , TheyWorkForYou.com and WhatDoTheyKnow.com in their own countries. This high level of interest is why we have set up DIY mySociety – to make it an easier to get started running your own similar sites.
One of the most common barriers to launching a new website is the lack of technology skills. This post is about finding those skills in your city, country or region.
The Basics
You don’t have to be a web developer to run a website based on mySociety code, but you do need access to someone with developer skills if you’re going to launch your own successful site.
When people are thinking about setting up a new mySociety-style website, they often assume that it can’t be done, because they aren’t technical, and they don’t know anyone who is.
Even if you’re not a coder, you might still be the right person (or group of people) to run this project. Are you good at motivating people, communicating, and organising? Then you have invaluable skills for this kind of enterprise. But that doesn’t remove the need for technical skills.
Why do you need a web developer?
You need a developer because it takes specialist skills to set up a website based on mySociety’s software.
Our software helps by saving developers months or years of work that they would have to do if starting from scratch. But it does not eliminate all the technical work – you will still need someone who knows how to build websites.
You need a web developer to:
- Install the software on a computer
- Configure the software to work in your local language(s)
- Make changes to the wording and graphic design of the site
- To add or remove features that are important to users in your country, city or region
You will need a developer to work on the project not only at the start, but for regular maintenance and improvements once it’s up and running.
What are my options for getting a developer to help?
If you are not a developer yourself, you have three basic choices for getting hold of some help.
- Pay a developer to set up your website
- Pay a web company to set up your website
- Ask a volunteer to set up your website
Obviously the third of these options looks the most attractive, but please be aware that when you are asking somone to volunteer on a website like this you are asking them to work for days and weeks for free – this is not a question of helping our for an hour or two once a week.
Working with an individual vs a company is also a decision. An individual will normally charge less per day worked, but an agency may reduce your risk of hiring someone who just can’t do the work at all. It is a balance. If you find someone who you think might be good, but you’re not sure, please don’t hesitate to ask mySociety for help in deciding.
What are you looking for in a web developer?
Judging web developers thoroughly is a tricky, expert task. However, the following rules of thumb will help you:
1. Check that they care about using the internet to help with democracy or transparency. If they are excited then they are more likely to stick with the project, and make it succeed. Do not assume that just because they know about computers, they do not care about the rest of the world!
2. Ask them to show you some kind of website or mobile app that they build entirely or mostly on their own. If they can’t show you anything then that might cause you concern. If they have something good, that’s a good sign.
3. If they really keen to win your business or volunteer, they might be willing to try setting up a basic version of the website before you start working together. If they can do this then that’s a pretty good sign that they might be the right person to help you in the longer run.
4. If they tell you “there’s no need to re-use mySociety’s software – I can build this new website from scratch more easily”, this is a bad sign. Less good web developers often underestimate how hard it is to build a website like mySociety runs, and saying this is a common give-away that you are talking to someone who might not be very likely to succeed in launching your website.
Where on earth can I find someone?
Whether they are being paid or are volunteering, the question remains: where do you find such people? The answers might surprise you – it’s not as simple as ‘put an advert in the newspaper’.
Linux User Groups are found in many countries around the world, and often contain people interested in working on interesting projects.
OpenStreetMap is a global map that is made by normal people who live around the world. It has many local branches, and can also be a good place to mail or meet to find local people to talk to.
Digital democracy and transparency mailing lists are email discussion forums for people with common interests. Developers often join mailing lists that deal with their areas of expertise or passions, so emailing a message to see if people are interested in working with you is a good way of reaching out quickly to people who are interested in the same kinds of projects as you are.
To find such mailing lists, try searching the internet for phrases like ”Digital democracy mailing list” and “transparency mailing list” in your own language, or try one of the mailing lists below to see if there is anyone interested in working with you in or near your country:
- Transparency Tech Google Group – for anyone using technology to aid transparency.
- Open Government mailing list – for those interested in open government data.
- DoWire.org – international exchange on e-democracy and online news.
- Sunlight Labs - for software developers interested in open government, government transparency, and civic hacking.
- Alaveteli mailing list – for those interested in starting a Right-to-Know website.
- FixMyStreet mailing list – for those interested in starting a street problem-reporting website.
- mySociety dev public – our general mailing list.
The Open Knowledge Foundation is a network of regional groups which bring together people interested in open data, including developers. Joining your local group – or starting a new one – will help you meet people with common interests. You’ll find a list of all local groups at the foot of this page.
BarCamps are workshop-style events, often focusing on web applications, open source technologies and open data – and are a great place to meet people with the skills you need. They happen all over the world – search for ‘barcamp‘ + the name of your city or region. If there are none near you, you can organise your own.
CityCamps are a specific type of Barcamp. They bring together local government officials, municipal employees, experts, programmers, designers, citizens and journalists to share perspectives and insights about the cities in which they live. You can check whether there’s a city camp near you here – and if there isn’t, you can start one.
If you try all the above approaches, and try to meet with local people from all these kinds of group where you live, it will not be long before you find some sympathetic people who may well be interested in your project.
Lastly, Ask here
If you’re looking for developers or any other people to join your project, feel free to leave a comment at the end of this post – make sure you say where you are and what you’re hoping to achieve.
We’re just beginning a new project here at mySociety to discover what steps we should be taking to make it easier for people to set up a site like FixMyStreet for their own country or city.
We’ve already put a lot of work into making the code base for the FixMyStreet platform generic and country-neutral, but we’d like to make the process of setting up such a site easier than it is at present.
The first step in this project is going to be contacting as many people across the world as we can who have thought about trying to set up their own version of FixMyStreet, or who’ve actually tried.
We’ll be talking to people ranging from those who have active, running sites, to those who never got past the stage of thinking it might be a nice idea. We want to find out what things presented particular difficulties, and which of the next steps we’re considering would make the greatest difference to international adoption of FixMyStreet.
Some of the things we’re interested in, for example, include:
- Would you be interested in a hosted version of FixMyStreet, or do you prefer to set up a version locally?
- How difficult did you find the process of finding administrative boundaries for your country? Are the boundaries in OpenStreetMap good enough for your use? As a last resort, would you want to draw the boundaries manually?
- Would you be interested in an automated setup procedure which deploys a new server for your locality and then just requires web-based setup?
If you have any views on this, please get in touch with Mark Longair and Tony Bowden.
