Manifesto Trackers - or Promise Trackers - are an essential part of any well-functioning democracy. So why are they not more common? Only a few make it to launch, and fewer still get any traction. The best trackers out there - the Trump-O-Meter, the Trudeau Meter and the now defunct Morsi Meter - are the exceptions to the rule. There have been some attempts in the UK too, and if our experience is anything to go by, the reason we don't see more of these is because they require a lot of work, lots of time, and lots of volunteers.
That's why, when we set up our first Tracker in 2017, we wanted to make sure that everything we do can be used by other people, and improved by other people. Part of that means being completely transparent in our process. This post explains how we did it. We're going to talk about The Process, and The Challenges...
The process of setting up and maintaining a policy tracker is quite logical and straight forward. The work has it's challenges of course (described later) but overall we found this was an interesting and useful process to go through, and was a good way to develop our knowledge and shape the website (lesson #1: get the policies ready before you design the website!)
The first step was to extract the policies from the manifesto of the party that won the the 2017 UK General Election - the Conservative Party.
A 'policy' in this context is anything that was promised in the manifesto. Typically this is identified by any kind of "we will..." statement. Of course, not all policies are as quantifiable or measurable as others, but they constitute a promise nonetheless so we have to track them in the same way (you'll notice we use the terms 'policy' and 'promise' interchangeably - this is why!). For each policy, we logged:
This is when the real work began. We have a group of brilliant volunteers Working as a team of volunteers, we analysed policy statements one at a time to provide a 'verdict' on their current status - either 'not started', 'in progress', 'done' or 'broken'. The verdict is a short summary, written in clear language. The aim is to:
All policy verdicts are peer reviewed before being published on the site.
Once the policies goes live, the actual 'tracking' work begins! Specific policies (or entire categories of policies) are allocated to volunteers, who each day look for any progress updates or significant news. We use a range of resources, and typically sign up for any available alerts from news or information sources relevant to the category. Google Alerts are useful for news mentions, and They Work For You and Parliament.uk also provide good notification services. We can also see what parliamentary business is coming up with the Parliament.uk calendar. We also get 'edit requests' from our website users, who provide valuable feedback and challenge our policy verdicts.
Now the site is live, this is the activity we will be continually learning about and improving. If you want to help in any way, we'd love to hear form you.
As with all things designed to be simple and useful, they are not easy to create! Working through the manifesto and making sure we captured everything in a fair and consistent way meant making key decisions along the way. Here is a summary of what we learnt and the decisions we made.
When you read the Conservative Party's 2017 general election manifesto, you'll see it opens with a an introduction and a foreword - neither of which contain policies or promises to track, and then a section called 'Five Giant Challenges'. Here, none of the statements of intent can be taken as ‘policies’ (e.g. “we need to invest in infrastructure and people”, or the heading “We will govern in the interests of ordinary, working families”), but they do provide the context for the policies that follow. In other words, this section describes the ‘why’ not the ‘what’.
Each section starts with a page of bullet points covering the key deliverables in that sections. These sentences were not taken as policies, but every promise within them was cross-referenced to ensure there was a corresponding policy (with more detail) later in the section or elsewhere in the manifesto. For example, at the start of the Economy section, the first bullet point reads:
“Theresa May’s Conservatives will deliver
- A strong economy built on sound public finances, low taxes, better regulation and free trade deals with markets around the world.”
We therefore checked that there were policies that pledged to create “sound public finances”, “low taxes”, “better regulation” and “free trade deals with markets around the world”. There were no circumstances where promises made here did not have a corresponding more detailed policy in that section.
We did this for each of the bullet points on each summary page at the start of the 5 main sections.
We also treated headings in the same way - for example “Keeping taxes as low as possible” (page 14). As long as there is a corresponding statement elsewhere, the heading itself was not extracted as a policy or promise.
Although there is an implied intent by making statements about what is best or right, they are not taken as policies or promises. An example of this is:
“Sound money and responsible public finances are the essential foundations of national economic success.” (page 13).
While this may or may not be true, there is no specified outcome that has been promised. Therefore we don't treat this as a policy.
Or on prison reform (page 45)...
“The £15 billion annual cost to society of reoffending shows we have so much more to do to make the penal system work better. Prisons must become places of safety, discipline and hard work, places where people are helped to turn their lives around. They should help prisoners learn English, maths and the work skills they need to get a job when they leave prison, whilst providing the help prisoners require to come off drugs and deal with mental health problems.”
These are statements of opinion with no clear outcome being promised, so therefore these are not policies to track. This kind of writing may be deemed mis-leading to some, and to others a helpful description of the government's vision. Either way, we're taking the language used at it's face value in order to be consistent and fair.
It's worth being clear that we can't consider past accomplishments as policies. For example,
"Stopping tax evasion
We have taken vigorous action against tax avoidance and evasion, closing the tax gap – the difference between the amount of tax due and the amount collected – to one of the lowest in the world."
This is not something we would track as it is not a promise, nor does it relate to this new term of government.
Seek to is a promise of action, so we track it. For example,
"We will seek to replicate all existing EU free trade agreements."
While at first glance it may be less clear how to measure "seek to replicate" but, as before, we interpret the language literally in order to remain consistent and fair. In this case however, "seek to" falls short of promising all trade agreements will be replicated, but there will be a clear 'attempt to' replicate them. The 'attempt' is therefore what we need to look for evidence for.
Where there is one main policy with several policies that are connected to it, we include just the main ‘summary’ policy. As example of this is “Update the rules on mergers and takeover”, which actually includes a number of other activities that describe how this will be achieved. On page 17 the manifesto states:
"We welcome overseas investment and want investors to succeed here but not when success is driven by aggressive asset-stripping or tax avoidance. We will update the rules that govern mergers and takeovers. This will require careful deliberation but we can state now that we will require bidders to be clear about their intentions from the outset of the bid process; that all promises and undertakings made in the course of takeover bids can be legally enforced afterwards; and that the government can require a bid to be paused to allow greater scrutiny."
In this case, “require bidders to be clear about their intentions…” or “all promises or undertakings made in the course of takeover bids can be legally enforced afterwards” are policies that have been promised as part of delivery of the overall goal of "update the rules that govern mergers and takeovers". To simplify things, we will initially use the related policies as 'acceptance criteria' for the overall summary policy. However, we're aware that some could be met while others are not. If this happens, we will split them out and track them individually.
So there you have it! We'll keep updating this blog with details of our approach and the learnings we find. This is an open source project, so all materials will be available - including the website source code and the policy analysis resources we have created - so that independent organisations in other countries can benefit from this kind of service. But we need your help to make sure this is the best service it can be. If you can offer feedback, help or support of any kind, we'd love to hear from you.