Our methodology

This page documents our approach to extracting and summarising the policies from the 2017 political manifestos for inclusion in the Vote for Policies 2017 UK general election survey.

Our method was discussed with and consulted on by Ben Worthy at Birkbeck College.

Objectives

For the 2017 survey, our objectives were based on improvements we wanted to make after feedback from our 2015 survey. These were:

  • Provide shorter, more succinct policy summaries that are easy for all to read (8-12 words ideally).
  • Focus on the measurable actions that are being promised.
  • Make it easy for users to easily compare policies for the same issue across parties.
  • Remove any potential for any unintentional bias in our selection process.

Method

For each manifesto:

  1. Reading through a manifesto, copy all policies and paste into a text document, under the relevant Issues headings (for some policies this may be the same or different to how the party categorises them). Maintain the relative order / priority in which they appear in the manifesto. A policy statement can only appear under one issue, not multiple issues.
  2. As policies are copied and pasted, take time to shorten each policy point by focussing on the action points (not the reported outcome) and removing extraneous words that don’t affect the meaning (e.g. “This party will commit to…”). Summarising can be quite ‘rough’ at this stage as we will revisit each policy several times during this process. For more information on this please see How to edit policies below.
  3. Once all policies have been extracted for the manifesto, paste these into the Working Document (word doc) under the relevant core headings, and add bullet numbers (for ease of counting policy numbers later). Further refine and shorten text as above.
  4. Repeat steps 2-4 for all manifestos.
  5. Create the First Draft document.
    We now have all policies from all parties in the working document. Take one Issue, and paste the policies from each party into the First Draft so they can be seen side by side. For example, the Crime policies from Conservatives are positioned next to the Crime policies for Greens, then for Labour, Lib Dems and so on.
  6. Repeat step 6 for all issues. This completes the First Draft as an issue-by-issue view of all of the policies in the Working Document.
  7. Duplicate the First Draft, renaming it the Second Draft. This is where most of the sorting and reorganisation happens. The First Draft is kept as a back-up.
  8. For the first Issue, we now look at the individual policies with the goal of reducing the number from each party down to a maximum of six or eight (see Issues section below). For each issue we look for the common or ‘core’ policies as well as priority order from the parties, and use this to refine the sub-issues that are to be included (and any that aren’t). We continue this until a sufficient number of policies remain for each parties, and the Issues list is updated to reflect those shortlisted.
  9. Repeat step 9 for each issue. We now have a set of policies to test and validate.
  10. Testing and validation.
    Several other organisations and the media publish their own manifesto policy summaries (like these). These provide a useful means of double-checking both the meaning / wording and the relative priority of each policy. If we decide to make any changes, the Issues list is updated to reflect this.
  11. To complete the refining process, a final pass is made to remove further extraneous wording where possible, so as to reduce the word count as much as possible. We now have our final set of policies from each party, for each issue.
  12. Approval from the parties.
    Create a separate document for each party, containing all of their policy summaries. Send to the direct contact for the party, and call to confirm receipt. We only send each party the policies we have summarised from their manifesto, we do not polices from other parties.
  13. Changes requested by a political party can be accepted or rejected, but should be documented in both cases and remain available for audit. You can read more about how we handle feedback from the political parties.
  14. Software testing. The policies are uploaded to a test version of the Vote for Policies (this is a complete, fully functional copy of the main site). We run test scripts to verify the policies are aligned with the correct issues and parties. We also perform manual tests ‘visually’ before final sign-off from the content owner. The policies are then made live.

How to edit policies

It’s important to make the policies as succinct as possible to reduce the amount of reading required, but without introducing any change to the meaning. Here are the principles we worked to, and some examples.

  1. Use the root form of verb.
    For consistency and ease of reading, all policies are written in the root form of the verb, not the present participle of future tense (e.g. Invest, rather than Investing, or We will invest).
  2. Remove extraneous ‘supporting verbs’ (e.g. Continue to invest becomes Invest).
  3. Shorten the wording without changing the meaning.
    Sometime the same meaning can be conveyed in fewer words, so we always attempt to do this in order to reduce the amount of reading for the user and make the survey quicker to do. For example:
    Recover the cost of medical treatment from people not resident in the UK
    can be reworded to:
    Recover the cost of medical treatment from non-UK residents. As long as we maintain the points that this applies to those who do not reside in the UK, the shorter wording is acceptable as the meaning remains unchanged.
  4. Focus on the action, not the justification.
    Often, the justification for a policy or the reported impact it will have – is particularly verbose. We have chosen not to include this aspect purely for reasons of brevity, and also to help users focus on the policy.
  5. Understand the policy meaning before making changes.
    This isn’t purely about language – sometime technical knowledge about specific policy areas is required when attempting to reduce the word count. For example:
    Comply fully with the 1951 UN Convention Relating to the Status of Refugees and honour our obligations to bona fide asylum seekers.
    Because the second part (and honour our obligations…) is essentially an explanation of the first part, we can remove this without remove without fear of being too hard…
    We also can provide explanations of some terms via a lexicon feature on the site, so don’t need to add explanations in the policy statements.

Feedback from political parties

We had great engagement from all of the political parties. All of them fed back and most had changes. Change requests are only be actioned if the proposed new wording or additional policy point satisfies the following:

  1. Is it in the manifesto? It must be included in the same manifesto – with similar wording and explanation.
  2. Is it a policy? It must meet our requirement of being ‘actionable’.
  3. Does it fit within our categorisation. It must be included in the relevant section, and isn’t one of the sub-issues we aren’t including in that issue.

If these criteria are satisfied, the change is made and a record of the change is kept for auditing and transparency purposes.

Issues / sub-issues

Below are the Issues we are covering in the 2017 survey, along with the core issues that we want to compare policies for. We developed our list of main issues and sub-issues based on a combination of:

  • what we have covered before
  • what the parties are leading with in their 2017 manifesto
  • cross-checking the above with what the media is focussing on.

The final list of issues and sub-issues is as follows:

  1. Crime (6 policies):
    Policing, terrorism and extremism, prisons, victim protection.
  2. Democracy (8 policies):
    Voting system, voting age, Scotland independence referendum, House of Lords, devolvement.
  3. Economy (8 policies):
    Deficit strategy, Industrial strategies, R&D, Infrastructure investment, connectivity, airport expansion, banking and finance sector-specific.
    Do not include: Energy tariffs.
  4. Education (8 policies):
    Strategy / approach / benefits, funding, nursery, school systems, further education, university fees.
    Do not include: Teacher pay / benefits.
  5. Environment (8 policies):
    Strategy / carbon reduction, renewables, clean air, fossil fuels, fracking / shale, marine.
    Do not include: Flood defences, bees, animal welfare.
  6. Equalities / Rights (8 policies):
    Human rights, privacy, discrimination (pay, gender, race, disability, mental health, LGBT).
  7. Europe / Brexit (8 policies):
    Brexit deal / negotiation points, UK & EU residents’ rights, single market / customs union, Erasmus programme, border with Ireland.
  8. Foreign Policy / Defence (6 policies):
    Strategic / ideological position, Trident, refugee crises, budget for international development / aid.
    Will not include: Veterans.
  9. Health / NHS (8 policies max):
    Funding, staff, service standards / approach (e.g. primary care), GPs, mental health.
  10. Housing (6 policies):
    House building targets, approach, social housing, buyers, renters, homelessness.
  11. Immigration (6 policies):
    Net migration target (plus how it’s counted), approach / integration, asylum.
  12. Jobs / Work (6 policies):
    Employment targets, quality of jobs, wages (living wage, minimum wage. pay ratios), workers’ rights.
  13. Retirement (6 policies):
    Pensions, social care, pensioner benefits.
    Do not include: Veterans
  14. Tax & Benefits (8 policies):
    Tax: Personal allowance, income tax rates, Corporation tax, specific taxes (bedroom tax, then VAT, council tax if maximum not reached).
    Do not include: Tax avoidance, HMRC staffing.
    Benefits 4: Strategic / funding, Universal Credit, housing benefit, nursery benefits.

Being consistent with issues

In order to provide a meaningful comparison across parties, it’s important to be consistent with the issues we include for each party.

For example, although policies for veterans are something we could have included in Retirement, we chose to include pensions, social care, and pensioner benefits as our priorities. Therefore, even if a party doesn’t have the maximum of 6 policies for the issues we are covering, (not the maximum of 8), we won’t add their policy for Veterans to make it up to six as this could have implied they are the only party to have a policy for veterans in their manifesto. The same is also true of ‘animal cruelty’ in the Environment issue, and also ‘tax avoidance’ in the Tax & Benefits issue.

The ‘Erasmus’ (EU student exchange) programme, while being strongly linked to Education, was referenced in the 2017 manifestos in the context of EU / Brexit negotiations. Therefore we only placed it in the Europe / Brexit issue for all parties that had a policy for it.