How do you class what you do as a success – or otherwise? All of us have our own individual metrics. For some advisers it’s purely to do with how much money they earn, for others it’s about constructing a sublime piece of financial planning for someone in need and seeing it come to fruition.
For the Money Advice Service, perhaps unsurprisingly, the way to measure its own achievements seems to be the number of people it succeeds in helping.
Last week, the MAS released new quarterly figures about its success in assisting people to access its service, as well as “what actions people are choosing to take as a result”. To dignify all this wonderful research further, the MAS has given this self-claimed index of people’s engagement with finance the title of “Quarterly Financial Capability Tracker”.
I’ll come back to this “tracker” and how useful it really is later. But it should first be acknowledged that its launch is a clever – if predictable – move from a PR perspective, because they appeal to journalists’ inherent love of regularly-published “statistics”, no matter how spurious they really are.
Surveys like this are part of any PR manager’s ABC: they allow scribes to create a succession of stories, which then appear the context of whether the figures revealed are going up or down at various time intervals, three-monthly in this instance, rather than the value of the statistics themselves.
A classic example to have entered the national consciousness is the Nationwide and Halifax house price indices. Not only do they tell us something useful about the economy, they also appeal to millions of homeowners’ natural curiosity about the value of their greatest asset.
This also helps explain why, if you are a company or a public body desperately in need of a helpful headline to help cement your place in the public consciousness, one of the first things you reach out for is a “survey”.
Ally it to the word “tracker” and with any luck credulous journalists will be writing about you for years to come. If you don’t believe me, check out the number of similar “surveys” to make an appearance across the press.
The real question, however, is that of what statistics you decide to pump out. In the case of the MAS, it is the claim that as a result of accessing its own service, 184,000 people have taken action to manage their finances in the period between April and June.
For new MAS chief executive Caroline Rookes the figures are proof positive that after several years of suffering from widely-publicised credibility issues, her organisation is moving in a positive direction.
Rookes claims: “The methodology we have created – in collaboration with behaviour change and research experts – allows us to dig deeper than ever before into this area which is extremely hard to analyse.” The results, she says are “encouraging”.
So how does the MAS arrive at these figures? It says it “worked with” pollsters Ipsos MORI, asking questions of 483 customers across a mixture of web, phone and face-to-face interviews. The precise mixture of each is not revealed, or the scientific basis of how they were chosen.
Nor have we been given any data in terms of different responses depending on the type of interview, each of the five specific categories chosen – protecting assets, providing for your family, saving regularly, preparing for retirement and managing debt – or of the specific questions used, or whether there was any follow-up to them. The latter is a particular impediment in arriving at any serious assessment of their usefulness.
For example, in the context of protecting your assets, the question to respondents was framed as follows: “Did your experience with the MAS influence your decision to take out home contents insurance?” What it did not ask is precisely what action was taken, or how influential the MAS was in the context of any decision taken.
Similarly, when it comes to saving, the MAS asks: “Did your experience with the Money Advice Service influence your decision to start saving?” We are not told what the proportion of people who answered positively to this particular question was, relative to the other categories, or whether they were influenced to save or not to save.
One the saddest features of all this pseudo-science is the fact that within it there may be genuinely useful nuggets of information. After all, the MAS now controls the £30m budget which allows face-to-face and telephone-based debt advice to be delivered.
Indeed, one of the few questions where there is any semblance of genuine response is data suggesting 78 per cent of people who received help from six debt advice delivery partners in England and Wales, including Citizens Advice, said they have started to reduce their debts as a result.
Not that these gaping lacunae in the MAS’s evidence base will trouble it very much. At the end of the day, these “statistics” are only there to provide a particular narrative: it wants us to believe that it is genuinely doing some good and that people are actually being helped by its very existence.
Unfortunately, it smells to me more of an organisation still trying to justify the inflated monthly salaries dropping into its employees’ bank accounts.
Nic Cicutti can be contacted at firstname.lastname@example.org