Financial advice is not the only paid service where researchers have asked how much it matters which practitioner you get. They have asked it of psychotherapists and of doctors, and in both cases the records were large enough to answer. Of the two, psychotherapy is the closer parallel, to professional financial advice rather than to advisers who act as salespeople. Both it and advice are a paid relationship between one professional and one client, conducted mostly by talking. Both depend on the client's trust, the professional's judgement, and what the client then does. In both, the outcome is the client's own position over months and years, not a product. The difference is that psychotherapy has measured its practitioners for forty years, doctors have been measured to find the worst among them but rarely and only sporadically the best, and financial advice has not measured its advisers at all.
The measurement of psychotherapists began from an uncomfortable finding. The pooled analysis of the antidepressant trials submitted to the United States drug regulator, published and unpublished alike, found the drugs' advantage over an inert placebo reaching the conventional criterion for clinical significance only for patients at the upper end of the very severely depressed category.1 If the medicine added little over a sugar pill, what did the person across the table add? That question produced the research this page reports. The answer, from the author's introduction to his own 2022 review of medical doctors, is that psychotherapists can have an effect on their patients' mental health that equals the strength of pharmaceutical interventions.2 This effect varies widely between individual psychotherapists.
Medicine supplies the second parallel, and the one study found in this context that can establish causation: Norwegian patients who did not choose their doctor but were allocated to a new one when their own doctor left the list. This page sets out how large the practitioner's effect is in each field, how far it ranges from one practitioner to another, how it was made visible, whether the people receiving the service could tell, and what that suggests for advice.
Sources used are the field's standard handbook and Wampold and Imel's book on what makes psychotherapy work, which map the research,3,4 the two systematic reviews of therapist effects, the two British studies of one health service, the three studies of whether therapists notice deterioration, and the four poolings of the feedback trials. The Norwegian study was read with the authors' replication code. The practice's own searches found no study of this kind in financial advice.19
Psychotherapy research calls this share the therapist effect. It is the proportion of the total variation in clients' outcomes that is associated with which therapist the client saw. A 2019 systematic review reports the earlier pooling and adds its own.5 In 2013, Baldwin and Imel reviewed 46 studies covering 1,218 therapists and 14,519 patients and found the therapist accounting for 5% of the variation overall. Five per cent sounds small, but it is not: an effect this large that is always present makes a big difference to clients over the course of a career. The share differed by setting. It was 7% in the 17 studies of ordinary practice and 3% in the 29 trials. A later, 2019 review,5 attributes that difference to the way trials are run. In its account, trials select their patients tightly, follow manuals and check that the manual was followed. They also supervise closely and use fewer therapists. Across those 46 studies the share of the variation in patients' outcomes that went with the therapist ranged from 0% to 55%.
The 2019 review searched for the studies published since. It found 20 that qualified: three randomised trials and seventeen studies of ordinary practice. Nineteen of the twenty found a therapist effect. Across all twenty the share ran from 0.2% to 29%, with a weighted average of 5%, the same figure as before. The seventeen practice-based studies averaged 5%, across a range of 0.2% to 21%. The three randomised trials averaged 8.2%, across a range of 1% to 29%. The review concludes that therapist effects are a robust phenomenon and that considerable heterogeneity exists across studies. It adds that "using an overall therapist effects statistic may lack precision", so the effects "might be better reported separately for specific clinical settings".5
Saxon and Barkham took records from the United Kingdom National Health Service for September 2000 to July 2008. The records came from its counselling and psychological therapy services in primary care. They cover 119 therapists and 10,786 patients. Each patient had at least two sessions, a planned ending, and a completed standard questionnaire before and after treatment. For a patient of average severity, the share of the variation in outcomes that went with the individual therapist was 6.6%. The share rose with severity, from 1% to 10% as patients' intake scores rose. When the model ignored how much risk each therapist's caseload carried, the share was 7.8%; allowing for it lowered the share to 6.6%. Risk here is the questionnaire's own risk items, harm to self or to others.6
Recovery rates are the more concrete measure, and the range in this study is a range of recovery rates. The average therapist's recovery rate was 58.8%, with a standard deviation of 13.7 percentage points. The standard deviation is the usual measure of how far individual figures sit from the average. Across the 119 therapists the rates ran from 23.5% to 95.6%. The paper does not publish the caseload beside each rate, so the two ends are the rates of individual therapists, not measured effects. The paper's own like-for-like comparison is narrower and can be checked. It identifies 19 therapists as less effective than the rest. It reports that of the 1,704 patients they saw, 786 recovered. The paper reports that 1,049 would have recovered at the average therapist's rate of 61.7%, a difference of 265 patients. The paper also reports that the deterioration rate among the least effective therapists was around three times that of the others.
Patients were not allocated to therapists at random. The study reports differences that remain after adjusting for the patient characteristics it measured, chiefly severity and risk at intake. The sample includes only therapists who returned a before-and-after questionnaire for at least 90% of their patients, which is a selection on record keeping. The service is British. The outcome is a score of symptoms and functioning over a course of therapy.
The same research group measured two further outcomes: whether a patient stopped attending, and whether a patient finished worse than they began. The study covered 85 therapists. It had 10,521 patients in the dropout comparison and 6,405 in the deterioration comparison. The individual therapist accounted for 12.6% of the variation in dropout and 10.1% of the variation in deterioration. Dropout rates for individual therapists ran from 1.2% to 73.2%, and deterioration rates from 0% to 15.4%. The paper also tested whether a therapist's dropout rate went with that therapist's deterioration rate. At the precision it had, it could not tell the relationship apart from zero. It reports a correlation of 0.07 with a p-value of 0.52. This means there was no discernible relationship between dropout rates and deterioration rates, a seemingly surprising result.7
The study used the same service and the same kind of records as the entry above, so the same limits apply. There was no random allocation, the adjustment covers only what was measured, and the setting is British primary care. Stopping attendance in a health service is a different act from ending a paid engagement. The study does not cover any setting outside psychological therapy.
One study took place at the counselling centre of a large university in the United States. Therapists were asked to predict which of their current clients would leave treatment worse than they arrived. The published summary states that clinicians rarely accurately predict who will not benefit from psychotherapy.8 A separate pair of studies reports that, unlike the questionnaire, therapists had considerable difficulty recognising client deterioration; one compared therapists' progress notes with recorded changes in scores, the other surveyed therapists.9 A survey of mental health professionals found that 25% rated their own clinical skill at the 90th percentile of their profession, and that none rated themselves below average; the professionals also overestimated their clients' improvement and underestimated deterioration against the published rates.10
The first study covers one counselling centre, so it cannot measure whether individual centres influence patient outcomes. That is less far-fetched than it seems: hospitals have a major influence on patients' physical health outcomes, according to the author of this page's experience. The survey figures are practitioners' self-reports. Two of the three papers are from the research group that built the questionnaire being compared.
There is currently no standardised, widely accepted way to measure the performance of financial advisers; only very crude tools exist to find out who the best, the average and the worst performing advisers are.
One reason is that the industry consists both of salespeople, who sell investment, life and disability insurance products, and of professionals, who provide a much wider range of benefits tailored to the individual.
Hence the question arises if there are any other industries or professions where the quality of individual practitioners has been successfully measured, and whether those methods could be used for financial advisers.
The answer is yes, for psychotherapists, who had an urgent need to measure practitioner performance once pharmaceuticals turned out to be of very limited use and the profession needed another way to justify its existence. What follows are further details about this successful way to measure performance.
Every figure in groups A to C rests on one instrument: a short standard questionnaire about symptoms and functioning, answered by the client at each session, most often the Outcome Questionnaire-45, which takes about five minutes.3 The client's scores are compared with those of other clients who started at the same level of being affected by mental health issues, and the questionnaire flags the client whose progress is less than expected. In the counselling-centre study the algorithm identified 100% of the clients whose condition had deteriorated by the end of treatment, and 85% of those that are deteriorating by their third session.8 As part of the trials, the flag goes back to the practitioner, sometimes with suggested tools attached.
Did psychotherapists become better at their work after receiving objective feedback about their patients' performance? A little, and the reviews disagree on how firm the improvement is. The largest pooling, 58 randomised and non-randomised studies with 21,699 patients, found a small effect, d = 0.15 standard deviations between the two groups.11 The 95% confidence interval runs from 0.10 to 0.20 – the exact definition of a confidence interval is disputed among statisticians. One explanation may be that there is a 95% probability or likelihood that the true value is between 0.10 and 0.20 and a 5% chance that the true value is lower than 0.10 or higher than 0.20, with the largest probability that the true value is fairly close to the measured value of 0.15. A 95% confidence interval that is wholly above or below zero typically denotes a statistically significant effect; an interval that straddles zero denotes that the measured effect may have been pure chance. For the clients flagged as off track the effect was 0.17. An earlier pooling of twelve studies found 0.10 (0.01 to 0.19) that did not persist at longer follow-up.12 The research group behind the questionnaire reports its own six trials, with 6,151 patients, as effective, especially for the flagged clients.13 The Cochrane review, admitting twelve randomised trials with 3,696 participants, found −0.07 (−0.16 to 0.01), indistinguishable from no effect, on evidence it graded low.14 The measurement's own effect as a treatment is therefore small or unproven. Its value is different: it is what made the therapist effect visible.
The four poolings differ in what they admit, which explains most of why they differ in what they find. Part of the trial evidence and one of the four poolings come from the research group that developed the system being tested. The psychologist Michael Lambert edited the chapters and wrote one of them;3 he also leads the research group behind the Outcome Questionnaire system, and he is an author of that pooling and of two of the detection studies. The outcomes are symptom and functioning scores over weeks to months.
One word of caution: it seems that the financial success of a financial adviser crucially depends on their ability to convince clients of their quality, regardless of the actual quality of their advice. The above can still be useful if that ability to persuade is kept in mind.
This is a study that gives proof that GPs (medical doctors) make a very large difference to their patients' survival and shows that the practitioner matters – a lot.
Norwegian law reallocates patients to a new general practitioner (GP) when their own doctor closes a practice or cuts the size of the list. A computerised procedure makes the reallocation, and about 26% of all changes of doctor in the system happen that way. Ginja, Riise, Willage and Willen used that reallocation and the whole country's register of patients and doctors: 92,205 patients aged 55 and over, moved to 2,023 new GPs between 2006 and 2015. They compared patients who had shared a doctor and lived in the same municipality, and who were then sent to different new doctors. Because the patient did not pick the doctor, the study can establish causation, which almost nothing else in this field can, and the authors show that the new GP's quality does not predict the patients' earlier health, income or education. It is the only study of its kind.15
Each GP was measured by whether the patients allocated to that GP were alive two years later, adjusted for the patients' age and sex, their previous GP and the year, with a correction for small numbers. Among these patients, 3.1% died within two years. A patient sent to a GP one standard deviation higher on that measure died 4.1 months later (interval 1.5 to 6.8 months), measured among the 13,695 patients who died within the ten-year window. For patients aged 55 to 61 at the move the gain was 8.3 months (4.8 to 11.8). The chance of dying within two years fell by 1.4 percentage points per standard deviation (from 3.0 points lower to 0.2 higher), so that comparison on its own does not exclude zero. The fall was in cancer deaths, 1.1 points on a base of 1.1% (2.1 to 0.1), which the authors attribute to earlier detection.
The spread between GPs, read on the paper's own distribution of the measure. From the 10th percentile to the 90th is 1.7 standard deviations: 7.0 months of life (2.5 to 11.5), 14 months for patients aged 55 to 61 (8.2 to 19.9), and a two-year death risk of 2.0% instead of 4.4% (from 5.1 points lower to 0.3 higher). From the 25th percentile to the 75th is 0.7 standard deviations: 2.8 months (1.0 to 4.6), 5.7 months at ages 55 to 61 (3.3 to 8.0), and 1.0 point of risk (2.0 lower to 0.1 higher). The paper's own two comparisons, made on the unadjusted spread of the measure, are larger: a GP at the 75th percentile rather than the median cuts the two-year risk by 4.9 points, and moving from the bottom quarter of GPs to the top quarter adds 3.5 months of predicted life expectancy, about 5% of the average 5.7 years from the move to death. The paper gives no intervals for those two.
The authors put a value on replacing the worst 5% of GPs with average ones: 12.1 extra months of life for each patient affected, worth $35,500 per patient and $1.2 billion in all at the Norwegian government's value of a year of life ($35,000), or $372,075 per patient and $12.4 billion at the value used by the United States Food and Drug Administration ($369,000). The paper makes no such calculation for the best 5%.
The better GPs' patients had slightly fewer consultations and cost the health system about 9% less, had fewer acute hospital contacts (2.4 points fewer on 35%), retired early less often (4.8 points fewer on 53%) and drew fewer welfare payments (2.5 points fewer on 75%), with no change in employment or earnings; the cost and hospital figures are significant at the 10% level only. The GPs' age, sex, specialisation, list size and recorded style of practice together explained about 1% of the differences between them. Male and female GPs did not differ measurably: men scored 0.016 standard deviations lower (from 0.11 lower to 0.08 higher), although the paper's text says that female GPs score higher. Patients who changed GP by choice did not move towards the better GPs on this measure, and patients' online ratings were unrelated to it.
The measure is a GP's estimated contribution to whether patients were alive two years later; it is not a measured effect of any named treatment. The two-year survival measure has a ceiling: a GP who loses none of about 45 patients has reached its top, which one average GP in four reaches by luck, so the measure understates the best GPs and the spans above are the furthest the paper's statistics can be read; the paper's own 4.9-point figure exceeds the 3.1% base rate, which is why the regression-based spans are given first. Everyone reallocated is counted whether or not they stayed with the new doctor. The setting is Norway, where the national health system puts every resident on a general practitioner's list.
The Norwegian study above went on to ask whether patients could pick the doctors its measure ranked highly. The authors merged their register data with a proprietary set of ratings that patients wrote about the same doctors. They found no relationship between the two. Their own reading of that, in the body of the paper, is that the result "is consistent with the notion that patients either are unable to identify high-quality GPs, or they value aspects of GPs other than their ability to save lives". They also report that patients who change doctor by choice move towards doctors who are more popular, more experienced, female and holding a specialisation but not better quality.15
The matching Australian measurement is in advice rather than in medicine. It is the shadow-shopping study of retirement advice by the Australian Securities and Investments Commission (ASIC). In that study, satisfaction with the advice did not vary with the grade the advice received from the regulator for the quality of the Statement of Advice. The study is treated on the page What is known about the effectiveness of financial advice?16
No study was found that reports, for financial advice, the share of the difference between clients' outcomes that goes with which adviser they had. None was found on any outcome, in any country. The measurements that do exist in advice are about what advisers' clients hold and what those holdings cost. They are set out with their sources and their limitations on the page What is known about the effectiveness of financial advice?16 That page also sets out the questions the field could answer, and what answering them would take.
What the findings show together. The practitioner's effect in psychotherapy is real, replicated across 66 studies, and large between therapists: in one service the best therapists' patients recovered at four times the rate of the worst therapists' patients, and the therapists themselves could not tell which of their own clients were getting worse. The effect appears only when outcomes are measured on a common scale and grouped by practitioner. Before that was done, the field assumed the method mattered and the practitioner did not. It turned out to be the opposite – the practitioner mattered, not the method used. In medicine, the one study with patients who did not choose their doctor finds the same shape, as a cause rather than an association: months of life, with intervals, between the doctors at the 10th and the 90th percentile.
Why psychotherapy may have parallels that matter for advice. Professional financial advisers, not sales-focused advisers, at times act in a role that has aspects of those of a therapist. A client in a market fall, a couple in a dispute about money, a retiree afraid of running out: the adviser's work in those hours is listening, steadying and judging, not calculating. Hence, the industry's own estimates of the value of advice put most of it in "behavioural coaching".17
If therapists differ as widely as the figures above show, on an outcome measured by a questionnaire, there is no reason to expect advisers to differ less on the outcomes a client would value, whether tangible or intangible. The one measurement that exists for advisers points the same way: the adviser's preferences explain more of a client's portfolio than the client's own characteristics do.18
What it would take. Two items were present for the therapy studies that do not exist in advice. One is records that group clients under the practitioner who served them. The other is an agreed measure of what a client's outcome is, recorded at intervals. Both exist in principle: platform records group accounts by adviser, a client's position can be measured in dollars, fees and risk, and well-being questionnaires such as the SF-36 already exist. The Norwegian study shows the further step that turns a difference into a cause, which is clients who did not choose their practitioner; the therapy studies did not have that step, so for them causality is very likely, not certain.
In the author's view. Having worked as an adviser for twenty years, the author expects the spread between advisers to be at least as wide as the spread between therapists, for the same reasons: the work is personal, but over a much longer timespan; the client may not easily be able to judge adviser performance, as patients may be unable to judge their doctors' performance; and there are no objective feedback mechanisms except a thorough audit, which does not happen even in cases of suspected wrongdoing. The therapy research shows that the practitioners could not see their own effect either. Measurement, not self-assessment, is what revealed it.
However, a survey of 505 adults found that people have encountered what they consider exceptionally good doctors and describe them differently from average doctors: a doctor who willingly listens to the patient to the end is taken to be not just empathic and a good communicator but also more competent.20
In psychotherapy the individual practitioner has a measured effect on clients' outcomes, of the order of a pharmaceutical medicine, and the range between practitioners is wide. In medicine, where patients did not choose, the effect is causal and measured in months of life. It took a common measure and records grouped by practitioner to see either. Financial advice has neither, and so has no measured adviser effect, in either direction. The parallels suggest the effect is there to be found.
Designed and produced by Dr Christoph Schnelle, In Your Interest Financial Planning. In Your Interest Financial Planning Pty Ltd, ABN 28 094 300 464 is Authorised Rep. No 308161 of Fiduciary Duty Advisers Pty Ltd AFSL No 527434. Nothing on this page is advice to you or to your clients, and nothing on it is a recommendation of any product, any strategy or any adviser. Corrections are welcome and wanted – contact us.