Financial advice is not the only paid service where researchers have asked how much it matters which practitioner you get. They have asked it of psychotherapists and of doctors, and in both cases the records were large enough to answer. Of the two, psychotherapy is the closer parallel, to professional financial advice rather than to advisers who act as salespeople. Both it and advice are a paid relationship between one professional and one client, conducted mostly by talking. Both depend on the client's trust, the professional's judgement, and what the client then does. In both, the outcome is the client's own position over months and years, not a product. The difference is that psychotherapy has measured its practitioners for forty years, doctors have been measured to find the worst among them but rarely and only sporadically the best, and financial advice has not measured its advisers at all.
The measurement of psychotherapists began from an uncomfortable finding. The pooled analysis of the antidepressant trials submitted to the United States drug regulator, published and unpublished alike, found the drugs' advantage over an inert placebo reaching the conventional criterion for clinical significance only for patients at the upper end of the very severely depressed category.1 If the medicine added little over a sugar pill, what did the person across the table add? That question produced the research this page reports. The answer, from the author's introduction to his own 2022 review of medical doctors, is that psychotherapists can have an effect on their patients' mental health that equals the strength of pharmaceutical interventions.2 This effect varies widely between individual psychotherapists.
Medicine supplies the second parallel, and the one study found in this context that can establish causation: Norwegian patients who did not choose their doctor but were allocated to a new one when their own doctor left the list. This page sets out how large the practitioner's effect is in each field, how far it ranges from one practitioner to another, how it was made visible, whether the people receiving the service could tell, and what that suggests for advice.
Sources used are the field's standard handbook and Wampold and Imel's book on what makes psychotherapy work, which map the research,3,4 the two systematic reviews of therapist effects, the two British studies of one health service, the three studies of whether therapists notice deterioration, and the four poolings of the feedback trials. The Norwegian study was read with the authors' replication code. The practice's own searches found no study of this kind in financial advice.19
In psychotherapy research, this proportion is referred to as the therapist effect. It represents the proportion of the total variation in clients' treatment outcomes that is related to which therapist the client was treated by. A systematic review from 2019 builds on the earlier summary and supplements it with its own findings.5 In 2013, Baldwin and Imel analysed 46 studies involving 1,218 therapists and 14,519 patients, and found that the therapist accounted for 5% of the variation overall. Five per cent may sound like a small figure, but it is not: such an effect, which is consistently present, makes a big difference to clients over the course of a career. The proportion varied depending on the setting. In the 17 studies of routine practice, it was 7%; in the 29 clinical trials, it was 3%. A later review from 20195 attributes this difference to the way in which clinical trials are conducted. According to this review, in clinical trials patients are rigorously selected, treatment manuals are followed and adherence to them is checked. Furthermore, there is close supervision, and fewer therapists are involved. In these 46 studies, the proportion of variation in patient outcomes attributable to the therapist ranged from 0% to 55%.
The 2019 review searched for studies published since then. It identified 20 suitable studies: three randomised clinical trials and seventeen real-world studies. A therapist effect was observed in 19 of the 20 studies. Across all 20 studies, the proportion ranged from 0.2% to 29%, with a weighted average of 5%, the same figure as previously. Among the 17 practice-based studies, the average was 5%, ranging from 0.2% to 21%. The three randomised trials yielded an average of 8.2%, ranging from 1% to 29%. The review concludes that therapist effects are a robust phenomenon and that there is considerable heterogeneity between the studies. It adds that "using an overall therapist effects statistic may lack precision", so that the effects "might be better reported separately for specific clinical settings".5
Saxon and Barkham took records from the United Kingdom National Health Service for September 2000 to July 2008. The data were drawn from counselling and psychological therapy services within primary care. They comprised 119 therapists and 10,786 patients. Each patient attended at least two sessions, had a scheduled end to treatment, and completed a standard questionnaire before and after treatment. For a patient with an average level of severity, the proportion of variation in outcomes attributable to individual therapists was 6.6%. This proportion rose with severity, from 1% to 10% as patients' intake scores rose. When the model disregarded the risk associated with each therapist's caseload, this proportion was 7.8%; when this was taken into account, the proportion fell to 6.6%. Risk here refers to the questionnaire's own risk items: self-harm or harm to others.6
Recovery rates are the more concrete measure, and the range in this study represents a spectrum of recovery rates. The average recovery rate among the therapists was 58.8%, with a standard deviation of 13.7 percentage points. The standard deviation is the standard measure of how far individual values deviate from the mean. Among the 119 therapists, the rates ranged from 23.5% to 95.6%. As the number of cases is not specified alongside the respective rates, the two extreme values represent the rates of individual therapists and not measured effects. The paper's own like-for-like comparison is narrower and can be checked. This identifies 19 therapists as being less effective than the rest. It is reported that, of the 1,704 patients they treated, 786 recovered. The study reports that, given an average success rate of 61.7% among the therapists, 1,049 rather than 786 patients would have recovered, an additional 265 patients. (The paper's two figures do not agree: 61.7% of 1,704 is 1,051, which gives its 265.) The study also reports that the rate of deterioration among the least effective therapists was approximately three times higher than among the others.
Patients were not randomly assigned to therapists. The study reports differences that persist even after adjusting for the measured patient characteristics, primarily severity and risk at intake. The sample comprises only therapists who returned a pre-post questionnaire for at least 90% of their patients, which constitutes a selection based on documentation. The organisation is based in the UK. The outcome is a score of symptoms and functioning over a course of therapy.
The same research group measured two further endpoints: whether a patient discontinued treatment and whether a patient's condition deteriorated compared with the start of treatment. The study involved 85 therapists. A total of 10,521 patients were included in the comparison of dropout rates, and 6,405 patients in the comparison of deterioration rates. The individual therapist accounted for 12.6% of the variation in dropout and 10.1% of the variation in deterioration. The dropout rates for individual therapists ranged from 1.2% to 73.2%, while the deterioration rates ranged from 0% to 15.4%. The study also investigated whether a therapist's dropout rate was correlated with their deterioration rate. Given the level of precision available, no correlation differing from zero could be detected. A correlation of 0.07 with a p-value of 0.52 is reported. This means that there was no discernible association between dropout rates and deterioration rates, an apparently surprising result.7
The study relied on the same service and the same type of records as the entry above, so the same limitations apply. There was no randomisation, the adjustment covers only what was measured, and the setting is UK primary care. Stopping attendance at a health service is a different act from ending a paid engagement. The study does not cover any context outside psychological therapy.
A study was conducted at a counselling centre at a major university in the United States. Therapists were asked to predict which of their current clients would leave treatment in a worse state than when they arrived. The published summary states that clinicians are rarely able to predict accurately who will not benefit from psychotherapy.8 Two separate studies report that, unlike the questionnaire, therapists had considerable difficulty in recognising a deterioration in their clients' condition; in one study, the therapists' progress notes were compared with the documented changes in assessment scores; in the other, therapists were surveyed.9 A survey of mental health professionals revealed that 25% rated their own clinical skills at the 90th percentile of their profession and that none rated themselves as below average; the professionals also overestimated their clients' improvement and underestimated any deterioration compared with published rates.10
The first study relates to a single counselling centre, so it is not possible to determine whether individual institutions influence patients' treatment outcomes. This is less far-fetched than it seems: in the author's experience, hospitals have a major influence on patients' physical health outcomes. The survey results are based on self-reported information from the professionals. Two of the three articles were written by the research group that developed the questionnaire compared here.
At present, there is no standardised, generally accepted method for measuring the performance of financial advisers; there are only very rough tools available to identify who the best, average and worst advisers are.
One reason for this is that the industry comprises both sales staff who sell investment, life and disability insurance products, and professionals who offer a much broader range of bespoke services.
This raises the question of whether there are other sectors or professions in which the quality of individual practitioners has been successfully measured, and whether these methods could also be applied to financial advisers.
The answer is yes, for psychotherapists, who had an urgent need to measure therapists' performance after medication had proved to be of only very limited effectiveness and the profession needed another way to justify its existence. Further details on this successful method of performance measurement are provided below.
Each figure in groups A to C is based on a single instrument: a short standard questionnaire on symptoms and functioning, which the client completes at each session, usually the Outcome Questionnaire-45, which takes about five minutes to complete.3 The client's scores are compared with those of other clients who exhibited the same level of mental health problems at the outset, and the questionnaire identifies those clients whose progress is falling short of expectations. In the counselling centre's study, the algorithm identified 100% of clients whose condition had deteriorated by the end of treatment, as well as 85% of those that are deteriorating by their third session.8 As part of the studies, the identification is reported back to the therapist, sometimes together with suggestions for appropriate measures.
Have psychotherapists improved in their work after receiving objective feedback on their patients' progress? To a small extent, and the systematic reviews disagree on how pronounced this improvement is. The largest systematic review, comprising 58 randomised and non-randomised studies involving 21,699 patients, found a small effect, d = 0.15 standard deviations between the two groups.11 The 95% confidence interval ranges from 0.10 to 0.20 – the precise definition of a confidence interval is a matter of debate among statisticians. One explanation could be that there is a 95% probability that the true value lies between 0.10 and 0.20, and a 5% probability that the true value lies below 0.10 or above 0.20, with the greatest probability being that the true value lies fairly close to the measured value of 0.15. A 95% confidence interval that lies entirely above or below zero typically indicates a statistically significant effect; an interval that spans zero suggests that the measured effect may have been purely due to chance. For the clients flagged as off track, the effect was 0.17. An earlier synthesis of twelve studies found 0.10 (0.01 to 0.19), which did not persist at longer follow-up.12 The research group behind the questionnaire reports that their own six studies involving 6,151 patients were effective, particularly among the identified clients.13 The Cochrane review, which considered twelve randomised trials involving 3,696 participants, yielded a value of −0.07 (−0.16 to 0.01), which is indistinguishable from no effect, and classified the evidence as low.14 The actual effect of measurement as a treatment is therefore small or not demonstrated. Its value lies elsewhere: it has brought the therapist effect to light.
The four meta-analyses differ in terms of the studies included, which largely explains why they arrive at different conclusions. Some of the study results and one of the four meta-analyses originate from the research group that developed the system under test. The psychologist Michael Lambert edited the chapters and authored one of them;3 he also heads the research group behind the Outcome Questionnaire system and is an author of that meta-analysis and of two of the identification studies. The results consist of symptom and functional scores over a period ranging from weeks to months.
A word of caution: it appears that a financial adviser's financial success depends crucially on their ability to convince clients of their quality, regardless of the actual quality of their advice. The above comments may nevertheless be useful if one bears this persuasive power in mind.
This is a study that gives proof that GPs (medical doctors) make a very large difference to their patients' survival and shows that the practitioner matters – a lot.
Norwegian law reallocates patients to a new general practitioner (GP) when their own doctor closes a practice or cuts the size of the list. A computerised procedure makes the reallocation, and about 26% of all changes of doctor in the system happen that way. Ginja, Riise, Willage and Willen used that reallocation and the whole country's register of patients and doctors: 92,205 patients aged 55 and over, moved to 2,023 new GPs between 2006 and 2015. They compared patients who had shared a doctor and lived in the same municipality, and who were then sent to different new doctors. Because the patient did not pick the doctor, the study can establish causation, which almost nothing else in this field can, and the authors show that the new GP's quality does not predict the patients' earlier health, income or education. It is the only study of its kind.15
Each GP was measured by whether the patients allocated to that GP were alive two years later, adjusted for the patients' age and sex, their previous GP and the year, with a correction for small numbers. Among these patients, 3.1% died within two years. A patient sent to a GP one standard deviation higher on that measure died 4.1 months later (interval 1.5 to 6.8 months), measured among the 13,695 patients who died within the ten-year window. For patients aged 55 to 61 at the move the gain was 8.3 months (4.8 to 11.8). The chance of dying within two years fell by 1.4 percentage points per standard deviation (from 3.0 points lower to 0.2 higher), so that comparison on its own does not exclude zero. The fall was in cancer deaths, 1.1 points on a base of 1.1% (2.1 to 0.1), which the authors attribute to earlier detection.
The spread between GPs, read on the paper's own distribution of the measure. From the 10th percentile to the 90th is 1.7 standard deviations: 7.0 months of life (2.5 to 11.5), 14 months for patients aged 55 to 61 (8.2 to 19.9), and a two-year death risk of 2.0% instead of 4.4% (from 5.1 points lower to 0.3 higher). From the 25th percentile to the 75th is 0.7 standard deviations: 2.8 months (1.0 to 4.6), 5.7 months at ages 55 to 61 (3.3 to 8.0), and 1.0 point of risk (2.0 lower to 0.1 higher). The paper's own two comparisons, made on the unadjusted spread of the measure, are larger: a GP at the 75th percentile rather than the median cuts the two-year risk by 4.9 points, and moving from the bottom quarter of GPs to the top quarter adds 3.5 months of predicted life expectancy, about 5% of the average 5.7 years from the move to death. The paper gives no intervals for those two.
The authors put a value on replacing the worst 5% of GPs with average ones: 12.1 extra months of life for each patient affected, worth $35,500 per patient and $1.2 billion in all at the Norwegian government's value of a year of life ($35,000), or $372,075 per patient and $12.4 billion at the value used by the United States Food and Drug Administration ($369,000). The paper makes no such calculation for the best 5%.
The better GPs' patients had slightly fewer consultations and cost the health system about 9% less, had fewer acute hospital contacts (2.4 points fewer on 35%), retired early less often (4.8 points fewer on 53%) and drew fewer welfare payments (2.5 points fewer on 75%), with no change in employment or earnings; the cost and hospital figures are significant at the 10% level only. The GPs' age, sex, specialisation, list size and recorded style of practice together explained about 1% of the differences between them. Male and female GPs did not differ measurably: men scored 0.016 standard deviations lower (from 0.11 lower to 0.08 higher), although the paper's text says that female GPs score higher. Patients who changed GP by choice did not move towards the better GPs on this measure, and patients' online ratings were unrelated to it.
The measure is a GP's estimated contribution to whether patients were alive two years later; it is not a measured effect of any named treatment. The two-year survival measure has a ceiling: a GP who loses none of about 45 patients has reached its top, which one average GP in four reaches by luck, so the measure understates the best GPs and the spans above are the furthest the paper's statistics can be read; the paper's own 4.9-point figure exceeds the 3.1% base rate, which is why the regression-based spans are given first. Everyone reallocated is counted whether or not they stayed with the new doctor. The setting is Norway, where the national health system puts every resident on a general practitioner's list.
The Norwegian study above went on to ask whether patients could pick the doctors its measure ranked highly. The authors merged their register data with a proprietary set of ratings that patients wrote about the same doctors. They found no relationship between the two. Their own reading of that, in the body of the paper, is that the result "is consistent with the notion that patients either are unable to identify high-quality GPs, or they value aspects of GPs other than their ability to save lives". They also report that patients who change doctor by choice move towards doctors who are more popular, more experienced, female and holding a specialisation but not better quality.15
The matching Australian measurement is in advice rather than in medicine. It is the shadow-shopping study of retirement advice by the Australian Securities and Investments Commission (ASIC). In that study, satisfaction with the advice did not vary with the grade the advice received from the regulator for the quality of the Statement of Advice. The study is treated on the page What is known about the effectiveness of financial advice?16
No study was found that specifies, in the context of financial advice, the proportion of the difference between clients' outcomes that can be attributed to the individual adviser. No study was found on any outcome in any country.19 The metrics that do exist in the advisory sector relate to the assets held by advisers' clients and the costs incurred by these portfolios. They are listed, together with their sources and limitations, on the page What is known about the effectiveness of financial advice?16 That page also sets out the questions that could be answered in this area, as well as the conditions required to answer them.
What the results show overall. The therapist effect in psychotherapy is real, has been confirmed in 66 studies, and varies greatly depending on the therapist: in one service, the patients of the best therapists recovered at four times the rate of those of the worst, and the therapists themselves were unable to identify which of their own clients were getting worse. The effect only becomes apparent when the results are measured on a standardised scale and grouped by therapist. Before this was done, the professional community assumed that the method mattered and that the therapist did not. However, the opposite turned out to be true – the therapist mattered, not the method used. In medicine, the only study involving patients who had not chosen their own doctor reaches the same conclusion, as a cause rather than merely a correlation: months of life, with intervals, between doctors at the 10th and 90th percentiles.
Why psychotherapy may have parallels that are relevant to advice. Professional financial advisers, not sales-oriented advisers, sometimes take on a role that bears similarities to that of a therapist. A client facing a market downturn, a couple in a dispute over money, a retiree who is afraid of running out of money: in these moments, the adviser's job is to listen, reassure and assess, not to calculate. Consequently, the industry's own estimates of the value of advisory services attribute most of it to "behavioural coaching".17
If, as the figures above show, therapists differ so greatly in terms of an outcome measured by a questionnaire, there is no reason to assume that advisers differ any less in terms of the outcomes a client would value, whether tangible or intangible. The only metric that exists for advisers points in the same direction: the adviser's preferences explain a larger proportion of a client's portfolio than the client's own characteristics.18
What would be required for this. The therapy studies included two elements that are missing from financial advice. One is records that group clients under the practitioner who served them. The other is an agreed measure of a client's outcome, recorded at regular intervals. Both exist in principle: platform records group accounts by adviser, a client's situation can be measured in dollars, fees and risk, and well-being questionnaires such as the SF-36 are already available.
In the author's view. Having worked as an adviser for twenty years, the author expects the spread between advisers to be at least as wide as the spread between therapists, for the same reasons: the work is personal, but over a much longer timespan; the client may not easily be able to judge adviser performance, as patients may be unable to judge their doctors' performance; and there are no objective feedback mechanisms except a thorough audit, which rarely happens. The therapy research shows that the practitioners could not see their own effect either. Measurement, not self-assessment, is what revealed it.
However, a survey of 505 adults found that people have encountered what they consider exceptionally good doctors and describe them differently from average doctors: a doctor who willingly listens to the patient to the end is taken to be not just empathic and a good communicator but also more competent.20
In psychotherapy the individual practitioner has a measured effect on clients' outcomes, of the order of a pharmaceutical medicine, and the range between practitioners is wide. In medicine, where patients did not choose their doctor, it could be shown that medical doctors are the cause of their patients living longer or shorter lives, depending on what is presumably the quality of the practitioner. To establish both of these points, a standardised metric and data grouped by practitioner were required. Neither of these exists in the field of financial advice, which is why there is no measured adviser effect in either direction except, in the author's experience, for reduced investment returns from advisers who act as investment product salespeople of their employer. The parallels between what financial advisers are doing and what psychotherapists and medical doctors are doing, individual, often long-term ongoing consultations that are very personal, suggest that it is very likely that an effect is there to be found and that differences between advisers can be identified.
Designed and produced by Dr Christoph Schnelle, In Your Interest Financial Planning. In Your Interest Financial Planning Pty Ltd, ABN 28 094 300 464 is Authorised Rep. No 308161 of Fiduciary Duty Advisers Pty Ltd AFSL No 527434. Nothing on this page is advice to you or to your clients, and nothing on it is a recommendation of any product, any strategy or any adviser. Corrections are welcome and wanted – contact us.