A national survey of 504 clinicians, published in 2018, asked what they made of ROMs, typically short questionnaires a client completes before or between sessions that the therapist can review to assess whether treatment is working. Their attitudes were generally positive. But only 13.9% reported giving a standardized measure as often as once a month, and 61.5% had never given one.
I interpreted that to indicate that clinicians would like to be able to engage in best practices, but there is friction, and the barriers may be too much to break through without help. In one of my research roles, I was responsible for getting the entire clinical staff to administer ROMs. It was like herding cats. That task did not help my popularity, even though I was largely responsible for collecting, entering, and analyzing the data for them. Every step that needed someone to hand out a form or score it was a step that got skipped when caseloads filled, and we all dropped the ball. That was one of the experiences that led me to develop a software solution.
How do I run routine outcome monitoring in a group practice?
Choose a small number of validated measures, send them to every client on a fixed schedule with reminders no one on staff has to remember, have each clinician read the score before the session and review the scores with the client, and have one director read the charts for the whole practice. A 2017 review of 51 articles found that feedback improved outcomes in nearly every randomized trial in which the provider saw the client's self-reported symptoms often, promptly, and during the session. Screening a client once, measuring rarely, or sending results to the provider outside the session did not improve outcomes.
Why clinicians who value outcome measures rarely use them
A 2019 review in JAMA Psychiatry put the share of behavioral health practitioners who have built measurement into their work at typically under 20%, and sorted the barriers into four levels. A client may worry about who will read her answers, and a practitioner may trust his own judgment over a questionnaire. An organization may have no resources for training, and the wider system may set competing requirements. In the 2018 survey, clinicians with a cognitive-behavioral orientation held more favorable attitudes. That describes me, and it did not make my own paper system work.
When I directed research at the Albert Ellis Institute, I made routine outcome measurement a requirement. Clients completed an outcome questionnaire, the OQ-45, on paper, and at intake they also rated their irrational beliefs and their satisfaction with life. The staff pushed back, and clients struggled with the forms. I collected the forms, scored them, tracked the results, and reminded the staff, which is the polite word for what I did. We believed it improved the care. It was not always efficient.
A 2016 study gave 26 psychotherapists an online progress-monitoring tool and four classes on using it. Right after the training, their use of the tool rose, and so did their use of progress monitoring of any kind. A year later, they were still monitoring progress at nearly the same rate, but they used the tool with far fewer clients (about 20%, down from about 45%). The authors suggest several reasons, including fading accountability, the friction of a new system, and uncertainty about keeping access. One hypothesis is that a tool outside a clinician’s usual workflow is one more step, and that could lead to drift.
The 2019 review describes a study in which clinicians rarely gave or discussed measures while the feedback tool sat apart from the EHR; when they documented inside the record, they reported giving measures in 90% of sessions and discussing scores in 75%. At the Albert Ellis Institute, the forms lived outside the record as well, and that was a significant impediment to compliance, saving time, and keeping stress levels low. Clinical work already does enough to challenge the emotional bandwidth of therapists, and adding to it should be avoided where possible.
Routine outcome monitoring reduces symptoms by a small amount and cuts dropout
A multilevel meta-analysis from 2021 pooled 58 studies, randomized and non-randomized, covering 110 effect sizes and 21,699 clients, and found that progress feedback reduced symptoms compared with control conditions by d = 0.15, with a 95% confidence interval of 0.10 to 0.20, which the authors call a small effect. For clients who were not on track, the figure was d = 0.17, barely above the overall one. Feedback also had a small effect on dropout, an odds ratio of 1.19.
A 2018 meta-analysis of 24 studies of the two most studied feedback systems, the Outcome Questionnaire System and the Partners for Change Outcome Management System, found that in two of every three studies, therapy with feedback did better than the usual care the same practitioners gave without it. The authors put the effects between small and moderate. Feedback lowered the rate at which clients deteriorated, and for clients predicted to do poorly it brought the rate of reliable or clinically significant change close to double. Clinical significance is the critical measure here. This is a meaningful improvement in the lives of clients.
Both analyses tested feedback systems built for the purpose, the Outcome Questionnaire System and the Partners for Change system, among them. Being able to see, in close proximity to the interventions, when and what is working, and when and what isn't, allows for flexible, optimal care. It can also be incredibly helpful for the clients to see and discuss, as they can provide even more information related to changes seen on objective measures and can also have their behaviors that are working reinforced, enhancing self-efficacy and motivation, as well as their emotional lives. Some of my clinical experiences have also surprised me in the other direction, with a client doing better than I thought. More than once, a client's scores for satisfaction with life, for meaning, and for happiness were higher than I had judged from our sessions, and we talked sooner than we otherwise would have about meeting less often or finishing treatment. There are times when what is reported in session subjectively differs from the scores on ROMs. Those moments are often important opportunities to probe deeper and learn about the client’s experience and also their style of communicating about what they are feeling and doing to the therapist in session.
Which EHR has PHQ-9 and GAD-7 built in?
My Best Practice was founded in large part because, regardless of the setting in which I worked, every workflow caused frustration, compliance problems, and stressful interactions for the therapists and clients. And I believed there was a software solution. Now its library of measures holds the PHQ-9, the GAD-7, the OCI-R (the Obsessive-Compulsive Inventory, revised), the AAQ-II (the Acceptance and Action Questionnaire), the DERS (the Difficulties in Emotion Regulation Scale), another 100 questionnaires, and an owner can add the practice's own measures with no limit on the number and no added fees. The routine outcome measures (ROMs) come with every plan rather than as a paid add-on.
The PHQ-9 was validated in a 2001 study of 6,000 patients at eight primary care clinics and seven obstetrics and gynecology clinics. Of those patients, 580 were also interviewed by a mental health professional, which let the researchers check the questionnaire against a clinical diagnosis. A score of 10 or higher proved a strong signal of major depression. It flagged 88% of the people who actually had major depression, and it correctly cleared 88% of the people who did not. The study also set severity ranges. Scores of 5, 10, 15, and 20 mark the start of mild, moderate, moderately severe, and severe depression.
The GAD-7 was developed in a 2006 study of 2,740 patients at 15 primary care clinics across the US. A mental health professional interviewed 965 of them by phone to confirm or rule out a diagnosis. At a score of 10 or higher, the GAD-7 flagged 89% of the people who had generalized anxiety disorder and correctly cleared 82% of the people who did not.
A clinician can set either measure to go out on whatever cadence suits the client: three hours ahead of each session, the following day, the third Thursday of every month, or once a quarter. The client can receive as many reminders as the practice thinks will be helpful. A clinician doesn’t have to score anything. She can just click the metrics tab and see the scores on a chart and a table. When a client could benefit, the therapist can share that screen, reinforce the client for all of the hard work they are doing, and show them how it is clearly paying off. For more specifics, here is an additional guide to measurement-based care in an EHR.
How a group practice makes outcome monitoring a habit
The 2019 review names implementation strategies including a measurement feedback system, a local champion, leaders who have been trained themselves, consultation with an expert, learning collaboratives, and incentives. Its authors also want feedback systems cut back to their key ingredients and better interoperability with the EHR. The 2017 review says aggregated scores can serve a clinician's professional development and the clinic's quality improvement. In a group practice, both of those often sit with a director or supervisor, so they may be the right people to interpret the scores and talk with clinicians about what they mean clinically. Supervisors may often open a client's chart and see self-reported depression scores that haven’t improved, while the therapist's impression was that the sessions were going well.
When the therapist is coached on this and follows up, they may be able to determine the reason for the disconnect, which could be a number of things, e.g., shame or a wish to please the therapist. These are common barriers to optimal care that can be overcome with progress monitoring. Those conversations can lead to more candid communication. What the client says in session may begin to more closely mirror the ROM scores.
About the author
Dr. J. Ryan Fuller is Chief Clinical Officer at My Best Practice. A licensed clinical psychologist, Dr. Fuller specializes in Cognitive Behavioral Therapy (CBT), practice management innovation for group behavioral health practices, and executive coaching. He was the Director of Research at the Albert Ellis Institute and the Director of Behavior for the Obesity Research Center Weight Loss Program in the Columbia Department of Medicine. He has published peer-reviewed research on anger treatment outcomes, irrational beliefs, interpersonal relationships and happiness, and loneliness and obesity.
He also founded New York Behavioral Health, a private-pay group practice. Dr. Fuller has been featured on Good Morning America, ABC World News with Diane Sawyer, and Katie Couric's talk show, and in The New York Times, and he serves as an expert consultant for group practice technology and clinical implementation. His clinical and business expertise spans behavioral health technology, practice operations, and the intersection of evidence-based clinical work and sustainable business models. He regularly consults with growing group practices on scaling, clinical leadership, and financial modeling.