Mark V. Johnston, PhD, professor, Department of Physical Medicine and Rehabilitation, UMDNJ-New Jersey Medical School; director, Outcomes Research, Kessler Medical Rehabilitation Research and Education Corporation; Karen Hwang, PHD, Post-doctorial fellow, rehabilitation outcomes research, njms, kmrrec

Outcomes research in medical rehabilitation by Mark V. Johnston

Whether a treatment is new or old, curative or behavioral, questions should be asked about its effectiveness and value in the real world. Did the treatment improve the health, function or quality of life of populations served? Is the improvement worth the expense? The answer to these questions requires systematic study of outcomes. Outcomes research (OR) is the name for a set of interrelated methodologies that address these questions.

OR begins by measuring actual outcomes of persons in the real world. Over the years, the Center for Outcomes Research at Kessler Medical Rehabilitation Research and Education Corporation (KMRREC) has done much work to develop more reliable and valid measures of health, function and quality of life in the community. Like other outcomes researchers, we have conducted many observational studies documenting the results of care, applying advanced correlational methods to identify reliable independent associations between likely causal factors and outcomes. Outcomes are usually determined by multiple factors, including treatments received, severity of illness, comorbid conditions, background variables, family support and the long-term environment to which the person is discharged. So we develop multivariate models to attain an overview of factors that affect valued outcomes.

Randomized controlled trials (RCTs) are usually (but not always) the best way to acquire rigorous evidence of the efficacy of a treatment. As director of the Center for Outcomes Research at KMRREC, I have emphasized the support of RCTs, working with clinical professionals who have the skills and ideas to develop new interventions that promise to improve patients’ lives. Because the most productive OR is collaborative, our contributions have been in a variety of diagnostic areas.

Controlled Trials
Neuropsychological rehabilitation is a mainstay of the rehabilitation of persons with traumatic brain injury (TBI), as the major impairment of people with TBI is neuropsychological – cognitive and psychosocial – rather than just physical. But until recently, there has been almost no rigorous RCT testing of the efficacy of defined treatment protocols for persons with TBI. (To be more humble as well as more precise, there have been two previous related RCTs, but they were badly flawed. One learns from past research.) As combination project director and project statistician, for years I worked with Lana Tiersky, PhD, now at Fairleigh Dickinson University, to provide the support needed for our trial of neuropsychological rehabilitation for people with “mild and moderate” TBI. The treatment involved a combination of both attention training and sensitive psychotherapy, administered by Vera Anselmi, PhD. We thought the trial would fail, due to difficulty of recruiting and numerous “real world” problems. But results were unequivocal: compared to the control group, the treatment group had less general psychological distress and depression (measured by the Symptom Check List-90R) than the control group. Performance on a neuropsychological test of divided auditory attention (PSAT) also improved. Few RCTs are done in rehabilitation, but they are possible with true cooperative teamwork.

Average General Syptoms Index Scores: Changes in treatment vs. control group over time.

Other recent controlled trials we have supported — statistically, in improvement and analysis of measures, and even in obtaining funding — include: an RCT of the efficacy of structured exercise for veterans with inoperable chronic venous insufficiency, working with Frank Padberg, Jr., MD, professor of surgery, NJMS, and the VA New Jersey Health Care System in East Orange; and an RCT of a behavior management program for persons with serious TBI in the community.

We have two more RCTs under development but we may not have results for another four years.

Measure development and correlational studies
RCTs are not the only method of gaining useful knowledge of outcomes. Here are two other recent works:

In TBI, community outcomes have long been thought to be multidimensional when you look at them in detail, but the number of dimensions has long been controversial. So we set out to determine the optimal number of outcome dimensions to measure. This project applied advanced methods of psychometric analysis to the largest existing database on TBI outcomes — that of the National TBI Model Systems, housed at KMRREC. We expected to find great complexity and multidimensionality — but we did not. Instead, we found that underlying all the complexity and variations of many community activities, there was a single, unidimensional, probabilistically-equal-interval latent dimension, which we labeled “general community functioning.” The result is important, because treatment research — and research on quality and outcomes in practice — have been inhibited by the uncertainty surrounding the use of dozens of conflicting outcomes scales. Research and practice will be facilitated if a valid scale can be employed.

• Long-term outcomes can depend on the knowledge and education of the person served, as much as on the technical quality of medical treatment. As in other groups, we found in SCI that health literacy is related to morbidity. A unique finding was confirmation of measurement “ceiling” artifact in the standard measure of health literacy, which constrained the size of correlations obtained. A measure that tapped higher levels of health literacy and knowledge would all-but-certainly show a higher association between the person's health knowledge-processing abilities and health outcome.

Evidence-based practice
Evidence-based practice (EBP) may be defined as the systematic identification of best evidence and its application to clinical practice. This application involves integration with patient values and clinical experience. In recent years, we have increasingly emphasized EBP — education in what it means, evidence synthesis to write guidelines (or to discover what we don't know) and research on the implementation of resulting knowledge.

In rehabilitation, EBP remains controversial for two reasons. One reason is that EBP advocates sometimes rely solely on RCTs, as if they were the only way of obtaining useful knowledge: They are not. The other reason is that there are comparatively few RCTs in rehabilitation (which one would expect, as it is a small part of the total biomedical enterprise). We have documented these in our recent state-of-the-art review. In the years 1999 through 2004, there were just five Level 1 studies in SCI rehabilitation, 15 in TBI, and 12 in burn rehabilitation. What was more surprising was the relatively limited number of Level II studies, which are also very good studies (a total of 32 during the 5 year period). As in many other areas of medical care, rehabilitation in practice faces the challenge of sparse evidence: the evidence supporting many, perhaps most, treatments currently employed is weak.

Although one could criticize research in the field, one can also point out that many sensible, promising rehabilitative interventions are not of the type that review committees fund. The interventions are multifaceted, complex and individualized, often based on biopsychosocial theories rather pharmacologic findings. With limited funding, few RCTs can be done. Even so, the review found that the number of RCTs in rehabilitation is increasing rapidly (especially internationally). There is also another way of looking at the “weak” evidence. We identified large numbers of studies demonstrating very promising but not proven (level 3 or 4) treatments. We need to move these promising interventions to larger, more rigorous trials — Phase II or Phase III clinical trials — to develop our toolbox of proven, highly effective therapies.

Mark V. Johnston, PhD, received his BA from the University of Chicago in economics and his PhD from Claremont Graduate University in social psychology specializing in research methodology for rehabilitation program evaluation. He is currently the director of Outcomes Research at KMRREC and a professor in the Department of Physical Medicine and Rehabilitation at UMDNJ-New Jersey Medical School.