Seeking Answers:

Questions Health Professionals Ask and the Evidence Needed to Answer Them

ÓCraig L. Scanlan, 2004 (Version 2, Sep 2005)

 

Objectives

After completion of this module, you will be able to:

1. differentiate between background and foreground questions asked by health professionals
2. identify the common types of foreground questions and the goal in asking them
3. retrieve examples of studies addressing a foreground question category of interest (therapy, diagnosis, harm/etiology or prognosis)
4. distinguish among the common types of original research and secondary or integrative studies, and the relative strength of the evidence they provide
5. relate the relative strength of the evidence provided by various types of studies to the major categories of foreground questions (therapy, diagnosis, harm/etiology or prognosis)

 

Background vs. Foreground Questions

 

Health professionals commonly ask two types of questions: background and foreground questions. The following table distinguishes between these two types of questions and provides an example of each:

 

.

 

As indicated in the Figure 1, students or novice practitioners ask primarily background questions. As one progresses from novice to expert clinician, more and more of the pertinent questions that need answering are of the foreground type. Of course, one generally needs to fully understand the background questions and answers before asking and answering foreground questions.

 

 

Foreground Question Types

The most common types of foreground questions heath professionals ask revolve around their clinical practice and patients. These question generally fall into one of four major categories, as summarized in the table below (adapted from Centre for Evidence-Based Medicine Type of Question, retrieved October 12, 2005 from http://www.cebm.net/question_types.asp):

 

 

Examples of Studies Addressing Foreground Questions (Self-Directed Activity)

There are as many examples of published articles addressing foreground questions as there are clinical problems encountered by health professionals. To get a 'feel' for the types of studies that address the above categories of questions, go to the National Library of Medicine's PubMed Clinical Queries page at: 

 

http://www.ncbi.nlm.nih.gov/entrez/query/static/clinical.html

 

As show in Figure 2, this page helps you find articles in the Medline database on a particular subject area that focus on one of the major categories of foreground questions described above. To obtain a list of citation for a subject area of interest to you that is based primarily on studies of therapy, diagnosis, harm (etiology) or prognosis:

 

1. Under Search by Clinical Study Category, type your desired subject into the Search box (show in the middle of Figure 2)
2. Under the Category choices, select the area of interest (etiology, diagnosis, therapy, or prognosis) by clicking its button
3. In terms of Scope:

1. if you want to narrow the search to retrieve mainly the most relevant studies but are willing to miss a few of these ('false negatives') select a narrow, specific search
2. if you want a broad search that will capture most of what you are looking for but also retrieve some irrelevant  articles ('false positives'), select a broad, sensitive search

 

 

For example, assume that my question involves identifying the best screening test for the diagnosis of occupational asthma (asthma associated with exposure to allergens in the work setting). I type occupational asthma into the Search text box, click on diagnosis for my Category and select narrow, specific search for my Scope. This search results in 94 citations, of which the following examples are representative of the many good 'hits':

 

9:  Anees W. Use of pulmonary function tests in the diagnosis of occupational asthma. Ann Allergy Asthma Immunol. 2003 May;90(5 Suppl 2):47-51.

 

29:  Weytjens K, Malo JL, Cartier A, Ghezzo H, Delwiche JP, Vandenplas O. Comparison of peak expiratory flows and FEV1 in assessing immediate asthmatic reactions due to occupational agents. Allergy. 1999 Jun;54(6):621-5.

 

33. Leroyer C, Perfetti L, Trudeau C, L'Archeveque J, Chan-Yeung M, Malo JL. Comparison of serial monitoring of peak expiratory flow and FEV1 in the diagnosis of occupational asthma. Am J Respir Crit Care Med. 1998 Sep;158(3):827-32. 

 

50:  Paggiaro PL, Giannini D, Moscato G, Bacci E, Bancalari L, Carrara M, Dente FL, Di Franco A, Di Pede F, Petrozzino M, et al. Peak expiratory flow monitoring in diagnosis and management of occupational asthma. Monaldi Arch Chest Dis. 1994 Dec;49(5):425-31.

 

68:  Perrin B, Lagier F, L'Archeveque J, Cartier A, Boulet LP, Cote J, Malo JL. Occupational asthma: validity of monitoring of peak expiratory flow rates and non-allergic bronchial responsiveness as compared to specific inhalation challenge. Eur Respir J. 1992 Jan;5(1):40-8.

 

All in all a good start on answering my question. Try out this tool and see what results you get!

 

Two important notes related to using these research methodology filters:

 

1. PubMed tries to automatically map your subject to its controlled vocabulary of Medical Subject Headings (MeSH). Use of the correct MeSH term(s) will improve your results.
2. These searches are based on pre-programmed PubMed filters that have been developed and tested by the National Library of Medicine staff and health informatics researchers. You can obtain similar results when searching Medline (or CINAHL) using a different gateway like OVID by manually including similar filters in your searches.

 

Other Question Types

 

Not all questions of importance to health-related professionals focus on patients and their therapy, diagnosis, prognosis or /harm. Typically, these 'other' types of questions focus on units of analysis broader than an individual patient or client, i.e., on policies/procedures, programs, systems, or organizations. A supplementary help document entitled 'Other' Types of Questions is available to assist those intent on pursuing these broader questions at:

http://www.umdnj.edu/idsweb/idst6400/other_questions.htm

Types & Levels of Knowledge

 

A careful review of all the pertinent articles I retrieved related to my foreground question on diagnosis of occupational asthma reveals several different types of article and studies. For example, citations #29 is a clinical trial, citation #68 is a prospective cohort study and citations #9, 33 and 50 are review articles. Although they are all related to the diagnosis of occupational asthma, they clearly represent different types of evidence. Which type of study or evidence provides the best answer?

 

Types of Studies

Before you can determine which type of evidence provides the best answer to one's clinical questions, it is essential that you understand the full scope of study types typically encountered in the literature. Then you will need to relate this knowledge back to the category of question being asked (i.e., therapy, diagnosis, harm/etiology or prognosis).

 

Health science studies commonly are divided into two major types: original research and secondary or integrative studies (reviews). Original research studies report research first-hand. Secondary, integrative studies or reviews summarize and draw conclusions from primary studies.

 

Original (Primary) Research

Original research can further be divided into two subcategories: experimental studies and observational studies (Gay, 1999). Generally, experimental studies are assumed to provide stronger empirical evidence than that resulting from observational studies. Two other unique types of original research, methodological studies and evaluation research, are discussed separately.

 

Experimental Studies. Experimental studies are prospective in nature, meaning that data collection and the events of interest occur after subjects are enrolled. Moreover, in these types of studies, the investigator manipulates the exposure or treatment by controlling the assignment or allocation of subjects (Gay, 1999). In relative order from those providing the strongest to those providing the weakest empirical evidence for clinical practice are the following types of experiments (some study type definitions based on National Library of Medicine Medical Subject Headings):

 

• Randomized Controlled Trial (RCT). An RCT is an experimental study involving random assignment of subjects to one or more test interventions and at least one control group, with data prospectively gathered on specified outcome measures. Randomization assures that the groups are 'probabilistically equivalent' on all variables not directly manipulated by the investigator (extraneous variables). Interventions may include drugs, devices, or procedures studied for diagnostic, therapeutic, educational or preventive effectiveness. Control measures may include placebos, active treatment, no treatment, dosage forms and regimens, historical comparisons, etc. Ideally, to avoid bias, both the subjects and investigators are kept unaware of who is actually getting which specific treatment (a 'double-blinded' study). A properly executed RCT is the most powerful type of experiment and is the only design that can truly establish cause and effect relationships. In general, the large the number of subjects in the trial, the more powerful the results. For this reason, RCTs that are planned and carried out by several cooperating institutions (multicenter clinical trials) tend to provide the strongest clinical evidence regarding diagnostic, therapeutic, educational or preventive effectiveness.
• Randomized Crossover Trial. A randomized crossover trial is an experimental study in which subjects are randomly allocated to one of two treatment groups, and, after a sufficient treatment period and often a washout period, are switched to the other treatment for the same period (Gay, 1999). In this manner, the subjects serve as their own controls, which strengthens the power of the study. However, biased results can occur if carry-over effects from the first treatment affect the second. A variation of the randomized cross-over trial is the "N of 1 " trial, In the N of 1 trial alternative treatments are administered to a patient in a random sequence while the patient's response is blindly observed to assess treatment effects. [Guyatt, G.H., Keller , J.L., Jaeschke, R., Rosenbloom, D., Adachi, J.D., & Newhouse, M.T. (1990). The n of 1 randomized controlled trial: clinical usefulness. Ann Intern Med, 112, 293-299.]
• Nonrandomized and Quasi-experimental Trials. There are times when either randomization or comparison to a control group is simply not possible. Since these type of studies fail to meet the strict definition of an experiment, they are often referred to quasi-experimental designs. When randomization is not possible, researchers commonly use one of two methods to minimize the effect of extraneous variables: matching or statistical control. In matching, subjects in both the experimental and control groups are ‘paired’ on the factor or factors of interest, e.g., age-matched controls. Alternatively, statistical methods can be applied to balance out differences between groups on key extraneous variables. The most common statistical method used to control for difference between groups is analysis of covariance. When a control group is not possible (e.g., due to ethical concerns) a single-group time-series design can be used. In this design, the subjects undergo serial, repetitive measurement or observation on the outcome measure before, during and after the intervention, with the goal being to detect whether or not and when a change occurs. 
• Laboratory Experiments. Laboratory experiments are studies conducted on humans or animal in the highly-controlled (and thus the artificial) laboratory setting. Laboratory studies are very powerful tools for doing basic research because all extraneous factors other than those of interest can be controlled or accounted for (high internal validity). However, this tight control over the internal aspects of the lab experiment severely limits it generalizability to the 'real world' circumstances of day-to-day clinical practice. For example, lab mice cannot cheat on a prescribed diet and humans receiving IV drugs under observation cannot 'forget their meds.'

 

In discussing clinical trials, one also may encounter terminology used by the FDA in its approval process for new drugs, devices or procedures. The FDA defines four phases of clinical trials, each with a different purpose (definitions based on National Library of Medicine Medical Subject Headings):

• Phase I Clinical Trials involve studies performed to evaluate the safety of diagnostic, therapeutic, or prophylactic drugs, devices, or techniques in healthy subjects and to determine the safe dosage range (if appropriate). These tests also are used to determine pharmacologic and pharmacokinetic properties (toxicity, metabolism, absorption, elimination, and preferred route of administration). They involve a small number of persons and usually last about 1 year.
• Phase II Clinical Trials involve studies that are usually controlled to assess the effectiveness and dosage (if appropriate) of diagnostic, therapeutic, or prophylactic drugs, devices, or techniques. These studies are performed on several hundred volunteers, including a limited number of patients with the target disease or disorder, and last about two years.
• Phase III Clinical Trials are comparative studies designed to verify the effectiveness of diagnostic, therapeutic, or prophylactic drugs, devices, or techniques determined in phase II studies. During these trials, patients are monitored closely by physicians to identify any adverse reactions from long-term use. These studies are performed on groups of patients large enough to identify clinically significant responses and usually last about three years.
• Phase IV Clinical Trials are planned post-marketing studies of diagnostic, therapeutic, or prophylactic drugs, devices, or techniques that have been approved for general sale. These studies are often conducted to obtain additional data about the safety/efficacy of a product.

 

Observational Studies. The second subcategory of original research is the observational study. In an observational study, the investigator cannot and does not manipulate the exposure or treatment. Instead, subjects are identified and compared (by historical record or survey) as either having or not having the exposure, treatment or attributes of interest or are followed forward in time to observe what occurs in the future (e.g., morbidity, mortality, contracting a disease, etc). In this regard, observational studies are essentially "experiments in nature." In general, the level of evidence provided by observational studies is lower/weaker than that provided by experimental studies due to the many confounding factors that cannot be fully controlled (Gay, 1999). However, for some clinical questions, observational studies are not only appropriate but also represent the best available evidence. In relative order from those providing the strongest to those providing the weakest empirical evidence for clinical practice are the following types of observational studies:

 

• Cohort (Incidence, Longitudinal) Study. A 'cohort' is a group of people clearly identified by one or more specific attributes. Typically, some of these individuals have been exposed to factors hypothesized to influence the occurrence or incidence of a particular disease or other outcome. Although retrospective cohort studies can be performed, most cohort studies follow exposed/unexposed groups forward (prospectively) to determine the association between exposure, other distinguishing group characteristics and the defined outcome(s). Cohort studies are susceptible to several biases, including differential loss to follow-up, the lack of control over risk assignment, and zero time bias (when enrollment of individuals results in unintended but systematic differences between groups at the beginning of the study) (Gay, 1999). Prosepctive cohort studies provide the highest level of evidence among observational methods, but can be very expensive to conduct (Mann, 2003).
• Case-Control Study. A case-control study retrospectively identifies persons with a disease or condition of interest and a control (comparison, referent) group without the disease or condition. The relationship between various factors and the disease are examined by comparing diseased and non-diseased individuals as to the incidence or magnitude of these factors in each group (Duke University Medical Center Library, n.d.). Case-control studies are commonly used for initial, quick and inexpensive assessment of risk factors and are particularly helpful when the condition is rare or the risk factors have long induction periods. Unfortunately, as a retrospective method, the case-control study is prone to several potential forms of bias and thus provides relatively weak empirical evidence, even when properly executed (Mann, 2003).
• Cross-Sectional Study (Survey). A cross-sectional study is a descriptive observational technique that uses the survey method (interviews, questionnaires, diagnostic tests) to determine the presence, absence or degree to which selected attributes exist in a study population or in a representative sample at one particular time. The researchers examine and report findings, without deliberately intervening in the course of events. If the goal of the survey is to determine the prevalence of a disease or condition in a population, the term prevalence study is used. Prevalence studies lack any information on timing of exposure and thus cannot establish a firm causal link to the measured outcome(s) (Mann, 2003).
• Case Series (Review of Cases). A case seriesdescribed the manifestations, clinical course, and prognosis of a disease or condition among a a series of clinical cases (Gay, 1999). Ideally, a good case series also cites the literature in which all other known cases were reported. Due to their lack of control, case series generally provides weak empirical evidence and are best used as a preliminary source for generating hypotheses to test with stronger study designs (Gay, 1999).
• Case Report. An anecdotal description of a single case, generated by the investigator's actual clinical encounters with a patient with a given disease or disorder. The report typically describes the manifestations of the disease or disorder and the patient's clinical course and prognosis (Gay, 1999). Due to large variations in the response to disease among individual patients, single case report provide relative weak evidence for clinical decision-making, but can be helpful in describing how others diagnosed and treated the disease or disorder and its possible outcomes (Gay, 1999).

 

Other Types of Original Research.  There are two other types of original research that do not nicely fit into either the experimental or observational category: methodological studies and evaluation research. What distinguishes these studies from the subject and treatment/exposure-oriented research methods described above is their focus. Methodological studies focus on assessing the tools or methods of research, whereas evaluation research involve drawing summary judgments about the impact of a process or program.

 

• Methodological Studies. Methodological studies involve the development and validation of measurement tools, methods or procedures used in practice and/or clinical research. Methodological studies may be designed to evaluate the accuracy of instrumentation or to assess the reliability and validity of psychological inventories (such as tests of motor development or intelligence tests) or surveys. The goal of a methodological study is not to evaluate the effects of treatment, but to develop valid and reliable instruments that can be used to answer clinical questions. If the study involves a small-scale test of methods or procedures to see if they will work on a larger scale, the term pilot study is used.
• Evaluation Research. Evaluation research studies assess the efficacy or efficiency of processes or programs. They may include the evaluation of cost-effectiveness, the extent to which program objectives are met, or impact (outcomes assessment). If the intent is to determine the advantages or disadvantages, practicability, or capability of accomplishing a projected plan or project, the term feasibility study is used.

 

Secondary or Integrative Studies

Secondary studies summarize and draw conclusions from primary studies. Secondary or integrative studies found in professional journals include systematic reviews and meta-analyses, decision analyses, practice guidelines, consensus statements and journalistic reviews/overviews. Outside the journal literature, books and book chapters are the most common secondary source materials. In relative order from those providing the strongest to those providing the weakest empirical evidence for clinical practice are the following types of secondary studies:

 

Systematic Reviews and Meta-Analyses. Systematic reviews provide summaries of related primary studies that have been searched for, evaluated, selected and reported according to a rigorous and predefined methodology. The most common methodology employed to conduct a systematic review is that developed by the Cochrane Collaboration. Often, a systematic review will employ meta-analysis to statistically combine the numeric results of several separate studies addressing the same question into a single estimate of their combined effect (commonly referred to as 'pooling data'). Typically, the results are presented in graphic form using a 'forest plot' (Figure 3). In this example of a forest plot, the individual odds ratios (with confidence intervals) of five studies (i. – v.) are depicted by the black blobs and lines, with the pooled odds ratios (and its confidence intervals) represented by the open diamond.

 

 

 

Decision Analyses. Decision analyses use the results of primary studies to develop quantitative measures or estimates of the risks and benefits of alternative diagnostic or therapeutic options. As described in Figure 4, these estimates are then used to develop a probability tree that can help health professionals and/or patients make informed choices about clinical management. Decision analysis also can be used to help develop clinical guidelines or consensus statements.

 

 

Practice Guidelines (Evidence-Based). An evidence-based practice guideline is a statement based on systematic review of primary studies that is designed to help clinicians make appropriate decisions about health care in specific clinical circumstances. The guideline is evidence-based to the extent that it employs the same methods and procedures used in developing systematic reviews and/or meta-analyses. Evidence-based guidelines may also involve quantitative assessment of alternative risks and benefits, as with decision analysis.

 

Practice Guidelines (Non-Systematic).  Like its evidence-based counterpart, a non-systematic practice guideline is a statement based on the review of primary studies that is designed to help clinicians make appropriate clinical decisions. However, a non-systematic practice guideline does not employ reproducible methods for evaluating, selecting and reporting the primary studies on which it is based.

 

Consensus Statements. A consensus statement is a statement developed by professionals via a group consensus process that is intended to advance health professional and/or public understanding of a targeted health problem, practice or issue. Typically, consensus statements are developed before the research literature reaches the 'critical mass' needed to conduct systematic reviews or write evidence-based practice guidelines. Notwithstanding this limitation, consensus statements should still be based on evaluation of the available scientific evidence, no matter how lean.

 

Journalistic Reviews/Overviews. Non-systematic or journalistic reviews provide a summary of evidence derived from primary studies that have been selected and synthesized according to the author's personal and professional perspective. Non-systematic reviews can cover a wide range of subject matter at various levels of completeness and comprehensiveness. Most books and book chapters provide a similar perspective and together with journal overviews generally represent the lowest level of secondary or integrative evidence.

 

Levels and Grades of Evidence

 

The prior discussion of study types provided a general hierarchy of evidence for experimental, observational and integrative studies. Now it's time to determine which type of study is best for answering each specific type of question, i.e., therapy, diagnosis, harm/etiology or prognosis.

 

Although the literature provides many systems for defining levels of evidence, the one most commonly cited and used is that developed Oxford Centre for Evidence-based Medicine. Below is an adapted version of the Oxford Levels of Evidence table with accompanying notes:

 

Levels of Evidence Based on Category of Question Being Asked

(adapted from Levels of Evidence and Grades of Recommendation, Oxford Centre for Evidence-based Medicine available online at: http://www.cebm.net/?o=1025

 

 

Notes

 

The Oxford table is the most-cited resource for assessing levels of evidence, but (as suggested by the notes) is a bit complex. A simplified 3-level table of evidence levels, adapted from the Oxford version, is provided below. This simplified table is acceptable for use in Evidence-Based Literature Review (IDST6400). Note the use of Roman numerals instead of Arabic numbers to designate the levels of evidence (I = highest; III = lowest).

 

 

Of course, levels of evidence speak only to the validity of studies and not their clinical applicability. Clinicians normally take other factors into account (such as cost, easy of implementation, impact of the disease when making choices for their patients). To address the broader issue of clinical applicability, the Oxford Centre also provides four grades of recommendations that can be derived from the literature and can be made by or for clinicians considering a diagnostic test, specific therapy, or long-term treatment plan for a patient:

 

Grades of Recommendations (Based on Oxford Levels 1-5)

 

 

More recently, the GRADE Working Group (Atkins, Best, Briss and others, 2004) have developed a system for judging the strength of a recommendation that includes not only the quality of the evidence but also the potential benefits and harms, the costs/ resource utilization, the translation of the evidence into specific circumstances, and the baseline risk.

 

Bibliography

 

Atkins, D., Best, D., Briss, P.A., Eccles, M., Falck-Ytter, Y., Flottorp, S., et al. (2004). GRADE Working Group. Grading quality of evidence and strength of recommendations. BMJ, 328, 1490. Available as full-text via UMDNJ EZProxy

 

Duke University Medical Center Library (n.d.). Study design. Retrieved August 4, 2005 from http://www.mclibrary.duke.edu/subject/ebm/studies.html

 

Gay, J. (1999). Clinical epidemiology & evidence-based medicine glossary: Clinical study design and methods terminology. Available online at: http://www.vetmed.wsu.edu/courses-jmgay/GlossClinStudy.htm

 

Greer, N., Mosser, G., Logan, G., Halaas, G.W. (2000). A practical approach to evidence grading. Jt Comm J Qual Improv, 26, 700-12.

 

Guyatt G., Drummond R. (Eds.) (2002). Users' guide to the medical literature. Chicago, IL: AMA Press.

 

Guyatt, G.H., Keller , J.L., Jaeschke, R., Rosenbloom, D., Adachi, J.D., & Newhouse, M.T. (1990). The n of 1 randomized controlled trial: Clinical usefulness. Ann Intern Med, 112, 293-9.

 

Guyatt, G.H., Sackett D.L., Sinclair, J.C., Hayward, R., Cook, D.J., Cook, R.J., et al. (1995). Users' guide to the medical literature IX: a method for grading health care recommendations. JAMA, 274, 1800-4.

 

Mann, C.J. (2003). Observational research methods. Research design II: cohort, cross sectional, and case-control studies. Emerg Med J, 20, 54-60.

 

Michigan State University - College of Human Medicine, Department of Family Practice Reading and doing decision analyses (Module 6 in An Introduction to Information Mastery). Retrieved May 17, 2005 from http://www.poems.msu.edu/InfoMastery/DecisionAnalysis/DA.htm

 

National Health and Medical Research Council. (2000). How to use the evidence: assessment and application of scientific evidence. Canberra: AusInfo. Available online at:

http://www.nhmrc.gov.au/_files_nhmrc/file/publications/synopses/cp69.pdf

 

Oxford Centre for Evidence-based Medicine (2001). Levels of evidence and grades of recommendation. Available online at: http://www.cebm.net/?o=1025

 

Richardson, W.S., Wilson, M.C., Nishikawa, J., Hayward, R.S. (1995). The well-built clinical question: A key to evidence-based decisions. ACP Journal Club, 123, A-12. Retrieved June 23, 2005 from http://www.mclibrary.duke.edu/training/pdaformat/articlepda.html

 

Sackett, D.L. (1993). Rules of evidence and clinical recommendations for the management of patients. Can J Cardiol, 9, 487-9.

 

University of Sheffield, School of Health and Related Research (n.d.). Appraisal of reviews. Retrieved November 15, 2004 from http://www.shef.ac.uk/scharr/ir/units/critapp/apprev.htm

 

West, S., King, V., Carey, T.S., Lohr, K.N., McKoy, N., Sutton, S.F., et al. (2002). Systems to rate the strength of scientific evidence. Rockville, MD: Agency for Healthcare Research and Quality.  (AHRQ publication No 02-E016.) Retrieved July 13, 2005 from http://www.ahrq.gov/clinic/epcsums/strengthsum.htm