202 Copyright © 2009 The Author(s)

Evidence-Based Practice: Critical Appraisal of Qualitative Evidence

Kathleen M. Williamson

One of the key steps of evidence-based practice is to critically appraise evidence to best answer a clinical question. Mental health clinicians need to understand the importance of qualitative evidence to their practice, including levels of qualitative evidence, qualitative inquiry methods, and criteria used to appraise qualitative evidence to determine how implementing the best qualitative evidence into their practice will influence mental health outcomes. The goal of qualitative research is to develop a complete understanding of reality as it is perceived by the individual and to uncover the truths that exist. These important aspects of mental health require clinicians to engage this evidence. J Am Psychiatr Nurses Assoc, 2009; 15(3), 202-207. DOI: 10.1177/1078390309338733

Keywords: evidence-based practice; qualitative inquiry; qualitative designs; critical appraisal of qualitative evidence; mental health

Evidence-based practice (EBP) is an approach that enables psychiatric mental health care practitioners as well as all clinicians to provide the highest quality of care using the best evidence available (Melnyk & Fineout-Overholt, 2005). One of the key steps of EBP is to critically appraise evidence to best answer a clinical question. For many mental health questions, understanding levels of evidence, qualitative inquiry methods, and questions used to appraise the evidence are necessary to implement the best qualitative evi- dence into practice. Drawing conclusions and making judgments about the evidence are imperative to the EBP process and clinical decision making (Melnyk & Fineout-Overholt, 2005; Polit & Beck, 2008). The over- all purpose of this article is to familiarize clinicians with qualitative research as an important source of evidence to guide practice decisions. In this article, an overview of the goals, methods and types of qualita- tive research, and the criteria used to appraise the quality of this type of evidence will be presented.

QUALITATIVE BELIEFS

Qualitative research aims to generate insight, describe, and understand the nature of reality in

human experiences (Ayers, 2007; Milne & Oberle, 2005; Polit & Beck, 2008; Saddler, 2006; Sandelowski, 2004; Speziale & Carpenter, 2003; Thorne, 2000). Qualitative researchers are inquisitive and seek to understand knowledge about how people think and feel, about the circumstances in which they find themselves, and use methods to uncover and decon- struct the meaning of a phenomenon (Saddler, 2006; Thorne, 2000). Qualitative data are collected in a natural setting. These data are not numerical; rather, they are full and rich descriptions from participants who are experiencing the phenomenon under study. The goal of qualitative research is to uncover the truths that exist and develop a complete understand- ing of reality and the individual’s perception of what is real. This method of inquiry is deeply rooted in descriptive modes of research. “The idea that multiple realties exist and create meaning for the individuals studied is a fundamental belief of qualitative research- ers” (Speziale & Carpenter, 2003, p. 17). Qualitative research is the studying, collecting, and understand- ing the meaning of individuals’ lives using a variety of materials and methods (Denzin & Lincoln, 2005).

WHAT IS A QUALITATIVE RESEARCHER?

Qualitative researchers commonly believe that indi- viduals come to know and understand their reality in

Kathleen M. Williamson, PhD, RN, associate director, Center for the Advancement of Evidence-Based Practice, Arizona State University, College of Nursing & Healthcare Innovation, Phoenix, Arizona; [email protected].

Journal of the American Psychiatric Nurses Association,Vol. 15, No. 3 203

Critical Appraisal of Qualitative Evidence

different ways. It is through the lived experience and the interactions that take place in the natural setting that the researcher is able to discover and understand the phenomenon under study (Miles & Huberman, 1994; Patton, 2002; Speziale & Carpenter, 2003). To ensure the least disruption to the environ- ment/natural setting, qualitative researchers care- fully consider the best research method to answer the research question (Speziale & Carpenter, 2003). These researchers are intensely involved in all aspects of the research process and are considered participants and observers in setting or field (Patton, 2002; Polit & Beck, 2008; Speziale & Carpenter, 2003). Flexibility is required to obtain data from the richest possible sources of information. Using a holistic approach, the researcher attempts to cap- ture the perceptions of the participants from an “emic” approach (i.e., from an insider’s viewpoint; Miles & Huberman, 1994; Speziale & Carpenter, 2003). Often, this is accomplished through the use of a variety of data collection methods, such as inter- views, observations, and written documents (Patton, 2002). As the data are collected, the researcher simultaneously analyzes it, which includes identi- fying emerging themes, patterns, and insights within the data. According to Patton (2002), quali- tative analysis engages exploration, discovery, and inductive logic. The researcher uses a rich literary account of the setting, actions, feelings, and mean- ing of the phenomenon to report the findings (Patton, 2002).

COMMONLY USED QUALITATIVE DESIGNS

According to Patton (2002), “Qualitative methods are first and foremost research methods. They are ways of finding out what people do, know, think, and

feel by observing, interviewing, and analyzing docu- ments” (p. 145). Qualitative research designs vary by type and purpose: data collection strategies used and the type of question or phenomenon under study. To critically appraise qualitative evidence for its valid- ity and use in practice, an understanding of the types of qualitative methods as well as how they are employed and reported is necessary.

Many of the methods are routed in the anthropol- ogy, psychological, and sociology disciplines. Many commonly used methods in the health sciences research are ethnography, phenomenology, and grounded theory (see Table 1).

Ethnography

Ethnography has its traditions in cultural anthropology, which describe the values, beliefs, and practice of cultural groups (Ploeg, 1999; Polit & Beck, 2008). According to Speziale and Carpenter (2003), the characteristics that are central to eth- nography are that (a) the research is focused on culture, (b) the researcher is totally immersed in the culture, and (c) the researcher is aware of her/ his own perspective as well as those in the study. Ethnographic researchers strive to study cultures from an emic approach. The researcher as a par- ticipant observer becomes involved in the culture to collect data, learn from participants, and report on the way participants see their world (Patton, 2002). Data are primarily collected through obser- vations and interviews. Analysis of ethnographic results involves identifying the meanings attrib- uted to objects and events by members of the cul- ture. These meanings are often validated by members of the culture before finalizing the results (called member checks). This is a labor-intensive method that requires extensive fieldwork.

TABLE 1. Most Commonly Used Qualitative Research Methods

Method Purpose

Research question(s)

Sample size (on average) Data sources/collection

Ethnography Describe culture of people

What is it like to live . . . What is it . . . 30-50 Interviews, observations, field

notes, records, chart data, life histories

Phenomenology Describe phenomena, the

appearance of things, as lived experience of humans in a natural setting

What is it like to have this experience? What does it feel like?

6-8 Interviews, videotapes, observations,

in-depth conversations

Grounded theory To develop a theory rather than

describe a phenomenon

Questions emerge from the data

25-50 Taped interview, observation,

diaries, and memos from researcher

Source. Adapted from Polit and Beck (2008) and Speziale and Carpenter(2003).

204 Journal of the American Psychiatric Nurses Association,Vol. 15, No. 3

Williamson

Phenomenology

Phenomenology has its roots in both philosophy and psychology. Polit and Beck (2008) reported, “Phenomenological researchers believe that lived experience gives meaning to each person’s percep- tion of a particular phenomenon” (p. 227). According to Polit and Beck, there are four aspects of the human experience that are of interest to the phe- nomenological researcher: (a) lived space (spatial- ity), (b) lived body (corporeality), (c) lived human relationships (relationality), and (d) lived time (tem- porality). Phenomenological inquiry is focused on exploring how participants in the experience make sense of the experience, transform the experience into consciousness, and the nature or meaning of the experience (Patton, 2002). Interpretive phenom- enology (hermeneutics) focuses on the meaning and interpretation of the lived experience to better understand social, cultural, political, and historical context. Descriptive phenomenology shares vivid reports and describes the phenomenon.

In a phenomenological study, the researcher is an active participant/observer who is totally immersed in the investigation. It involves gaining access to participants who could provide rich descriptions from in-depth interviews to gather all the informa- tion needed to describe the phenomenon under study (Speziale & Carpenter, 2003). Ongoing analyses of direct quotes and statements by participants occur until common themes emerge. The outcome is a vivid description of the experience that captures the meaning of the experience and communicates clearly and logically the phenomenon under study (Speziale & Carpenter, 2003).

Grounded Theory

Grounded theory has its roots in sociology and explores the social processes that are present within human interactions (Speziale & Carpenter, 2003). The purpose is to develop or build a theory rather than test a theory or describe a phenomenon (Patton, 2002). Grounded theory takes an inductive approach in which the researcher seeks to generate emergent categories and integrate them into a theory grounded in the data (Polit & Beck, 2008). The research does not start with a focused problem; it evolves and is discovered as the study progresses. A feature of grounded theory is that the data collection, data analysis, and sampling of participants occur simulta- neously (Polit & Beck, 2008; Powers, 2005). The

researchers using ground theory methodology are able to critically analyze situations, not remove themselves from the study but realize that they are part of it, recognize bias, obtain valid and reliable data, and think abstractly (Strauss & Corbin, 1990).

Data collection is through in-depth interview and observations. A constant comparative process is used for two reasons: (a) to compare every piece of data with every other piece to more accurately refine the relevant categories and (b) to assure the researcher that saturation has occurred. Once saturation is reached the researcher connects the categories, pat- terns, or themes that describe the overall picture that emerged that will lead to theory development.

ASPECTS OF QUALITATIVE RESEARCH

The most important aspects of qualitative inquiry is that participants are actively involved in the research process rather than receiving an interven- tion or being observed for some risk or event to be quantified. Another aspect is that the sample is pur- posefully selected and is based on experience with a culture, social process, or phenomena to collect infor- mation that is rich and thick in descriptions. The final essential aspect of qualitative research is that one or more of the following strategies are used to collect data: interviews, focus groups, narratives, chat rooms, and observation and/or field notes. These methods may be used in combination with each other. The researcher may choose to use triangulation strategies on data collection, investigator, method, or theory and use multiple sources to draw conclusions about the phenomenon (Patton, 2002; Polit & Beck, 2009).

SUMMARY

This is not an inclusive list of qualitative methods that researchers could choose to use to answer a research question, other methods include historical research, feminist research, case study method, and action research. All qualitative research methods are used to describe and discover meaning, understand- ing, or develop a theory and transport the reader to the time and place of the observation and/or inter- view (Patton, 2002).

THE HIERARCHY OF QUALITATIVE EVIDENCE

Clinical questions that require qualitative evi- dence to answer them focus on human response and

Journal of the American Psychiatric Nurses Association,Vol. 15, No. 3 205

Critical Appraisal of Qualitative Evidence

meaning. An important step in the process of apprais- ing qualitative research as a guide for clinical prac- tice is the identification of the level of evidence or the “best” evidence. The level of evidence is a guide that helps identify the most appropriate, rigorous, and clinically relevant evidence to answer the clinical question (Polit & Beck, 2008). Evidence hierarchy for qualitative research ranges from opinion of authori- ties and/or reports of expert committees to a single qualitative research study to metasynthesis (Melnyk & Fineout-Overholt, 2005; Polit & Beck, 2008). A metasynthesis is comparable to meta-analysis (i.e., systematic reviews) of quantitative studies. A meta- synthesis is a technique that integrates findings of multiple qualitative studies on a specific topic, pro- viding an interpretative synthesis of the research findings in narrative form (Polit & Beck, 2008). This is the strongest level of evidence in which to answer a clinical question. The higher the level of evidence the stronger the evidence is to change practice. However, all evidence needs be critically appraised based on (a) the best available evidence (i.e., level of evidence), (b) the quality and reliability of the study, and (c) the applicability of the findings to practice.

CRITICAL APPRAISAL OF QUALITATIVE EVIDENCE

Once the clinical issue has been identified, the PICOT question constructed, and the best evidence located through an exhaustive search, the next step is to critically appraise each study for its validity (i.e., the quality), reliability, and applicability to use in practice (Melnyk & Fineout-Overholt, 2005). Although there is no consensus among qualitative researchers on the quality criteria (Cutcliffe & McKenna, 1999; Polit & Beck, 2008; Powers, 2005; Russell & Gregory, 2003; Sandelowski, 2004), many have published excellent tools that guide the process

for critically appraising qualitative evidence (Duffy, 2005; Melnyk & Fineout-Overholt, 2005; Polit & Beck, 2008; Powers, 2005; Russell & Gregory, 2003; Speziale & Carpenter, 2003). They all base their cri- teria on three primary questions: (a) Are the study findings valid? (b) What were the results of the study? (c) Will the results help me in caring for my patients? According to Melnyk and Fineout-Overholt (2005), “The answers to these questions ensure rele- vance and transferability of the evidence from the search to the specific population for whom the practi- tioner provides care” (p. 120). In using the questions in Tables 2, 3, and 4, one can evaluate the evidence and determine if the study findings are valid, the method and instruments used to acquire the knowl- edge credible, and if the findings are transferable.

The qualitative process contributes to the rigor or trustworthiness of the data (i.e., the quality). “The goal of rigor in qualitative research is to accurately represent study participants’ experiences” (Speziale & Carpenter, 2003, p. 38). The qualitative attributes of validity include credibility, dependability, confirm- ability, transferability, and authenticity (Guba & Lincoln, 1994; Miles & Huberman, 1994; Speziale & Carpenter, 2003).

Credibility is having confidence and truth about the data and interpretations (Polit & Beck, 2008). The credibility of the findings hinges on the skill, competence, and rigor of the researcher to describe the content shared by the participants and the abil- ity of the participants to accurately describe the phenomenon (Patton, 2002; Speziale & Carpenter, 2003). Cutcliffe and McKenna (1999) reported that the most important indicator of the credibility of findings is when a practitioner reads the study find- ings and regards them meaningful and applicable and incorporates them into his or her practice.

Confirmability refers to the way the researcher documents and confirms the study findings (Speziale

TABLE 2. Subquestions to Further Answer, Are the Study Findings Valid?

Participants

Sample

Data collection

How were they selected?

Was it adequate?

How were the data collected?

Did they provide rich and thick descriptions?

Was the setting appropriate to acquire an adequate sample?

Were the tools adequate?

Were the participants’ rights protected?

Was the sampling method appropriate?

How were the data coded? If so how?

Did the researcher eliminate bias?

Do the data accurately represent the study participants?

How accurate and complete were the data?

Was the group or population adequately described?

Was saturation achieved?

Does gathering the data adequately portray the phenomenon?

Source. Adapted from Powers (2005), Polit and Beck (2008), Russell and Gregory (2003), and Speziale and Carpenter (2003).

206 Journal of the American Psychiatric Nurses Association,Vol. 15, No. 3

Williamson

& Carpenter, 2003). Confirmability is the process of confirming the accuracy, relevance, and meaning of the data collected. Confirmability exists if (a) the researcher identifies if saturation was reached and (b) records of the methods and procedures are detailed enough that they can be followed by an audit trail (Miles & Huberman, 1994).

Dependability is a standard that demonstrates whether (a) the process of the study was consistent, (b) data remained consistent over time and conditions, and (c) the results are reliable (Miles & Huberman, 1994; Polit & Beck, 2008; Speziale & Carpenter, 2003). For example, if study methods and results are depend- able, the researcher consistently approaches each occurrence in the same way with each encounter and results were coded with accuracy across the study.

Transferability refers to the probability that the study findings have meaning and are usable by oth- ers in similar situations (i.e., generalizable to others in that situation; Miles & Huberman, 1994; Polit & Beck, 2008; Speziale & Carpenter, 2003). To deter- mine if the findings of a study are transferable and can be used by others, the clinician must consider the potential client to whom the findings may be applied (Speziale & Carpenter, 2003).

Authenticity is when the researcher fairly and faithfully shows a range of different realities and develops an accurate and authentic portrait for the phenomenon under study (Polit & Beck, 2008). For example, if a clinician were to be in the same

environment as the researcher describes, they would experience the phenomenon similarly. All mental health providers need to become familiar with these aspects of qualitative evidence and hone their criti- cal appraisal skills to enable them to improve the outcomes of their clients.

CONCLUSION

Qualitative research aims to impart meaning of the human experience and understand how people think and feel about their circumstances. Qualitative researchers use a holistic approach in an attempt to uncover truths and understand a person’s reality. The researcher is intensely involved in all aspects of the research design, collection, and analysis pro- cesses. Ethnography, phenomenology, and grounded theory are some of the designs that a researcher may use to study a culture, phenomenon, or theory. Data collection strategies vary based on the research question, method, and informants. Methods such as interviews, observations, and journals allow for information-rich participants to provide detailed lit- erary accounts of the phenomenon. Data analysis occurs simultaneously as data collection and is the process by which the researcher identifies themes, concepts, and patterns that provide insight into the phenomenon under study.

One of the crucial steps in the EBP process is to critically appraise the evidence for its use in practice

TABLE 3. Subquestions to Further Answer, What Were the Results of the Study?

Is the research design appropriate for the research question?

Is the description of findings thorough?

Do findings fit the data from which they were generated?

Are the results logical, consistent, and easy to follow?

Was the purpose of the study clear?

Were all themes identified, useful, creative, and convincing of the phenomena?

Source. Adapted from Powers (2005), Russell and Gregory (2003), and Speziale and Carpenter (2003).

TABLE 4. Subquestions to Further Answer, Will the Results Help Me in Caring for My Patients?

What meaning and relevance does this study have for my patients?

How would I use these findings in my practice?

How does the study help provide perspective on my practice?

Are the conclusions appropriate to my patient population?

Are the results applicable to my patients?

How would patient and family values be considered in applying these results?

Source. Adapted from Powers (2005), Russell and Gregory (2003), and Speziale and Carpenter (2003).

Journal of the American Psychiatric Nurses Association,Vol. 15, No. 3 207

Critical Appraisal of Qualitative Evidence

and determine the value of findings. Critical appraisal is the review of the evidence for its validity (i.e., strengths and weaknesses), reliability, and usefulness for clients in daily practice. “Psychiatric mental health clinicians are practicing in an era emphasizing the use of the most current evidence to direct their treatment and interventions” (Rice, 2008, p. 186). Appraising the evidence is essential for assurance that the best knowledge in the field is being applied in a cost-effective, holistic, and effective way. To do this, one must incorporate the critically appraised findings with their abilities as clinicians and their clients’ preferences. As professionals, clinicians are expected to use the EBP process, which includes appraising the evidence to determine if the best results are believable, useable, and dependable. Clinicians in psychiatric mental health must use qualitative evidence to inform their practice deci- sions. For example, how do clients newly diagnosed with bipolar and their families perceive the life impact of this diagnosis? Having a well done meta- synthesis that provides an accurate representation of the participants’ experiences, and is trustworthy (i.e., credible, dependable, confirmable, transferable, and authentic), will provide insight into the situational context, human response, and meaning for these cli- ents and will assist clinicians in delivering the best care to achieve the best outcomes.

REFERENCES

Ayers, L. (2007). Qualitative research proposals—Part I. Journal Wound Ostomy Continence Nursing, 34, 30-32.

Cutcliffe, J. R., & McKenna, H. P. (1999). Establishing the credibil- ity of qualitative research findings: The plot thickens. Journal of Advanced Nursing, 30, 374-380.

Denzin, N. K., & Lincoln, Y. S. (2005). The Sage handbook of qualitative research (3rd ed.). Thousand Oaks, CA: Sage.

Duffy, M. E. (2005). Resources for critically appraising qualitative research evidence of nursing practice clinical question. Clinical Nursing Specialist, 19, 288-290.

Guba, E. G., & Lincoln, Y. S. (1994). Competing paradigms in qualitative research. In N. K. Denzin & Y. S. Lincoln (Eds.), Handbook of qualitative research (pp. 105-117). Thousand Oaks, CA: Sage.

Melnyk, B. M., & Fineout-Overholt, E. (Eds.). (2005). Evidence-based practice in nursing and healthcare. Philadelphia: Lippincott Williams & Wilkins.

Miles, M. B., & Huberman, A. M. (1994). An expend sourcebook qualitative data analysis (4th ed.). Thousand Oaks, CA: Sage.

Milne, J., & Oberle, K. (2005). Enhancing rigor in qualitative description: A case study. Journal Wound Ostomy Continence Nursing, 32, 413-420.

Patton, M. Q. (2002). Qualitative research & evaluation methods (3rd ed.). Thousand Oaks: Sage.

Ploeg, J. (1999). Identifying the best research design to fit the question. Part 2: Qualitative designs. Evidence-Based Nursing, 2, 36-37.

Polit, D. F., & Beck, C. T. (2008). Nursing research: Generating and assessing evidence fro nursing practice. Philadelphia: Lippincott Williams & Wilkins.

Powers, B. A. (2005). Critically appraising qualitative evidence. In B. M. Melnyk & E. Fineout-Overholt (Eds.), Evidence-based practice in nursing and healthcare (pp. 127-162). Philadelphia: Lippincott Williams & Wilkins.

Rice, M. J. (2008). Evidence-based practice in psychiatric care: Defining levels of evidence. Journal of the American Psychiatric Nurses Association, 14(3), 181-187.

Russell, C. K., & Gregory, D. M. (2003). Evaluation of qualitative research studies. Evidence-Based Nursing, 6, 36-40.

Saddler, D. (2006). Research 101. Gastroenterology Nursing, 30, 314-316.

Sandelowski, M. (2004). Using qualitative research. Qualitative Health Research, 14, 1366-1386.

Speziale, H. J. S., & Carpenter, D. R. (2003). Qualitative research in nursing: Advancing the humanistic imperative. Philadelphia: Lippincott Williams & Wilkins.

Strauss, A., & Corbin, J. (1990). Basics of qualitative research: Grounded theory procedures and techniques. London: Sage.

Thorne, S. (2000). Data analysis in qualitative research. Evidence- Based Nursing, 3, 68-70.

For reprints and permission queries, please visit SAGE’s Web site at http://www.sagepub.com/journalsPermissions.nav.