Research Theory, Design, and Methods Walden University

© 2016 Laureate Education, Inc. Page 1 of 3

Trustworthiness (Lincoln & Guba, 1985)

Trustworthiness is 1. The extent to which one can have confidence in the study’s findings 2. Parallel of reliability, validity, and objectivity in traditional “quantitative”

research Trustworthiness Criteria Credibility

Findings and interpretations are plausible to the “researched” (the participants) Do findings accurately reflect reality as seen by participants?

Transferability

Applicability of findings based on comparability of contexts Are conditions similar enough to make findings applicable?

Dependability

Account for factors of instability and change within the natural context Document naturally occurring phenomena (stability and change)

Confirmability

Capacity to authenticate the internal coherence of data, findings, interpretations, and recommendations Document “researcher as instrument” and potential sources of bias

Research Theory, Design, and Methods Walden University

© 2016 Laureate Education, Inc. Page 2 of 3

Insuring Trustworthiness Action Description Insures Prolonged engagement

Investing sufficient time to learn the culture, build trust with stakeholders, understand the scope of target phenomena, and test for misinformation/misinterpretation due to distortion by the researcher or informant

Credibility (internal validity)

Persistent observation

Continuing data collection process to permit identification and assessment of salient factors, and investigation in sufficient detail to separate relevant (typical) from irrelevant (atypical)

Credibility (internal validity)

Triangulation

Data collection and analysis interpretation based on multiple sources, methods, investigators, and theories

Credibility (internal validity)

Peer debriefing

Engage in analytic discussions with neutral peer (e.g., colleague not involved in the project)

Credibility (internal validity)

Member checks

Test veracity of the data, analytic categories (e.g., codes), interpretations, and conclusions with stakeholders to ensure accurate representation of emic perspectives

Credibility (internal validity)

Thick description

Describe procedures, context, and participants in sufficient detail to permit judgment by others of the similarity to potential application sites; specify minimum elements necessary to “recreate” findings

Transferability (external validity)

Audit trail

Records that include raw data; documentation of process and products of data reduction, analysis, and synthesis; methodological process notes; reflexive notes; and instrument development/piloting techniques

Dependability Confirmability (reliability and objectivity)

Negative case analysis

Investigate “disconfirming” instance or outlier; continue investigation until all known cases are accounted for so that data reflects range of variation (vs. normative portrayal)

Credibility (internal validity)

Research Theory, Design, and Methods Walden University

© 2016 Laureate Education, Inc. Page 3 of 3

Action Description Insures Reflexive journal

Researcher’s personal notes; documentation of researcher’s thinking throughout the research process

Credibility (internal validity) Transferability (external validity) Dependability Confirmability (reliability and objectivity)

Referential adequacy

Archiving of a portion of the raw data for subsequent analysis and interpretation, for verification of initial findings and conclusions

Credibility (internal validity)

References Lincoln, Y. S., & Guba, E. G. (1985). Naturalistic inquiry. Thousand Oaks, CA:

Sage.

Educational Evaluation and Policy Analysis Winter, Vol. 6, No. 4, 1984, pp. 379-391

The Problem of Criteria for Judging Interpretive Inquiry

John K. Smith University of Northern Iowa

Over the years various interpretive (or qualitative, naturalistic, etc.) investiga- tors have been concerned with develop- ing criteria for judging, and defining pro- cedures for doing, this type of investiga- tion. Apparently the aim is to establish some set of foundational criteria to sepa- rate the good interpretive study from the not so good. These criteria and procedures would help advocates of this approach refute charges of subjectivity and allow them to lay claim to that certitude that is available, at least as it is generally pre- sumed, to their quantitatively oriented counterparts. Given the assumptions con- ventionally made to distinguish interpre- tive from quantitative inquiry, this paper takes the position that the quest for foun- dations may be a misguided one. If the assumptions and their implications are taken seriously, arguments that attempt to establish such nonarbitrary criteria and/or procedures will, perhaps inevita- bly, be marked by confusion and incon- sistency. To defend this contention, this paper will begin with a brief historical overview of the issue. This discussion will then be followed by a specific example of the problem and an examination of why inconsistency mars our attempts to de- velop criteria given certain assumptions.

T h e a u t h o r wishes to t h a n k Lous Heshusius, Jack Yates, Joe Blase, and three a n o n y m o u s reviewers for their insightful c o m m e n t s of earlier versions of this paper.

Historical Context In that this issue of assumptions and

criteria (and thus the problem of certi- tude) is not a new one for this area of interpretive inquiry, a brief historical note will help to clarify the nature of the problem. One might easily argue that the tenor of our present discussions was es- tablished around the turn of this century by, among others, Dilthey and Weber. Dil- they deserves mention in that he pre- sented one of the first serious and sus- tained challenges to positivism and, in doing so, elaborated a different approach to social inquiry based on assumptions quite different from those of positivism (Hodges, 1944, 1952; Smith, 1983). Weber merits attention in this instance not for his numerous important achievements, but for the legacy left by his substantial, but nonetheless unsuccessful, attempt to develop a social science that combined a focus on subjective meaning with the de- sire for universally valid judgments (Aron, 1967, pp. 185-216).

Dilthey argued that social inquiry should not be conducted with the meth- ods of the physical sciences because there was a fundamental difference in subject matter. Whereas the physical sciences dealt with a series of inanimate objects that could be seen as existing independ- ently of us (a world of independent but objectively knowable facts), the cultural sciences focused on the products of hu- man minds. Those products were inevit-

379

380 John K. Smith

ably and intimately connected to human minds—including those of the investiga- tors—with all of their emotions, values, subjectivity, and so on. Accordingly, since society was the result of conscious human intention, the interrelationships of what was investigated and the investigator were impossible to separate. For Dilthey, there simply was no objective social real- ity as such, that is, divorced from the people who participate with others in con- structing or interpreting reality. Bergner (1981), in his discussion of Dilthey, has put the situation as follows: "We cannot meaningfully stand outside of the events and occurrences of life and say with cer- tainty what they signify. What we can know, however, is that historical and so- cial events do signify things to those who enact them and to those who endure them" (p. 64).

Dilthey clearly believed that positivism erred by making the discovery of social laws, analogous to physical laws, the prin- ciple goal of social inquiry. He said that, if nothing else, the complexity of the so- cial makes it impossible to find such laws. More important, he also argued that to find laws should not even be the point of the cultural studies. The primary concern in this case was understanding (verste- hen). This point was the basis for his dis- cussion of the ideas of lived experience and that understanding required a her- meneutical perspective. In his discussion of the latter idea he noted that, at least at one level, to understand the experiences of another required a constant movement or interplay between the individual expression and the context (hermeneuti- cal circle)—a process with no absolute beginning or absolute ending points.

It is here, however, that Dilthey faced a problem that he was unable to resolve. If meaning must be taken in context and understanding must be hermeneutical, clearly the interpretation of expressions could vary. Given this position, could there be any such thing as a correct inter- pretation? Dilthey could find no founda- tional standards to apply for sorting out conflicting possibilities and, at the same time, he rejected an appeal to transcen- dental values or to a metaphysical solu- tion. Whereas positivism had developed standards against which to judge true

from false or correct from incorrect, Dil- they was caught between assumptions that are epistemologically antifounda- tional and a desire for criteria that are foundational. In other words, his idealist tendencies led to a relativism which he found unacceptable, but from which he could find no escape (Hughes, 1958, p. 199).

Even though Weber obviously disa- greed with Dilthey in various ways (e.g., Weber felt the two sciences differed not because of an inherent subject matter dif- ference, but rather because of a different interest taken in the subject matter), he was influenced by Dilthey's ideas of un- derstanding and interpretation (Roche, 1973; Wrong, 1970). However, Weber re- alized that this idealist-oriented position, in contrast to a realist-oriented positivist position, brought with it a problem of certitude. He therefore attempted to find a middle ground, in the sense of a synthe- sis, between the two approaches. In his conception the problem was that a Dil- thean approach paid too little attention to existent social reality, whereas the other perspective devoted too little attention to the ideas of subjective meaning and social reality as an interpreted reality. Aron (1967) has concisely summarized Weber's project in this area: "How can there be an objective science—one not distorted by our value judgments—of the value- charged productions of man? This is the central question Weber asked himself and to which he tried to provide an answer" (p. 193).

The answer Weber provided cannot be discussed in all of its detail and richness. Suffice it to say that one of the central tenets of his argument (along with the concept of "ideal type") was the separa- tion of the idea of value relevance from that of value judgment. Weber accepted that social reality is meaningful only to the extent that it is based on values. More- over, he said that "without the investiga- tor's evaluative ideas, there would be no principle of selection and no meaningful knowledge of the concrete reality" (1949, p. 82). Yet, at the same time he argued that universally valid knowledge is pos- sible because "all research in the cultural sciences . . . once it is oriented to a given subject matter . . . and has established its

Judging Interpretive Inquiry 381

methodological principles, will consider t h e analysis of data as a n end in i t s e l f (1949, p. 112). In other words, Weber held that even though values are t h e basis for m e a n i n g a n d are crucial to t h e content of inquiry, this does not m e a n that t h e in- quirer must evaluate or m a k e value judg- m e n t s about w h a t is investigated.

That Weber's arguments for a distinc- tion b e t w e e n value relevance and value judgment, and m o r e generally for t h e sep- aration of facts and values, cannot be sus- tained (and even ends in contradiction), is not unrecognized. Various scholars, ar- guing from a r a t h e r wide variety of differ- ent positions, h a v e m a d e substantial cri- tiques of Weber's distinctions and thereby, at least indirectly, of his attempt to synthesize the two perspectives (see, e.g., Habermas, 1968; Hindess, 1977; Lu- kacs, 1971; Rorty, 1982; Strauss, 1953). Simey (1969), however, has put the situ- ation as clearly as anyone: "Weber, t h e r e - fore, was attempting to find a via media between, on the one hand, realism (in its positivistic form), and, on the other, phil- osophical idealism. He accepted both al- ternatives, in t h e sense that h e sought to establish a synthesis b e t w e e n t h e m . But t h e synthesis in fact eluded h i m " (p. 83). T h e question of importance for our pur- poses is, w i t h w h a t consequences for the development of social inquiry?

Two things, very m u c h related, follow from Weber's "failure of synthesis." First, his inability to find a via media m a y well indicate that such a position cannot be found. Certainly since his time social in- quiry has b e e n characterized far less by synthesis and far more by "an e n d e m i c conflict b e t w e e n competing frameworks" (Outhwaite, 1983, p. 2). T h e two ap- proaches have moved along separate lines, differing not only in t e c h n i q u e , b u t also in terms of their philosophical tem- p e r a m e n t s or "logics of justification." What w e refer to as t h e quantitative ap proach is based on an external perspec- tive—facts separate from values, t r u t h as a m a t t e r of correspondence to an inde- p e n d e n t l y existing reality, and t h e idea that w e may adopt a "God's eye" point of view. T h e qualitative perspective is based on an internalist position—truth as so- cially and historically conditioned agree- ment, social reality as m i n d constructed,

and the idea that t h e only point of view w e can take is that of various people based on their various interests, purposes, and so on (Putnam, 1981, pp. 49-74).

Second, in one way Weber's failure is an expression, albeit in different form, of the Dilthean problem of combining as- sumptions that are epistemologically an- tifoundational (meaning as subjective, etc.) with the attempt to find epistemolog- ically foundational criteria (certitude). Weber's via media was really a matter of bringing together a frame of reference based on subjective choice and proce- dures that would yield universally valid judgments. In other words, how can one maintain or have the kind of certitude claimed by and generally granted to the realist perspective in the face of a concern over meaning and intention? Dilthey's problem became, even if in a different form, Weber's problem and, more impor- tant for our purposes, it is the problem that continues to be at the center of any discussion of interpretive approaches to social inquiry.

Finally, one additional c o m m e n t is needed about this historical context. Over the years n u m e r o u s schools of thought h a v e b e e n put forth as t h e basis for t h e interpretive forms of inquiry. T h e n a m e s of these schools are quite well-known (phenomenological, ethnomethodologi- cal, h e r m e n e u t i c a l , etc.). Clearly it would be a mistake not to recognize that t h e r e are differences among these schools. However, it would also be a mistake, pos- sibly even a more serious one, not to rec- ognize their similarities. A good case can be m a d e that the " i n s i g h t s . . . from [these] schools of t h o u g h t . . . stand close to phil- osophical idealism" (Giddens, 1976, p. 155). They all, in at least some important aspects such as the idea that meaning must be grasped hermeneutically, are part of a broad m o v e m e n t away from empiri- cism along the lines of an idealist orien- tation. As Benton (1977) s u m m a r i z e s the situation, "The philosophical sources of h u m a n i s t approaches in the social sci- ences at first glance appear to be very heterogeneous—phenomenological, eth- nomethodological, existentialist. . . But it t u r n s out that most of these philosophical traditions are related, through either his- torical origin, or conceptual affinities or

382 John K. Smith

both, to the work of. . . Kant" (p. 101; see also, Bernstein, 1976; Bleicher, 1982; Hughes, 1980; Levy, 1981; O u t h w a i t e , 1975, 1983; Psathas, 1973; Roche, 1973).

A Contemporary Example

To clarify, in more tangible terms t h a n have b e e n employed u p to this point, the n a t u r e of this problem of criteria, an ex- ample of some length and detail is neces- sary. While m a n y people have discussed, in different forms and with different de- grees of aggressiveness, the topic of crite- ria a n d / o r procedures for interpretive in- quiry (e.g., Bogdan & Biklen, 1982; Bruyn, 1966; Glaser & Strauss, 1967; LeCompte & Goetz, 1982; Johnson, 1975; Wolcott, 1975), an article by Guba (1981) merits special attention (see also, Guba, 1978; Guba & Lincoln, 1981, 1982a, 1982b, 1982c). Unlike some authors, Guba dis- cusses some of the specific assumptions underlying a naturalistic approach and t h e n elaborates the criteria to be used and the procedures to be employed for this mode of inquiry. His article is one of the more precise and concise t r e a t m e n t s of this topic to be found, especially in the educational research literature. Hence, Guba's work will serve as an excellent vehicle with w h i c h to demonstrate the problems that can occur w h e n one ac- cepts certain assumptions and t h e n at- tempts to elaborate certain criteria for judging naturalistic research.

Guba (1981) begins by noting that the rationalist and naturalist paradigms are based on different assumptions (pp. 7 6 - 79). Rationalism assumes that a single reality exists, w h e r e a s naturalism as- sumes there are multiple realities. T h e relationship of the investigator to w h a t is investigated is a n o t h e r area of contrast. In rationalism the object of study is seen as existing i n d e p e n d e n t l y of the investiga- tor. Reality is assumed to be self-subsis- tent, a separate entity w h e t h e r or not it is an object of interest or study. In natural- ism, the investigator and w h a t is investi- gated are seen as i n t e r d e p e n d e n t . Accord- ing to this view, w h a t we treat as real is in large m e a s u r e m i n d - d e p e n d e n t as op- posed to m i n d - i n d e p e n d e n t . As Guba puts it, "The rationalistic paradigm rests on t h e assumption that the inquirer can main- tain a discrete (and discreet) distance from

the objects of t h e inquiry . . . the relation- ship . . . is essentially one of independ- ence" (p. 77), w h e r e a s for naturalism, "the inquirer a n d t h e r e s p o n d e n t . . . are inter- related, with each influencing the other" (p. 77).

Another important difference b e t w e e n the two paradigms concerns t h e n a t u r e of "truth statements." In this case the ration- alist paradigm assumes that generaliza- tions—"enduring t r u t h statements that are context free" (p. 77)—are possible. Naturalism, however, holds that such generalizations are not possible and "that at best w h a t one can hope for are 'working hypotheses' that relate to a particular con- text" (p. 77). T h e implication of this dif- ference in position is that while a n e u t r a l (in the sense of nonevaluative) posture is possible from a rationalist perspective, es- pecially in regard to physical science, from a naturalist perspective neutrality is "impossible w h e n the objects of investi- gation are people" (p. 78). T h e reasons for this impossibility are that h u m a n behav- ior is never context free and that both respondent and investigator exert a recip- rocal influence on one another.

Guba has found the two paradigms dif- fer significantly in terms of their basic assumptions. He s u m m a r i z e s the under- lying assumptions of the naturalistic ap- proach quite well w h e n he says, "Social/ behavioral p h e n o m e n a exist chiefly in the minds of people, and t h e r e are as m a n y realities as persons" (p. 77). Social reality is m i n d - d e p e n d e n t in the sense that people (including investigators) shape or construct reality and people (in- cluding investigators) may construct their realities in their own different ways at different times and places.

Given such a world of multiply con- structed realities, how is it possible to find some versions of reality "trustworthy" and others not? Although he does not specifically define the term trustworthy, Guba does discuss four major aspects of it (pp. 79-82). First, the findings of a trust- worthy study exhibit "isomorphism to the respondents' perceptions" (p. 80). In other words, the investigator's statements ac- curately reflect the respondents' percep- tions. Second, the findings are neutral in that they are a "function solely of subjects (respondents) and conditions of t h e in-

Judging Interpretive Inquiry 383

quiry and not of the biases, motivations, interests, perceptions, and so on of the inquirer" (p. 80). Third, if the inquiry is repeated with the same or similar sub- jects, the findings should be consistent with those of the first inquiry. Fourth, the results must be transferable to other sim- ilar situations. Taken together these points seem to represent an attempt to separate inquiry that tells us how people really perceive their own situations from inquiry that distorts "reality" by the in- volvement of the investigator's percep- tions, biases, and interests. Using this interpretation of trustworthiness, Guba then describes a series of procedures, such as member checks, structural corrobora- tion, and dependability audits, which are used in the naturalist mode to deal with this problem.

Given his assumptions, what Guba is attempting to do by specifying criteria is not possible. The assumptions of multiple realities and reality as mind-involved se- riously undermine the notion of applying foundational criteria to distinguish trust- worthy results from untrustworthy ones. These assumptions and foundational cri- teria are, in a word, incompatible. To ac- cept that social reality is mind-con- structed and that there are multiple real- ities is to deny that there are any "givens" upon which to found knowledge. If one accepts these assumptions, different claims about reality result not from incor- rect procedures but may simply be a case of one investigator's interpretation of real- ity versus another's. In a world of multi- ply constructed, mind-dependent reali- ties, there may be no "court of last resort" to appeal to to sort out trustworthy inter- pretations from untrustworthy ones.

Some examples of the inconsistencies that result from accepting antifounda- tional assumptions while attempting to pose foundational criteria can now be pre- sented. Guba's problems with consistency are clearly evident in his varied use and definition of the term criterion. By formal definition criterion can mean slightly dif- ferent things. Among other possibilities, the term may refer to a "characterizing trait." In this usage it has, at best, mild implications as a prescription for inquirer behavior and does not necessarily refer to something that is held to be foundational.

Criterion may also, however, mean "a standard against which to make a judg- ment." Here the term brings with it a very strong implication of something founda- tional—it is that touchstone that can be employed to sort out the good from the bad, the correct from the incorrect.

In some instances Guba seems to be employing the former definition and seems aware of the antifoundational im- plications of his assumptions. Toward the end of the article he says his criteria are not prescriptions and that he does not want them to become an orthodoxy (p. 90): "But—most emphatically—the [cri- teria] are not prescriptions of how inquiry must be done" (p. 90). If this is the case, then he is discussing simply the charac- terizing traits of a particular approach to inquiry and saying that these criteria are merely the way inquirers seem to do it. The elements he discusses cannot there- fore be thought of in such terms as mini- mums, absolutes, and essentials.

Throughout the paper, however, there is considerable evidence that Guba de- sires criteria to carry more weight than as simply characterizing traits. His first sen- tence demonstrates that he is concerned with "criteria for judging the trustworthi- ness of inquiries conducted with the na- turalistic inquiry paradigm" (p. 75). In fact, the entire paper is directed at dem- onstrating that naturalistic researchers also have standards. For example, Guba poses his criteria as equivalent to or par- allel to those of a rationalist paradigm. The tone is one of "we are equally well protected from the problems of subjectiv- ity." More specifically, he uses terms such as "newer rules" (p. 90), implies that his efforts are an attempt to "codify the safe- guards" (p. 76), and says that he has elab- orated criteria that are to be generally applied by editors, referees, and research- ers to sort out the trustworthy from the untrustworthy: "When a naturalistic study is to be judged, it is these criteria that ought to be brought to bear" (p. 88). This is all rather strongly put and defi- nitely goes beyond describing simply how inquirers proceed through this type of study. Even though at times his use of language makes it difficult to discern his exact meaning or intention, it is clear that he desires standards that are neutral and

384 John K. Smith

nonarbitrary. In the end, his antifounda- tionalist assumptions push his use of cri- teria in one direction, while his desire for "something to hang on to" pushes it in the other direction.

A second major area of inconsistency involves the status of t h e procedures Guba lists. At one point he seems to rec- ognize the epistemological consequences of his assumptions, but at other points he does not. He contends that procedures such as triangulation, peer debriefing, and prolonged engagement will not guarantee that a naturalistic study will be trustwor- thy. These procedures do not constitute, as Guba feels is the case for their ration- alist counterparts, a "theoretically com- plete w h o l e " and will therefore not in- sure, but only increase, the probability of a study's trustworthiness (p. 88). How- ever, shortly after discussing these pro- cedures in terms of "ambiguity" and "in- d e t e r m i n a n c y " (p. 88), he gives some of the m e m b e r s on the list the status of sine qua nons (p. 88). As he notes, "It is incon- ceivable that one would be persuaded of t h e trustworthiness of a study that in- volved no triangulation and no m e m b e r checks" (p. 88). This is quite clearly an attempt to pose an absolute (a necessary if not sufficient condition) on w h i c h to found naturalistic inquiry. T h e establish- m e n t of absolute conditions for trustwor- thiness negates the fact that only indeter- minate procedures can follow from the relativism of multiply constructed reali- ties. T h e assumptions on w h i c h natural- istic research is based are violated by pos- iting procedures that are indispensable. T h e r e is a great deal of difference b e t w e e n describing w h a t one might do and man- dating w h a t one must do.

Inherent in Guba's assumptions and specifically stated by him (p. 78), is the significant idea that the investigator can- not be neutral. This idea immediately pro- vokes problems since earlier h e argues for neutrality as one of the key elements for trustworthiness. This point aside, this is- sue of neutrality appears w h e n Guba at- tempts to preserve the idea of inquirer "distance" from the subject matter and even claims there is such a thing as "op- timal distance" (p. 77). Once it is granted that reality is constructed by people (among w h o m are investigators), this

prospect of maintaining distance is very difficult to defend. To do so, the object of study must exist i n d e p e n d e n t l y of the in- quirer, and the inquirer must have inde- p e n d e n t access to that object. Once the investigator is seen as mind-involved with the construction of reality that he or she investigates, there is no way for the m i n d to stand apart (that God's eye view) and measure distance. The mind, after all, cannot be distanced from itself. Once one grants that reality is m i n d - d e p e n d e n t and that there are as m a n y realities as t h e r e are persons, it follows that it may be im- possible to sort out w h o has maintained proper distance and w h o has not.

Guba's work therefore provides an ex- cellent example in tangible terms of w h a t happens w h e n antifoundational assump- tions are combined with the attempt to pose foundational criteria. T h e inevitable result is inconsistency of argument. It is now time to discuss w h y this result is inevitable.

The Issue of Assumptions and Criteria

Given the assumptions of naturalistic or interpretive inquiry, w h y is it impos- sible to develop and apply foundational criteria to this type of investigation? To answer this question, it is necessary to review briefly how the dominant mode of inquiry—quantitative—has dealt with this issue of assumptions and criteria. One reason for this digression is that interpre- tive inquiry, at least since the time of the challenge initiated by Dilthey and others to positivism, has very often been para- sitic (i.e., taken its strength from and de- veloped through critique of the quantita- tive approach). As Rorty (1979) notes, "Hermeneutics is always parasitic upon the possibility of epistemology" (p. 365). Therefore, to u n d e r s t a n d the claim that there can be no criteria to reconcile dis- course (to sort out the trustworthy from the u n t r u s t w o r t h y results) for interpre- tive inquiry, we must first e x a m i n e the opposing claim that t h e r e are indeed foundations on w h i c h to base epistemo- logical claims.

A n u m b e r of assumptions, some of w h i c h are often implicit, are crucial to the assertion that t h e r e are nonarbitrary criteria for judging and nonarbitrary pro- cedures for doing quantitative inquiry.

Judging Interpretive Inquiry 385

Even though other terms, such as exter- nalism, objectivism, and so on, could be used here (Putnam, 1981, pp. 49-50; Ru- binstein, 1981, pp. 24-25), for our pur- poses the assumptions of this approach can be located within the general philo- sophical perspective or temperament known as realism. A most critical feature of a realist perspective is the idea that "what exists does so whether men con- ceive of it or not" (Trigg, 1980, p. vii). In other words, reality is self-subsistent; it exists "out there" independently of us and of our interest in it. Following the lead of Durkheim (1938), social and educational facts are considered external to us, exist- ing prior to an investigator's study of them, awaiting discovery. While this po- sition grants that minds are necessary to conceptualize reality (by definition only minds can do so), it does not accept the idea that reality is mind-dependent in the sense of being created or constructed. As Trigg notes, "The realist must hold that the mind and the world are separate items" (p. 93).

Given this separation of the knower from what is or can be known, the basic epistemological thrust of realism is that to know reality is to be able to describe or reflect it accurately. "According to most realists . . . a judgment is true when it corresponds to an independent reality and false when it does not" (Ewing, 1974, p. 195). Not surprisingly, this position is la- beled as the correspondence theory of truth. True statements are judged to have accurately reflected the qualities and characteristics of what is out there, whereas false ones are those judged to have in some way distorted the nature of that independently existing reality. For example, if one says the cat is on the mat and the cat is observed to be there, we accept that one has produced a true state- ment. Similarly, if one says that school performance and self-concept are posi- tively related in first graders and on ob- servation this turns out to be the case, one has made a statement we judge to be true. The significant point is that truth has its source in an independently existing real- ity—a reality that can be known for what it really is.

As these examples demonstrate, the process of observation, or empirical veri-

fication, is essential to our ability to judge whether or not statements are true. How- ever, there are many ways of observing the world, and some are considered less subject to error and distortion than others. From the quantitative perspective, one of the greatest dangers to sound empirical verification is subjectivity. Paraphrasing from Nagel's (1981) discussion of the topic, to be subjective is to see the world from one's own particular place within it (pp. 196-213). To approach the world in this way is to let one's personal values and dispositions distort or bias an accurate description of what really exists. The re- sult of such subjectivity, if allowed to dominate our views of the world, would allow that everyone could create his or her own little private reality. To avoid this danger, the demand of quantitative inquiry is for objectivity—that we take a God's eye point of view (Putnam, 1981, p. 49)—or see the world from no particular place within it. How this approach is able to insure objectivity requires that inves- tigators adhere to method, in particular the scientific method.

First and foremost, the methods we con- ventionally think of as scientific serve to constrain the investigator and hence pro- tect him or her from the perils of distort- ing the qualities and characteristics of the object of study. At the level of the practice of inquiry, adherence to these procedures will insure that the study has been prop- erly conducted. And if this is the case, we must accept the results, regardless of how we might feel about them. In other words, these are results to which all "rational" people must accede, and disagreements can be reconciled by a properly done ap- peal to an independent reality. These pro- cedures are neutral and nonarbitrary in that they stand over and beyond any one individual's interests, dispositions, or place in the world. The assumptions of this approach allow for a foundational epistemology, one that holds we can know reality for what it really is and that we have criteria for judging, and procedures to employ as we go about discovering, that reality.

A brief restatement of these issues in slightly different terms will be helpful. The ideal of verification and of a founda- tional epistemology is dependent on the

386 John K. Smith

possibility of building knowledge free from a never-ending circle of interpreta- tion (Taylor, 1971). The nonarbitrary ver- ification of propositions requires that there be such a thing as "brute data" or "givens." These data, and the inferences based on them, stand on their own in the sense that they are "data whose validity cannot be questioned by offering another interpretation or reading, data whose va- lidity cannot be founded on or under- mined by further reasoning" (Taylor, 1971, p. 8). Therefore, to have founda- tional criteria requires one to assume that there is an independent reality out there that can be read for what it is, and, if this is the case, then the claim of certitude may follow.

The assumptions of naturalistic in- quiry—that reality is mind-dependent and that there are multiple realities—are incompatible with the idea of an inde- pendently existing reality that can be known through a neutral set of proce- dures. While various labels, such as inter- nalism, subjectivism, and so on, can be employed here, this approach to inquiry can be discussed within the context of an idealist philosophical temperament. Even though there are several forms of ideal- ism, for our purposes suffice it to say that they all share one point in common: "To the idealist it seems inconceivable that anyone should talk of knowledge in any intelligible sense if the supposed object of knowledge is wholly independent of the knower—in which there is not in some way and in some degree mutual involve- ment of mind and object" (Urban, 1949, p. 16). There is no independent reality or data free from interpretation on which to found our knowledge.

The most striking epistemological con- sequence of this perspective is that cor- respondence or accuracy of reflection is not an acceptable way to define truth. The idea of a mind-dependent reality is in- compatible with the correspondence the- ory. For interpretive inquiry, the basis of truth or trustworthiness is social agree- ment; what is judged true or trustworthy is what we can agree, conditioned by time and place, is true or trustworthy.1 This

1 A coherence theory of t r u t h is often associated with an idealist perspective. Simply put, for this

appeal to common understanding as op- posed to correspondence results in defi- nitions of such basic terms as objective and subjective that are quite different from those used in the quantitative ap- proach to inquiry. To be objective is not a matter of representing things as they really are. As Rorty (1979) notes, "The application of such honorifics as 'objec- tive' and 'cognitive' is never anything more than an expression of the presence of, or the hope for, agreement among in- quirers" (p. 335). Similarly, to be subjec- tive does not mean representing things influenced by the emotions, interests, and so on; it is to "bring in considerations which others think beside the point" (Rorty, 1979, p. 339). What we agree with and on is objective; what we find unfa- miliar we label subjective.

The implications of this idea of agree- ment can be seen if we imagine a situation in which competing claims have been made. From a realist/quantitative per- spective this conflict can be resolved by referring to the extralinguistic facts—that external, independently existing reality. From an idealist/interpretive perspective, however, reference to the facts will not provide a solution, since disagreement is not founded on "error of reflection or cor- respondence" but is simply the result of interpreting things in a different way. While the different interpretations may be subject to questions of internal consis- tency (coherence), there is no nonarbi- trary set of criteria that can be employed to sort out the difference and no brute data to appeal to to reconcile the issue. In a world where "social/behavioral phe-

theory a proposition is judged true if it coheres (is connected and consistent) with other propositions in a scheme or network. Coherence is t h u s a matter of internal relations. One question arises, however, that leads back to the idea of agreement as the basis for truth. That is, how can one choose from among various coherent schemes? As Grayling (1982) puts the problem, "It is logically possible for there to be any n u m b e r of internally coherent systems of be- liefs, and since there is no external criteria for choos- ing among them, it cannot be k n o w n which is the 'right' one" (p. 137). Since idealism holds that there is no independent reality to w h i c h w e have inde- p e n d e n t access (no facts of an extralinguistic nature), a coherence approach must ultimately respond to the idea of common understanding. A scheme or network is held to be true only to the extent we have come to agree, conditioned by time and place, that it is true.

Judging Interpretive Inquiry 387

nomena exist chiefly in the minds of peo- ple, and there are as many realities as persons" (Guba, 1981, p. 77), we are faced with the prospect of one interpretation of reality coexisting with another interpre- tation of reality. No safeguards, rules, or absolute minimums are available to work out our differences. Clearly this position is epistemologically antifoundational.

What does all of this mean for the prac- tice of naturalistic inquiry? According to Taylor (1971), the sciences of man are hermeneutical. A hermeneutical science is directed at interpreting the intentional, meaningful expressions of people and pro- vides interpretations of interpretations people have already given to their own situations. Furthermore, a defining char- acteristic of a hermeneutical science is that it is, in principle, impossible to de- velop a precise and nonarbitrary way to verify results. All that an investigator can do is offer an interpretation (of interpre- tations) and hope others will "catch on" or accept those readings. If they do not, nothing outside of the hope for common understanding can be used to compel agreement. As Taylor puts it, "We can only convince an interlocuter if at some point he shares our understanding of the language concerned. If he does not, there is no further step to take in rational ar- gument; we can try to awaken these in- tuitions in him, or we can simply give up; argument will advance us no further" (p. 6).

Hermeneutics and the idea of the her- meneutical circle have been subject to various interpretations over the years.2

Generally speaking, these terms mean that any reading by the inquirer (or lay- person for that matter) is the product of an interpretive movement back and forth

2 As used here, h e r m e n e u t i c s is not a successor to method as m e t h o d is defined for the quantitative mode of inquiry. H e r m e n e u t i c s is not an alternative, but equally foundational way to obtain the truth. Even though there have been attempts (from Dilthey to Betti) to m a k e h e r m e n e u t i c s the method of the cultural sciences, I think these efforts have not been successful. Thus, the position h e r e is that of Gada- mer (1975): "Hermeneutics . . . is not . . . a m e t h o d - ology of the h u m a n sciences, but an attempt to u n d e r s t a n d w h a t the h u m a n sciences truly are, be- yond their methodological self-consciousness" (p. xiii). See Palmer (1969) for an excellent discussion of these issues.

between parts and whole. In the interpre- tation of a text, for example, individual sentences can only be understood in re- lation to the whole text, and the whole text can be understood only in relation to the individual expressions. To give a read- ing to an expression requires the appli- cation of mental categories to provoke an understanding of the expression and of the context of the expression. As Lessnoff (1974) notes in a discussion based on the work of Melden (1961), "In order to see an event as a human action, it is necessary to interpret its empirically observable fea- tures in terms of mental categories, to assume the applicability of these cate- gories to what is observed" (p. 37). In interpretive understanding, a constant movement back and forth between parts and whole is necessary. However, this process responds to no formal rules of procedure and has no absolute beginning point and no absolute ending point. There is no time at which one can say with assurance, or in a foundationalist manner, "It is over and I have got it right."

The implications of this hermeneutical- interpretive approach to inquiry can be seen in a situation in which one person does not accept the readings of another. Will it not be easier to convince the other person if the inquirer has used some pro- cedures as opposed to others in achieving the result? Not necessarily. If the results are based on particular procedures, such as Guba's member checks and so on, these will be convincing only if these proce- dures make sense to the reader. The prob- lem is that what makes sense depends on how one reads the situation—not only the general situation of interpretive inquiry, but the specific situation of any particular inquiry; not only in terms of the actual interpretation, but also in terms of how the interpretation was obtained. As Tay- lor (1971) notes in a slightly different con- text, "Ultimately, a good explanation is one which makes sense of the behavior; but then to appreciate a good explanation, one has to agree on what makes good sense; what makes good sense is a func- tion of one's readings; and these in turn are based on the kind of sense one under- stands" (p. 14).

What all of this means is that we have no way, in a foundational sense, to know a good interpretation from a bad interpre-

388 John K. Smith

tation. One can try to convince another of the appropriateness of a particular read- ing, but this can only be done through an appeal to common understanding. If such an understanding does not exist, there is no "higher authority" to which one can appeal to resolve the issue. Naturalistic inquiry then, given these idealist assump- tions, begins to look very relativistic. And, once accepted, the idea of relativism can- not be suspended in the quest for foun- dational criteria by which to judge the results of this form of inquiry.

Member Checks as Absolute Minimum A more specific discussion of one pro-

cedure advocated by Guba should clarify some of the preceding points. Guba (1981) says that member checks are an absolute minimum for naturalistic inquiry (p. 88) and that they are the most important thing a naturalistic inquirer can do (p. 85). This is the same as saying the procedure is foundational in that it is a necessary, if not sufficient, condition for trustworthi- ness. If an inquirer has confirmed his or her interpretations with the subjects (and produced a record of how this process has influenced the results), the study has a chance of being judged trustworthy. If not, it is unlikely the results will be granted such status.

Is this claim defensible? If we accept the assumptions of interpretive inquiry, then we must accept that the need for member checks will depend on the per- ceptions of the people involved. If a reader feels member checks are essential, he or she will not accept a study in which this procedure has not been done. Someone else, however, may believe either in gen- eral or for a specific study that they make no difference at all. For example, it is very possible for a person to argue that the most important and original insights come during the initial contacts with subjects. To bring one's interpretations back to them will only serve to attenuate these insights. The point is, since we are not dealing with brute data there is no un- ambiguous way to know what makes sense in this regard.

The question of member checks must also be seen in regard to what we might call a form of reflexivity. Since it can be inferred that the purpose of member

checks, to those who advocate them, is to verify the extent to which the inquirer has accurately described or reflected what the subjects did or said, this proce- dure looks very much at home with the correspondence theory. In other words, implicit in the procedure is the assump- tion of an independent, external reality (the subjects' expressions) that can be known for what it really is. However, when subjects are asked about their re- sponses, it is always possible that this is new information which will lead them to change their self-definition and how they interpret their own past and present sit- uations. This may in turn lead the in- quirer to reinterpret his or her findings in what can be a never-ending process. Sup- pose, for example, that the subject tells the inquirer that the latter's interpreta- tion of what was said is not correct and what was really meant was . . . . The prob- lem is that there is no definitive way to decide between (a) the inquirer did not originally reflect the subject's true mean- ing but now is getting closer, and (b) the inquirer's comments provoked a change in the member's definitions of the situa- tion and hence a change in his or her interpretations. Further questioning will only push the question back, not answer it.

A simple, but illustrative, hypothetical example will be helpful.3 Suppose that a researcher has been a nonparticipant ob- server of a number of "staffings" in which school personnel discuss with parents the special education placement of their chil- dren. The inquirer notes that after about 30 minutes the principal will often say to the parents, "I do not mean to rush you, but there are only two spots left in the resource room and you must make a de- cision soon." The inquirer interprets this to mean, "We have spent a half hour on this, we are busy, and we must get this meeting over." When the inquirer returns for a member check, he or she says, "You said the former but could have really meant the latter." The principal insists he meant the former and that the latter never entered his mind. In the name of iso-

3 T h e idea for, but not the reality of, this example is based on a study by Law (1981).

Judging Interpretive Inquiry 389

morphism or an accurate reflection of reality, must the inquirer accept what the principal said or can he or she continue to offer the alternative interpretation? There is simply no clear-cut way to decide the issue. And if we have no foundational basis from which to judge how to locate the information resulting from a member check, it seems unlikely that this proce- dure, or any other by extension, can be raised to the level of absolute minimum.

One objection to this example requires comment.4 One can argue that an inquirer would ask the questions both directly and indirectly and would ask related ques- tions so that a pattern of responses, inter- preted within the context that occasioned them, could be developed. This process would then yield results that have truth value or are scientifically relevant. Two things can be said in response to this sit- uation. First, that it is a good description of actual fieldwork practice almost goes without saying. Second, that it will pro- vide for results that have truth value or are scientifically relevant, if these terms are taken in a foundationalist sense, is debatable. This approach to the interpre- tation of responses is in essence an expres- sion of the idealist injunction that mean- ing must be grasped hermeneutically. And, if this is the case, than the Dilthean problem returns: Is there any point at which interpretation ends and one can say, "I have got it right?" or, more gener- ally, can there be any such thing as a "correct" interpretation? One might say that the process ends when one has a coherent interpretation that makes sense of the situation. However, given the idealist assumption that there is no inde- pendent reality to which we have inde- pendent access, there can be numerous, yet different, coherent interpretations, and what makes sense depends on the kind of sense one understands. In other words, the antifoundationalist nature of interpretive inquiry means that no epis- temological privilege can be attached to any procedure for doing or criterion for judging this approach to inquiry.

4 This objection was raised by an anonymous re- viewer of a previous version of this paper.

Summary and Implications

This paper has attempted to describe the inconsistencies that occur and to dem- onstrate why they must occur, when the assumptions of naturalistic or interpretive inquiry meet with a desire to establish neutral, nonarbitrary criteria for separat- ing trustworthy from untrustworthy re- sults. This problem of consistency arises from the incompatibility of assumptions that are epistemologically antifounda- tional and the desire for criteria (defined as standards) that are epistemologically foundational. Given the idea that social reality is mind-dependent, that facts can- not be separated from values, that the only point of view we have is that of various people based on various interests and purposes, and so on, it is impossible to be foundational or to "ground" our find- ings in the sense of "getting it right." In this situation what is trustworthy or true is nothing more than what we can agree, at any given time and place, is trustwor- thy or true.

The question now is, What does this mean for the nature of social inquiry? Presuming that consistency is important, we are left with a choice between two options: either give up the assumptions or give up the desire for "something to hang on to." In the first instance, if the claim to certitude is compelling, then we must re- consider, or simply not heed, the assump- tions of this perspective. To some extent there has been movement, even though not necessarily intended, in this latter direction. Guba was clearly drawn by this possibility with his emphasis on iso- morphism, neutrality, accurate represen- tation of respondents' perceptions, and so on. All of these points are more at home as terms of correspondence in that they are sustained by the prospect that an in- vestigator can stand apart from or inde- pendent of social reality and discover and describe that reality as it really is. Le- Compte and Goetz (1982), in another ex- ample, take Guba one step further. For them the difference between the two ap- proaches is one of technique or investi- gator strategy with both based on the same general "logic of justification." They define validity in the same way for both perspectives (as accuracy of representa-

390 John K. Smith

tion) and hold that the difference is sim- ply one of procedure to achieve such accuracy. Thus, one way out of the con- sistency problem is to ease aside the interpretive assumptions and, given the emphasis on "grounding" inquiry, indi- rectly accept those of a realist orientation.

However, if the interpretive assump- tions are more satisfying, then we must put aside the desire to be foundational. And if this is the case, then we will be compelled to rethink the nature of social inquiry. What this rethinking will lead to is impossible to specify. At this stage only a most general sort of speculation is pos- sible. Taking the lead from Rorty (1979, 1982), maybe the situation of assumptions and criteria indicates that we should dis- pense with the traditional ideas of objec- tivity and truth and realize that we are "beyond method." There are no proce- dures or criteria exclusive to or particu- larly appropriate for social inquiry and, accordingly, one cannot simply tell an- other it must be done in this way or that way because to do so will insure objectiv- ity and lead to truth (in any sense other than as a description for a sentence with which one agrees). From this antifounda- tionalist perspective social inquiry might be best seen as "continuous with litera- ture—as interpreting other people to us, and thus enlarging and deepening our sense of community" (Rorty, 1982, p. 203). Sociologists interpret the interpretations of the poor and other marginal peoples, educational psychologists interpret the interpretations of children, and educa- tional evaluators do the same for those affected by particular educational policies and practices. Yet, these interpretations are not about certitude or the discovery of how things really are—they are an at- tempt to enlarge the conversation and to keep it going.

All of this, of course, is very speculative. Not only the solutions, but even the pre- cise nature of the problems, associated with such an antifoundationalist position are extremely unclear. What is clear, however, is that the assumptions of in- terpretive inquiry are incompatible with the desire for foundational criteria. How we are to work out this problem, one way or another, would seem to merit serious attention.

References

A R O N , R. (1967). Main currents in sociological thought 2 (R. Howard & H. Weaver, Trans.). New York: Penguin.

BENTON, T. (1977). Philosophical foundations of the three sociologies. London: Routledge and Kegan Paul.

BERGNER, J. (1981). The origins of formalism in social science. Chicago: University of Chicago Press.

BERNSTEIN, R. (1976). The restructuring of social and political theory. New York: Harcourt, Brace, Jov- anovich.

BLEICHER, J. (1982). The hermeneutic imagination. London: Routledge and Kegan Paul.

BOGDAN, R., & BIKLEN, S. (1982). Qualitative research for education: An introduction to theory and practice. Boston: Allyn and Bacon.

BRUYN, S. (1966). H u m a n perspective in sociology. Englewood Cliffs, NJ: Prentice-Hall.

DURKHEIM, E. (1938). Rules of the sociological method (S. Solovay & J. Mueller, Trans., & G. E. G. Catlin, Ed.). Chicago: University of Chicago Press.

EWING, A. C. (1974). Idealism: A critical survey. Lon- don: M e t h u e n .

GADAMER, H-G. (1975). Truth and method (G. Borden & J. Cumming, Trans.). New York: Seabury Press.

GIDDENS, A. (1976). New rules of sociological method. New York: Basic Books.

GLASER, B., & STRAUSS, A. (1967). Discovery of grounded theory. Chicago: Aldine.

GRAYLING, A. C. (1982). An introduction to philosoph- ical logic. Totowa, NJ: Barnes and Noble.

GUBA, E. (1978). Toward a methodology of naturalis- tic inquiry in educational evaluation (Mono- graph Series No. 8). Los Angeles: Center for the Study of Evaluation, University of California, Los Angeles.

GUBA, E. (1981). Criteria for assessing the trust- worthiness of naturalistic inquiry. Educational Communication and Technology Journal, 29, 7 5 - 9 1 .

GUBA, E., & LINCOLN, Y. (1981). Effective evaluation. San Francisco: Jossey-Bass.

GUBA, E., & LINCOLN, Y. (1982a, March). Causality versus plausibility: Alternative stances for in- quiry into h u m a n behavior. Paper presented at the a n n u a l meeting of the American Educa- tional Research Association, New York.

GUBA, E., & LINCOLN, Y. (1982b, March). Establishing dependability and confirmability in naturalistic inquiry through an audit. Paper presented at the a n n u a l meeting of the American Educational Research Association, New York.

GUBA, E., & LINCOLN, Y. ( 1 9 8 2 C , March). The nature

of naturalistic inquiry. Paper presented at the a n n u a l meeting of the American Educational Research Association, New York.

HABERMAS, J. (1968). Knowledge and h u m a n interests (J. Shapiro, Trans.). Boston: Beacon Press.

HINDESS, B. (1977). Philosophy and methodology in the social sciences. Atlantic Highlands, NJ: Hu- manities Press.

HODGES, H. (1944). Wilhelm Dilthey: An introduction. London: Routledge and Kegan Paul.

HODGES, H. (1952). The philosophy of Wilhelm Dil- they. London: Routledge and Kegan Paul.

Judging Interpretive Inquiry 391

HUGHES, H. (1958). Consciousness and society. New York: Knopf.

HUGHES, J. (1980). The philosophy of social research. London: Longman.

JOHNSON, J. (1975). Doing field research. New York: Free Press.

LAW, S. (1981). The staffing: A naturalistic study of the educational placement process. Unpublished Ed.S. thesis, University of N o r t h e r n Iowa, Cedar Falls.

LECOMPTE, M., & GOETZ, J. (1982). Problems of relia- bility and validity in ethnographic research. Re- view of Educational Research, 52, 31-60.

LESSNOFF, M. (1974). The structure of social science. London: George Allen and Unwin.

LEVY, D. (1981). Realism: An essay in interpretation and social reality. Atlantic Highlands, NJ: Hu- manities Press.

LUKACS, G. (1971). History and class consciousness (R. Livingstone, Trans.). London: Merlin Press.

MELDEN, A. (1961). Free action. London: Rout- ledge and Kegan Paul.

NAGEL, T. (1981). Mortal questions. New York: Cam- bridge University Press.

OUTHWAITE, W. (1975). Understanding social life: The method called verstehen. London: Allen and Un- win.

OUTHWAITE, W. (1983). Concept formation in social science. London: Routledge and Kegan Paul.

PALMER, R. (1969). Hermeneutics. Evanston, IL: Northwestern University Press.

PSATHAS, G. (Ed.). (1973). Phenomenological sociol- ogy. New York: Wiley.

PUTNAM, H. (1981). Reason, truth, and history. Cam- bridge: Cambridge University Press.

ROCHE, M. (1973). Phenomenology, language, a n d the social sciences. London: Routledge and Kegan Paul.

RORTY, R. (1979). Philosophy and the mirror of na- ture. Princeton, NJ: Princeton University Press.

RORTY, R. (1982). Consequences of pragmatism. Min- neapolis: University of Minnesota Press.

RUBINSTEIN, D. (1981). Marx a n d Wittgenstein. Lon- don: Routledge and Kegan Paul.

SIMEY, T. (1969). Social science and social purpose. New York: Schocken.

SMITH, J. (1983). Quantitative versus qualitative re- search: An attempt to clarify the issue. Educa- tional Researcher, 12, 6-13.

STRAUSS, L. (1953). Natural right and history. Chi- cago: University of Chicago Press.

TAYLOR, C. (1971). Interpretation and the sciences of man. Review of Metaphysics, 25, 3 - 5 1 .

TRIGG, R. (1980). Reality at risk. Totowa, NJ: Barnes and Noble.

URBAN, W. (1949). Beyond realism a n d idealism. Lon- don: George Allen and Unwin.

WEBER, M. (1949). The methodology of the social sciences (E. Shils & H. Finch, trans, and eds.). Glencoe, IL: Free Press.

WOLCOTT, H. (1975). Criteria for an ethnographic approach to research in schools. H u m a n Orga- nization, 34, 111-127.

WRONG, D. (Ed.). (1970). Max Weber. Englewood Cliffs, NJ: Prentice-Hall.

Author JOHN K. SMITH, Associate Professor of

Education, Department of Educa- tional Psychology and Foundations, University of Northern Iowa, Cedar Falls, IA 50614. Specializations: Edu- cational research, educational soci- ology.

The Qualitative Report

Volume 8 | Number 4 Article 6

12-1-2003

Understanding Reliability and Validity in Qualitative Research Nahid Golafshani University of Toronto, [email protected]

Follow this and additional works at: https://nsuworks.nova.edu/tqr

Part of the Quantitative, Qualitative, Comparative, and Historical Methodologies Commons, and the Social Statistics Commons

This Article is brought to you for free and open access by the The Qualitative Report at NSUWorks. It has been accepted for inclusion in The Qualitative Report by an authorized administrator of NSUWorks. For more information, please contact [email protected].

Recommended APA Citation Golafshani, N. (2003). Understanding Reliability and Validity in Qualitative Research. The Qualitative Report, 8(4), 597-606. Retrieved from https://nsuworks.nova.edu/tqr/vol8/iss4/6

Understanding Reliability and Validity in Qualitative Research

Abstract The use of reliability and validity are common in quantitative research and now it is reconsidered in the qualitative research paradigm. Since reliability and validity are rooted in positivist perspective then they should be redefined for their use in a naturalistic approach. Like reliability and validity as used in quantitative research are providing springboard to examine what these two terms mean in the qualitative research paradigm, triangulation as used in quantitative research to test the reliability and validity can also illuminate some ways to test or maximize the validity and reliability of a qualitative study. Therefore, reliability, validity and triangulation, if they are relevant research concepts, particularly from a qualitative point of view, have to be redefined in order to reflect the multiple ways of establishing truth.

Keywords Reliability, Validity, Triangulation, Construct, Qualitative, and Quantitative

Creative Commons License

This work is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 4.0 License.

This article is available in The Qualitative Report: https://nsuworks.nova.edu/tqr/vol8/iss4/6

The Qualitative Report Volume 8 Number 4 December 2003 597-607 http://www.nova.edu/ssss/QR/QR8-4/golafshani.pdf

Understanding Reliability and Validity in Qualitative Research

Nahid Golafshani University of Toronto, Toronto, Ontario, Canada

The use of reliability and validity are common in quantitative research and now it is reconsidered in the qualitative research paradigm. Since reliability and validity are rooted in positivist perspective then they should be redefined for their use in a naturalistic approach. Like reliability and validity as used in quantitative research are providing springboard to examine what these two terms mean in the qualitative research paradigm, triangulation as used in quantitative research to test the reliability and validity can also illuminate some ways to test or maximize the validity and reliability of a qualitative study. Therefore, reliability, validity and triangulation, if they are relevant research concepts, particularly from a qualitative point of view, have to be redefined in order to reflect the multiple ways of establishing truth. Key words: Reliability, Validity, Triangulation, Construct, Qualitative, and Quantitative

This article discusses the use of reliability and validity in the qualitative research

paradigm. First, the meanings of quantitative and qualitative research are discussed. Secondly, reliability and validity as used in quantitative research are discussed as a way of providing a springboard to examining what these two terms mean and how they can be tested in the qualitative research paradigm. This paper concludes by drawing upon the use of triangulation in the two paradigms (quantitative and qualitative) to show how the changes have influenced our understanding of reliability, validity and triangulation in qualitative studies.

What is Quantitative Research? Researchers who use logical positivism or quantitative research employ experimental

methods and quantitative measures to test hypothetical generalizations (Hoepfl, 1997), and they also emphasize the measurement and analysis of causal relationships between variables (Denzin and Lincoln, 1998). To illustrate the meaning of quantitative research for its use of explaining social problems, Bogdan and Biklen (1998) note:

Charts and graphs illustrate the results of the research, and commentators employ words such as ‘variables’, ‘populations’ and ‘result’ as part of their daily vocabulary…even if we do not always know just what all of the terms mean…[but] we know that this is part of the process of doing research. Research, then as it comes to be known publicly, is a synonym for quantitative research. (p. 4) Quantitative research allows the researcher to familiarize him/herself with the problem or

concept to be studied, and perhaps generate hypotheses to be tested. In this paradigm: (1) the emphasis is on facts and causes of behaviour (Bogdan & Biklen, 1998), (2) the information is in

Nahid Golafshani 598

the form of numbers that can be quantified and summarized, (3) the mathematical process is the norm for analysing the numeric data and (4) the final result is expressed in statistical terminologies (Charles, 1995).

Generally, quantitative research “…supported by the positivist or scientific paradigm, leads us to regard the world as made up of observable, measurable facts” (Glesne & Peshkin, 1992, p. 6) though their assumption that “social facts have an objective reality” and “variables can…be identified and relationships measured” (p. 7) is problematic. The notion of ‘measuring’ means to understand, say, educational issues by performing an operation called ‘measurement’ on the physical world by the observer (Crocker & Algina, 1986). Stevens (1946) defines measurement as the assignment of numerals to objects or events according to rules. From these definitions, one may perceive measurement as necessarily objective, quantitative and statistically relevant. Simply put, measurement can be about numbers, objective hard data.

A quantitative researcher attempts to fragment and delimit phenomena into measurable or common categories that can be applied to all of the subjects or wider and similar situations (Winter, 2000). In his/her attempts, the researcher's methods involve the "use of standardised measures so that the varying perspectives and experiences of people can be fit into a limited number of predetermined response categories to which number are assigned" (Patton, 2001, p.14). For example, a quantitative researcher may prepare a list of behaviour to be checked or rated by an observer using a predetermined schedule or numbers (scales) as an instrument in his/her method of research. Thus, a quantitative researcher needs to construct an instrument to be administered in standardised manner according to predetermined procedures. But the question is if the measuring instrument measures what it is supposed to measure. In the broadest sense, devising a test (Crocker & Algina, 1986) or the validity of an instrument is on focus. The significance of this test is to ensure replicability or repeatability of the result.

Reliability and Validity in Quantitative Research

“Reliability and validity are tools of an essentially positivist epistemology.”

(Watling, as cited in Winter, 200, p. 7) Reliability

Joppe (2000) defines reliability as: …The extent to which results are consistent over time and an accurate representation of the total population under study is referred to as reliability and if the results of a study can be reproduced under a similar methodology, then the research instrument is considered to be reliable. (p. 1)

Embodied in this citation is the idea of replicability or repeatability of results or observations.

Kirk and Miller (1986) identify three types of reliability referred to in quantitative research, which relate to: (1) the degree to which a measurement, given repeatedly, remains the same (2) the stability of a measurement over time; and (3) the similarity of measurements within a given time period (pp. 41-42).

Charles (1995) adheres to the notions that consistency with which questionnaire [test] items are answered or individual’s scores remain relatively the same can be determined through

599 The Qualitative Report December 2003

the test-retest method at two different times. This attribute of the instrument is actually referred to as stability. If we are dealing with a stable measure, then the results should be similar. A high degree of stability indicates a high degree of reliability, which means the results are repeatable. Joppe, (2000) detects a problem with the test-retest method which can make the instrument, to a certain degree, unreliable. She explains that test-retest method may sensitize the respondent to the subject matter, and hence influence the responses given. We cannot be sure that there was no change in extraneous influences such as an attitude change that has occurred. This could lead to a difference in the responses provided. Similarly, Crocker and Algina (1986) note that when a respondent answer a set of test items, the score obtained represents only a limited sample of behaviour. As a result, the scores may change due to some characteristic of the respondent, which may lead to errors of measurement. These kinds of errors will reduce the accuracy and consistency of the instrument and the test scores. Hence, it is the researchers’ responsibility to assure high consistency and accuracy of the tests and scores. Thus, Crocker and Algina (1986) say, "Test developers have a responsibility of demonstrating the reliability of scores from their tests" (p. 106).

Although the researcher may be able to prove the research instrument repeatability and internal consistency, and, therefore reliability, the instrument itself may not be valid.

Validity

The traditional criteria for validity find their roots in a positivist tradition, and to an extent, positivism has been defined by a systematic theory of validity. Within the positivist terminology, validity resided amongst, and was the result and culmination of other empirical conceptions: universal laws, evidence, objectivity, truth, actuality, deduction, reason, fact and mathematical data to name just a few (Winter, 2000).

Joppe (2000) provides the following explanation of what validity is in quantitative research:

Validity determines whether the research truly measures that which it was intended to measure or how truthful the research results are. In other words, does the research instrument allow you to hit "the bull’s eye" of your research object? Researchers generally determine validity by asking a series of questions, and will often look for the answers in the research of others. (p. 1)

Wainer and Braun (1998) describe the validity in quantitative research as

“construct validity”. The construct is the initial concept, notion, question or hypothesis that determines which data is to be gathered and how it is to be gathered. They also assert that quantitative researchers actively cause or affect the interplay between construct and data in order to validate their investigation, usually by the application of a test or other process. In this sense, the involvement of the researchers in the research process would greatly reduce the validity of a test.

Insofar as the definitions of reliability and validity in quantitative research reveal two strands: Firstly, with regards to reliability, whether the result is replicable. Secondly, with regards to validity, whether the means of measurement are accurate and whether they are actually measuring what they are intended to measure. However, the concepts of reliability and validity are viewed differently by qualitative researchers who strongly consider these concepts

Nahid Golafshani 600

defined in quantitative terms as inadequate. In other words, these terms as defined in quantitative terms may not apply to the qualitative research paradigm. The question of replicability in the results does not concern them (Glesne & Peshkin, 1992), but precision (Winter, 2000), credibility, and transferability (Hoepf, 1997) provide the lenses of evaluating the findings of a qualitative research. In this context, the two research approaches or perspectives are essentially different paradigms (Kuhn, 1970).

What is Qualitative Research?

Qualitative research uses a naturalistic approach that seeks to understand phenomena in

context-specific settings, such as "real world setting [where] the researcher does not attempt to manipulate the phenomenon of interest" (Patton, 2001, p. 39). Qualitative research, broadly defined, means "any kind of research that produces findings not arrived at by means of statistical procedures or other means of quantification" (Strauss and Corbin, 1990, p. 17) and instead, the kind of research that produces findings arrived from real-world settings where the "phenomenon of interest unfold naturally" (Patton, 2001, p. 39). Unlike quantitative researchers who seek causal determination, prediction, and generalization of findings, qualitative researchers seek instead illumination, understanding, and extrapolation to similar situations (Hoepfl, 1997).

Qualitative analysis results in a different type of knowledge than does quantitative inquiry because one party argues from the underlying philosophical nature of each paradigm, enjoying detailed interviewing and the other focuses on the apparent compatibility of the research methods, “enjoying the rewards of both numbers and words” (Glesne & Peshkin, 1992, p. 8). This means such methods like interviews and observations are dominant in the naturalist (interpretive) paradigm and supplementary in the positive paradigm, where the use of survey serves in opposite order. Although it has been claimed (Winter, 2000) that quantitative researchers attempt to disassociate themselves as much as possible from the research process, qualitative researchers have come to embrace their involvement and role within the research. Patton (2001) supports the notion of researcher's involvement and immersion into the research by discussing that the real world are subject to change and therefore, a qualitative researcher should be present during the changes to record an event after and before the change occurs. However, both qualitative and quantitative researchers need to test and demonstrate that their studies are credible. While the credibility in quantitative research depends on instrument construction, in qualitative research, “the researcher is the instrument" (Patton, 2001, p. 14). Thus, it seems when quantitative researchers speak of research validity and reliability, they are usually referring to a research that is credible while the credibility of a qualitative research depends on the ability and effort of the researcher. Although reliability and validity are treated separately in quantitative studies, these terms are not viewed separately in qualitative research. Instead, terminology that encompasses both, such as credibility, transferability, and trustworthiness is used.

Reliability and Validity in Qualitative Research

To understand the meaning of reliability and validity, it is necessary to present the

various definitions of reliability and validity given by many qualitative researchers from different perspectives.

601 The Qualitative Report December 2003

Reliability

Although the term ‘Reliability’ is a concept used for testing or evaluating quantitative research, the idea is most often used in all kinds of research. If we see the idea of testing as a way of information elicitation then the most important test of any qualitative study is its quality. A good qualitative study can help us “understand a situation that would otherwise be enigmatic or confusing” (Eisner, 1991, p. 58). This relates to the concept of a good quality research when reliability is a concept to evaluate quality in quantitative study with a “purpose of explaining” while quality concept in qualitative study has the purpose of “generating understanding” (Stenbacka, 2001, p. 551). The difference in purposes of evaluating the quality of studies in quantitative and quantitative research is one of the reasons that the concept of reliability is irrelevant in qualitative research. According to Stenbacka, (2001) “the concept of reliability is even misleading in qualitative research. If a qualitative study is discussed with reliability as a criterion, the consequence is rather that the study is no good” (p. 552).

On the other hand, Patton (2001) states that validity and reliability are two factors which any qualitative researcher should be concerned about while designing a study, analysing results and judging the quality of the study. This corresponds to the question that “How can an inquirer persuade his or her audiences that the research findings of an inquiry are worth paying attention to?" (Lincoln & Guba, 1985, p. 290). To answer to the question, Healy and Perry (2000) assert that the quality of a study in each paradigm should be judged by its own paradigm's terms. For example, while the terms Reliability and Validity are essential criterion for quality in quantitative paradigms, in qualitative paradigms the terms Credibility, Neutrality or Confirmability, Consistency or Dependability and Applicability or Transferability are to be the essential criteria for quality (Lincoln & Guba, 1985). To be more specific with the term of reliability in qualitative research, Lincoln and Guba (1985, p. 300) use “dependability”, in qualitative research which closely corresponds to the notion of “reliability” in quantitative research. They further emphasize “inquiry audit” (p. 317) as one measure which might enhance the dependability of qualitative research. This can be used to examine both the process and the product of the research for consistency (Hoepfl, 1997). In the same vein, Clont (1992) and Seale (1999) endorse the concept of dependability with the concept of consistency or reliability in qualitative research. The consistency of data will be achieved when the steps of the research are verified through examination of such items as raw data, data reduction products, and process notes (Campbell, 1996).

To ensure reliability in qualitative research, examination of trustworthiness is crucial. Seale (1999), while establishing good quality studies through reliability and validity in qualitative research, states that the “trustworthiness of a research report lies at the heart of issues conventionally discussed as validity and reliability” (p. 266). When judging (testing) qualitative work, Strauss and Corbin (1990) suggest that the "usual canons of ‘good science’…require redefinition in order to fit the realities of qualitative research" (p. 250).

In contrast, Stenbacka (2001) argues that since reliability issue concerns measurements then it has no relevance in qualitative research. She adds the issue of reliability is an irrelevant matter in the judgement of quality of qualitative research. Therefore, if it is used then the “consequence is rather that the study is no good” (p. 552).

To widen the spectrum of conceptualization of reliability and revealing the congruence of reliability and validity in qualitative research, Lincoln and Guba (1985) states that: "Since there can be no validity without reliability, a demonstration of the former [validity] is sufficient to

Nahid Golafshani 602

establish the latter [reliability;]" (p. 316). Patton (2001) with regards to the researcher's ability and skill in any qualitative research also states that reliability is a consequence of the validity in a study. Validity

The concept of validity is described by a wide range of terms in qualitative studies. This concept is not a single, fixed or universal concept, but “rather a contingent construct, inescapably grounded in the processes and intentions of particular research methodologies and projects” (Winter, 2000, p.1). Although some qualitative researchers have argued that the term validity is not applicable to qualitative research, but at the same time, they have realised the need for some kind of qualifying check or measure for their research. For example, Creswell & Miller (2000) suggest that the validity is affected by the researcher’s perception of validity in the study and his/her choice of paradigm assumption. As a result, many researchers have developed their own concepts of validity and have often generated or adopted what they consider to be more appropriate terms, such as, quality, rigor and trustworthiness (Davies & Dodd, 2002; Lincoln & Guba, 1985; Mishler, 2000; Seale, 1999; Stenbacka, 2001).

The discussion of quality in qualitative research initiated from the concerns about validity and reliability in quantitative tradition which “involved substituting new term for words such as validity and reliability to reflect interpretivist [qualitative] conceptions” (Seale, 1999, p. 465).

The issue of validity in qualitative research has not been disregarded by Stenbacka (2001) as she has for the issue of reliability in qualitative research. Instead, she argues that the concept of validity should be redefined for qualitative researches. Stenbacka (2001) describes the notion of reliability as one of the quality concepts in qualitative research which "to be solved in order to claim a study as part of proper research" (p. 551).

In searching for the meaning of rigor in research, Davies and Dodd (2002) find that the term rigor in research appears in reference to the discussion about reliability and validity. Davies and Dodd (2002) argue that the application of the notion rigor in qualitative research should differ from those in quantitative research by “accepting that there is a quantitative bias in the concept of rigor, we now move on to develop our reconception of rigor by exploring subjectivity, reflexivity, and the social interaction of interviewing” (p. 281).

Lincoln and Guba (1985) argue that sustaining the trustworthiness of a research report depends on the issues, quantitatively, discussed as validity and reliability. The idea of discovering truth through measures of reliability and validity is replaced by the idea of trustworthiness (Mishler, 2000), which is “defensible” (Johnson 1997, p. 282) and establishing confidence in the findings (Lincoln & Guba, 1985).

If the issues of reliability, validity, trustworthiness, quality and rigor are meant differentiating a 'good' from 'bad' research then testing and increasing the reliability, validity, trustworthiness, quality and rigor will be important to the research in any paradigm.

Testing Validity and Reliability

So far, the concepts of reliability and validity as they have been redefined for their usefulness in qualitative research have been presented. Now, the question which remains to be answered is ‘How to test or maximize the validity and as a result the reliability of a qualitative study?’

603 The Qualitative Report December 2003

If the validity or trustworthiness can be maximized or tested then more “credible and defensible result” (Johnson, 1997, p. 283) may lead to generalizability which is one of the concepts suggested by Stenbacka (2001) as the structure for both doing and documenting high quality qualitative research. Therefore, the quality of a research is related to generalizability of the result and thereby to the testing and increasing the validity or trustworthiness of the research.

In contrast, Maxwell (1992) observes that the degree to which an account is believed to be generalizable is a factor that clearly distinguishes quantitative and qualitative research approaches. Although the ability to generalize findings to wider groups and circumstances is one of the most common tests of validity for quantitative research, but Patton (2001) states generalizability as one of the criteria for quality case studies depending on the case selected and studied. In this sense the validity in quantitative research is very specific to the test to which it is applied – where triangulation methods are used in qualitative research. Triangulation is typically a strategy (test) for improving the validity and reliability of research or evaluation of findings. Mathison (1988) elaborates this by saying:

Triangulation has risen an important methodological issue in naturalistic and qualitative approaches to evaluation [in order to] control bias and establishing valid propositions because traditional scientific techniques are incompatible with this alternate epistemology. (p. 13)

Patton (2001) advocates the use of triangulation by stating “triangulation strengthens a

study by combining methods. This can mean using several kinds of methods or data, including using both quantitative and qualitative approaches” (p. 247). However, the idea of combining methods has been challenged by Barbour (1998). She argues while mixing paradigms can be possible but mixing methods within one paradigm, such as qualitative research, is problematic since each method within the qualitative paradigm has its own assumption in “terms of theoretical frameworks we bring to bear on our research” (p. 353). Even though triangulation is used in quantitative paradigm for confirmation and generalization of a research, Barbour (1998) does not disregard the notion of triangulation in qualitative paradigm and she states the need to define triangulation from a qualitative research’s perspective in each paradigm. For example, in using triangulation of several data sources in quantitative research, any exception may lead to a disconfirmation of the hypothesis where exceptions in qualitative research are dealt to modify the theories and are fruitful.

In this view, Healy and Perry (2000) explicate on the judging validity and reliability within the realism paradigm which relies on multiple perceptions about a single reality. They argue the involvement of triangulation of several data sources and their interpretations with those multiple perceptions in the realism paradigm.

Another paradigm in qualitative research is constructivism which views knowledge as socially constructed and may change depending on the circumstances. Crotty (1998) defined constructivism from the social perspectives as "the view that all knowledge, and therefore all meaningful reality as such, is contingent upon human practices, being constructed in and out of interaction between human beings and their world, and developed and transmitted within an essentially social context" (p. 42). In any qualitative research, the aim is to "engage in research that probes for deeper understanding rather than examining surface features” (Johnson, 1995, p. 4) and constructivism may facilitate toward that aim. The constructivist notion, that reality is changing whether the observer wishes it or not (Hipps, 1993), is an indication of multiple or

Nahid Golafshani 604

possibly diverse constructions of reality. Constructivism values multiple realities that people have in their minds. Therefore, to acquire valid and reliable multiple and diverse realities, multiple methods of searching or gathering data are in order. If this calls for the use of triangulation in the constructivism paradigm, then the use of investigators, method and data triangulations to record the construction of reality is appropriate (Johnson, 1997). An open-ended perspective in constructivism adheres with the notion of data triangulation by allowing participants in a research to assist the researcher in the research question as well as with the data collection. Engaging multiple methods, such as, observation, interviews and recordings will lead to more valid, reliable and diverse construction of realities. To improve the analysis and understanding of construction of others, triangulation is a step taken by researchers to involve several investigators or peer researchers’ interpretation of the data at different time or location. In a related way, a qualitative researcher can “use investigator triangulation and consider the ideas and explanations generated by additional researchers studying the research participants” (Johnson, 1997, p. 284).

Triangulation may include multiple methods of data collection and data analysis, but does not suggest a fix method for all the researches. The methods chosen in triangulation to test the validity and reliability of a study depend on the criterion of the research.

Our Understanding

From the foregoing discussion, the association of quantitative paradigm with qualitative research through validity and reliability have changed our understanding of the traditional meaning of reliability and validity from the qualitative researchers’ perspectives. Reliability and validity are conceptualized as trustworthiness, rigor and quality in qualitative paradigm. It is also through this association that the way to achieve validity and reliability of a research get affected from the qualitative researchers’ perspectives which are to eliminate bias and increase the researcher’s truthfulness of a proposition about some social phenomenon (Denzin, 1978) using triangulation. Then triangulation is defined to be “a validity procedure where researchers search for convergence among multiple and different sources of information to form themes or categories in a study” (Creswell & Miller, 2000, p. 126).

Therefore, reliability, validity and triangulation, if they are to be relevant research concepts, particularly from a qualitative point of view, have to be redefined as we have seen in order to reflect the multiple ways of establishing truth.

References Babour, R. S. (1998). Mixing qualitative methods: Quality assurance or qualitative quagmire?

Qualitative Health Research, 8(3), 352-361. Bogdan, R. C. & Biklen, S. K. (1998). Qualitative research in education: An introduction to

theory and methods (3rd ed.). Needham Heights, MA: Allyn & Bacon. Campbell, T. (1996). Technology, multimedia, and qualitative research in education. Journal of

Research on Computing in Education, 30(9), 122-133. Charles, C. M. (1995). Introduction to educational research (2nd ed.). San Diego, Longman. Clont, J. G. (1992). The concept of reliability as it pertains to data from qualitative studies.

Paper Presented at the annual meeting of the South West Educational Research Association. Houston, TX.

605 The Qualitative Report December 2003

Creswell, J. W. & Miller, D. L. (2000). Determining validity in qualitative inquiry. Theory into Practice, 39(3), 124-131.

Crocker, L., & Algina, J. (1986). Introduction to classical and modern test theory. Toronto: Holt, RineHart, and Winston, Inc.

Davies, D., & Dodd, J. (2002). Qualitative research and the question of rigor. Qualitative Health research, 12(2), 279-289.

Denzin, N. K. (1978). The research act: A theoretical introduction to sociological methods. New York: McGraw-Hill.

Denzin, N. K., & Lincoln, Y. S. Eds.). (1998). The landscape of qualitative research: Theories and issues. Thousand Oaks: Sage Publications.

Denzin, N. K., & Lincoln, Y. S. (1998) (Eds). Collecting and interpreting qualitative materials. Thousand Oaks: Sage Publication.

Eisner, E. W. (1991). The enlightened eye: Qualitative inquiry and the enhancement of educational practice. New York, NY: Macmillan Publishing Company.

Glesne, C., & Peshkin, P. (1992). Becoming qualitative researches: An introduction. New York, NY: Longman.

Healy, M., & Perry, C. (2000). Comprehensive criteria to judge validity and reliability of qualitative research within the realism paradigm. Qualitative Market Research, 3(3), 118- 126.

Hoepfl, M. C. (1997). Choosing qualitative research: A primer for technology education researchers. Journal of Technology Education, 9(1), 47-63. Retrieved February 25, 1998, from http://scholar.lib.vt.edu/ejournals/JTE/v9n1/pdf/hoepfl.pdf

Hipps, J. A. (1993). Trustworthiness and authenticity: Alternate ways to judge authentic assessments. Paper presented at the annual meeting of the American Educational Research Association. Atlanta, GA.

Johnson, B. R. (1997). Examining the validity structure of qualitative research. Education, 118(3), 282-292.

Johnson, S. D. (1995, Spring). Will our research hold up under scrutiny? Journal of Industrial Teacher Education, 32(3), 3-6.

Joppe, M. (2000). The Research Process. Retrieved February 25, 1998, from http://www.ryerson.ca/~mjoppe/rp.htm

Kirk, J., & Miller, M. L. (1986). Reliability and validity in qualitative research. Beverly Hills: Sage Publications.

Lincoln, Y. S., & Guba, E. G. (1985). Naturalistic inquiry. Beverly Hills, CA: Sage. Mathison, S. (1988). Why triangulate? Educational Researcher, 17(2), 13-17. Maxwell, J. A. (1992). Understanding and validity in qualitative research. Harvard Educational

Review, 62(3), 279-300 Patton, M. Q. (2002). Qualitative evaluation and research methods (3rd ed.). Thousand Oaks,

CA: Sage Publications, Inc. Seale, C. (1999). Quality in qualitative research. Qualitative Inquiry, 5(4), 465-478. Stenbacka, C. (2001). Qualitative research requires quality concepts of its own. Management

Decision, 39(7), 551-555 Strauss, A., & Corbin, J. (1990). Basics of qualitative research: Grounded theory procedures

and techniques. Newbury Park, CA: Sage Publications, Inc. Wainer, H., & Braun, H. I. (1988). Test validity. Hilldale, NJ: Lawrence Earlbaum Associates.

Nahid Golafshani 606

Winter, G. (2000). A comparative discussion of the notion of validity in qualitative and quantitative research. The Qualitative Report, 4(3&4). Retrieved February 25, 1998, from http://www.nova.edu/ssss/QR/QR4-3/winter.html

Author’s Note

Nahid Golafshani is a doctoral student in the department of Curriculum, Teaching and

Learning at the Ontario Institute for Studying Education\University of Toronto where she learns and gains experiences by working with the diverse group of students and faculties. Her area of specialization is Mathematics Education. Her researches include multicultural teaching and learning, teachers’ beliefs, and problem-based mathematics learning and learning. Ms. Golafshani may be contacted at Ontario Institute for Studying Education\University of Toronto Curriculum, Teaching and Learning, 252 Bloor Street West, Toronto, Ontario, Canada M5S 1V6; Telephone: 519-747-4559; Fax Telephone: (253) 322-8961; E-mail: [email protected] Copyright 2003: Nahid Golafshani and Nova Southeastern University

Author’s Citation

Golafshani, N. (2003). Understanding reliability and validity in qualitative research. The Qualitative Report, 8(4), 597-606. Retrieved [Insert date], from http://www.nova.edu/ssss/QR/QR8-4/golafshani.pdf