IR 515

IR 515, Fall 1998	Professor John Odell
Office VKC 300B	odell@usc.edu, 740-4298
Tues & Wed 3 to 5 pm	Web page www-rcf.usc.edu/~odell

QUALITATIVE RESEARCH DESIGN

Choosing among research methods depends on which types of risk the scholar is most comfortable taking, among other things. Quantitative and qualitative methods have different strengths, and every research method ever invented is poor--as a means to at least one legitimate objective. Operating under this assumption, this seminar offers to help advanced social science students get the greatest possible value from qualitative empirical research designs, whether or not these are combined with mathematical or statistical techniques. Illustrative publications tend to come primarily from political science and political economy especially at the international level, and to lesser degrees from education, history, and sociology.

The term "qualitative" is used by diverse scholars who disagree on epistemological basics. Assuming some prior exposure to these debates, this seminar works within what could be called broad or pragmatic positivism, the mainstream in the USA. This inclusive position will be discussed briefly at the outset. This year this seminar does not attempt to teach how to use methods that regard themselves as alternatives to this tradition. IR 513 has introduced historical and humanistic methods and discussed prospects for synthesizing methods. Appendix 3 lists additional references from interpretivist traditions.

Prerequisite: IR 513 or another basic course in epistemology and research design, including some exposure to statistical methods. This course is most appropriate after completing other graduate courses where you will have read scholarship on your particular subject and begun to identify your research topics.

Requirements: Throughout the semester the student works on two levels--studying general works introducing methods ideas, and using them to develop an original proposal for a specific project. Exercises will provide occasions for experimenting with the techniques on your particular subject. If you have not already selected a topic for dissertation research, you may select any subject temporarily for the purpose of learning the techniques.

At our meetings members are expected to be prepared to answer questions about the required readings or exercises for that week, or to discuss results of your work or give suggestions to a colleague. Two books (King et al. and Maxwell) have been ordered at the University Bookstore and placed on reserve. Other titles marked [R] are also on reserve in Leavey Library. Originals for other required readings will be available for copying in VKC 300. I will loan optional titles when I have them. In VKC 42D you may read a file of successful past proposals, including USC winners of national fellowship competitions. Please get an email account and send your address and phone numbers to seetal@usc.edu.

Grades: Ninety percent will correspond to the quality of the student’s final research proposal and ten percent will reflect earlier short papers.

To refresh your memory of elementary research design, you could peruse a textbook such as E. Babbie, 1992, The Practice of Social Research, 6th ed., which is on reserve in Leavey Library, or another comparable text.

1. 9/3 Objectives, assumptions, and tips for success.

Supplement:

Diesing, P. 1991. How does social science work: reflections on practice. Rather than prescribing, the author describes several different epistemologies in use in contemporary social science as well as logical positivism, which has virtually disappeared from actual practice.

I. PURPOSES, RESEARCH QUESTIONS AND RESEARCH PROGRAMS

2. 9/10

Your purposes and your research program

1. Study:

King et al. 1994. Designing Social Inquiry (KKV), pp. 3-33
J. Maxwell. 1996. Qualitative research design, pp. ix-62
Imre Lakatos, "Falsification and the Methodology of Scientific Research Programmes," in Lakatos, ed., Criticism and the Growth of Knowledge (Cambridge University Press, 1970), pp. 91-138 and 173-180. [R]

2. Write Maxwell’s exercise 2.1, reflecting on your personal purposes in doing this study (assuming it is one you really want to do). Your memo can be in outline form or text. It can be as brief or as long as you find useful, but at least make it honest and omit no significant goal. Refer to Maxwell, pp. 17-24 on the particular strengths of qualitative methods. Do your purposes match any of his list? (Naturally we are not obliged to swallow Maxwell whole; your teacher will in fact pick a few significant bones with him.) Turn in a copy at class. This document will not be graded or shown to anyone else without your permission.

3. Write Maxwell’s exercise 3.2, creating a first primitive concept map for your prospective study. Pull this first version out of your own mind--as it is right now--rather than by turning to additional published studies or evidence. Turn it in at class. It will not be graded or circulated without your permission.

4. Select a particular research program within your discipline for your personal evaluation and proposal-writing. Come prepared to give a preliminary oral description of the research program you have selected for the semester. That is, tell us the main questions or puzzles and the main ideas or approach for answering them that unify this body of scholarship, and name some of the most influential studies. (Later, on 10/1 you will need to write a critical evaluation of a research program.) I would be happy to help evaluate your options in the office.

5. Select the three or more specialized articles or books that come the closest to answering your top research question as it stands today, as far as you know. For each study, draw a concept map representing the main concepts and main relations among them in the author’s arguments. A simpler diagram is better, for this purpose. Indicate the direction in which causality or influence is thought to run between concepts, when the author makes this clear. Record your evaluations of the study in some way. Turn in copies of two or more of your diagrams at class.

3. 9/17

Focusing our research questions

1. Continue studying specialized literature in your research program, recording concept maps and evaluative notes on each important work, preparing for the short paper due on 10/1. No need to turn in these maps or notes.

2. Complete Maxwell’s exercise 4.1, developing your research questions (pp. 49-62). Follow his six steps as faithfully as possible, and bring to class enough copies of your research questions alone for each member to have one. Bear in mind the wise advice (p. 60) that "a good set of research questions will evolve, over time, after you have considered and reconsidered your broad research theme. . . Be wary of the desire to push forward before going through this process." In this spirit be prepared to pose orally at least one plea for assistance arising from this assignment, challenging our group to suggest next steps. This might be a report of a dilemma between conceptual options, a confusion, or another uncertainty you feel at this particular moment.

3. Experiment with mental tricks from the 1985 Odell memo on research questions, and the list of ways to make a contribution in KKV, pp. 16-17, generating at least two additional candidate research questions starting directly from published literature on your subject. Generate at least one question by first picking a weakness in scholarship on your subject. Generate at least one by borrowing and proposing to apply a causal idea or hypothesis from some field of scholarship other than your primary one. While a more plausible idea is better than a wildly irrelevant one, again let creativity fly without having to defend the idea in depth, for now. Turn in a one-page outline listing tactics you used and whatever questions you generated that way, at class time. A simple outline that is intelligible on its face will be sufficient.

4. 9/24

Private tutorials:

5. 10/1

Critical essay (graded) on your research program due

A. What are the program’s primary puzzles or questions today? B. What are its primary conceptual and methodological approaches to solving the puzzles? C. How have the puzzles and solutions evolved over the decades? Have they changed because earlier puzzles have been solved or for other reasons? D. Today what are this program’s main accomplishments and its most significant failures--confusions, empirical anomalies, gaps? Maximum 3000 words.

II. METHODS FOR STRENGTHENING CAUSAL INFERENCES

AND MAKING OTHER THEORETICAL CONTRIBUTIONS

6. 10/8 Causal inference and comparative methods

1. Study:

Maxwell, J. 1996. Qualitative Research Design. Methods: what will you actually do? and validity: how might you be wrong? pp. 63-98
KKV, Descriptive inference, causality and causal inference, pp. 75-114
Mill, J.S. 1888 [1970]. Two Methods of Comparison. In A. Etzioni and F. Dubow, eds., Comparative Perspectives: Theories and Methods, 205-213. (Taken from Mill, A System of Logic, book 3, chap. VIII, "of the four methods of experimental inquiry.")
Lijphart, A. 1971. Comparative politics and the comparative method. APSR 65: 689-693.
George, A. and T. McKeown. 1985. Case studies and theories of organizational decision-making. Advances in Information Processing in Organizations 2: 21-58.

2. Exercise: Devise a simple original illustration of each of the following techniques, drawing on your project or any subject. Describe each illustrative research project in a paragraph or two and turn in the memo at class time (to be graded).

Mill’s method of agreement;
Mill’s method of difference;
how in my own project I might conceivably address the "many-variables-small n" problem by conceptually collapsing two variables or variable classes into one.

3. Select the three articles, one of the following books, or another you propose to me, and prepare to summarize your assessment orally, briefly answering these five questions: what is the study’s main thesis? how would you describe its research design (goals and methods)? does the method it illustrates have any appeal for future research? what concerns does this method raise for you? what strengths or shortcomings are due to this particular application of the method, rather than the method itself?

Lieber, R. 1970. British politics and European unity: parties, elites, and pressure groups.

George, A. and R. Smoke. 1974. Deterrence in American Foreign Policy: Theory and Practice.
Skocpol, T. 1979. States and social revolutions: a comparative analysis of France, Russia, and China.

Bates, Robert. 1981. Markets and States in Tropical Africa: The Political Basis of Agrarian Policies.

Janis, Irving L. 1982. Groupthink : psychological studies of policy decisions, 2nd ed.
Milner, Helen. 1988. Resisting Protectionism: global industries and the politics of international trade.

Goodman, J. 1992. Monetary Sovereignty: the politics of central banking in Western Europe.

Three articles: R. Mitchell. Summer 1994. Regime design matters: intentional oil pollution and treaty compliance," IO 48:425-458. G. M. Luebbert. July 1987. Social foundations of political order in interwar Europe. World Politics 39: 449-478. L. J. Schoppa. Summer 1994. Two-level games and bargaining outcomes: why gaitsu succeeds in Japan in some cases but not others. IO 47: 353-386.

4. Write yourself a one-page private memo where you choose and state your own conception of causality and your position as to what ideal your research aspires to, in this respect.

Supplements:

Weber, M. Objective possibility and adequate causation in historical explanation. In Weber, Methodology of the social sciences.

Moore, Jr., Barrington. 1966. Social origins of dictatorship and democracy: lord and peasant in the making of the modern world.

Przeworski, A. and H. Teune. 1970. The logic of comparative social inquiry.

Smelser, N. 1976. Comparative methods in the social sciences.

George, A. 1979. Case studies and theory development: the method of structured, focused comparison.

Katzenstein, P. 1985. Small states in world markets: industrial policy in Europe.

McCloskey, D. N. 1990. If You’re so smart: the narrative of economic expertise.

Collier, D. 1991. The Comparative method: two decades of change. In Comparative Political Dynamics: Global Research Perspectives, ed. D. A. Rustow and K. P. Erickson.

7. 10/15 Single-case designs

1. Study:

Review A. Lijphart. 1971.
Review George and McKeown. 1985.
Geertz, C. 1973. The Interpretation of Cultures, Thick description: toward an interpretive theory of culture, pp. 3-30.
Eckstein, H. 1975. Case study and theory in political science. In Handbook of Political Science, vol. 7, ed. Fred Greenstein and Nelson Polsby, 94-137. [R]

2. Choose one of the types of single-case study identified by Lijphart, and consider how you might use that research design as the primary method for your proposed project. Turn in a one to two page memo (to be graded) identifying which type or purpose your study would serve and selecting a case for that purpose. Assume (contrary to Lijphart) that an "interpretive" case study could make a valuable contribution by showing that a known hypothesis or theory helps to explain or interpret some new event, even if it does not generate a new hypothesis. (Is Geertz relevant here?) Also note from Eckstein that a case will be especially powerful for confirming a theory if it is a "least likely" case, and vice versa for infirming a theory. You may, if you prefer, substitute George/McKeown for Lijphart.

3. Select one of the following books or three articles, and prepare to assess it orally, answering the following questions: what is the study’s main thesis? which type of single-case study (as defined above under 1) is this? what are the strengths of this type of study? the type’s weaknesses? what strengths and shortcomings are due to this particular application of the method?

Type A: Morse, E. L. 1970. Foreign Policy and Interdependence in Gaullist France.

Type B

Lijphart, A. 1975. The politics of accommodation: pluralism and democracy in the Netherlands.
LeoGrande, W. July1979. Cuban dependency. CubanStudies.
Haggard, S. 1988. The institutional foundations of hegemony: explaining the Reciprocal Trade Agreements act of 1934. International Organization 42: 91-120.

Type C : Russett, B. 1967. Pearl harbor: deterrence theory and decision theory. Journal of Peace Research 2: 81-106. Rpt. in Theory and research on the causes of war, ed. D. Pruitt and R. Snyder, pp. 127-135.

Type D

Winham, G. October 1977. Negotiation as a management process. World Politics.
Deudney, D. Spring 1995. The Philadelphian system. International Organization 49: 191-228.

Type E

George, A. and J. George. 1956. Woodrow Wilson and Colonel House: a personality study.
Khong, Y. F. Analogies at war: Korea, Munich, Dien Bien :Phu, and the Vietnam decisions of 1965.

8. 10/22

Selecting cases, other headaches, and remedies

(how much prestructuring, selection and other biases, too few observations, reciprocal causation, other threats to validity)

1. Study:

Review Maxwell, pp. 63-73
KKV 1994, pp. 115-230

2. Exercise: In a brief memo (graded) due at class time, outline one possible example from your subject of each of the following five practical remedies suggested by KKV:

selecting observations to insure variation in the explanatory variable (p. 140),
selecting a range of values of the dependent variable (141),
selecting observations so the dependent variable is constant (147),
selecting observations to control for omitted variables (179-182), and
one remedy to avoid endogeneity (185-196).

3. Study C Lave and J. March. An Introduction to Models in the Social Sciences, an introduction to speculation, pp. 9-49 [ok to skip through 25-34]. They illustrate which one of KKV’s suggested remedies for the "small n" problem? I suggest you take seriously the notes saying "stop and think." Exercise: Problem 12 p. 49 in Lave and March. Turn in your brief memo at class time.

4. Study J. Fearon. January 1991. Counterfactuals and hypothesis testing in political science. World Politics 43: 169-95. Exercise: Prepare to present orally to the class one illustration of the use of counterfactuals that might strengthen a causal claim about your subject.

Supplements:

Campbell, Donald. 1975. "‘Degrees of Freedom’ and the Case Study." Comparative Political Studies 8: 168-93. Rpt. in D. Campbell 1988. Methodology and Epistemology for Social Science.

Geddes, B. 1990. How the cases you choose affect the answers you get: selection bias in comparative politics. Political Analysis, vol. 2, J. A. Stimson, ed.

The qualitative-quantitative disputation: review symposium on KKV. June 1995. American Political Science Review 89: 454-478.

Tetlock, P. and A. Belkin, eds. 1996. Counterfactual Thought Experiments in World Politics.

9. 10/29 Combining quantitative and qualitative methods

1. Study S. Tarrow. June 1995. Bridging the quantitative-qualitative divide in political science. APSR. 89: 471-474.

2. Select one of the following items and write a one-page memo for distribution to the seminar. What is the book’s main thesis? the author’s reasons for using quantitative methods (the purposes they serve)? the author’s reasons for using qualitative methods or contributions they make? your personal assessment of the result and the utility of this particular model of combining methods, for future research?

Bunce, V. 1981. Do new leaders make a difference? exclusive succession and public policy under capitalism and socialism.

Janis, Irving L. 1982. Groupthink : psychological studies of policy decisions. 2nd ed., plus G. M. Herek, I. L. Janis, and P. Huth. 1987. Decision-making during international crises: is quality of process related to outcome? Journal of Conflict Resolution 31: 203-226. Also recommended: M. Schafer and S. Crichlow. September 1996. Antecedents of groupthink: a quantitative study. JCR 40: 415-435.
Laba, R. 1991. The roots of solidarity: a political sociology of Poland working-class democratization.

Martin, L. 1992. Coercive cooperation: explaining multilateral economic sanctions.

Putnam. R. D. 1993. Making democracy work: civic traditions in modern Italy.

Simmons, B. 1994. Who Adjusts? Domestic Sources of Foreign Economic Policy during the Interwar Years.

3. With your research question and research program in mind, draft a first one-page outline of the empirical methods section of your project proposal. Select at least one research design from the above menu, for attacking your main research question. You may adjust or change your mind later, but commit a preliminary decision to paper. Turn it in at class.

Supplements:

McAdam, D. 1981. Freedom summer.

Tilly, C. 1993. European revolutions, 1492-1992.

Cook, T. D. and C. S. Reichardt, eds. 1979. Qualitative and quantitative methods in evaluation research.

III. TECHNIQUES OF OBSERVATION AND

STRONGER DESCRIPTIVE INFERENCES

This part provides brief initial exposure and further references to selected techniques of observation and classification that can be used after proposal writing. A proposal need not discuss these issues in great detail. But if you have not grasped these basic issues, your text will probably show it. This reading and experimentation should help you see more clearly how you could implement one of the designs on the menu.

10. 11/5

Turning information into data:

making descriptive claims less arbitrary and enhancing cumulation

1. Study:

Babbie, E. 1992. The practice of social research, 6th ed., pp. 126-146 and 165-189. A review of elementary measurement and scale construction. [R]
KKV, pp. 34-71 & 150-168

2. After reading the first two general works, choose three of the following exemplars and look in particular to discover how the authors made their descriptive statements more precise, more consistent, and more reliable by specifying operational definitions and following them faithfully. The cited pages identify the most concrete rules for sorting evidence. While many of these studies also assign numbers to various categories, all illustrate techniques whose logic can be used without this quantification as well.

Cox, R. W. and H. K. Jacobson. 1974. The anatomy of influence: decision-making in international organization. Pp. 437-465 [indicators for state power, state power in three special issue-areas, and classifications of countries according to type of political regime and level of economic development]
Hopmann, P. T. Summer 1974. Bargaining in Arms control negotiations: the Seabeds denuclearization treaty. International Organization 28: 313-344 [bargaining strategies, styles, and outcomes]
Butterworth, R. L. 1976. Managing interstate conflict, 1945-74: data with synopses, pp. 481-98 [an example of a coding manual for creating both quantitative and qualitative data for questions such as "which type of powers exercised leadership in conflict management?" "how wide was agreement in the international organization on the management actions it took?" "what was the strongest action it took?"
Odell, J. 1985. The outcomes of international trade conflicts: the US and South Korea 1960-1981. International Studies Quarterly 29:263-286. [rates outcomes of 13 trade disputes comparably to reflect who gained how much]
Destler, I. M. and J. Odell, 1987. Anti-Protection: Changing Forces in US Trade Politics, pp. 1-7, skim 9-21, read 23-26, 143-157. [comparing the level of political activity against import protection in different episodes]
Hudec, R. E. 1993. Enforcing International Trade Law. The appendix reports reliable, consistent descriptions of all 207 legal complaints brought under the GATT between 1948 and the end of 1989, including indicators of how well the legal system functioned. Included are the data entry form used to create case descriptions (pp. 588-89). Concepts include the procedural outcome (legal ruling by a panel v. settled or conceded by the defendant v. withdrawn or abandoned), substantive outcomes of rulings (defendant violated v. did not violate the rules), and how countries responded to adverse rulings against them (gave full satisfaction v. partial satisfaction v. negative outcome). Which country most frequently resisted legal rulings, would you guess?

3. Exercise due at class: After studying the foregoing material, write a memo drafting preliminary operational definitions for three key concepts in your study. For this exercise try to express your concepts as matters of degree rather than qualities that can only be present or absent, and rather than nominal categories, unless doing so seems misleading. For example, "democratic" and "authoritarian" might be replaced with "degree of political openness." Your memo should give me, your research assistant, rules answering all possible questions on how to sort the evidence you collect into conceptual "bins." Try to anticipate as many difficult boundary cases as possible, since otherwise I will rely on my intuitions, which might not be consistent over time. To discover my possible questions, experiment with your rules on several concrete examples. The memo should contain:

for each concept, your best nominal definition;
for each, your best operational definition (classification rules). The set of categories ("bins") constituting your indicator may be either nominal, rank-ordered but not quantitative, or quantitative. Include at least one ordinal and one quantitative indicator.
Your reaction to the proposal that you use explicit operational definitions like these to discipline the descriptive statements you will draw from your evidence. (Any reply can be correct.)

4. Now that you have a more concrete sense of recognized methods, write a preliminary draft of the first half of your final proposal--not describing your empirical method, but stating your research question, identifying the larger research program and conceptual context it will address, and saying generally how your project will go beyond available specialized literature.

Supplements:

Webb, Eugene et al. 1966. Unobtrusive measures: Non-reactive research in the social sciences

Lazarsfeld, Paul. 1972. Development of a test for class-consciousness. In Continuities in the Language of Social Research, ed. Lazarsfeld, Pasanella, and Rosenberg, pp. 41-43. [The famous scholar reveals how he created this indicator for an early study of American voters by recalling what he and others believed as young members of the Austrian Social Democratic party]

Gurr, T. R. 1974. Persistence and change in political systems, 1800-1971. Am.Pol.Sci. Review, 68: 1482-1582. [Classifies each into one of six ordered types, based on index combining several regime characteristics]

Snyder, G. and P. Diesing. 1977. Conflict among nations, pp. 340-361 [types of decision-making structure]

Putnam, R. and N. Bayne. 1984, 1987. Hanging together: cooperation and conflict in the seven-power summits, chap 11 [the degree of cooperation achieved in summit meetings].

Underdal, Arild. 1993. Measuring and explaining regime effectiveness. In Complex cooperation, ed. H. Hveem, pp. 92-124.

Henning, C. R. 1994. Currencies and politics in the United States, Germany, and Japan, chap. 3. [classifies the degree of central bank independence from the government in three countries]

George, A., and W. Simons, eds. 1994. The limits of coercive diplomacy, 2d ed., pp. 267-294 for summary, and passim for concepts and definitions [coercive diplomacy, its success and failure, and presence or absence of conditions favorable for success]

11. 11/12 Observing indirectly through interviews

1. Study either A or B depending on your interests:

A. Interviewing ordinary people

Spradley, James. 1979. The ethnographic interview, pp. vii-39
Hochschild, J. L. 1981. What’s fair? American Beliefs about distributive justice, pp. 17-45 and 148-191 [R]
Scott, James. 1985. Weapons of the weak: everyday forms of peasant resistance, pp. 1-27.

B. Interviewing elites

Dexter, L. A. 1970. Elite and specialized interviewing, pp. 3-138 [R]

2. Improve the first draft outline of your empirical methods section, in light of the preliminary first half drafted last week. Is this project feasible in a reasonable time? What can you do to find out?

Supplements:

Aberbach, J. et al. 1975. Exploring elite political attitudes: some methodological lessons. Political Methodology 2: 1-27.

Taylor and Bogdan, Introduction to qualitative methods, pp. 76-105

Seldon, A. and J. Pappworth. 1983. By word of mouth: elite oral history.

Mishler, E. 1986. Research interviewing: context and narrative.

Kvale, Steinar. 1996. InterViews: an introduction to qualitative research interviewing.

12. 11/19

Private tutorials. Please bring at least a sentence outline of a complete research proposal, as well as any questions you would like to raise.

IV. DISCUSSION OF RESEARCH PROPOSALS

Classes 13 & 14

Appendix 1. Other textbooks and practical advice

Becker, H. W. 1986. Writing for social scientists: how to start and finish your thesis, book or article. University of Chicago Press. How bad writing really is not required to make social science good--contrary to the impression you get from too many journals.

Patton, Michael Quinn. 1990. Qualitative Evaluation and Research Methods, 2d. ed. Sage.

Miles, M. B. and A. M. Huberman. 1994. Qualitative Data Analysis, 2d ed. Sage.

Booth, W. C., et al. 1995. The Craft of Research. University of Chicago Press. The writing process--asking questions, making a claim and supporting it, pre-drafting and drafting, revising, and introductions.

Becker, H. 1996. Tricks of the Trade: Thinking about your Research while Doing It

Appendix 2. Additional interpretivist references

Mead, G. H. 1934. Mind, Self, and Society.

Abel, T. 1948. The operation called Verstehen. American Journal of Sociology 54: 211-218.

Blumer, H. 1969. Symbolic interactionism: perspective and method.

Dallmayr, F. R. and T. A. McCarthy, eds. 1977. Understanding and Social Inquiry.

Neufeld, M. 1993. Interpretation and the ‘science’ of international relations. Review of International Studies 19: 39-61.

Denzin, N. K. and Y. S. Lincoln, eds. 1994. Handbook of Qualitative Research

Puchala, D. J. 1995. The pragmatics of international history. Mershon International Studies Review 39: 1-18.

Alker, H., Jr. 1996. Rediscoveries and reformulations: humanistic methodologies for international studies.