Becoming a Behavioral Science Researcher

학부생 또는 초급 석사생 정도를 대상으로 하는 리서치 관련 (쉬운) 기본서 정도 되는 책이다. 이 책 역시 다른 책을 읽던 중 발견해서 대출한 책인데, 아는 내용도 많았지만 어렴풋 알던 사실을 확실하게 확인할 수 있었고, 몇몇 부분은 모르던 내용도 있어서 알차게 읽은 책이다. 특히, 프리젠테이션 할 경우 마련하면 좋은 핸드아웃 같은 경우는, 아주 참신한 아이디어이고 이제까지 짧은 경험으로는 접하지 못한 사례이어서 나중에 꼭 활용해야겠다는 생각을 갖게 한다.

 

쉽게 씌여져 있어서, 부담없이 읽을 수 있는 점이 가장 좋은 점이고, Recommended Readings나 References 를 통해 또 다른 좋은 책을 알게 된 점 또한 책 읽은 후 거둔 큰 수확이다.

 

 

---

 

  

Becoming a Behavioral Science Researcher: a guide to producing  research that matters

by Rex B. Kline

 

 Chapter 2. The Good, the Bad, and the (Really) Ugly of Behavioral Science Research

 

Wide Gap between Research and Policy or Practice

. . . Miller (1999) described the “black hole” of educational research into which vast resources invested in empirical studies disappear with little impact on efforts to reform schools. . . . Even when numerous empirical studies in some area exist, the results are often contradictory or based on research methods that are so weak that no clear conclusion can be drawn. (24)

Lack of Relevance for Practitioners

. . . researchers communicate poorly with practitioners. This can happen when researcher report their findings using language that is pedantic or unnecessarily technical. (25) . . . research topics are sometimes too narrow or specific to be of much practical value. (26)

Lack of Cumulative Knowledge

The quote from Kmetz (2002) presented next poignantly expresses a similar idea – note that the term “s³m” below refers to the “soft social science model”:

            After publishing nearly 50 year’s worth of work in s³m, the three terms most commonly seen in the literature are “tentative,” “preliminary,” and “suggest.” As a default, “more research is needed.” After all these years and studies, we see nothing of “closure,” “preponderance of evidence,” “replicable,” or “definitive.” (p. 62)

The lack of cumulative knowledge also means that it is difficult to point to clear examples of progress in many areas of behavioral science research. (29)

Paucity of Replication

. . . replication is grossly undervalued in the behavioral sciences. This is a devastating flaw of our research literature. . . . one must sift through a lot of uninteresting rubble before finding that rare and precious nugget that rewards the search. (30)

 

Why?

Soft science is hard

. . . true experimental designs are simply impossible to use in many types of studies with human research participants. (30)

1.      Human behavior may be much more subject to idiographic factors than to nomothetic factors than physical phenomena. The latter refers to general laws or principles that apply to every case and work the same way over time. In contrast idiographic factors are specific to individual cases. . . . If human behavior is more controlled by idiographic factors (e.g., experiences and environments) than nomothetic factors (e.g., genetics and common neural organization), then there is less potential for prediction. (31)

2.      Context effects tend to be relatively strong for many aspects of human behavior. That is, how a behavior is expressed often depends on the particular familial or social context. . . . strong context effects tend to reduce the chance that a result will replicate across different situations, samples, or times. (31)

4.  The “soft” behavioral sciences lack a true paradigm, and a paradigm in science is necessary for theoretical cumulativeness. . . . there is little agreement in the “soft” behavioral sciences about just what the main problems are and exactly how to study them. This disagreement reflects the preparadigmatic (i.e., prescientific) state of much of the behavioral sciences. (32)

 

Overreliance on Statistical Tests

Specifically, increasing numbers of authors in many different fields now argue that we (1) rely too much on statistical significance tests and (2) typically misinterpret outcomes of statistical tests. (32)

 

Chapter 3. The Research Trinity

This adage attributed to the physician and professor Martin Henry Fischer (1879-1962) is an apt starting point: Knowledge is a process of piling up facts; Wisdom lies in their simplification. (39)

 

Design

Trochim and Land (1982) noted that a best possible design is:

            1. Theory-grounded because theoretical expectations are directly represented in the design.

            2. Situational in that the design reflects the specific setting of the investigation.

            3. Feasible in that the sequence and timing of events, such as measurement, is carefully planned.

            4. Redundant because the design allows for flexibility to deal with unanticipated problems without invalidating the entire study (e.g., loss of one outcome measure is tolerable).

            5. Efficient in that the overall design is as simple as possible, given the goals of the study. (43)

Design must also generally guarantee the independence of observations, which means that the score of one case does not influence the score of another. . . . the requirement for independence is generally met through design and measurement, not analysis. (44)

Three general conditions must be met before one can reasonably infer a cause-effect relation:

            1. Temporal precedence: The presumed cause must occur before the presumed effect.

            2. Association: There is observed covariation, that is, variation in the presumed cause must be related to that in the presumed effect.

            3. Isolation: There are no other plausible alternative explanations (i.e., extraneous variables) of the covariation between the presumed cause and the presumed effect. (45)

 

Measurement

. . . (1) the identification and definition of variables of interest. In human studies, these variables often correspond to hypothetical constructs that are not directly observable. . . . (2) an operational definition, which specifies a set of methods or operations that permit the quantification of the construct. . . . (3) scores (i.e., the data), which are the input for the analysis. (46)

 

Analysis

. . . the researcher must consider the degree of support for the hypotheses, explain any unexpected findings, relate the results to those of previous studies, and reflect on implications of the results for future work in the area. These are all matters of human judgment, in this case based on the researcher’s substantive expertise about the research problem. (47)

If an intervention or program has been poorly carried out (e.g., due to lack of protocol standardization), then its observed covariance with outcome may be inaccurate. In this view, conclusion validity does not fall exclusively within the realm of statistical analysis; instead, it is also part of the correct implementation of the design, here the independent variable. (48)

 

Internal Validity

The critical issue about internal validity is the requirement that there should be no other plausible explanation of the results other than the presumed causes measured in your study. . . . there are six basic ways to do so.

            1. Direct manipulation.

            2. Random assignment (randomization).

            3. Elimination or inclusion of extraneous variables.

            4. Statistical control (covariate analysis).

            5. Through rational argument.

            6. Analyze reliable scores. (55)

In nonexperimental studies where causal hypotheses are evaluated, threats to internal validity are dealt with mainly through the direct measurement of alternative explanations and the use of statistical control in the analysis. However, this process relies heavily on the researcher to specify these alternative explanations in the first place and then to measure them with precision. Both of these steps require strong knowledge about which alternative explanations are plausible. (57)

 

Table 3.1. Descriptions of Major Threats to Internal Validity (58)

Threat	Description
	General
Ambiguous temporal precedence	Lack of understanding about which of two variables occurred first (i.e., which is cause and which is effect?)
History	Specific events that take place concurrently with treatment
Maturation	Naturally occurring changes are confounded with treatment
Testing	Exposure to a test affects later scores on outcome variable
Instrumentation	The nature of measurement changes over time or conditions
Attrition	Loss of cases over conditions, groups, or time
Regression	When cases selected for extreme scores obtain less extreme scores on the outcome variable
	Multiple-group studies
Selection	Groups differ before treatment is given
Treatment diffusion or imitation	Control cases learn about treatment or try to imitate experiences of treated cases
Compensatory rivalry	Control cases learn about treatment and become competitive with treated cases
Compensatory equalization of treatment	Cases in one condition demand to be assigned to the other condition or be compensated
Resentful demoralization	Control cases learn about treatment and become resentful or withdraw from study
Novelty and disruption effects	Cases respond extraordinarily well to a novel treatment or very poorly to one that interrupts their routines

 

 Construct Validity (63)

Threat	Description
Unreliable scores	Scores are not precise nor consistent
Poor construct definition	Construct may be mislabeled or defined at the wrong level (e.g., too general or too specific)
Construct confounding	Definition or study operational definitions confounded with other construct
Monomethod bias	Measurement of different outcome variables all rely on the same method (e.g., self-report) or informant (e.g., parents)
Mono-operation bias	Refers to the use of a single operationalization of the independent or dependent variables
Evaluation apprehension	Anxiety about measurement and evaluation adversely affects performance
Reactive self-report changes	Motivation to be in a treatment condition affects responses, and this motivation can change over time
Researcher expectancies	The researcher conveys (consciously or otherwise) expectations about desirable responses

 

 

Summary

Without a clear and meaningful question, all that follows may be for naught, so first think long and hard about the rationale of your hypotheses before considering specifics of design, measurement, or analysis. . . . In planning the number of groups or conditions, sample size, the schedule for interventions or measurement, and other design details, you must balance what is ideal against what is possible, given limitations on resources but still respecting the hypotheses. . . . Because most samples studies in the behavioral sciences are nonprobability samples, concern about generalizability is usually warranted. Consequently, always describe in detail the characteristics of your sample, and be cautious about the potential generalizability of your findings without evidence from replication studies. (70)

 

 Chapter 4. Design and Analysis

From Question to Design

            1. Descriptive: . . . the simple description of a sample of cases (people or animals) on a set of variables of interest. . . . it is relatively rare when research questions are solely descriptive.

            2. Relational: This most common kind of question concerns the covariance between variables of interest. . . . it is rare researchers have absolutely no idea in advance about whether two variables related. Instead, a relational question is more typically one about the direction and degree of covariance.

            3. Causal: A causal question concerns how one or more independent variables affects one or more dependent (outcome) variables. (74-75)

 Table 4.1. Major Types of Experimental Designs (77)

Type	Representation
Basic	R		X	O
	R			O
Factorial	R		XA1B1	O
	R		XA1B2	O
	R		XA2B1	O
	R		XA2B2	O
Pretest-posttest	R	O1	X	O2
	R	O1		O2
Solomon Four Group	R	O1	X	O2
	R	O1		O2
	R		X	O2
	R			O2
Switching replications	R	O1	X	O2		O3
	R	O1		O2	X	O3
Crossover	R	O1	XA	O2	XB	O3
	R	O1	XB	O2	XA	O3
Longitudinal	R	O . . . O	X	O	O . . . O
	R	O . . . O		O	O . . . O

Note. R, random assignment; O, observation; X, treatment

 

Randomized Longitudinal Design

Several different kinds of statistical techniques are applied to longitudinal data, especially in studies with at least three measurement occasions. Some advanced techniques estimate a latent growth model, which captures the initial level and trajectory of behavior change, variation in both initial level and trend, and the covariation between initial level and subsequent change. The latter concerns whether cases that start out at a higher (or lower) level at the initial assessment show a higher (or lower) rate of change across subsequent occasions. Two major techniques used to estimate such models are hierarchical linear modeling (HLM) and structural equation modeling (SEM). The former is basically a variation of MR that is especially well suited for analyzing hierarchical (nested) data structures where data points at a lower level are clustered into larger units, such as siblings within families. Repeated-measures datasets are also hierarchical in that multiple scores are clustered under each case, and these scores are probably not independent. The term “SEM” actually refers to a family of techniques for modeling presumed covariance structures and mean structures, which concern, respectively, patterns of presumed relations between variables (observed or latent) and their means. (91)

Indirect effects concern a mediator effect in which on variable is specified to affect another only through a third variable. . . . In contrast, a moderator effect refers to an interaction effect in which the association of one variable with another changes across the levels of a third variable, and vice versa. . . . Moderator (interaction) effects can be estimated in ANOVA and standard MR with no special problem, but no so for mediator effects. (92)

 

Chapter 9. Writing

Simple, Clear, Active

Good writing is efficient and concise. This means that, in general, a shorter text is better than a longer one, if the two convey the same basic information. Use words that are familiar and organized in sentences that are short and direct instead of long and complicated. . . . Also avoid inflated diction, which results from choosing pretentious words instead of simpler ones, such as “utilize,” “facilitate,” or “subsequent to” instead of , respectively, “use,” “help,” or “after.” (258)

Good writers also use words correctly. . . . Finally, avoid anthropomorphism, the attribution of uniquely human characteristics to abstract entities or inanimate objects. . . . the phrase “This research found . . . “ is an anthropomorphism because “research” is an abstract entity that cannot “find” anything-only people (i.e., researchers) can do so. For the same reason, the phrase “The theory says . . . “ is also an anthropomorphism. It is better to say something like “Smith (2007) claimed . . . “ or “An implication of the theory is . . . ,” neither of which are anthropomorphisms. (259)

Read and Listen

. . . the writing quality of articles in the best journals tends to be good, and a well-written article in the same area as your own research can be a helpful example. It also helps to read aloud a draft of your own writing, or listen as someone else reads it to you.  (261)

 

Principles of Good Scientific Writing

The single most important requirement for good scientific writing is good content. This means that the (1) hypotheses tested are meaningful and based on solid ideas, and (2) the design, methods, and analyses are all technically correct and appropriate for the problem.

            1. Keep it brief and concise.

            2. The title must be descriptive, and the abstract should convey all cardinal findings.

            3. Avoid excessive literature review and excessive use of tables and figures.

            4. Make clear the weaknesses of the study, and avoid superlatives in describing your own work. (261-262)

You should write for a general reader, that is, your manuscript should be understandable by someone who is intelligent but not really familiar with your specific research area. (262)

 

Writing Sections of Empirical Studies

In rank order, the six most important things in writing manuscripts are (1) title, (2) title, (3) title, (4) abstract, (5) first paragraph, (6) last paragraph; begin strong, end strong!

Title and Abstract

The abstract should clearly and briefly convey to a general reader both the rationale and the results of your study. An abstract that mentions the results only in passing at the end with the empty phrase, “The findings and implications of this study are discussed,” and gives no further detail, is incomplete. (263)

Introduction (Literature Review)

. . . just the story of the rationale of your particular study should be told in the introduction. . . . (1) a statement of the basic research problem, (2) explanation of why the problem is important, and (3) an outline of what solution or step toward a solution is proposed. (265)

Discussion

Remind readers in the very first paragraph about the basic purpose(s) of your study. Some authors next begin to summarize the main findings, but I prefer next to outline possible limitations to the generalizability, credibility, or robustness of the results. (271) “Before specific results are discussed, some possible limitations to their generalizability are considered.” (272)

Connecting the results of the present study to those reported by others also builds a context to help readers better understand the research problem. . . . However, do not in the discussion mention any result or pattern from your study that was not already described in the results section. That is, the discussion is no place for surprises concerning your findings. (272)

Please do not include the discussion section with a variation on the typically banal, hackneyed call for future research on the topic. . . . Sometimes more research is not needed, especially when it is quite clear after many studies that some result is nil (i.e., it is time to move on). Otherwise, it is more helpful to offer specific suggestions about how to improve research in the area, or to articulate new hypotheses (questions for future studies).

 

Chapter 10. Presentations

IF / BP / 10, 20, 30, 3 X 5 / NT / DDS / NC

Ideas First

. . . write three sentences that summarize the essential take-home message of your presentation. These are the critical points that, even if an audience member forgets everything else, communicate something of substance if he or she remembers them. . . . These three sentences should say something about the core research problem, the methods used, and the results. (290)

Be Prepared

. . . to prepare a printed handout that accompanies the presentation. (292) . . . A handout for a scientific presentation should contain text, a few tables or figures (as needed), and a reference list. . . . The handout can also be used to present more detail than could be shown in a PowerPoint slide. . . . The on-screen portion could concern some basic pattern of result, but the rest of the details from the table are given in the handout. . . . However, the handout should not be printed in PowerPoint or similar computer tools. You should view the handout as a distinct product that is to be produced in a word processor, not a presentation computer tool. (294)

 

 

 

 

 

 10, 20, 30, 3X5

. . . a PowerPoint presentation should have 10 slides, last no more than 20 minutes, and have a minimum font size in text slides of at least 30 points. . . . once the number of slides reaches 20 or so, the speaker can wind up interacting more with the computer than the audience, due to the need to advance the slides so often. . . . The 3X5 part of the rule is for text slides, and it says that there should be no more than three bullet points per slide each expressed in about five words. . . . If more text details are needed, then put them in the handout, not in your text slides. (295)  

 

 

Becoming a Behavioral Science Researcher.pdf

 

Becoming a Behavioral Science Researcher.pdf

0.19MB