Skip over navigation

Journal Issue: America's High Schools Volume 19 Number 1 Spring 2009

U.S. High School Curriculum: Three Phases of Contemporary Research and Reform
Authors: Valerie E. Lee Douglas D. Ready

The Problem of Selectivity Bias in Research about Curriculum

Although each of these hypothetical issues deserves more discussion, space limitations allow us to expand only on the first. The body of research on curriculum effects from Phase II is extensive, and the evidence from these studies is generally quite positive: in schools where students typically complete rigorous course sequences in high schools, students learn more, and the learning is more equitably distributed. However, the conclusions drawn by much of the Phase II research—that schools should extend the college-preparatory curriculum to all students—may have been premature. The studies are likely plagued by selectivity bias, which, as noted earlier, potentially operates at two distinct levels: student allocation to coursework is a phenomenon that occurs both within and between schools.

The Phase II studies cited above were generally conducted using nationally representative samples of high schools, the majority of which were comprehensive public high schools that offered students choice within broad curricula. Moreover, the fact that these studies used multi-level analytic methods means that students were compared with other students in the same high schools (that is, schools with the same sets of courses available). The concern with selectivity bias here is that students who select rigorous courses (or have them selected for them) are also more likely to come from socioeconomically advantaged families, to be more motivated than other students in their school, and to possess unmeasured personal characteristics beyond academic ability that allowed them to do well enough in previous courses to move on to advanced coursework. Statistical controls for students' academic and social backgrounds—the typical methods of addressing student selection to courses—would not capture unmeasured characteristics, such as student motivation, personality traits, or access to social and academic supports. However, these characteristics affect student outcomes independent of course selection. For example, important differences likely distinguish a low-income, average-achieving student who completes trigonometry from a low-income, average-achieving student who stops at geometry. These unmeasured differences are probably also related to how much mathematics each student learns during high school. In addition, more effective teachers often teach the advanced courses within schools, which could influence student outcomes above and beyond the effects of curricular content. When schools are mandated to teach Algebra I or English to all ninth graders, the people who teach these courses will likely not change, even if the course offerings are revised.

Selection bias in student course-taking likely exists not only within schools but also between schools. The background of students in different types of public high schools and of students in public and private schools varies widely. Selection bias at the school level may thus occur because high schools offer a broad array of courses in response to the perceived needs and desires of their students and families. Schools serving large proportions of students with disadvantaged backgrounds often do not offer a full array of challenging and advanced courses, either because of a lack of demand from parents or perceptions that their students cannot handle rigorous material. Schools that do provide access to advanced courses may be college-oriented in other ways beside the curriculum. They may enroll larger proportions of motivated students, have greater "academic press," and have teachers and staff who know how to prepare students for college. The Phase II studies that compared the link between course taking and achievement in Catholic and public schools must therefore contend with considerable self-selection in who chooses to attend such schools.58 Studies focused on public high schools must also consider self-selection, as schooling is typically tied to residential location. Thus, in addition to within-school selection bias, between-school biases may exist, in that students who typically complete demanding course sequences are those who also attend schools where such courses are available and have families that have selected particular schools for them to attend. In this regard, it would appear that almost all research on curriculum effects on student outcomes is—almost by definition—plagued by some form of selection bias.

Differences between Chicago schools and those in much of the United States raise still more selection bias issues. In Chicago, many students enter high school with low achievement. Although some students in Chicago are similar to the students in the Phase II studies, in that they choose college-preparatory coursework, many are quite different from that group of students. Most important, the curriculum structures of schools in the Phase II studies also differ, in that students in those schools were allowed to choose from a wide array of courses, some of which were rigorous, some of which were not, and some of which were remedial in nature. In Chicago, such choices were eliminated during the early high school grades. In theory at least, all courses are rigorous, none are remedial, and students have no choice (at least not in English and mathematics at grade nine). At this level, there is no selectivity bias in the Chicago study, precisely because all students— regardless of their social or academic background— are enrolled in college-preparatory coursework. This fact may well explain the conflicting Chicago and Phase II results. Both the Phase II and Phase III studies take background characteristics into account, but the organizational context (that is, the structure of the curriculum) in the Phase II and Phase III research is quite different.

The pervasiveness of these concerns about selection bias suggests that the findings of the Phase II studies may not be generalizable to schools that enroll high proportions of low-performing students. Students who would be affected by ending remedial coursework would predominantly be low-achieving students who would have otherwise been counseled into low-level courses, if such options were available (as they were in Chicago before 1997). In the Phase II studies, those low-ability students who took college-prep coursework would have been a very select group and may not be representative of most low-ability students (including those in Chicago). It is not clear from the Phase II research whether curricular effects differ for low-ability and high-ability students, who should be better able to handle higher-level content.

Whatever the explanation for the Chicago evaluation findings, they call into question the conclusion of the Phase II research that the constrained academic curriculum reduces social stratification in educational outcomes and raises achievement across the board.59 It is possible that the findings from the Phase II studies simply cannot be generalized to the Chicago context because of selection bias. It is also possible that the "College Prep for All" curriculum cannot be successfully implemented without attending to many other issues plaguing high schools, such as unmotivated and unprepared students, lackluster instruction, or teachers unprepared to instruct heterogeneous classes. In any case, the Chicago evaluation requires policy makers, practitioners, and researchers concerned about issues of excellence and equity in secondary education to focus anew on what is the most appropriate high school curriculum, especially for initially low-performing and possibly unmotivated adolescents.