Designing Curriculum Effectiveness Studies

by Catherine A. Little and Joyce VanTassel-Baska

Originally published in the spring 2002 issue of Systems, the newsletter of the Center for Gifted Education at the College of William and Mary.

Curriculum products in every subject area and for every grade level abound in today’s educational contexts. In addition to textbooks adopted by school systems and locally determined curriculum, multiple resource books and on-line lesson plans are available for teacher use. However, many of the curriculum products on the market have not been tested with students for effectiveness in promoting student learning, and although they may be interesting or attractive, there is limited documentation supporting their use. In a society increasingly focused on accountability in education, there is an ever greater need to demonstrate the effectiveness of the materials and instructional methods students are experiencing in classrooms.

Over the last decade, the Center for Gifted Education has been engaged in ongoing curriculum effectiveness research investigating various outcomes related to the use of our curriculum units in classrooms across the country. Results of these studies, demonstrating learning gains for students in science and language arts, have been published in several journals (e.g., VanTassel-Baska, Bass, Ries, Poland, Avery, 1998; VanTassel-Baska, Johnson, Hughes, & Boyce, 1996; VanTassel-Baska, Zuo, Avery, & Little, 2002), and additional results of our most recent work appear in this issue of Systems.

In addition to contributing to the ongoing curriculum development agenda, work to refine our research efforts has also helped us develop a template for designing curriculum effectiveness research, whether that research is to be conducted on a small scale, such as an action research project conducted by one or two teachers, or on a large scale across a school system or multiple systems. At the 2001 convention of the National Association for Gifted Children, Dr. Joyce VanTassel-Baska and Dr. Catherine Little from the Center, with Dr. Karen Rogers of the University of St. Thomas and Virginia Tucker of Montgomery County Public Schools, presented a Master Class exploring this template with participants interested in conducting curriculum effectiveness research. A summary of the highlights of that presentation follows.

The first step in any research study is the determination of an area of focus and tentative questions of interest. This should be followed by an investigation of the existing educational literature related to the subject, including emphasis on related theories and previous research conducted. Research questions may then be formalized, growing from the tentative questions of interest and the exploration of existing literature. At this point in the study planning process, one important challenge is the need to specify the scope of the assessment planned and to ensure that the scope is manageable – a single study does not need to answer every possible question about a curricular program’s effectiveness. Sample research questions that may underlie a curriculum effectiveness study include the following:

• Do gifted learners participating in a specific curriculum intervention outperform gifted learners not participating on relevant measures?
• Does the grouping approach affect performance on relevant measures?
• Do gender, age, or socioeconomic status relate to performance on relevant measures?
• Do observed teacher behaviors relate to student performance on relevant measures?
• What developmental or other patterns are evident in student products?

In addition to these types of questions, school-based curriculum research studies also frequently involve an evaluative component, exploring curriculum in use to support maintaining or changing it or investigating to what degree a curriculum deemed effective in other contexts is appropriate for a given local population. Some sample questions that might frame an evaluation study are as follows:

• To what extent are the learning goals and outcomes of the curriculum addressed and fulfilled in classroom implementation?
• To what extent is the curriculum meeting the needs of identified students, as perceived by relevant groups?
• What evidence exists to document positive student performance trends for students participating in specialized curriculum?
• What are the strengths and weaknesses of the existing curriculum in relation to the state of the art or best practices in gifted education?

The remaining aspects of the study design then grow out of the research and/or evaluation questions. Depending on the results to be explored (e.g., student outcomes, teacher behaviors, stakeholder perspectives), the researcher must identify specific curriculum outcomes, teacher behaviors, or perspectives to be assessed. Particularly when designing research studies around curriculum for the gifted, student outcomes should be focused at higher levels of thinking and must be measurable. Teacher behaviors should also include methods that support higher level thinking and demonstrate attention to differentiation. Moreover, the specific outcomes to be assessed should be linked directly with the goals and outcomes of the curricular program being studied.

The research design emerges from the questions of interest and the types of outcomes to be assessed, keeping in mind that the design selected must be feasible, given the resources available, and appropriate to answer the types of questions asked. In measuring curriculum effectiveness in real-world classroom settings, the most powerful design that is practical for use is a pre-post quasi-experimental one. In this design, two similar groups of students are defined and established as the treatment group and comparison group. Both groups take pre- and post-assessments related to the outcomes of interest; between the assessments, the treatment group participates in the curriculum under study, while the comparison group does not. Student gain scores are then compared statistically. If the groups are sufficiently similar and if the treatment group’s gains are larger than the comparison group’s gains, then student growth on the outcomes of interest may be attributed to the curriculum treatment. This type of design has been the main focus of the Center’s curriculum effectiveness studies.

However, other types of studies are also useful in exploring aspects of curriculum effectiveness, depending on the research questions. Researchers may conduct an observation study of teacher behaviors, again relying on a pre-post design or perhaps using a qualitative observational approach instead. A qualitative study of student products developed in the course of a curriculum unit or a case study of a classroom or school using a given curriculum are also design possibilities, as is a descriptive study of student learning effects or other perceived effects.

A critical part of preparing a curriculum effectiveness study is the determination of data sources and instrumentation. Studies are stronger if they utilize multiple sources of data and if the instruments selected are demonstrably valid and reliable. Measures of student and teacher performance, perception surveys, and interview protocols are among the sources possible to tap. Wherever possible, existing instruments aligned with the intended outcomes should be used, assuming that these instruments have been published with relevant technical adequacy data. The issues of instrument reliability and validity are key questions to consider, because any findings will rely on the strength of the instruments used. On the other hand, the instruments chosen must be appropriate for the questions of interest and logistically practical for use in the study, so sometimes no appropriate instrument exists. If instruments must be created for a study, they should be piloted and revised to strengthen technical adequacy.

With these elements in place, study procedures, participants, and timeline should be determined, and then an action plan moved forward. In addition, with any educational research study, proper approval for human subjects research should be sought from relevant authorities; most school districts and universities have existing procedures and committees for review prior to study implementation, and generally most require attention to alignment with existing programs and to obtaining parental permission.

The implementation of a curriculum effectiveness study may encounter many challenges and delays along the way, and researchers must be prepared for the impact of extraneous variables. Several recommendations to smooth the path of a curriculum effectiveness study follow. These recommendations apply primarily to studies to be conducted across multiple classrooms and perhaps multiple schools, but aspects of them also apply to smaller action research studies:

• Assign a research coordinator to facilitate the process.
• Pilot the curriculum with a small number of teachers and schools and examine the data from the smaller sample before a large implementation study.
• Communicate the purposes and intended use of the data collection efforts to relevant audiences, using multiple modes and venues for communication.
• Provide appropriate professional development on the curriculum intervention, including follow-up opportunities during implementation.
• Assign sufficient resource time for monitoring (to allow 2-3 visits per classroom)
• Assess selected outcomes within curriculum and use performance-based measures that meet technical adequacy requirements.
• Administer pre-post measures directly.
• Institute double-blind scoring procedures, but provide teacher training in the process as well.
• Schedule a student exhibit day for products.
• Use external reviewers for product review.

Curriculum effectiveness research, whether conducted on a small scale or a large scale, should be promoted as a critical aspect of the educational process. If student learning is the ultimate goal of education, then discovering whether what we do is helping to achieve that goal is fundamental to the educator’s role. Teachers, school and central office administrators, and university personnel should work together to conduct curriculum research in order to strengthen the effectiveness of what happens in classrooms each day.

References

VanTassel-Baska, J., Bass G. M., Ries, R. R., Poland, D., & Avery, L. (1998). A national pilot study of science curriculum effectiveness for high-ability students. Gifted Child Quarterly, 42, 200-211.

VanTassel-Baska, J., Johnson, D. T., Hughes, C., & Boyce, L. N. (1996). A study of language arts curriculum effectiveness with gifted learners. Journal for the Education of the Gifted, 19, 461-480.

VanTassel-Baska, J., Zuo, L., Avery, L. D., & Little, C. A. (2002). A curriculum study of gifted student learning in language arts. Gifted Child Quarterly, 46, 30-44.