DIR, through an ED-IES contract, directed development of three new reports focused on design-based methods for estimating intervention effects.

DIR is pleased to announce that today (August 1, 2017) the National Center for Education Evaluation and Regional Assistance (NCEE) announced the release of three new reports:

  • “What is Design-Based Causal-Inference for RCTs and Why Should I Use It?”
  • “Multi-Armed RCTs: A Design-Based Framework”
  • “Comparing Impact Findings From Design-Based and Model-Based Methods: An Empirical Investigation”

These reports can help education researchers use new design-based methods to analyze data from impact evaluations. The novel design-based approach uses the building blocks of experimental designs to develop impact estimators with minimal assumptions, and has important advantages over “model-based” methods that have typically been used in education research. The reports were authored by DIR’s partner, Mathematica Policy Research, Inc., commissioned by DIR, and funded through our Analytic Technical Assistance and Development (ATA&D) contract with the Institute of Education Sciences (IES) in the U.S. Department of Education. DIR, with Dr. Russell Jackson, DIR’s president and CEO as project director, leads the team providing support to Regional Educational Laboratories and other National Center for Education Evaluation (NCEE) contractors for IES.
The IES Newsflash announcing release of this new guide, as well as direct links to each full report are below.

 

ies
fa
tw
n

Three Reports Examine New Design-Based Methods for Analyzing Data From Impact Evaluations

The Institute of Education Sciences (IES) has released three reports focused on design-based methods for estimating intervention effects. The novel design-based approach uses the building blocks of experimental designs to develop impact estimators with minimal assumptions, and has important advantages over “model-based” methods that have typically been used in education research. The methods apply to randomized controlled trials and quasi-experimental designs with treatment and comparison groups.

What is Design-Based Causal-Inference for RCTs and Why Should I Use It?

This brief aims to broaden knowledge of design-based methods by describing their key concepts and how they compare to model-based methods. Rudiments of the design-based approach are presented using simple mathematical notation, and the intuition underlying the theory is discussed for designs where individuals or groups are randomized. The brief explains in simple terms the advantages of the design-based approach relative to commonly-used model-based approaches, such as hierarchical linear modeling (HLM).

Multi-Armed RCTs: A Design-Based Framework

Multi-armed designs are becoming increasingly common in social policy research to simultaneously examine the effects of multiple interventions in a single study. This report describes how design-based estimators for the two-group design need to be modified for the multi-armed design when comparing pairs of research groups to each other. It also discusses multiple comparison adjustments when conducting hypothesis tests across pairwise contrasts to identify the most effective interventions. Finally, it discusses the complex assumptions required to identify and estimate the complier average causal effect (CACE) parameter in the multi-armed context.

Comparing Impact Findings From Design-Based and Model-Based Methods: An Empirical Investigation

This study investigates how much of a practical difference it makes to use design-based methods versus more conventional model-based methods. The study re-analyzes data from nine past education RCTs covering a wide range of evaluation designs. Impacts are estimated using design-based, HLM, and robust standard error methods. The study finds that the design- and model-based methods yield very similar impact estimates and levels of statistical significance, especially when the underlying analytic assumptions (such as the weights used to aggregate clusters and blocks) are aligned. Furthermore, the differences between the design- and model-based methods are no greater than the differences between the two considered model-based methods.

·······················································································································································

The Institute of Education Sciences, a part of the U.S. Department of Education, is the nation’s leading source for rigorous, independent education research, evaluation and statistics.

Comments are closed.

www.dir-online.com