methods, program evaluation

When the intervention affects the entire population: Interrupted Time Series

May 02, 2018 | Pieta Blakely

Most program evaluations depend on comparing people who did participate in our program to people who didn't. But how can we measure the effects of a program that affects everyone in a population? How can we understand whether installing murals in our downtown area has increased traffic to local businesses or whether a public health information campaign has changed smoking behavior in our state? An Interrupted Time Series design is one solution.

Blurred crowd in the street

Interrupted Time Series Design

An interrupted time series makes use of an established trajectory and observes the differences between the historic trajectory and the new one after the program is implemented. We will call the program an intervention here. Ideally, you'll need measurements from at least three points in time before the intervention and three points in time afterwards. The assumption you are making is that the population was on a trajectory and that it would have continued on that path without the intervention. This accounts for all the other things that were going on in the environment when your intervention went into effect. Your analysis looks for a change in level and slope between the trajectory that had been established before the intervention and the trajectory that is observed after the intervention.

Interrupted time series-1

Interrupted Time Series with a Control Group

An Interrupted Time Series design can use a control group, too, so that the comparison is not just a projection of historic data, but the comparison group's actual data. For example, if you are measuring the effect of a public health campaign in your state, you might look at neighboring states that did not implement the campaign.

Non-statisticians can apply this method with a graph and a ruler, as I have above. Statisticians might want to read more detail about the design about how to use statistical software to run this model. For example, you can learn how to do it it Stata.

A Couple of Caveats

Seasonality Beyond having three points in time, you want to make sure that those are the right points in time to show a real trajectory. If there is seasonality in your data (for example, sales would be expected to be higher in the summer when people go out more), then you need to select your time points to that they really represent the trend from year to year, rather than seasonal fluctuations. You might use the average for the year, for example. 

Gradual Implementation A second challenge is implementations that are gradual, rather than all of a sudden. Policy changes are all of a sudden, but some programs will be rolled out over time. For example, the State of Oregon implemented a training program for bartenders in order to reduce drunk driving. All the bartenders could not be trained at once, and because of turnover in the industry, the program never reached a level where all of the working bartenders were trained. However, evaluators were able to show a relationship between the percentage of bartenders who had been trained and a reduction in drunk driving incidents.

Delayed Effects Some policy changes or programs would not have immediate effects. For example an anti-smoking campaign won't reduce instances of lung cancer right now, but might in a few years. In this case, you'll look for a change a fixed period of time after the implementation, by adding a lag to your model. 

Questions on using this in your program? Click the button below to set up a phone call.

 

 

Tags: methods, program evaluation