An interrupted time series (ITS) design involves collecting data consistently before and after an interruption. This means introducing and withdrawing your digital product or service, or some part of it. It is a useful design to demonstrate cause and effect of your digital product without randomising to different conditions – for example, as you would do in a randomised controlled trial.
What to use it for
Using ITS can help you to find out whether your digital product or service achieves its aims, so it can be useful when you have developed your product (summative evaluation). To use an ITS design, you need 2 things:
- an intervention that begins at a specific time point
- to collect time series data (sequential observations from before and after the intervention) to evaluate the effects of introducing the intervention
Benefits of ITS include:
- it is a strong design to use to estimate effects of your product when randomisation is not suitable or possible
- many ITS designs involve comparing participants to themselves, which means that the design is more sensitive to differences in the effects of the intervention
- it could be conducted with a small sample size
Drawbacks of ITS include:
- lack of randomisation means that drawing definitive answers about the effects of your digital product will be limited
- some ITS designs might be more likely to be influenced by a novelty factor, for example, when you introduce and withdraw the product in a short space of time
How to carry out an interrupted time series study
Traditionally, ITS designs have been used to assess effects of interventions when assigning people to different groups is not possible, such as with mass media campaigns or policy change. For example, ITS lets us assess the impact of introducing smoke-free policies by comparing the data before and after the introduction of the policies.
However, ITS can also be very useful for evaluating digital health products. You could use ecological momentary assessment to collect your data. There are 2 main types of design useful for assessing products delivered through digital means:
ABA design (also called reversal or removal design)
In an ABA design, outcomes are collected consistently but the intervention is added and removed. This assesses if the outcome changes when the intervention is received and the effect diminishes after it is removed.
This design is only appropriate if the digital product or service aims to produce an effect and the effect is expected to diminish when the product is withdrawn (see carryover effects in crossover randomised controlled trials).
Multiple baseline design
In a multiple baseline design, the start of the intervention is staggered across participants. This design is particularly useful if you cannot remove the digital product after introducing it. For example, it is useful when you are progressively rolling out a new digital service to different organisations. You can measure the outcome before and after the introduction of the service in different places at different times. You can then compare the change over time accounting for any factors that may influence the effects, such as events that impact all organisations.
Analysing ITS data
Analysis of an ITS series takes into account the underlying trends when estimating intervention effects. It does not just report the effects of pre-intervention post-intervention periods as in a before and after study. There are various potential biases you need to account for when analysing the data, for example:
- random fluctuations in the outcomes
- seasonal effects
- autocorrelation – that is, individual responses tend to be more similar when assessments are carried out close together in time
Example: Using incentives to increase blood glucose testing in teenagers with Type 1 diabetes
Blood glucose testing is critical in preventing the progression to diabetes-related illnesses, such as retinopathy. Raiff and Dallery (2010) used an ABA design to assess the feasibility and acceptability of a web-based incentive intervention to increase blood testing in teenagers with Type 1 diabetes. They included 4 participants.
There were 3 phases, each lasting 5 days:
- baseline (A)
- intervention (B)
- return to baseline (A)
In the intervention, participants received financial incentives when they posted videos of themselves performing at least 4 blood glucose tests per day. At the end of the intervention, participants were asked to stop posting videos but to continue blood testing. For both A-phases, data was collected from the participant’s personal glucometer.
The authors reported:
- an increase in the frequency of glucose tests when incentives were introduced
- a reduction when the incentives were removed
There were some carryover effects.
As this was a feasibility study with a small sample, the authors did not draw firm conclusions about the efficacy of the incentives. However, the study suggested web-based incentive programme may be a promising digital tool to increase blood glucose monitoring in teens with Type 1 diabetes.
More information and resources
Ewusie and others (2020), ‘Methods, Applications and Challenges in the Analysis of Interrupted Time Series Data: A Scoping Review’. Information about various methods used in analysing time series data, and their strengths and limitations.
Bernal and others (2018), ‘The use of controls in interrupted time series studies of public health interventions’. One method for strengthening the findings of a study using an ITS design is to add a control. This describes the wide range of controls that can be used.
Ramsay and others (2003), ‘Interrupted Time Series Designs in Health Technology Assessment: lessons from two systematic reviews of behavior change strategies’. This paper critically evaluates analysis of various ITS studies that assessed health technology, points to the issues in their analysis, and suggests a framework for analysing the data.
Examples of interrupted time series studies in digital health
Perski and others (2020), ‘Influence of the SARS-CoV-2 Outbreak on the Uptake of a Popular Smoking Cessation App in UK Smokers: Interrupted Time Series Analysis’. The authors assessed whether the coronavirus outbreak influenced the downloads of a smoking cessation app.
Brydon and others (2020), ‘Transitioning to full field digital mammography in Nova Scotia: using interrupted time series methods to study the impact of technology change on mammography volumes’. The authors used ITS in a real-world setting to evaluate the transition from analogue to digital diagnostic imaging.
Chereni and others (2017), ‘Effect of adding a mobile health intervention to a multimodal antimicrobial stewardship programme across three teaching hospitals: an interrupted time series study’. The researchers assessed the impact of adding an app intervention to decrease antibiotic prescribing.