GOVERNMENT RESEARCH INTO
FANTASTIC PHONICS

Download >>> Liberia Program Evaluation Report - Executive Summary

Summary:
In 2008, USAID began a research trial in Liberia, to improve literacy outcomes in Liberian schools. The exploration tool was the Early Grade Reading Assessment (EGRA) Plus, which applies a standardised measurement method to benchmark children in grades 2 and 3.

180 schools were randomly selected, and divided into three different groups; no treatment, light treatment, and full EGRA treatment, using a reading program (Fantastic Phonics) of 60 graded stories, with teacher guides.

The children were bench marked before commencement, and tested on several literacy criteria. It was found that students in Grade 2 (on average) read close to 18 correct words per minute. In comparison, the United States benchmark for “severe risk” is less than 70 correct words per minute.

After 18 months, the children in the full treatment group showed a 250% improvement, compared to their nil- and light-treatment peers.

Program Impact
Table 1 also shows the increase in scores over the baseline for each group. Note that these estimates come from a simple tabulation of the data.

Letter naming fluency
At the baseline, Liberian children were capable of identifying the names of letters, with the average control child identifying 60.7 letters in a minute. At the baseline, the letter naming scores were good, which suggests that program impacts were not likely to be very large. At the final assessment, students in full treatment schools showed a 59.2% increase in letters read, while Control schools increased in letter fluency by 39.0%.

Phonemic awareness
Program impact on phonemic awareness was also large.The combined scores for grades 2 and 3 show that the number of sounds identified increased by 67.4% in full treatment schools compared to 26.4% for control schools.

Familiar word fluency
For familiar words, children in full treatment schools increased by 247.8% and control-school children increased their scores by 121.2%.

Unfamiliar word fluency
For unfamiliar words, in full treatment schools the increase was 485.7% (from 2.5 words per minute to 17.3 words per minute) while control schools increased by just 42%.

Oral reading fluency
The impact was also quite large for fluency in oral reading of connected text. Compared against baseline, full treatment children increased the number of words read correctly by 138.2%, and control schools by 39.0%.

This means that at the final assessment, children in full treatment schools were reading nearly two and a half times as fluently as they were at the baseline.

Reading comprehension
Comparing the final and baseline assessment scores in reading comprehension, we find that full treatment schools increased their scores by 130.1% over baseline, while light treatment scores increased by 33.4% and control schools by 32.9%. This means that, at the final assessment, children in full treatment schools more than doubled their reading comprehension percentage rates.

Listening comprehension
For listening comprehension, the increases for full and light treatment schools were 148.8% and 108.0%, respectively. It should be noted that control schools increased their scores by 112.7%, so only by taking into account the baseline scores can a true program effect be estimated. Substantively, full treatment schools increased by 49.9% and light treatment schools by 37.3% over baseline, an effect size of 0.39 SD and no effect, respectively.

Overall Program Impact
This report presents the effect sizes from a more sophisticated analysis using differences-in-differences analyses. These are presented in Table 1 above, in the effect size column.These analyses show that the full treatment group increased student achievement for every section of the EGRA, often with quite large impacts on student achievement. In fact,the overall EGRA Plus effect size was 0.79 standard deviations, which is enormous in social science.

When the program impacts are expressed in terms of grade effects, the full treatment increased letter naming fluency by 1.2 times the effect of being in school for one year. Amazingly, this was the smallest effect size for any of the skills assessed. The EGRA Plus full treatment effect was the equivalent of 1.9 school years in phonemic awareness, 1.8 school years in familiar word reading, a remarkable 8.0 years i unfamiliar word fluency, 1.9 years in oral reading fluency, 2.0 years in reading comprehension, and 1.8 years in listening comprehension.

This shows that the EGRA Plus program did not simply increase the learning outcomes for children; it dramatically accelerated children’s learning to an extent seldom found in educational or social science research.

In summary, given the existing literature on effect sizes in literacy interventions, EGRA Plus: Liberia far exceeded expectations with respect to impact on student achievement, particularly in the full treatment schools. Note that the effects were most often large in full treatment schools, with some moderate effect sizes.

Oral reading fluency scores at the final assessment were 238% of what they were at the baseline, reading comprehension scores were 230% of what they were at baseline, and decoding fluency scores were 585% what they were at the baseline assessment.

These impressive results were found on all of the essential early reading components. Most critically, the EGRA Plus program accelerated the learning of children so much that children learned the equivalent of three years of schooling in one year.