Are Countries Really Being Compared Fairly?

Are Countries Really Being Compared Fairly?

Why comparability is something to investigate — not something to assume

International learning assessments are built on a simple and powerful idea: children in different countries complete comparable assessments, allowing their learning outcomes to be compared. These comparisons have transformed education policy. They have revealed the scale of the learning crisis, highlighted inequalities between countries and provided a common framework for monitoring progress.

Yet an important question is rarely asked: What makes two countries truly comparable?

It is tempting to assume that if two children are both in Grade 2 and complete the same assessment, their results can be compared directly. In practice, however, comparability is much more complex. Children who appear similar on paper may have experienced very different educational journeys before they ever sit the assessment.

One of the central arguments of our recent report is that comparability should not be treated as an assumption built into an assessment. It is an empirical question that researchers should investigate using the data themselves.

The hidden role of instructional exposure

Perhaps the clearest example concerns the timing of assessment.

School calendars differ substantially across Africa. Some countries begin their academic year in January, others in September or October, and the timing of fieldwork also varies widely. As a result, children classified as being in the same grade may have received very different amounts of instruction when they complete an assessment.

This matters because learning accumulates over time. During the early grades, even a few additional months of schooling can make a meaningful difference to children's reading and mathematics skills.

Yet international comparisons often ignore these differences. In our data, children in Nigeria were assessed after just 0.8 months of the school year on average — well under a month of instruction — while children in Comoros had already completed 12.1 months, the better part of an additional school year, by the time they were assessed. Both groups of children were classified as being in the same grade.

The question is not whether either assessment is wrong. Rather, the two assessments are measuring children at very different points in their learning journey.

Our report maps assessment timing across the MICS and ICAN-ICAR household assessments and shows that the amount of instructional exposure varies considerably between countries. These differences are large enough that they deserve explicit consideration when interpreting cross-country comparisons.

Looking beyond administrative grades

One response might be that differences in assessment timing are unavoidable. That is true. The more important point is that publicly available microdata allow researchers to account for them.

Using the timing of fieldwork together with children's grade, we estimated learning trajectories aligned to a common point in the school year. This does not produce a definitive 'corrected' ranking of countries, nor is that the intention. Instead, it provides a way of asking a different question: how would countries compare if children had received similar amounts of instruction?

The answer is revealing. Zimbabwe, for example, moves from second-lowest to seventh among the countries studied once instructional exposure is taken into account — a substantial change driven entirely by when, not how well, children were assessed. In other cases, differences become smaller or larger than suggested by the original comparisons. The exercise illustrates that part of the observed variation between countries reflects differences in when children were assessed rather than differences in learning alone.

More importantly, it demonstrates a broader methodological principle. Administrative grades are convenient categories, but they are not perfect measures of educational experience.

Comparability extends beyond timing

Assessment timing is only one example of a broader challenge.

Throughout the report we identify other factors that influence whether countries are genuinely comparable. Children may differ in their age distribution within grades. Household surveys may exclude younger children because of age eligibility rules. School participation varies considerably between countries, meaning that school-based assessments may represent different populations. Language of instruction and assessment language differ across education systems, shaping both participation and performance.

Each of these factors has the potential to influence observed learning outcomes. None invalidate international comparisons. They simply remind us that comparisons are strongest when we understand the populations and educational experiences being compared.

Learning trajectories tell a richer story

One of the unexpected benefits of adjusting for instructional exposure is that it shifts attention away from rankings and towards learning trajectories.

Rather than asking only which country performs best in Grade 2, we can begin asking how learning accumulates across the primary school years.

This perspective reveals that countries differ not only in learning levels but also in rates of progress.

Some education systems begin with relatively weak performance but make rapid gains during the early grades. Others start from a stronger position but progress more slowly. These differences become visible only when we examine learning across grades rather than focusing on a single assessment point.

The scale of this churn can be striking. In our analysis, Zimbabwe rises from seventh place in Grade 2 to first place by Grade 6, while Tunisia — the strongest performer in Grade 2 — falls to fourth by the same point. Neither country's underlying assessment changed; what changed was how much instruction each additional grade represented, and how quickly learning accumulated once it began.

These trajectories also differ by subject. Kenya and Tanzania both reach close to 90% mathematics proficiency by Grade 7, but their reading proficiency plateaus much lower, around 70%, while in Uganda it reaches only around 38% over the same period. Understanding why mathematics and reading trajectories diverge so sharply — particularly in Uganda — turns out to be closely connected to the language in which children are assessed, a question we take up in the next AFLEARN Insight.

For policymakers, this is a much richer way of thinking about performance. Education systems are dynamic. Understanding how learning develops over time often provides more useful information than comparing countries at one point in the primary school cycle.

Better comparisons require better evidence

The objective of international assessment is not simply to rank countries. It is to understand how education systems differ, identify where progress is being made and generate evidence that supports better policy.

Achieving that objective requires careful attention to comparability.

One of the great strengths of publicly available microdata is that they allow researchers to investigate the assumptions underlying international comparisons rather than taking those assumptions for granted. They make it possible to ask whether countries are measuring similar populations, whether children have received similar amounts of instruction and whether differences in observed performance might reflect features of assessment design as well as differences in learning.

This does not make comparisons more complicated for the sake of it. It makes them more meaningful.

From rankings to understanding

International comparisons will always attract attention because they provide simple answers to important questions. But education systems are rarely simple.

The greatest value of comparative assessment lies not in producing rankings but in improving understanding. Open microdata allow researchers to move beyond headline comparisons and investigate why countries differ, how learning develops and what factors shape children's educational experiences.

That is one of the most important contributions of secondary analysis. It shifts the conversation from asking 'Which country performed better?' to asking 'Are we making the fairest possible comparison?'

The answer to that second question ultimately determines how much we can learn from the first.

AFLEARN Insight #1       AFLEARN Insight #2       AFLEARN Insight #3      AFLEARN Insight #4     AFLEARN Insight #5     Read the report