International comparative studies of achievement facilitate national reflection on educational practice and experience. Such studies use sophisticated assessment designs, detailed questionnaires and complex scaling methodologies to produce comparable data. This data enables international ranking of countries based on a common assessment, the identification of indicator variables, and the modeling of achievement. At the heart of these studies is a belief that the “tests are equally unfair” to participating countries. It is believed that the removal of a small number of items unsuitable for any particular country will not appreciably change achievement scores, international ranking, or achievement model. This study investigates this claim using grade 8 science achievement TIMSS 2003 for South Africa. The paper extends this claim by progressively removing items that are not covered by the South African eighth grade science curriculum.

A panel of South African science education experts reviewed the TIMSS 2003 science items, dividing them into three groups: covered in the grade 8 curriculum, covered in a previous year’s curriculum, and not covered. We then used the published TIMSS item response theory parameters to produce conditioned scale scores by progressively removing items pertaining to curriculum content that South African students had not been exposed to.

The findings show how the South African score increases as items not covered are removed. Further, we show that the distribution of scores changes, with more students reaching international benchmarks. Finally, we report that models of achievement change according to how closely the international assessment matches the nationally implemented curriculum. We acknowledge that the deletion of items selectively changes the assessment framework and therefore limits league-table like comparisons of achievement. However, we argue the achievement distribution for a country may be misleading unless there is a very close match between the nationally implemented curriculum and the internationally assessed curriculum.