Governments make policy decisions based on the large volume of data and analyses emanating from The OECD Programme for International Student Assessment (PISA). Certain countries’ education systems are vilified in the media; others are deified. Absent from the public discourse are individual test items and information about student−item interactions. In a study that analysed French students’ results on the PISA 2000 Literacy Test, Rochex (2005) criticised the conception of skills as context-independent. In a study that examined context as a source of item difficulty on the PISA 2006 Science Test, Matters (2008) concluded that although the real-life context was problematic the issue was not context per se but the disjuncture between the construct being assessed, which is grounded in (presumably manifold) reality, and the ‘unreal’ neutrality (Kelly, 2008) to which item context seems to aspire, and which would not necessarily have the same effect on the mindset of all students. This issue and others about the contribution to item difficulty of student perceptions and design impositions will be raised during the presentation. Questions will be posed about test validity, particularly from Cronbach’s (1988) ‘explanatory’ and ‘functional’ perspectives. Thoughts on what might be the implications for countries or jurisdictions planning to join the ‘PISA club’ will be offered. Scenarios for students whose assessment experience is of content-specific multiple-choice tests that do not locate items in unfamiliar contexts will be described. Alternative explanations for the remarkable gender neutrality overall in PISA will be offered. Policy makers and educationists will be advised to approach test results at the item level before moving into the realm of abstraction of ability and construction of variables related to items. And they will be advised to proceed cautiously from that realm to the use of statistics, their interpretation, stunning presentation, and translation into a language called journalese.