This research project aims to improve and simplify the assessment design process by building an intelligent information system “IQDIS” (Intelligent Question Design Information System) for generating assessments. The system’s key features include rationalisation of assessment tasks based on a modified cognitive taxonomy approach and a built in mechanism for assessment quality assurance that includes an analysis of the allocation of marks. Here we describe the first three phases of the project, firstly determining the conceptual model, secondly developing generic questioning patterns and building question templates based on the selected model, and finally developing a quantitative approach to quality assurance of assessments.

To determine the conceptual foundation, we considered a layered approach to the design of questioning strategies based on the hierarchical model for the cognitive domain, commonly known as ‘Bloom’s taxonomy’. We have further developed the cognitive model, introducing a third dimension into the taxonomy. This modification allows for reflection on the relative difficulty of questions belonging to different cognitive levels, and also allows for the quantitative evaluation of an assessment. A set of relevant generic questioning patterns has been developed and applied to specific domains within a number of information systems and technology courses.

We have also developed an algorithm for evaluating the distribution of marks for assessment tasks. It allows the analysis of trends in mark allocation in relation to the various cognitive categories and the impact of each category on assessment design. The algorithm enabled us to make a significant step towards the quantification of the quality assurance of assessment question sets.

The suggested approach enables assessment designers to perform accurate quantitative comparisons and evaluations of assessments, that include such key elements as the coverage of learning objectives, addressing different levels of the cognitive process, the variety of question types, and ensuring fairness of mark allocation.