Welcome to the testmaster site for
Student Assessment Research

and discussion

3D Multi-Choice Theoretical Concepts

Contents of this Section:

• The Manner of the Performance of a Skill - Theoretical Concept
• Multi-Choice Testing - Explanation and Critical Comment
• The 3D Multi-Choice Concept - Theoretical Concept

The Manner of the Performance of a Skill

In order to illustrate a basic premise behind the discussion to follow, I would like you to consider the following situation:

You are watching two men at work. Each is building an identical straight wall from the same pile of bricks, on the same piece of ground. It is 3pm and they finish at 4.30pm. Both wish to finish on time.

Man A lays bricks at a steady rate, the work is very neat. He finishes the wall at 4.20, has time to pack up, and he makes a leisurely exit at 4.30.

Man B lays bricks at a varying rate, some of the work has to be undone and re-made because it was below standard. He finishes the wall at 4.26 rushes to pack up his tools and makes a hurried exit at 4.30.

If we compare the performance of both men, much is similar. Both walls were built, to standard and on time.

However, I maintain that it would be unfair to say that both men displayed the same degree of skill. All variables were as equal as could be made humanly possible. But, the performance of Man A could be said to be superior to Man B. Why?

I believe that most people would recognise that the manner in which the tasks were accomplished affects this judgement. Man A was more skilful on the day.

This may not be a very good example of the situation I am trying to explore, but I believe it may suffice. What I am trying to illustrate is that expert performance is usually more than just an excellent end result. It is the end result and the manner of the performance that make it excellent. 

This concept is I believe, important, and I will refer to it later.

Multi-Choice Testing.

The Multi-Choice test is a much maligned technique, but one that is adopted by many educational and vocational courses for student assessment. I have provided a very brief, hopefully balanced, comment of these issues the reader may wish to view.

The use of MCQ type assessment materials is convenient at present as the analysis and marking of freely entered script, in the form of typed essays, is still not a practical option.

MCQ's lend themselves to delivery and analysis by computer. But, the computer allows a dimension for delivery and analysis that paper, or any other system currently, does not.

For an example of some good MCQ web-based software, see halfbakedsoftware.com
 

The 3D Multi-Choice Concept.

The MCQ on paper allows control of the test items, and it provides the identified response in the form of some mark - a quantitative outcome. What it does not do is tell us anything about the 'performance' of the candidate during the completion of the exam. In other words, we do not know if the test was completed in reverse order or in any other sequence, quickly in parts - slowly in others or with any / a single or many corrections and changes of answers. We do not record if the paper was completed in 15, 30, 45, or 60 minutes, although all of these factors could be highly significant when assessing overall performance, and would lead to a much greater understanding of the qualitative aspects of the level of attainment.

The MCQ administered by computer allows, if required, a very high level of control over the administration of the test. It also provides an enormous range of options regarding the nature of the 'question' delivery method:-

These options may provide an opportunity to test higher level thinking skills.

Re-consider The Manner of the Performance of a Skill concept above and let us now apply it to the computer based MCQ test situation.

The use of a computer administered MCQ test allows a record of qualitative aspects to be kept ie. we can record: 

1. when the candidate started the test;
2. if the questions were reviewed before answering;
3. the order in which the questions were answered;
4. the content of the questions that were answered first/last;
5. the accuracy of the answer;
6. any changes made to answers and duration between changes;
7. the number of times an answer was changed;
8. the accuracy of the changed answer (on each occasion);
9. the duration spent on each question;
10. the total duration of the test. 

Item 1 is only potentially important if greatly different to other candidates taking the exam. 

Items 2 and 3 could indicate good examination technique, but are fairly unimportant by themselves. Taken together with 4 and 5, this could be indicative that certain questions or topics are considered easier / harder than others. This inference may be supported by the time taken to answer the items and the accuracy of the score for those items and whether, and in what way, the items were changed.

Item 6 could indicate an unsure knowledge, especially if supported by (1) 2, 3, 4 & 5. Or, alternatively, data from 2-6 could indicate that a response was simply miss-keyed / clicked. 

Item 7 suggests great uncertainty if many changes were made ( or cheating ) 

Item 8 could suggest unsure knowledge - item incorrect changed to correct later, or

guessing - item correct changed to incorrect later, or

item incorrect changed to incorrect later, or

the duration between changes, the frequency of the changes and the final accuracy of the final response all going to inform the final judgement.

 Item 9 is a good indicator. 

If answered quickly - accurately = sure knowledge.

If answered quickly - inaccurately ( and unchanged ) = sure incorrect knowledge.

If answered slowly - accurately = difficult knowledge

If answered slowly - inaccurately ( and unchanged ) = difficult incorrect knowledge 

Combinations of higher items will produce different judgements.

Item 10 is a good indicator ie. for a 1 hour text: 

If all answered correctly - in 15 minutes = expert knowledge

If all answered correctly - in 30 minutes = very high knowledge

If all answered correctly - in 45 minutes = high knowledge

If all answered correctly - in 60 minutes = knowledge

 

If all answered incorrectly - in 15 minutes = Guessing / non-serious attempt

If all answered incorrectly - in 30 minutes = Guessing

If all answered incorrectly - in 45 minutes = No knowledge

If all answered incorrectly - in 60 minutes = No knowledge

 

If 75% answered incorrectly - in 15 minutes = Guessing / non-serious attempt

If 50% answered incorrectly - in 30 minutes = Some knowledge

If 25% answered incorrectly - in 45 minutes = Fair knowledge / Genuine attempt

If 10% answered incorrectly - in 60 minutes = Good knowledge / Genuine attempt

 Combinations of higher items will produce different judgements. 

( Clearly this is a simplified / incomplete analysis. Judgements may be influenced by pass level ) 

The speed of response consideration may appear arbitrary. If, however, it was set as an average of all this individual's responses made during all exams ever sat, or all candidates taking the test, identified for each item (in class and/or nationally), then this factor becomes more objective and reliable. Likewise, total duration and other 'arbitrary' comparisons could be treated in this way.