ELT - J Issue #2 Preview Article: Devising Multiple-choice Questions, Quizzes and Tests
Issue 2 of ELT in Japan will consist of four articles: (1) "Teaching English [l] sounds vs. [r] sounds: Integrating applied phonology into the EFL classroom"; (2) "Devising multiple-choice questions, quizzes and tests"; (3) "Semantic mapping activities for the speaking class"; and (4)"Semantic mapping activities for the writing class".
Here is a special preview of Issue 2, article (2) from the list above. All four articles will be published together as one issue later this month (February 2010):
ELT - J Issue #2 Preview Article: Devising Multiple-choice Questions, Quizzes and Tests
by Charles Jannuzi, University of Fukui, Japan
Introduction
Chances are that, if you teach EFL at a tertiary level in Japan, your students are most familiar with relatively 'conservative' types of assessment tasks. However, these tasks are also further limited by an overall orientation of turning L2 (EFL) into L1 (Japanese). For example, they have often been tested with questions that asked them to put a word, sentence or phrase of L2 (English, EFL) into L1 (standard written Japanese).
If this sort of task is extended to 'reading comprehension', it is usually an underlined word, phrase or sentence of the L1 that they are tasked to put into the correct Japanese. Or they may be tested on their ability to figure out what words like 'this' or 'that' or 'it' refer back to in discourse.
There are many problems with such a limited approach to assessment. For one, the constant framework is from English (L2, SL, FL) into Japanese (L1, standard dialect), which seems to re-inforce the idea that understanding and communication only take place in Japanese, not English. A second weakness, especially with the reading comprehension questions, is that students are asked to translate parts of a text the whole of which they have no hope of reading with much comprehension. That is, the text being tested is too difficult for the majority of the students taking the test, thus rendering any tasks derived from it invalid for norm-referencing (which is why criterion-referenced tests are safer on the issue of validity if you can not norm a task or test on large populations). A third deficiency is that it doesn't prepare the students for a wider world of testing where the 'English-only' multiple-choice question is prevalent.
It could also be pointed out that the multiple-choice question itself is often seen as a 'conservative' type question, even a bane in the world of ELT and education. However, it does dominate important language tests, such as the TOIEC, TOEFL and Eiken/Step (a multiple-level, criterion-referenced EFL test given in Japan).
In this article we will look at how to introduce students to 'English-only' multiple-choice questions and then how to use such questions to assess students and evaluate courses. The focus here will be specifically on using multiple-choice questions to construct tests that assess vocabulary learning drawn from the syllabus, materials, and content of specific EFL classes--i.e., the classes you have to teach and your students have to attend.
It is hoped that these explanations and examples will serve, for example, teachers who have to give grades to hundreds of students each semester and have limited time to do other types of testing (such as oral interviews or projects, which can be too time-consuming to manage if you teach hundreds of students).
Some suggestions for this type of testing
1. Choose the most frequent and most useful vocabulary from the syllabus, the textbooks, lesson plans and class content. When you devise and select test questions, target items, and distractors, try to stay with the most useful and most frequently used words and phrases of English. It would be better for students to learn and be tested on a new use or meaning of a core word than an obscure item, and this principle will help you to help your students better to prepare for standardized tests.
2. I find it best to draw a very large set of vocabulary from the course content and then to put it on a list. Then, I identify the most frequent items and choose from these at random if I need to reduce the list of words on the test down to a smaller number.
2. Work on making a good question/test items first, and then worry about the distractors. To quite an extent, your distractors are already there--in the textbook, on your large list of vocabulary, and even on the smaller list from which you will chose to write the test questions.
3. Distractors can take too much time to prepare if you simply try to call them up from your own English. Rely on the textbook, textbook glossaries, your list of chosen words and the most frequent words of English instead. If having three distractors proves too difficult, try coming up with two good ones. Two good distractors are better than three bad ones. Choose distractors from your 'short list' of items and then, if these are used up, draw from the larger list or from a list like Ogden's Basic English or a list of, for example, the 1000 most frequent words of English.
4. Try holding several vocabulary tests throughout the term (which at my university is 15 class meetings once a week for 90 minutes).
5. Recycle. Vocabulary items can be recycled onto following tests, as can distractors. It is fairly easy to recycle an item by re-writing the question around it.
6. Work at making your distractors plausible, but remember, what is plausible for a fluent user of English might be completely different for a beginner. Also, language background can be a factor. For example, if I wrote, "She is a 'safety worker", that to most fluent users of English would be a rather obviously wrong and not a good distractor for 'safe'. But in Japanese, the word for 'safe' and for 'safety' is often the same word ('anzen'). So this is a plausible distractor for beginning EFL students in Japan, and most likely one that the makers of the TOEIC know about and put on their tests.
Try a variety of different distractors. Use semantic distractors. Use grammatical distractors. Use phonetic distractors. Use distractors that are about the same length and of around the same frequency of use. Use distractors that come from the same texts as the test items. Remember, what is a plausible distractor for someone who is at the beginning level of EFL may be hard for you to anticipate if you are a fluent user of English.
7. Keep accumulating word lists, distractors and test questions so that you can:
-give quizzes or tests of at least 35 questions or more (preferably 50);
-give quizzes or tests at least several times throughout the term.
8. Increase your total collection of test questions by paraphrasing, since students need to get familiar with the most common ways of basically saying the same thing.
9. Make the questions and the tests 'organic' to the type of class you are teaching and the content of its syllabus. For example, if the class is a very low-level 'General English' class, try using the simplest English possible when writing your questions. If the course is a listening one, try using some short dialogues instead of single sentences. Look at current TOEIC Listening questions to see relevant examples. If the course focuses on reading, try short paragraphs instead of just single sentences. Also, make the questions more realistic, relevant and communicative by basing them on things like: facts about your local city, facts about your university campus, current events, etc. For example, if a course is Business English, why not try writing questions based on the week's business news (e.g., Toyota in trouble over safety, recalls millions of cars)?
10. Some argue that we should avoid cultural bias in our test items. However, the TOEIC, for example, assumes that students are quite familiar with American culture (often dressed up as 'global business culture'). If multiple-choice questions are given as vocabulary practice, the questions might also help introduce students to cultural learning points and help them to build up background knowledge about many aspects of American culture--or other important anglophone cultures, such as those of the UK, Canada, Australia, Ireland, New Zealand, etc. It might depend on the goals and study-abroad opportunites your students have. I object to the American-bias of the TOIEC, but that doesn't mean it is going to go away.
11. The week before giving a multiple-choice quiz or exam, practice a set of examples with the students. Also, if they are new to identifying and studying key vocabulary on their own, give them a larger list of words to study from which you will randomly select your smaller set of test items.
Mutiple-choice Question Examples
(Note: the correct answer will always be 'a' in these examples, the distractors b, c, d.)
In this next section we will look at basic types of multiple-choice questions and use them as an opportunity to show distractor types as well (e.g., semantic, grammatical, phonetic, inflectional, derviational, etc.).
A. Multiple-choice basic sentence
ex. 1
The weather at the beginning of this week started very cold, but it should ______________ milder by the weekend.
a. become b. became c. begin d. complete
ex. 2
After work I like to watch T.V. programs in order to ____________.
a. relax b. look c. discuss d. describe
ex. 3
(Note: the TOEIC tests this sort of item a lot.)
To save electricity, when you leave the room, please turn ______________ the light.
a. off b. in c. around d. at
B. Multiple-choice dialogue
ex. 1
A: How is it _______________?
B: Oh, not bad. Yourself?
a. going b. doing c. meeting d. go
ex. 2
A: Do you think we should take our umbrellas?
B: Yes, the weather ______________ says it is going to rain soon.
a. forecast b. forest c. fortune d. cloud
C. Multiple-choice single sentence, match the underlined word with its synonym
ex. 1
Sony Corporation is a famous electronics manufacturer that was __founded__ in 1945.
a. established b. forced c. focused d. ordered
ex. 2
Sony Corporation __manufactures__ many different types of consumer electronics in its factories in Japan and overseas.
a. makes b. removes c. explains d. thinks
D. Multiple-choice, match the synonym, dialogue
ex.
A: Mommy! Mommy! I'm so hungry I can't stand it!
B: O.K. Calm down. I'm __preparing__ lunch right now.
a. making b. putting c. causing d. crying
E. Multiple-choice definition sentence
ex. 1
If you ______________ something, it means your habit is to like it.
a. prefer b. defer c. refer d. infer
ex. 2
_____________ is an overall economic condition of falling commodity and asset prices.
a. Deflation b. Inflation c. Decision d. Exhaustion
F. Multiple-choice definition sentence, matching synonym
ex.
Deflation is an overall economic condition of 'falling' commodity and asset prices.
a. declining b. increasing c. deciding d. protesting
G. What is ________________?
(Note: Here you give three examples illustrating the meanings, uses, nuances and common collocations of a key vocabulary item. This type of question could be very difficult to write distractors for if you try to avoid distractors that might work in one example. But remember, the correct answer has to work in all three examples, so you can use other words that work in only one sentence as distractors (for example, fix, as in 'fix a flat' instead of 'change a tire').
ex.
-Some people like to eat the same food every day for lunch, but others like to ______________ it and eat a variety of dishes.
-On the way to work today, my bicycle had a flat tire, so I had to _____________ it.
-The weather forecast in the newspaper says today will be fine but will then ____________ to rain later tonight.
What is _______________?
a. change b. turn c. fix d. return
Other question types are possible. For example, you could use a short paragraph instead of a sentence or a dialogue.
Conclusion
Many EFL teachers dislike the multiple-choice question and tests made from it. However, if you want a manageable way to assess and recycle vocabulary in large EFL classes, it is one of the best types of tests. It can be applied to all types of EFL classes, such as speaking/conversation, listening (especially standardized test listening), reading, vocabulary study, grammar review, etc. Moreover, if you teach EFL in Japan you might find that your students are not that familiar with alternative varities of multiple-choice questions for EFL, or tests, questions and tasks written only in English. .
If you write a variety of questions and stick to the syllabus and the most frequent vocabulary of English, you will probably be helping your students more than using methods of assessment that can not be scored objectively or test (punish or reward) language abilities that fall outside the scope of a course, its syllabus and textbooks. Multiple-choice quizzes and tests in English-only will also help prepare your students for important English proficiency exams like TOEIC.
Note: All content is copyright (c) 2010 of Charles Jannuzi and ELT in Japan, but is available for re-use, re-distribution and dissemination under the Creative Common license ( http://creativecommons.org/licenses/by-nc-nd/3.0/ ).
Here is a special preview of Issue 2, article (2) from the list above. All four articles will be published together as one issue later this month (February 2010):
ELT - J Issue #2 Preview Article: Devising Multiple-choice Questions, Quizzes and Tests
by Charles Jannuzi, University of Fukui, Japan
Introduction
Chances are that, if you teach EFL at a tertiary level in Japan, your students are most familiar with relatively 'conservative' types of assessment tasks. However, these tasks are also further limited by an overall orientation of turning L2 (EFL) into L1 (Japanese). For example, they have often been tested with questions that asked them to put a word, sentence or phrase of L2 (English, EFL) into L1 (standard written Japanese).
If this sort of task is extended to 'reading comprehension', it is usually an underlined word, phrase or sentence of the L1 that they are tasked to put into the correct Japanese. Or they may be tested on their ability to figure out what words like 'this' or 'that' or 'it' refer back to in discourse.
There are many problems with such a limited approach to assessment. For one, the constant framework is from English (L2, SL, FL) into Japanese (L1, standard dialect), which seems to re-inforce the idea that understanding and communication only take place in Japanese, not English. A second weakness, especially with the reading comprehension questions, is that students are asked to translate parts of a text the whole of which they have no hope of reading with much comprehension. That is, the text being tested is too difficult for the majority of the students taking the test, thus rendering any tasks derived from it invalid for norm-referencing (which is why criterion-referenced tests are safer on the issue of validity if you can not norm a task or test on large populations). A third deficiency is that it doesn't prepare the students for a wider world of testing where the 'English-only' multiple-choice question is prevalent.
It could also be pointed out that the multiple-choice question itself is often seen as a 'conservative' type question, even a bane in the world of ELT and education. However, it does dominate important language tests, such as the TOIEC, TOEFL and Eiken/Step (a multiple-level, criterion-referenced EFL test given in Japan).
In this article we will look at how to introduce students to 'English-only' multiple-choice questions and then how to use such questions to assess students and evaluate courses. The focus here will be specifically on using multiple-choice questions to construct tests that assess vocabulary learning drawn from the syllabus, materials, and content of specific EFL classes--i.e., the classes you have to teach and your students have to attend.
It is hoped that these explanations and examples will serve, for example, teachers who have to give grades to hundreds of students each semester and have limited time to do other types of testing (such as oral interviews or projects, which can be too time-consuming to manage if you teach hundreds of students).
Some suggestions for this type of testing
1. Choose the most frequent and most useful vocabulary from the syllabus, the textbooks, lesson plans and class content. When you devise and select test questions, target items, and distractors, try to stay with the most useful and most frequently used words and phrases of English. It would be better for students to learn and be tested on a new use or meaning of a core word than an obscure item, and this principle will help you to help your students better to prepare for standardized tests.
2. I find it best to draw a very large set of vocabulary from the course content and then to put it on a list. Then, I identify the most frequent items and choose from these at random if I need to reduce the list of words on the test down to a smaller number.
2. Work on making a good question/test items first, and then worry about the distractors. To quite an extent, your distractors are already there--in the textbook, on your large list of vocabulary, and even on the smaller list from which you will chose to write the test questions.
3. Distractors can take too much time to prepare if you simply try to call them up from your own English. Rely on the textbook, textbook glossaries, your list of chosen words and the most frequent words of English instead. If having three distractors proves too difficult, try coming up with two good ones. Two good distractors are better than three bad ones. Choose distractors from your 'short list' of items and then, if these are used up, draw from the larger list or from a list like Ogden's Basic English or a list of, for example, the 1000 most frequent words of English.
4. Try holding several vocabulary tests throughout the term (which at my university is 15 class meetings once a week for 90 minutes).
5. Recycle. Vocabulary items can be recycled onto following tests, as can distractors. It is fairly easy to recycle an item by re-writing the question around it.
6. Work at making your distractors plausible, but remember, what is plausible for a fluent user of English might be completely different for a beginner. Also, language background can be a factor. For example, if I wrote, "She is a 'safety worker", that to most fluent users of English would be a rather obviously wrong and not a good distractor for 'safe'. But in Japanese, the word for 'safe' and for 'safety' is often the same word ('anzen'). So this is a plausible distractor for beginning EFL students in Japan, and most likely one that the makers of the TOEIC know about and put on their tests.
Try a variety of different distractors. Use semantic distractors. Use grammatical distractors. Use phonetic distractors. Use distractors that are about the same length and of around the same frequency of use. Use distractors that come from the same texts as the test items. Remember, what is a plausible distractor for someone who is at the beginning level of EFL may be hard for you to anticipate if you are a fluent user of English.
7. Keep accumulating word lists, distractors and test questions so that you can:
-give quizzes or tests of at least 35 questions or more (preferably 50);
-give quizzes or tests at least several times throughout the term.
8. Increase your total collection of test questions by paraphrasing, since students need to get familiar with the most common ways of basically saying the same thing.
9. Make the questions and the tests 'organic' to the type of class you are teaching and the content of its syllabus. For example, if the class is a very low-level 'General English' class, try using the simplest English possible when writing your questions. If the course is a listening one, try using some short dialogues instead of single sentences. Look at current TOEIC Listening questions to see relevant examples. If the course focuses on reading, try short paragraphs instead of just single sentences. Also, make the questions more realistic, relevant and communicative by basing them on things like: facts about your local city, facts about your university campus, current events, etc. For example, if a course is Business English, why not try writing questions based on the week's business news (e.g., Toyota in trouble over safety, recalls millions of cars)?
10. Some argue that we should avoid cultural bias in our test items. However, the TOEIC, for example, assumes that students are quite familiar with American culture (often dressed up as 'global business culture'). If multiple-choice questions are given as vocabulary practice, the questions might also help introduce students to cultural learning points and help them to build up background knowledge about many aspects of American culture--or other important anglophone cultures, such as those of the UK, Canada, Australia, Ireland, New Zealand, etc. It might depend on the goals and study-abroad opportunites your students have. I object to the American-bias of the TOIEC, but that doesn't mean it is going to go away.
11. The week before giving a multiple-choice quiz or exam, practice a set of examples with the students. Also, if they are new to identifying and studying key vocabulary on their own, give them a larger list of words to study from which you will randomly select your smaller set of test items.
Mutiple-choice Question Examples
(Note: the correct answer will always be 'a' in these examples, the distractors b, c, d.)
In this next section we will look at basic types of multiple-choice questions and use them as an opportunity to show distractor types as well (e.g., semantic, grammatical, phonetic, inflectional, derviational, etc.).
A. Multiple-choice basic sentence
ex. 1
The weather at the beginning of this week started very cold, but it should ______________ milder by the weekend.
a. become b. became c. begin d. complete
ex. 2
After work I like to watch T.V. programs in order to ____________.
a. relax b. look c. discuss d. describe
ex. 3
(Note: the TOEIC tests this sort of item a lot.)
To save electricity, when you leave the room, please turn ______________ the light.
a. off b. in c. around d. at
B. Multiple-choice dialogue
ex. 1
A: How is it _______________?
B: Oh, not bad. Yourself?
a. going b. doing c. meeting d. go
ex. 2
A: Do you think we should take our umbrellas?
B: Yes, the weather ______________ says it is going to rain soon.
a. forecast b. forest c. fortune d. cloud
C. Multiple-choice single sentence, match the underlined word with its synonym
ex. 1
Sony Corporation is a famous electronics manufacturer that was __founded__ in 1945.
a. established b. forced c. focused d. ordered
ex. 2
Sony Corporation __manufactures__ many different types of consumer electronics in its factories in Japan and overseas.
a. makes b. removes c. explains d. thinks
D. Multiple-choice, match the synonym, dialogue
ex.
A: Mommy! Mommy! I'm so hungry I can't stand it!
B: O.K. Calm down. I'm __preparing__ lunch right now.
a. making b. putting c. causing d. crying
E. Multiple-choice definition sentence
ex. 1
If you ______________ something, it means your habit is to like it.
a. prefer b. defer c. refer d. infer
ex. 2
_____________ is an overall economic condition of falling commodity and asset prices.
a. Deflation b. Inflation c. Decision d. Exhaustion
F. Multiple-choice definition sentence, matching synonym
ex.
Deflation is an overall economic condition of 'falling' commodity and asset prices.
a. declining b. increasing c. deciding d. protesting
G. What is ________________?
(Note: Here you give three examples illustrating the meanings, uses, nuances and common collocations of a key vocabulary item. This type of question could be very difficult to write distractors for if you try to avoid distractors that might work in one example. But remember, the correct answer has to work in all three examples, so you can use other words that work in only one sentence as distractors (for example, fix, as in 'fix a flat' instead of 'change a tire').
ex.
-Some people like to eat the same food every day for lunch, but others like to ______________ it and eat a variety of dishes.
-On the way to work today, my bicycle had a flat tire, so I had to _____________ it.
-The weather forecast in the newspaper says today will be fine but will then ____________ to rain later tonight.
What is _______________?
a. change b. turn c. fix d. return
Other question types are possible. For example, you could use a short paragraph instead of a sentence or a dialogue.
Conclusion
Many EFL teachers dislike the multiple-choice question and tests made from it. However, if you want a manageable way to assess and recycle vocabulary in large EFL classes, it is one of the best types of tests. It can be applied to all types of EFL classes, such as speaking/conversation, listening (especially standardized test listening), reading, vocabulary study, grammar review, etc. Moreover, if you teach EFL in Japan you might find that your students are not that familiar with alternative varities of multiple-choice questions for EFL, or tests, questions and tasks written only in English. .
If you write a variety of questions and stick to the syllabus and the most frequent vocabulary of English, you will probably be helping your students more than using methods of assessment that can not be scored objectively or test (punish or reward) language abilities that fall outside the scope of a course, its syllabus and textbooks. Multiple-choice quizzes and tests in English-only will also help prepare your students for important English proficiency exams like TOEIC.
Note: All content is copyright (c) 2010 of Charles Jannuzi and ELT in Japan, but is available for re-use, re-distribution and dissemination under the Creative Common license ( http://creativecommons.org/licenses/by-nc-nd/3.0/ ).
I can find no references to the average level of proficiency in speaking, listening, reading and/or writing at which various groups of EFL-learners(users?) tend to be functioning. There are well-established and validated, behaviorally anchored scales for rating samples of behavior in various domains. And there is evidence that when such scales are adapted for self-assessment, EFL-learners/users are able to place themselves in much the same levels as they are likely to be placed by trained and experienced raters.
ReplyDeleteKen Wilson
ETS (USA) retired.
Note: http://search.japantimes.co.jp/cgi-bin/fl20090811zg.html
provides some interesting information about the TOEIC.
kw
These are good points Ken. And this is a major concern if you teach EFL.
ReplyDeleteOne issue with 'scales' is, On what population(s) were they normed?
The second issue is: In Japan it's possible for the 'average' to be below the range of validity for assessment. I encountered this with a popular 'writing assessment' tool that was normed on North American high school populations. Most EFL students at my university simply came in under and outside the scale.
I can give you an indicator of the level of EFL at my university. A couple years ago several hundred volunteered to take the TOEIC. The average score was around 270. Now generalizing that to the entire university population or to EFL students in Japan or at Japanese universities is problematic. Was the sample dragged down by weak students simply taking the TOEIC offered because it was free? Or was it elevated because this self-selected population actually represented a sub-group that is more serious about EFL study than the 'average' university student? Or did it comprise a large and random enough sample for me to say that the average student at my university scores under 300 on the TOEIC?
I get the feeling that a lot of my students would score below the valid range of the TOEIC and that is why the 'Bridge TOEIC' has been devised.
Now with the popular EIKEN/STEP, a test 'native' to Japan. It is a classic 'criterion-referenced' test, so 'norming' may not be such an issue. The issue, though, is students electing to take it but chosing the wrong level of EIKEN.
Finally, about 'around Asia' or 'world comparison' averages for Japan and the TOEIC. These are always problematic because we are talking about 'self-selected' populations, and we have to ask what is the/what are the dynamic(s) behind people choosing to take the TOEIC. In the case of Japan, it tends to be more affluent than most EFL markets in Asia or Latin America, and so it is quite possible that many students here take the TOEIC as a basic diagnostic but have not prepared much for it.
Still, overall, the results are not impressive.
See my piece on why ELT fails in Japan at this site.
Hi CEJ,
ReplyDeleteThank you for introducing my blog on your blog; I greatly appreciate it. I'm also looking forward to your upcoming article on semantic mapping activities for the speaking class.
Cheers,
eisensei
Hi Eisensei
ReplyDeleteIt's so nice to get comments that are not a spammer. Let's keep our 'ELT in Japan' blogs in synch and working together. I get the feeling you and I are experiencing a lot of the same sort of teaching situations, etc.
I hope to get the full issue (#2) out by the end of this month. What to do about graphics/figures within the blog format is slowing me down a bit. But I've got two solutions in mind.