Best Principles in
the Use of On-Line Quizzing
Thomas Brothen
University of
Minnesota
David B. Daniel
University of Maine
at Farmington
Diane L. Finley, Ph.D.
Prince George's Community College
Prepared for the
Society for the Teaching of Psychology
Pedagogical
Innovations Task Force
December 2004
With
today’s Internet-based courseware, instructors have been presented with many
new options for quizzing students. This new array of choices can be daunting
even for experienced computer users. In this section of the report, we hope to
provide useful information to guide productive use of computerized quizzing in
both face-to-face and asynchronous courses. Computerized quizzes can reside on
an individual computer, on a local area network, or on the Internet. They are
variously known as Internet quizzes, web quizzes, online quizzes, electronic
quizzes, computer-based quizzes, computer quizzes, or computerized quizzes.
Most quiz delivery software today is Internet based but many of the principles
and recommendations discussed in this section can refer to more than one method
of delivery. Throughout this section we will use the general term computerized
quizzes while recognizing that most applications will be delivered via the
Internet.
Why Should
Instructors Use Computerized Quizzes?
In general, and especially at the
introductory level, short quizzes at the start of a traditional class combined
with prompt feedback have proven to be a very effective teaching strategy
(Connor-Greene, 2000). Frequent quizzing reduces massed practice and
procrastination (Connor-Greene; Maki & Maki, 2000), keeps students focused
on what they need to do (Salomon & Gavriel, 1998), and helps them decide
how their studying is proceeding (Rosenthal & McKnight, 1996). But in-class
quizzes have limited utility because of the need to be short and focused (cf.,
Brothen, Wambach, & Hansen, 2002) and they cannot practically be given more
than once without sacrificing too much class time. Further, the time it takes
to generate in-class quizzes, grade them, and provide timely and effective
feedback can place significant demands on the instructor.
Administering
computerized quizzes outside of class via the Internet can be a tool for
efficiently providing the benefits of quizzing (Brothen & Wambach, 2001,
2004; Daniel & Broida, 2004, Daniel and King, 2003). Computerized quizzes
are more easily scored so they reduce grading time once created. Lectures may
be more effective if students have done the reading before hand and, if students
are required to complete quizzes before coming to class, they may be more
likely to have read the chapters (Connor-Greene; Daniel & Broida). Students
can complete computerized quizzes multiple times over the same material and
this mastery approach (Bloom, 1976) may produce greater learning. Many
computerized quiz programs are adaptable to a mastery learning approach and the
right kind of quiz structure facilitates this (Maki & Maki, 2001). Taking
multiple quizzes can increase fluency with the material (Johnson & Layng,
1992) and is ultimately related to course success (e.g., Daniel & King).
Quizzes delivered asynchronously that are self-grading and give feedback to the
learner also free up instructors’ time to interact with students in ways that
have been shown to be particularly effective in both distance courses and
standard face-to-face courses (Bernard et al., 2004).
Functions of Computerized
Quizzes
Computerized quizzes can be used as
formative or summative assessments. If quizzes are to function in a summative
way, there are several issues to consider. If used to determine grades, they
must be “secure” and resistant to what the instructor might consider to be
cheating. An obvious way to deliver them is in monitored computer classrooms.
Another way is to conceive of them as “open book” quizzes or to set time limits
for delivery off site. These and other techniques can increase instructors’
confidence that students actually know something about the material they are
being tested on. An additional possibility is to integrate quizzes into a
sequence of activities that increase instructor confidence that students have
mastered the material. Making accessing the summative quiz contingent on
achieving mastery scores on open book, formative practice exercises that have
little or no point value is one way to do this. For example, students may first
complete a fill in the blank exercise with key words from important sentences
that good students should highlight, and then take a multiple choice practice
quiz. Both would require mastery scores to go on to take the chapter quiz.
Most commonly, computerized quizzes
are conceived of as a formative assessment to help students assess their
current mastery of the material and guide studying. Many publisher-provided
computerized quiz programs are marketed as formative assessments and actually
seem to function in this manner. However, instructors must structure their
courses to require students to use such tools for them to be effective. It is
important to point out, however, that the mere availability of quizzes does not
guarantee their effective use. Even though, when encouraged, students tend to
prefer the availability of computerized quizzes and may even use them more than
other available pedagogical tools (Gurung, 2003), their use does not
automatically lead to improved exam scores (Brothen & Wambach, 2001, 2004).
In summary, students may not even use such tools (Daniel & King, 2003) and
if they do, their use does not necessarily lead to improved learning,
especially using the default settings of many publisher-provided course
management cartridges (Daniel & Broida, 2004).
The most important feature
necessary for quizzes as formative assessment is their ability to provide
prompt and accurate feedback. By its very nature, feedback is designed to
influence further learning. With that in mind, the useful structuring of
feedback is no simple matter. Bangert-Drowns, Kulik, Kulik, and Morgan (1991)
have shown that giving students access to practice exercise answers not contingent
on their answering the questions, is worse for their learning than giving them
no study aids. Thus, the typical study guide that has quiz answer keys at the
back may be attractive to students but counterproductive to learning.
Similarly, lenient quizzing parameters may allow students to utilize strategies
that do not optimize learning (Brothen & Wambach, 2001, 2004; Daniel &
Broida, 2004). Students may simply look up answers in the glossary or textbook,
take the test in groups, or use other work-avoiding strategies. In addition,
students who avoid mastering the text in favor of repeatedly taking quizzes and
finding answers (a “quiz to learn” strategy) tend to score lower on exams than
students who use quizzes as a formative assessment to guide their own learning
(Brothen & Wambach, 2001)
Computerized quizzes can deal with
potential student misuse if they are structured appropriately (see below).
Early versions of computerized study guides featured multiple choice quizzes
that allowed students to click on” a,” then “b” and then “c,” etc. until they
got the correct answer. This guessing game likely convinced students that they
either really knew the answer or had learned something about the material after
stumbling upon the correct answer. The typical practice quiz delivered today
via publishers’ Internet sites does not improve on this very much. Such quizzes
usually consist of 10 to 15 items that do not vary. Students can eventually
score 100% on them without doing any studying. The “magical” moment for students
of seeing a multiple choice item that could be on their exam with the correct
answer highlighted is a powerful incentive. It is not easy to convince them
that the hard work of reading, studying, and restudying after receiving
feedback is the better approach.
Of course, a quiz can function as
both summative and formative assessments as some hybrid uses have shown. The
key to quizzes functioning in this way is feedback. For example, if students
get three attempts at a quiz with their best score counting towards their
grade, helpful feedback and effective use of it by students to restudy can
result in improved learning at the same time their learning is evaluated for
assigning grades.
Practical Suggestions for Creating and Using Computerized
Quizzes
Computerized quizzes offer
instructors numerous options for improving and assessing learning but students
can subvert the instructional purposes by using online search engines,
glossaries, and other methods to get answers. We recommend the application of
psychological principles to structure the learning environment. Taking a
structured approach to using technology in an educational design is the most
significant technology available to instructors.
Based on
the research cited above and the combined experience of this task force’s
members, we make several suggestions for psychology instructors considering
using quizzes delivered electronically, especially those delivered via course
management systems over the Internet. Each of these suggestions and others can
be obtained from the references cited below.
How Many Items Do I Have to Create for My Computerized
Quizzes?
Correct answers on quizzes are very
important to students. Hindsight bias is one operative concept here. Seeing a
correctly keyed answer, no matter how obtained, is highly attractive to
students. They are too often more interested in getting the correct answer than
even understanding the question. Instructors should try to ensure that students
engage in the appropriate behaviors to utilize the quizzes to improve concept
mastery. Those behaviors are proper reading, studying, a good effort on
quizzes, asking questions, etc. In order to greatly reduce the likelihood of
students memorizing or sharing answers on a quiz, large pools of objective
items from which quizzes are randomly chosen are crucial to good learning
results. The pool should cover all the concepts from the unit being tested.
This might require instructors to devote nearly all of a textbook publisher’s
test item bank to their computerized quizzes. It is also possible to use test
banks from other textbooks as long as you verify that the concepts taught are
covered in the text or in class. There are also a number of commercial
computerized quizzing products that allow you to import quiz questions created
in word processing software, etc. In most course management systems, it is
possible to delete questions that are irrelevant or poorly constructed from an
imported test-bank. Further, some instructors prefer to delete the exact
questions used on exams from the quiz options although if the pool is large
enough this is not absolutely necessary.
Setting the quiz delivery software to randomize question
choice eliminates some concerns about cheating. A potential drawback to
randomization is that students may get quizzes of unequal difficulty or an
uneven sampling of concepts. Some course management systems, (e.g., WebCT)
allow the instructor to create blocks of related questions from which a
designated number will be randomly selected. For example, if I designate 10
questions of medium difficulty on a particular concept, I can have the program
randomly choose 2 of these for each quiz.
How Much Time Should I Allow For My Quizzes?
Time limits discourage using the
unread textbook as a primary reference. More specifically, they are necessary
to keep students from taking inordinate amounts of time searching the textbook
to get the correct answers for one quiz of 10 items and thus convincing
themselves they know the chapter or section of material. Combining time limits
with increased question difficulty is another effective strategy. The time-worn
metric of one minute for each item in a multiple-choice test is clearly too
long. We recommend 30-40 seconds per question, at most, depending upon item
difficulty. As an alternative to discouraging the use of the text on quizzes,
instructors might make the quizzes difficult enough that answering the
questions correctly, even with the textbook, is indicative of effective student
learning.
Of course, disability issues are a
concern. Because a common accommodation for students with disabilities is time
and a half on tests, instructors should decide upon the metric they will employ
so they have a rational for adjusting it for some students. Instructors should
also recognize that some ISPs may timeout students who take too long online.
Some rules addressing this possibility need to be considered when setting up
online quizzing.
Should My Quizzes Count Toward Students' Grades? If so,
How Much?
Requiring students to achieve a
minimum score to receive points (e.g., scores of less than 6 out of 10 count as
0) also increases students’ sense that the stakes are high enough that they
should be prepared for the quiz. Making the quizzes count for a relatively
small part of the grade and having high value other forms of assessment such as
term papers or a monitored final exam are other possibilities.
What Do I Want My Quizzes to Accomplish? Feedback and
Practice Quizzing
As discussed above, computerized quizzes
provide an excellent opportunity for students to receive formative evaluation
of their learning progress. Thomas and Rohwer (1986) in their executive
monitoring model for studying, suggest that students appraise their need for
further study, deploy strategies to meet those needs, and assess their learning
progress. Instructors who provide practice quizzes give students a tool to
improve their learning in this way. In addition, the availability (and maybe
the initial requirement) of practice quizzes allows the student to become familiar
with the format of the quizzes you are employing. Many students have not used
computerized quizzes and will not do so effectively if they seem difficult or
time-consuming to use, even though they might prove to be useful. We also
recommend that you provide some guidelines for how students can utilize quiz
results to guide studying.
As discussed above, we feel that
instructors need to do some thinking about whether they want to use quizzes in
a formative way, summative way, or both. If used as a formative assessment,
"practice" quizzes must provide feedback. The best feedback gets
students into the textbook, lecture notes, etc. Giving students the right
answer is not optimal. Simply reproducing the question with the student's
answer marked right/wrong and instructions for students to go find the right
one is better (and probably realistic for large item pools). To benefit most,
students should be doing the work of learning. More informative feedback
telling students they got an item about a particular concept wrong and a page
number where this concept is explained is a good and feasible technique.
Summative uses can also be effective teaching tools if students get more than
one attempt. Of course, feedback is necessary in these hybrid situations as
well. It is important to note that students often see the task differently and
interact with the quizzes accordingly. For example, some students will take
practice quizzes on a chapter 30 times if it is just practice. But if the quiz
is worth points, students may use as many minutes as they have available and
may employ techniques that compromise learning to optimize their scores. In any
case, focusing on the quiz as a learning tool only will not be very effective for
students if they do not also return to the text or notes to master the content
before the next quiz.
How Many Attempts Should Students Get?
Multiple tries on quizzes are
important if the quizzes are to serve as a formative assessment to guide
learning. For quizzes that count towards grades, the best score out of several
attempts is a way of getting students to keep trying for mastery and to
ameliorate the effects of a “bad draw” in the random selection of items. To
reduce the chances of a “good draw” of all easy items, instructors should avoid
including too many easy items in the pool. Some computer-based instructional
systems are designed to structure quizzes by difficulty of items, etc. (e.g.,
Maki & Maki, 2001) but most are not. A high percentage of very difficult
items in the pool is justified if one assumes that students will essentially
make the quizzes open book.
Costs of Using
Computerized Quizzing
Depending
upon a number of variables, the considerable benefits of computerized quizzing
may be balanced by the potential costs to be incurred. While these costs will
vary depending on your particular circumstance and resources, the most common
considerations are listed below.
Instructor Time
The
highest cost to the average instructor will be time. It takes time to develop
the quizzes and time to work with students who have difficulty interacting with
the programs. Further, as mentioned above, most publishers do not deliver the
quizzes in a format proven to positively impact learning, so adequate time must
be devoted to properly structuring the quiz parameters. We believe that the
greatest investment of time is in the very beginning. Once the quizzes are
created and structured properly, most course management systems allow them to
be administered fairly effortlessly so they can be used in subsequent terms.
Lack of Technology and Support
A
course management system provides the necessary interface and delivery method
for the quizzes outlined in this document. However, not every campus provides a
course management system and even those that do may not have staff properly
trained in the intricacies of these programs. For most introductory level
textbooks, publishers gladly provide access to such systems with support via a
toll-free number. It is important to note that, although publishers provide
access to such systems, they seldom structure their offerings in a
pedagogically sound manner. Most likely, you will have to negotiate for the
availability of the entire test-bank within the system and appropriately
structure the quizzes yourself.
Student Start-Up
Students
unfamiliar with such systems often need to be taught strategies to navigate
through them. It is encouraging that, most often, college-level students of all
ages and levels of familiarity with computers quickly develop the skills to
interact with the quizzes. We recommend a hand-out and devoting some class-time
to the basics. If possible, it may be beneficial to bring your students to a
computer classroom to allow them each to go through the motions in a supportive
environment. Whichever strategy you employ to help students familiarize
themselves with the procedures, setting aside extra office-hours to help
students interact with the system for the first few quizzes is a very good
idea. But, be prepared for exceptions throughout the semester. It is common for
students having technical difficulties accessing the quizzes to contact the
instructor. If the issue cannot be diagnosed online or on the phone, it might
be a good idea to ask students to try taking the quiz at a computer in the
classroom or in your office.
Conclusion
With the above suggestions in mind,
the first thing psychology instructors might consider before relying on any
pedagogical device is if what they are planning to do is based on sound
educational practice applied to their particular situation. The best way to
proceed in the use of computerized quizzing, then, is to first clarify your
personal teaching objectives. Clear objectives help guide you in constructing
good quizzes and procedures that follow sound guidelines. This will set the
stage for you to evaluate the effect of those quizzes in the context of your
own class.
References
Bangert-Drowns,
R., Kulik, C., Kulik, J., & Morgan, M. (1991). The instructional effect of
feedback in test-like events. Review of Educational Research, 61,
213-238.
Bernard, R. M.,
Abrami, P. C., Lou, Y., Borokhovski, E., Wade, A. Wozney, L. et al. (2004). How
does distance education compare to classroom instruction? A meta-analysis of
the empirical literature. Review of Educational Research.
Bloom, B. (1976).
Human characteristics and school learning. New York: McGraw-Hill.
Brothen, T.,
& Wambach, C. (2001). Effective student use of computer-based quizzes. Teaching
of Psychology, 28, 292-294.
Brothen, T.,
& Wambach, C. (2004). The value of time limits on Internet quizzes. Teaching
of Psychology, 31, 62-64.
Brothen, T.,
Wambach, C., & Hansen, G. (2002). Accommodating students with disabilities:
PSI as an example of universal instructional design. Teaching of Psychology,
29, 239-240.
Connor-Greene, P. A. (2000). Assessing and promoting student
learning: Blurring the
line between teaching and testing. Teaching
of Psychology, 27, 84-88.
Daniel, D. B., & Broida, J. P. (2004). Using web-based
quizzing to improve exam
performance: Lessons learned. Teaching of Psychology.
Daniel, D. B., & King, J. (2003, January). What Works
in General Psychology? The
differential effects of study
guides, web-based activities and web-based quizzing on exam scores. Poster
presented to the meeting of the National Institute of the Teaching of
Psychology, St. Petersburg Beach, FL.
Gurung, R. A. R. (2003). Pedagogical aids and student
performance. Teaching of Psychology, 30, 92-95.
Gurung, R A. R., & Daniel, D. B. (2005). Evidence-based
pedagogy. Best Practices for
Teaching
Introductory Psychology. Lawrence Erlbaum & Associates Press,
Mahwah,
NJ.
Johnson, K. &
Layng, J. (1992). Breaking the structuralist barrier: Literacy and numeracy
with fluency. American Psychologist, 47, 1475-1490.
Maki, W. S., & Maki, R. H. (2001). Mastery
quizzes on the Web: Results from a
Web-based introductory psychology course. Behavior
Research Methods,
Instruments, & Computers, 33, 212-216.
Maki, W. S., & Maki, R. H. (2000). Evaluation of
a Web-based introductory
psychology course: II. Contingency management to increase
use of on-line study aids. Behavior Research Methods, Instruments, &
Computers, 32, 240-245.
Rosenthal, G. T., & McKnight, R. (1996). What do
introductory psychology students know about their examinations, and when do they
know it? Journal of Instructional Psychology, 23, 137-144.
Salomon, G., & Almog, T. (1998). Educational psychology
and technology: A matter of reciprocal relations. Teachers College Record,
100, 222-242.
Thomas, J. W.
& Rohwer, W. D. (1986). Academic studying: The role of learning strategies.
Educational Psychologist, 21, 19-41.
ไม่มีความคิดเห็น:
แสดงความคิดเห็น