Generating Good Tests from the Testbank
I give several multiple choice exams in my personality course each semester, and composing those exams is challenging, time consuming, and, to be honest, not my favorite activity. At the same time, creating an exam that seems unfair or is otherwise problematic costs me in terms of class time and, if I am unfortunate enough to administer a couple of problematic exams in a semester, it can damage the class atmosphere. So, I need to be pretty good at this.
This semester (Fall, 2006), the first complete TestGen test bank for the textbook is now available, and I am using it – cautiously.
Here are some of the things I have learned to do when using the book's test bank to make my exams go more smoothly.
Testing from the Textbook And Other Sources
I start with the idea that the book generally covers much of the central material I would like students to learn. For that reason, I want to assess students' learning from the book (and supporting class lectures).
In addition, however, I like to test from the lecture. I think students who are conscientious about attending class appreciate that, and sometimes we (the class) veer off into useful topics not covered in the book.
I also use Readings in Personality Psychology, so I often want to add in a few questions from that source as well.
So, I start with the idea that my 60+ item multiple choice exam will begin with 55 questions drawn straight from the book, include about 7-10 more questions that deal specifically with in-class material, and include several more questions focused on Readings in Personality Psychology.
With that in mind, the steps I take to create the exam are to:
- get 55 good questions from the test bank in TestGen
- export them into a word file, and
- write 10-15 more questions and add them to the exam
Getting Good Questions from TestGen
Getting 55 good questions from TestGen, I found, was not entirely straightforward. For my first exam, I decided I would draw 15 questions from Chapter 1, 20 from Chapter 2, and 20 from Chapter 3. Ideally, you should be able to go into the TestGen Wizard and request that, and obtain a good test.
I did that, and the first exam TestGen returned looked pretty good, actually. But then I decided I wanted to re-weight the chapters, hoping I might get an even better test – but I did worse. I ended up with a test that greatly over-weighted the concept of structural validity in Chapter 2.
So, now I had 55 questions that were okay, but that I didn’t think were definitely good enough to use as a group. I explored several different methods to fix this. I'll describe three of them below, recommending only the last method, but I'll explain the other alterantives as well, and why I don't think they are as good.
Improving TestGen Tests: Method 1 (not recommended!)
The simplest method for getting a better TestGen test, I suppose, is to randomly reselect items and generate a second (and third, and fourth...) test. The problem with this is that the test may get better in one area and worse in another. I don’t recommend this because life is short, and our standards are high, and you may find yourself on a losing streak in terms of the test items that are generated.
Improving TestGen Tests: Method 2 (okay, but not my favorite)
The opposite of the gambling approach is to become a control freak. You can go ahead and hand-pick the items you want from the full test-bank. TestGen will let you do this, and you can get as good a test as you can make using the many questions you choose.
I think many instructors will like this approach (I have used it myself) and there is certainly nothing wrong with it.
At this point, however, I can’t recommend it because it means I have to read too many test questions to select the ones I want, and then re-read the questions after I select them to make sure I have done a good job.
Improving TestGen Tests: Method 3 (okay -- but still time consuming)
In a moment of desperation, I considered this approach (but fortunately found a better way later).
Rather than run the program again (i.e., throw the dice), I took the exam I had generated and, using the export function (under File), saved it as an RTF document (which you can edit in Word). Then, I generated a second, shorter test, and also exported that. I had then planned to substitute items from the second test to replace the items in the first, target test, that I hadn’t liked.
Note that this has the drawback of changing the items in relation to the answer key that TestGen supplies – so you need to alter the answer key as you do this as well. I hoped there was a better way.
Improving TestGen Tests: Method 4 (Recommended)
The best method, I think, is (relatively) simple. In this approach, I generated a test in TestGen with a few more questions in each chapter than I ultimately wanted. For example, I wanted 15 questions for Chapter 1, and 20 questions each for Chapters 2 and 3. So, I generated a test with 20 questions for Chapter 1, and 25 each for Chapters 2 and 3. Then, while still examining the test in TestGen, I selectively deleted my least favorite and the most redundant-seeming questions from each chapter.
The considerable advantage to this approach is that I obtained a test relatively free of irksome questions, and TestGen dynamically renumbered the test and adjusted the list of correct answers as I edited it.
Once the changes were complete, I exported the test, saved as an RTF document (which I could then edit in Word) and I was done with this portion of the test.
Exporting and Finishing the Test
Once the test is in RTF, I open it with my editor, and I then added in about 10 more items of my choice covering the course lectures, amplifying any underweighted material in the questions from TestGen, and adding in a few questions on the Readings book. Done!
Still not my favorite activity, but probably the best, most efficient way to go about this and get a good result in the classroom.
The test appeared to work well in class -- I had no complaints in a class of about 65 students.
