How We Build A Practice Exam In Six Easy Steps

September 1, 2009 at 4:15 pm | Posted in Transcender news | Leave a comment
Tags: ,

Several people have asked me recently, “How do you go about creating a practice exam for a certification test?” Do we pull questions out of a dirty basement with a pitchfork? Absolutely not. The process is very simple in concept (if not execution).  First, the vendor announces that there is a new certification exam and publishes a prep guide that describes the test’s objectives and sub-objectives and, if we’re lucky, gives a ratio of how many questions per objective we can expect (such as 9% storage solutions, 24% WAN security, 8% installation and upgrades, etc.). We use that to create a blueprint of what our practice test will cover.

The prep guides from the vendor are invaluable for you when studying for an exam. They are also extremely important for us when creating our product. If the prep guide is very detailed, with lengthy sub-objectives and many topics listed, it’s easier to build a practice exam. I can create a group of questions that relate to each topic, then adjust the item counts after the vendor publishes their exam to match our product with the vendor’s topic coverage ratios.

In the past, this was not so easy.  Most vendors listed hardly any sub-objectives, and the objectives were so vague you would have thought they were written by a minimalist bohemian artist in Greenwich Village. When you take an exam, you want to know what you are going to be tested on. You don’t want to guess.  You’re paying upwards of $125 testing fee and devoting many, many hours of your time to this endeavor. The least, and I do mean least, the vendor can do is tell you what you’ll be tested on, because there can be a LOT of territory to cover.

After the prep guide is released, we begin writing questions.  We do not create all the questions for the test during this phase; instead, we try to write at least one question for each subobjective’s topic.  Once we have made the first pass through all the test topics, we begin the third phase of the exam creation procedure, which is taking the beta test (when available).

All content developers working on the practice test (and some who aren’t) take the beta test for the exam. This test gives us an indication if we are on the right track in our practice exam creation.  We look at the types of questions being asked: Are there hotspot-clicks or drag-and-drops? What is the average length of a question?  Are there any graphics?  This information is critical in giving our customer a practice test that delivers the look and feel of the actual exam. We also check the beta for any topic that was not listed on the prep guide, so we can incorporate those knowledge areas into our practice test. We DO NOT – hang on, that’s worth repeating – DO NOT recreate, republish, copy, or plagiarize the exact questions on the test. Ever. What we do is to ensure that we cover each topic asked on the test as thoroughly as possible.  As much as I want you to pass the test, and I do, just take a look at our money back pass guarantee, I also want you to learn something along the way that may help you AFTER you pass your exam.

Once we have created 3 to 4 times as many practice test questions as you’ll see on the live exam, we are ready for the fourth phase, the technical edit. During this phase, other SMEs examine the questions to ensure technical accuracy. They also ensure that our practice items are testable and current. Very often, when a new exam is announced, only a beta version or a certain release candidate of the software is actually available (such as Windows 2008 R2 RC). Sometimes the vendor may change or delete a feature in the final market release, thus killing some of the greatest questions ever written about a beta feature. When that happens, we pick up the shattered pieces of our heart and write new questions to ensure every topic is covered (sticking the old questions in a vault in case the vendor resurrects the feature).

(Incidentally, for the flip-side look at how Microsoft develops their certification exam, read Liberty Munson’s series in the Born to Learn blog.)

The fifth phase is language edit, in which a language editor rips your hard work to pieces. Just because they majored in English, wrote their master thesis on Western literature, and can recite the works of Shakespeare, they censor and cut your work to ribbons.

For example, take a seemingly harmless and, shall I add, extremely creative item such as this:

“You are the administrator for an Active Directory domain in post apocalyptic wasteland. You have a remote site that has a single sub-domain that is manned by a few survivors. You have two domain controllers for the domain in the site. As the survivors in this domain fight off flesh eating zombies, the survivors need to ensure that they can log into the domain if a domain controller failed because of a zombie attack. How would one of the survivors ensure that the FSMO roles for the domain are available for the other survivors of the site?”

Generally, this question would be shredded and some not-so-nice comments would follow.  I thought I lived in a world without censorship, where your imagination could soar. I was wrong. So after some “constructive criticism,” a lot of dirty looks and a reluctant compromise or two, the question will be re-written, re-submitted, and re-edited.  The comments and changes are approved by the content developer and everybody is happy. A war has been averted….this time. (Cue the menacing music.)

After the language edit phase, we import the documents that contain questions into our database system.  The questions are integrated into our test engine and the sixth phase begins, the quality control phase. Each question must display correctly, correct answer must be tagged, and all reference links must work. After QA, the practice test is published for public consumption.

Is this the end of our involvement with the practice test? Sadly, no. If it’s, hypothetically speaking of course, a Microsoft practice test that required you to know how to use another Microsoft product that just got a radically different functionality due to a new service pack, we may have to update the exam in the near future. We regularly examine what topics and questions in the practice exam may have changed due to updates of the technology, or updates on objectives by the vendors. We also address any feedback you, the customer, alerts us to regarding areas that may need a shift in content coverage, or we tinker with reference links that may have moved. It is all in a day’s work – and that was just my Monday.

-George Monsalvatge

Leave a Comment »

RSS feed for comments on this post. TrackBack URI

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Blog at WordPress.com.
Entries and comments feeds.

%d bloggers like this: