The Transcender Team Weblog

Multiple options beyond multiple choice: an overview of Microsoft’s new exam item types

As technologies evolve, so do the means of testing your technical knowledge. While the multiple choice standard still has its place, Microsoft and other major vendors are rapidly evolving beyond such mechanical (and easily braindumped) question formats. Microsoft has even released a catchy YouTube video on the subject:

An awful lot of research goes into the most effective question format. In the past few years we’ve seen an explosion of new item types and testing techniques. Some have been rolled out, some have been rolled back, and some are newly announced but haven’t yet been sighted in the wild.  Here are the ones encountered by the Transcender Team, with our notes on each.

Extended Matching

This item type was announced in early 2011 (Something New, Something Borrowed, Something Blue: Introducing a New Item Type on Certification Exams), but we didn’t encounter it in an exam until recently. George, our Microsoft Windows Server and SQL Server expert, first tackled extended matching on the beta exams for SQL Server 2012. Here’s what he had to say:

I encountered Build-list and reorder questions that required you to know the exact sequence in which tasks should be performed. There were also Active Screen items that required you to answer questions based on a scenario. I also saw the new “Extended Matching” questions. The Extended Matching questions looked kind of like case studies, because they were a set of multiple choice questions answered in one time frame. However, these did not have the usual four or five answer choices. No, each question had the same 14 choices.  The questions were slightly different, but the choices were the same. These question types caught me off guard and I found them completely confusing until I realized you could actually have the same correct answer for more than one question in the set.

The Extended Matching questions were like someone put a long multiple choice question, a matching question, and a pint of buttermilk in a blender, pulsed it,  left the horrible concoction on the kitchen table overnight, and then tricked you into drinking it in the morning. 

We’re hoping that you go into your exams a little more prepared than George, so we’re in the process of revising our 70-667 practice exam  and our 70-432 practice exam to include this new item type. This will give you the chance to get comfortable with how Extended Matching items are put together, and not be caught off guard on exam day.

Case studies and code case studies

Case studies (mini-tests that are timed separately from the SAMC/MAMC questions) are nothing new in the Microsoft world, but they did vanish from the testing scene for a few years, until recently (see George’s post, The Case Study Gets Its Groove Back). Because each case study has its own clock, the trick is not to let them eat into your overall exam time. However, the Code case study was a new twist on the concept. It was touted in Born To Learn last year (Code case studies: test drive our new item type for developer exams). Josh reported on this item type extensively here a few months back (They’re back: the return of the developer exam case study). They’ve been incorporated into all of our practice tests for these technologies.

As of this writing, you can still access Microsoft’s mockup code case study here: 
http://mcppoc.rangers.ms/

Short Answer Code

This item type will incorporate live coding into the exam, and as far back as our SMEs can remember, this item type is a first for Microsoft. Short answer code items were announced in October 2011  (Check Out the Short Answer Code Item Type). While we haven’t encountered this item type on a certification exam yet, here’s what we know about it so far: the item will have a field in which the candidate writes a short code segment to accomplish the task in the scenario. All the standard tools that would be available to a developer in real life (such as syntax checking) will be reproduced on the test, so in theory, you can’t trip yourself up with a simple mis-key or typo.

Have you encountered this item type yet? If so, we’d love to hear about it.

MAMC: Choose All that Apply

“Wait,” you say. “That’s not a new item type. That’s the same old multiple choice question that Microsoft (and Transcender) has been doing all along.”

Well, yes. But in the course of reviewing the most test-worthy item types, psychometricians made a surprising discovery: this classic structure is actually one of the hardest to answer without a thorough knowledge of the subject being tested.  You can read about the methods used by psychometrician Liberty Munson here, Investigating the Psychometric Performance of Our Item Types.

How many times have you encountered a multiple choice question where you weren’t sure of all the answers, but the fact that the question said “choose two” or “choose four” let you safely guess the parts that you weren’t sure about? If Microsoft has anything to say about it – and, let’s face it, they do – then this guessing technique will be ruled out. Fortunately, Transcender has used this MAMC structure in all of our practice tests, so users should be prepared to answer them on exam day.

What about simulations?

A few years ago, simulation exams were the item type of the future; almost impossible to braindump, and representing a real-world test of the user’s skills. Microsoft introduced the simulation format with the 83-640 Windows Server Configuring exam. Problems with exam delivery, though, sidelined this particular format, which reverted to the conventional 70-640.

While there may have been some setbacks, this was an excellent testing format, and it certainly shouldn’t be ruled out of future Microsoft exams. We think the live coding exams for developers represent one new direction in which to take simulations – the goal of which, after all, is to have the user perform real-life tasks.

For one last obsessive look at this subject, check out Liberty Munson’s Born To Learn post on Microsoft’s changing attitudes towards the building of certification exams (Exams Grow Up)

Happy testing!

–the Transcender Team