The World According to Mitch

Because of my involvement I am often asked how certification exams are created. For the most part I have not answered the questions because of the secrecy behind it, but recently I came across this great outline on the Microsoft MCP site. I hope it helps! -M

At the heart of the Microsoft Certified Professional (MCP) program are carefully developed, computer-administered exams that measure your ability to perform a specific job function or set of tasks in the real world.

Certification exams measure real-world skills

Every job function requires different skills. MCP exams test the specific skills required to perform the necessary job functions in areas like systems engineering, database administration, and solutions development. These rigorous exams go well beyond testing your knowledge of a product or terminology. That's because in the real world, you're rarely called upon to recite a list of facts. Instead, you need to apply your knowledge to a situation, analyze technical solutions, solve problems, and make decisions.

Top of page

The exam development process

To ensure the validity and reliability of our certification exams, Microsoft adheres to a rigorous exam-development process, consisting of eight mandatory phases:

The following paragraphs describe the eight phases of exam development.

Phase 1: Job analysis
The job analysis is a breakdown of all the tasks that make up a specific job function, based on tasks performed by people who are currently performing that function. This analysis includes identifying the knowledge, skills, and abilities that relate specifically to the performance area being certified.

Phase 2: Objective domain definition
The results of the job analysis provide the framework used to develop objectives. The development of objectives involves translating the job function tasks into a comprehensive set of more specific and measurable knowledge, skills, and abilities. The resulting list of objectives, or the objective domain, is the basis for the development of both the certification exams and the training materials.

Phase 3: Blueprint survey
The final objective domain is transformed into a blueprint survey in which contributors—information technology (IT) professionals who are performing the applicable job function—are asked to rate each objective. Contributors may be selected from lists of past MCP candidates, from appropriately skilled exam development volunteers, and from within Microsoft. Based on the contributors' input, the objectives are prioritized and weighted.

The actual exam items are written according to the prioritized objectives. Contributors are queried about how they spend their time on the job, and if a contributor doesn't spend an adequate amount of time actually performing the specified job function, his or her data is eliminated from the analysis. The blueprint survey phase helps determine which skills to measure, as well as the appropriate number and types of items to include on the exam.

Phase 4: Item development
A pool of items is developed to measure the blueprinted objective domain. The number and types of items to be written are based on the results of the blueprint survey. During this phase, items are reviewed and revised to ensure that they are:

Items that meet these criteria are included in the initial item pool.

Phase 5: Alpha review and item revision
During this phase, a panel of technical and job function experts reviews each item for technical accuracy. The panel then answers each item, reaching consensus on all technical issues. Once the items have been verified as technically accurate, they are edited for clarity.

Phase 6: Beta exam
The reviewed and edited items are collected into a beta exam pool. During the beta exam, each participant has the opportunity to respond to all the items in the pool. Based on the responses of all beta participants, Microsoft performs a statistical analysis to verify the validity of the exam items and to determine which items will be used in the certification exam. Once the analysis is complete, the items are distributed into multiple parallel forms, or versions, of the final certification exam.

Phase 7: Item selection and cut-score setting
The results of the beta exam are analyzed to determine which items should be included in the certification exam. This analysis is based on many factors, including item difficulty and relevance. Generally, the desired items are those that were answered correctly by anywhere from 25 percent to 90 percent of the beta exam candidates. This helps ensure that the exam consists of a variety of difficulty levels, from somewhat easy to extremely difficult.

Also during this phase, a panel of job function experts determines the cut score (minimum passing score) for the exam. The cut score differs from exam to exam because it is based on an item-by-item determination of the percentage of candidates who answered the item correctly and who would be expected to answer the item correctly. The cut score is determined by a group of experts.

Phase 8: Live exam
MCP exams are administered by Prometric, an independent testing company. The exams are available at testing centers worldwide. For more information, e-mail MCPHelp@microsoft.com or call (800) 636-7544.