Examinee and Item Reports

Informative reports are essential to the management of an assessment program. The fourth tab (Figure 8.1) of FastTest provides functionality to analyze important aspects, providing the information needed to make decisions. For example, a summary report of the item bank can be easily produced, which highlights areas of the bank most in need of new items. Reports that are used often can be saved for quick access at later date. Continuing with the same example, the item bank summary report could be saved, and re-run weekly as item development continued to fill in weaker areas of the bank. Also, examinee and item reports that have recently been generated can be found and downloaded under Recent Exports.

Figure 8.1: Report Manager Tab

There are four categories of reports available for use:

• Examinees
• Items
• Tests
• Users
• Workspaces

Each category holds a number of reports, which are presented when the user clicks on the “Reports” button in the upper left (as shown in the image above). These are described below.

• The following table provides descriptions for various reports

Examinee Reports

Examinee Test Detail

This report gives an item-by-item summary of an examinee’s test. For conventional tests, each item is shown along with the score that the examinee received for that item. For CAT tests, the values of the θ estimate and standard error on an item-by-item basis are also displayed. The IRT parameters for each item will be shown, as well. If the CAT test uses Content Constraints, the subscore that each item was selected from will appear as well.

Figure 8.3: Examinee Test Detail Snippet

Examinee Group

The Examinee Group report summarizes all of the examinee results in a given group across all tests. It is separated into two tables. In the first table, each subgroup-test pair is listed alongside the number of examinees, the average score, and the passing rate. In the second table, data is aggregated over all examinees in the selected group and its subgroups. Each test that any of the examinees have taken appears alongside the number of examinees, the average score, and the passing rate.

Examinee Group and Test Summary

The Examinee Group and Test Summary Report summarizes the scores of all examinees in a given group that have taken a given test. If the test has a cut-score, it will show whether each examinee passed and calculate the overall passing rate. If the test has sub-scores, it will report the average score for each sub-score.

Note: This report will include examinees from the selected group as well as all subgroups of the selected group.

Item Reports

Cross-Test Item Analysis

In the classical test theory paradigm, item statistics are usually analyzed within a single test. However, in some situations it might be useful to know some statistics for an item across all tests on which it has been used. This report provides a snapshot of the item, including statistics that are recorded in the bank, as well as the number and proportion of examinees selecting each option across all tests. The Cross-Test Item Analysis Report can be especially useful when an item has been slated for review or possible retirement. For example, if the actual P value was 0.68 it would be much lower than the recorded value of 0.97, suggesting that review is appropriate.

Figure 8.4: Cross-Test Item Report Snippet

Bank Summary

The Bank Summary Report is designed to provide a complete summary of the items in a bank, which describes the status of the bank and helps to direct future development. The first two tables in this report are counts of item status and item type in each of the categories, as shown below. This helps to identify categories that have the fewest items and are in the greatest need of new items. It can also identify the categories with the most items up for review, and therefore used to direct the reviewers.

Figure 8.5: Bank Summary Report Snippet

Additional tables in this report provide frequency distributions of item usage (number of tests on which the item is used), item owners (to see which item writers are most productive), and item statistics. The final section of this report provides a simple list of the folder structure. This is useful because it can be used to copy an item bank structure without all of the items. For example, a previous year’s item bank structure could be copied from this report, and pasted into a new bank for the next year, which would automatically create all the same folders.

Item History

The Item History report provides a history of major actions on the item, such as comments and changes that bring about new versions (text and key changes). This is also useful during the item review process.

Figure 8.6: Item History Report Snippet