24981 - Perform item analysis for multiple choice tests

Sample 24981: Perform item analysis for multiple choice tests

Item analysis for multiple choice tests

Contents:

Purpose / History / Requirements / Usage / Limitations

PURPOSE:

The %ITEM macro computes descriptive statistics for analysis of data from a multiple-choice test. Each observation contains the answers from one subject to a set of questions ("items"). The data are compared to an answer key to determine which answers are correct. The score for each subject is computed as the number of correct answers. The output is very similar to that from the ITEM procedure in the SUGI Supplemental library, but several incorrect statistics have been fixed.

HISTORY:

Last updated: 26Jun92

REQUIREMENTS:

Only Base SAS software is required.

USAGE:

Follow the instructions in the Downloads tab of this sample to save the %ITEM macro definition. Replace the text within quotes in the following statement with the location of the %ITEM macro definition file on your system. In your SAS program or in the SAS editor window, specify this statement to define the %ITEM macro and make it available for use:

   %inc "<location of your file containing the ITEM macro>";

Following this statement, you may call the %ITEM macro. See the Results tab for an example.

The following arguments may be listed within parentheses in any order, separated by commas:

   DATA=        SAS data set to analyze. The default is _LAST_.
                Data set options may be used.

   VAR=         List of variables representing items. Only the first
                character of the formatted value of each variable is
                used to compare with the answer key. The variables may
                be numeric or character or a mixture of both. The usual
                forms of abbreviated lists (e.g., X1-X100, ABC--XYZ,
                ABC:) may be used.

                The default is _ALL_; BY variables and ID variables
                are not automatically removed from the default list.

   FORMAT=      An optional SAS format to be applied to all the VAR
                variables.

                If FORMAT= is not specified and if formats have been
                assigned to the VAR variables in a previous DATA step,
                then those formats are used.

   ID=          List of variables to be copied to the OUT= and OUTBIN=
                data sets and printed in the grade report. These
                variables are not used in the analysis. The usual forms
                of abbreviated lists (e.g., X1-X100, ABC--XYZ, ABC:)
                may be used.

                If you specify ID=, you must also specify VAR=.

   COPY=        List of additional variables to be copied to the OUT=
                and OUTBIN= data sets. The usual forms of abbreviated
                lists (e.g., X1-X100, ABC--XYZ, ABC:) may be used.

                If you specify COPY=, you must also specify VAR=.

   BY=          List of variables for BY groups. Abbreviated variable
                lists (e.g., X1-X100, ABC--XYZ, ABC:) may NOT be used.

                If you specify BY=, you must also specify VAR=.

   SUBTEST=     Series of one or more subtest specifications separated
                by one of the characters given in the SUBSEP= argument.

                Each subtest specification contains either a variable
                list that is a subset of the variables in the VAR= list,
                or the keyword _ALL_ indicating all of the variables
                in the VAR= list. A separate analysis is performed for
                each of these variable lists.

                Each subtest specification may also contain a quoted
                string of up to 40 characters providing a title to
                identify the subtest on the printout. This title is
                also written to the output data sets in a variable
                named by the SUBNAME= argument. This string may not
                contain any of the characters in the SUBSEP= argument,
                nor may it contain two consecutive quotation marks.

                If no subtests are specified, only one analysis is
                performed using all the variables in the VAR= list.

   SUBSEP=      One or more characters used to separate subtest
                specifications in the SUBTEST= argument. The default
                is a /. Do not use blanks or commas.

   SUBNAME=     Name of the variable in the output data sets containing
                the subtest title. The default is _SUBTEST.

   RESPONSE=    Quoted string containing all the valid single-character
                responses to all the items. Any answer not in this list
                is declared invalid. There cannot be more than 200
                valid responses because of the limit on the length of
                quoted strings in the SAS system.

                The default is RESPONSE='12345'

   KEY=         Quoted string specifying the correct single-character
                answers for each item in the same order as specified
                in the VAR= argument. If you specify a quoted string
                for the KEY= argument, then there cannot be more than
                200 items because of the limit on the length of
                character strings in the SAS system. You may specify
                several quoted strings separated by the concatenation
                operator || as long as the total length does not
                exceed 200.

                If you do not specify KEY=, or if you specify
                KEY=_FIRST_, then the first observation read from
                the data set is assumed to contain the answer key,
                and this observation is omitted from other
                computations. This method of specifying the answer key
                must be used when there are more than 200 items. If
                there are BY groups, they are assumed all to have the
                same key, so only the first BY group should have a
                record giving the answer key.

   OUT=         SAS data set containing the ID variables, the score for
                each subject, the number of missing and invalid answers,
                and the score as a percentage.
                Data set options may NOT be used.

   OUTBIN=      SAS data set containing the ID variables, the score for
                each subject, and one variable for each item coded as
                1 for a correct answer or 0 for an incorrect answer.
                The variable names are BIN1, BIN2, ..., BINn, where n
                is the number of items.
                Data set options may NOT be used.

   OUTITEM=     SAS data set containing item statistics.
                Data set options may NOT be used.

   SCORE=       Name of the score variable in the OUT= and OUTBIN=
                data sets. The default is SCORE=SCORE.
                The name should not begin with an underscore.

   MISSING=     Name of the variable giving the number of missing values
                in the OUT= data set. The default is MISSING=MISSING.
                The name should not begin with an underscore.

   INVALID=     Name of the variable giving the number of invalid values
                in the OUT= data set. The default is INVALID=INVALID.
                The name should not begin with an underscore.

   PERCENT=     Name of the variable giving the percentage score
                in the OUT= data set. The default is PERCENT=PERCENT.
                The name should not begin with an underscore.

   PROPHECY=    The desired reliability level to be used in the
                Spearman-Brown prophecy formula. The default is
                PROPHECY=.9.

   UPLO=        The proportion of subjects to be included in each of
                the two subsets of subjects when comparing the
                percentage of occurrence of each response in the
                highest-scoring subset and the lowest-scoring subset.
                The default is UPLO=.3334.

   OPTIONS=     List of additional options separated by blanks:

                NOGRADE        Suppress printing the grade report.
                NOG
                NG

                NOHISTOGRAM    Suppress printing the bar chart of
                NOHIST         score frequencies.

                NOITEM         Suppress printing statistics for
                               each item.

                NOTES          Do not suppress notes in the SAS log
                               for various preliminary procedure
                               and data steps. This option can be
                               useful for diagnosing mysterious error
                               messages.

                NOTOTAL        Suppress computation of item-total
                               correlations.

                NOUPLO         Suppresses computation of percentage
                               of responses by subjects in the
                               highest-scoring and lowest-scoring
                               subsets.

                NOX            Suppress printing extra columns for
                               omitted and invalid responses for
                               each item.

LIMITATIONS:

Number of responses <= 200.
Variable names should not begin with an underscore.
Spurious warning from PROC CONTENTS may occur when OPTIONS OBS= is specified.

These sample files and code examples are provided by SAS Institute Inc. "as is" without warranty of any kind, either express or implied, including but not limited to the implied warranties of merchantability and fitness for a particular purpose. Recipients acknowledge and agree that SAS Institute shall not be liable for any damages whatsoever arising out of their use of this material. In addition, SAS Institute will provide no support for the materials contained herein.

EXAMPLE:

These examples include the CORR procedure to validate the summary statistics for the test score and the item-total correlations. The ALPHA option is also used to validate KR20 (which is equivalent to Cronbach's alpha) and the item-remainder correlations.

 PROC FORMAT; VALUE $Q
    '0'='1' '1'='2' '2'='3' '3'='4' '4'='5'
    '5'='1' '6'='2' '7'='3' '8'='4' '9'='5';
 RUN;

 DATA TEST1;
 INPUT SSN $ 12-20 (Q1-Q33) (33*$CHAR1. );
 BY = 1 + (_N_ > 21);
 FORMAT Q1-Q33 $Q.;
 CARDS;
101        999999990372715151617250838251535163517182
101        200465598350507250617350638261536171517183
101        233868923373525163817250738063517251538173
101                 372527163618252828251516161517352
101        236909800362505173617250718263837161537361
101        235884995360505282717350836051715161538181
101        270606129370835151817153736051835071537171
101        188447596372517162617252738251535151537152
101        054389259372515250817350738351535071538153
101        130463838373825172618250838050818171537183
101        165463286371517253616253838251516253836251
101        2176696113728 5383608353836261737051538153
101                 3528351526151508382518151618171 2
101        228761635382825261717272835151525171538351
101        236869018372525062817251725251715250538182
101        206422209372825263817250838053515170537182
101        081481581363815151717250825251635161517172
101        236562064372635263618170725252525271537363
101        296549834373515050817153738061815273518151
101        234908052372515162817150836280535071517151
101        407681435372815163617150738261815071517151
101                 271535281817251 38251515161537161
101        21728907337261"151618 50725281535171507352
101                 37252"151617252728151737161807360
101        234171262370735162817152725150737071518373
101                 360505062817052838081815251537361
101        059486239372818262617252838051515061517152
101        232721382372718051617253828051735173537083
101        17638728 372835173617270838050516361537152
101                 371527162617250838251816371707383
101        194381114372537151617252828251715161537162
101        198424787372835061617272838051738161517162
101        190521297372718162817153718251835171537152
 ;

%inc "<location of your file containing the ITEM macro>";

title 'ITEM macro: Original test data';

title2 'Test 1';
%item(
      data=test1,
      var=q1-q33,
      id=ssn);

title2 'Test 2';
%item(
      data=test1(firstobs=2),
      var=q1-q33,
      response='12345',
      key='43332121222331144'||
          '4312141224123243',
      id=ssn,
      out=outtest,
      outitem=outitem,
      outbin=outbin,
      score=right,
      missing=blank,
      invalid=bad,
      percent=pcnt,
      prophecy=.75,
      uplo=.27);

proc print data=outitem; id _item_ _name_ _respon_; run;
proc print data=outtest; run;

proc corr data=outbin alpha nomiss nocorr nosimple; var bin:; run;
proc corr data=outbin nomiss; var bin:; with right; run;


*****************************************************************;


title2 'ITEM macro: Subtests';
%item(
      data=test1,
      var=q1-q33,
      subtest=_all_ 'English Final'/
              q1-q10 'Spelling Proficiency'/
              q11-q33 'Grammar Proficiency',
      subname=_sub_,
      out=outsub,
      id=ssn);

proc print data=outsub; id _sub_ ssn; run;

proc corr data=outbin alpha nomiss nocorr nosimple;
   var bin1-bin10; run;

proc corr data=outbin alpha nomiss nocorr nosimple;
   var bin11-bin33; run;


*****************************************************************;


title 'ITEM macro: BY groups';

title2 'Two BY groups';
%item(data=test1,var=q1-q33,by=by,id=ssn);


*****************************************************************;


title 'ITEM macro: Subtests and BY groups';

%item(by=by,
      data=test1,
      var=q1-q33,
      subtest=_all_ 'English Final'/
              q1-q10 'Spelling Proficiency'/
              q11-q33 'Grammar Proficiency',
      id=ssn);

RESULTS:

Following is abbreviated output from the first run of the %ITEM macro in this example. The output would continue through item Q33.

                              ITEM macro: Original test data
                                          Test 1

                                       Grade Report

                      SSN        Score  #Missing  #Invalid   Percent

                      200465598     19       0         0        58%
                      233868923     14       0         0        42%
                                    20       0         0        61%
                      236909800     17       0         0        52%
                      235884995     16       0         0        48%
                      270606129     17       0         0        52%
                      188447596     23       0         0        70%
                      054389259     19       0         0        58%
                      130463838     18       0         0        55%
                      165463286     16       0         0        48%
                      217669611     13       1         0        39%
                                    22       1         0        67%
                      228761635     15       0         0        45%
                      236869018     16       0         0        48%
                      206422209     21       0         0        64%
                      081481581     24       0         0        73%
                      236562064     13       0         0        39%
                      296549834     16       0         0        48%
                      234908052     20       0         0        61%
                      407681435     20       0         0        61%
                                    19       1         0        58%
                      217289073     20       1         1        61%
                                    18       0         1        55%
                      234171262     12       0         0        36%
                                    12       0         0        36%
                      059486239     22       0         0        67%
                      232721382     23       0         0        70%
                      17638728      20       0         0        61%
                                    17       0         0        52%
                      194381114     23       0         0        70%
                      198424787     22       0         0        67%
                      190521297     20       0         0        61%


                              ITEM macro: Original test data
                                          Test 1

                                Summary of Test Statistics

                         Number of Items                  33
                         Number of Subjects               32

                         Mean Score                       18.344

                         Variance of Scores               11.523
                         Standard Deviation                3.395

                         Kuder-Richardson 20               0.433
                         Kuder-Richardson 21               0.279

                         Standard Error (from KR20)        2.555
                         Standard Error (from KR21)        2.883

                         Minimum Score                    12
                         Maximum Score                    24

                         Total Number of Answers        1056
                               Number Missing              4
                               Number Invalid              2

                         Spearman-Brown Prophecy (from KR21):
                               To obtain a reliability of 0.90
                               the test should contain 769 items


                              ITEM macro: Original test data
                                          Test 1

                               Test Frequency Distribution

         Raw    Standard    Per-                   Cum
        Score     Score   Centile  Percent  Freq  Freq   Each X represents 1 score

          24       667       98      3.1%      1    32   X
          23       637       92      9.4%      3    31   XXX
          22       608       82      9.4%      3    28   XXX
          21       578       76      3.1%      1    25   X
          20       549       65     18.8%      6    24   XXXXXX
          19       519       51      9.4%      3    18   XXX
          18       490       43      6.3%      2    15   XX
          17       460       35      9.4%      3    13   XXX
          16       431       25     12.5%      4    10   XXXX
          15       401       17      3.1%      1     6   X
          14       372       14      3.1%      1     5   X
          13       343        9      6.3%      2     4   XX
          12       313        3      6.3%      2     2   XX


                              ITEM macro: Original test data
                                          Test 1

     Item    1: * is keyed       1       2       3       4*      5      Omit Invalid
     Q1            Responses    0.0%    0.0%    3.1%   96.9%    0.0%    0.0%    0.0%
                   Upper 33%    0.0%    0.0%    0.0%  100.0%    0.0%    0.0%    0.0%
                   Lower 33%    0.0%    0.0%    0.0%  100.0%    0.0%    0.0%    0.0%

              Item-Total:
                   Point Bis   0.000   0.000   0.035  -0.035   0.000
                   Biserial    0.000   0.000   0.087  -0.087   0.000

              Item-Remainder:
                   Point Bis   0.000   0.000   0.087  -0.087   0.000
                   Biserial    0.000   0.000   0.215  -0.215   0.000
                   t test      0.000   0.000   0.479  -0.479   0.000
                   Prob>|t|   1.0000  1.0000  0.6356  0.6356  1.0000

     Item    2: * is keyed       1       2       3*      4       5      Omit Invalid
     Q2            Responses    6.3%   12.5%   78.1%    3.1%    0.0%    0.0%    0.0%
                   Upper 33%   12.5%   12.5%   75.0%    0.0%    0.0%    0.0%    0.0%
                   Lower 33%    0.0%   20.0%   70.0%   10.0%    0.0%    0.0%    0.0%

              Item-Total:
                   Point Bis   0.167  -0.124   0.077  -0.180   0.000
                   Biserial    0.328  -0.199   0.108  -0.444   0.000

              Item-Remainder:
                   Point Bis   0.227  -0.035  -0.047  -0.138   0.000
                   Biserial    0.448  -0.057  -0.065  -0.341   0.000
                   t test      1.279  -0.194  -0.256  -0.763   0.000
                   Prob>|t|   0.2106  0.8474  0.7994  0.4513  1.0000

      ... etc.

Type:	Sample
Topic:	Analytics ==> Nonparametric Analysis Analytics ==> Descriptive Statistics SAS Reference ==> Procedures ==> CORR

Date Modified:	2007-08-11 03:03:05
Date Created:	2005-01-13 15:02:38

Product Family	Product	Host	SAS Release
			Starting	Ending
SAS System	SAS/STAT	All	n/a	n/a

Support

Sample 24981: Perform item analysis for multiple choice tests

Item analysis for multiple choice tests

Operating System and Release Information