H. Frequently Asked Questions

• A. General questions
• B. Data preparation questions (e.g., SPSS)
• C. Investigation of Dimensionality questions (e.g., SYSTAT)
• D. IRT Models questions (e.g., BILOG, BILOG-MG, MULTILOG7)
• E. Model-data Fit questions (e.g., MODFIT)
• F. DIF/DTF questions (e.g., ITERLINK)

A. General questions

Q1. I have trouble downloading the dataset. Where can I download the whole dataset?
A1: You may download the whole dataset (dataset.zip) here.

Q2. Where can I obtain the manual for DETECT, DIMTEST, or SIBTEST?
A2: Please contact Educational Measurement Laboratory (also known as Statistical Laboratory for Educational and Psychological Measurement) or Assessment Systems Corporation.

Q3. Where can I obtain a copy of the EQUATE99 program?
A3: Please contact Frank Baker at University of Wisconsin at Madison.

Q4. How can I obtain a copy of this unpublished manuscript by Levine, M. V., & Williams, B.A. (1993). Nonparametric models for polychotomously scored item responses: Analysis and integration?
A4: The technical report eventually became part of the 1995 Applied Psychological Measurement article.

Q5. Do you provide consulting or training workshops?
A5: Please contact us through e-mail.

Back to Top

B. Data Preparation questions

Q1. I have trouble making my own dataset work. What should I do?
A1. We suggest you first try to download the dataset from our web site. If you can follow the tutorial and have no trouble using the dataset, it is likely that there is something wrong with your dataset. For example, the file format or the dataset format is not recognized by the program.

Q2. I still have trouble making my own dataset work. What should I do?
A2. At http://io.psych.uiuc.edu/irt/prep_main.asp, we suggested using SPSS to format the data; although other statistical programs may also be used. The data must eventually be saved in ASCII format as a *.DAT file in order for the IRT programs to access it. Adjust your SPSS datafile such that each response option is only a single digit wide without any decimals. Otherwise, your data format statement will have to be specified differently from the examples provided on the tutorial. For instance, we used the first four characters as the Subject ID in the tutorial. If your dataset does not  begin with a 4-digit ID (e.g., 1001 or 0001), your *.DAT file will be formatted differently.

Back to Top

C. Investigation of Dimensionality questions

Q1. Can I use other statistics software to compute tetrachoric correlations?  
A1: Of the three most popular statistical analyses packages, SYSTAT is the easiest one to compute tetrachoric correlations. SAS's PROC FREQ procedure can compute tetrachoric correlations. You may download a macro provided by SAS at http://support.sas.com/ctx/samples/index.jsp?sid=512, or refer to a FAQ provided by UCLA's Academic Technology Services' web site at http://www.ats.ucla.edu/stat/sas/faq/tetrac.htm. While SPSS does not have built-in syntax to compute tetrachoric correlations, an SPSS macro, r_tetra, is available for computing tetrachoric correlation.

Q2. While investigating dimensionality, why am I getting multiple factors?
A2: At http://io.psych.uiuc.edu/irt/dim_main.asp, we suggested using Principle Axis Factoring (PAF), rather than Principal Components Analysis (PCA), to determine the number of factors in a measure. In SYSTAT, the default factor analysis method is PCA, not Iterated Principle Axis (IPA). You may refer to http://io.psych.uiuc.edu/irt/dim_dich1.asp for more details.

Back to Top

D. IRT Models questions

BILOG for DOS/Windows questions

Q1. I have trouble formatting my BILOG syntax. What should I do?
A1: At http://io.psych.uiuc.edu/irt/modeling_dich2.asp, we provided a sample BILOG syntax and a brief description for each command. Mistakes are more likely to happen when the syntax doesn't read like "English" to you. For instance, in the tutorial, we used the first four characters as the Subject ID, followed by Responses from the first item. If your dataset is formatted differently, the numbers in the line ">INPUT SAMPLE=99999; (4A1,10A1)" have to be changed accordingly.

Q2. My BILOG run does not provide meaningful outputs like you provided on the tutorial. What should I do?
A2: At http://io.psych.uiuc.edu/irt/modeling_dich2.asp, we have suggested that you may want to look at .ph1, .ph2, and .ph3 output files. For instance, you have to make sure that BILOG reads the dataset correctly and the statistics are reasonable (.ph1), and the estimation procedure converged (.ph2).

Q3. Do I need to provide a title or comments in my BILOG syntax?
A3: It's up to you. Providing title and comment may help you identify which dataset you are running.

Q4. Do I have to use file extensions like .3PL, .PAR, etc.?
A4: You may use any file extensions you want. We used those special file extensions such that it will be easier to tell the content of the file. In addition, any text editor such as Notepad provided by Windows can read those text files easily. Just remember that if you change your file extension, you may have to modify your syntax as well.

BILOG-MG questions

Q1. What about running BILOG-MG for a single group?
A1: The IRT tutorial web site currently provides sample syntax that can be used on BILOG for DOS and BILOG for Windows (e.g., version 3.1) only. As BILOG-MG uses a slightly different syntax from the previous two versions of BILOG, you may use the examples provided here and modify the syntax by referring to the examples provided within the Help file in BILOG-MG. A sample BILOG-MG syntax is provided below.

Q2. What about running BILOG-MG for multiple groups (e.g., DIF)?
A2: When using BILOG-MG to perform multiple groups DIF analysis, the dataset has to be formatted differently. For instance, group identification code should be inserted between the Subject ID and item scores. We may have sample BILOG-MG syntax codes updated here in the future.

Q3. I got an error message that "the key for item XXX is not the same as its key in an earlier form", but I have checked the key file a million times and the key answer is similar in both forms!
A3: Please check not only your key file, but also your data file.

Q4. The *.COV files I got from BILOG-MG is different from the files you provided in the tutorial. What do those numbers mean?
A4: The format of BILOG-MG's estimated covariance file is formatted slightly different from BILOG for DOS/Windows's estimated covariance file. That is, the order for each record remains the same, but the location for each record is formatted differently. Please refer to BILOG-MG manual pages 248 and 249 for details.

Lines 1 & 2 contains the title records of the BILOG-MG run that created the covariance file. Afterwards, there are two lines for each item. The first record contains
(1-8) item name,
(9-16) subtest name,
(17-21) group indicator,
field 1: (22-33) slope estimate (a),
field 2: (34-45) threshold estimate (b),
field 3: (46-57) lower asymptote estimate (c),
field 4: (58-69) estimation of error variance for slope (SE_a²), and
field 5: (70-81) estimation of error covariance for slope and threshold (Cov_a,b).
The second record contains
(1-17) blank filler,
field 6: (18-29) estimation of error variance for threshold (SE_b²),
field 7: (30-41) estimation of error covariance for slope and asymptote (Cov_a,c),
field 8: (42-53) estimation of the error covariance for threshold and asymptote (Cov_b,c), and
field 9: (54-65) the estimate of the error variance for lower asymptote (SE_c²).

Q5. How can I convert BILOG-MG generated *.PAR file for further analysis (e.g., ITERLINK) or other purposes (e.g., free-field format)?
A5: You may use a new program called PAR3PLMG. See Section F. DIF/DTF questions below.

Q6. How can I convert BILOG-MG generated *.COV file for further analysis (e.g., ITERLINK) or other purposes (e.g., BILOG for DOS/Windows format)?
A6: You may use a new program called COVMG.

Q7. Can you provide a sample BILOG-MG syntax?
A7: BILOG-MG syntax may look like this:

> AGREEABLENESS CALIBRATION FOR IRT TUTORIAL
> COMMENT
> GLOBAL DFNAME='AGR2_CAL.DAT', NPARM=3, SAVE;
> SAVE SCORE='AGR2_CAL.SCO', PARM='AGR2_CAL.PAR', COV='AGR2_CAL.COV';
> LENGTH NITEMS=(10);
> INPUT SAMPLE=99999, NIDCHAR=4, OFNAME='OMIT.KEY';
> ITEMS INAMES=(AGR1(1)AGR10);
> TEST1 INUMBER=(1(1)10);
(4A1,10A1)
> CALIB NQPT=40, CYCLES=100, NEWTON=30, CRIT=.001, PLOT=0;
> SCORE MET=2, IDIST=0, RSC=0, NOPRINT;

MULTILOG7 questions

Q1. Can you provide a sample MULTILOG7 syntax?
A1: MULTILOG7 syntax may look like this:

CALIBRATION OF AGREEABLENESS GRADED RESPONSE MODEL
>COMMENT
>PROBLEM RANDOM, INDIVIDUAL, NITEMS=10, NGROUPS=1, NEXAMINEES=1500, NCHAR=4, DATA='AGR1_CAL.DAT';
>TEST ALL, GRADED, NC=(5,5,5,5,5,5,5,5,5,5);
>ESTIMATE NCYCLES=50;
>SAVE;
>END;
5
12345
1111111111
2222222222
3333333333
4444444444
5555555555
(4A1,10A1)

Q2. I am getting confused about Fortran format statement. Can you provide an example?
A2: The first chapter of IRT from SSI provided several examples of Fortran format statement. Here are some important format codes:

"A" stands for alphanumeric, "F" stands for floating point, "I" stands for integer, "X" stands for skip one or more columns, "T" stands for the column position to tab to, "/" stands for go to the next line. Thus, (4A1) stands for 4 columns of alphanumeric characters of length one.

Q3. MULTILOG could not find the dataset file, but I have got the correct file name and path. What happened?
A3: Please make sure each line does not exceed 80 characters (columns).
 

Back to Top

E. Model-data Fit questions

MODFIT questions

Q1. How can I get more information about MODFIT?
A1: Please refer to: Drasgow, F., Levine M.V., Tsien, S., Williams B.A., & Mead, A.D. (1995). Fitting polytomous item response theory models to multiple-choice tests. Applied Psychological Measurement, 19, 143-165.

Q2. Where can I get instruction for MODFIT?
A2: You may review MODFIT instructions here.

Q3. Why isn't my MODFIT working?
A3: MODFIT uses Microsoft Excel Macros for analysis. On certain systems, such as Windows Server 2003, the macros are disabled by default because the security level was set to High and digitally signed Trusted Certificate is not attached to the macros. Please use the following procedure to enable the macro:

1. Select the Tools menu option and then select Macro and Security. In the resulting Security dialog, set the security level to Medium by clicking the Medium radio button.
   
2. Close the file and any other instances of the application currently running on the computer (close all applications that also use the application you are currently running).
   
3. Open the file again and examine the certificate of trust details and set the Always trust macros from this publisher box if you trust the certificate issued by the publisher. Click the Enable button to allow the macro to run.

Q4. Do I need a password to run MODFIT?
A4: No. As you do not want to change the macros and formulas in MODFIT, you do not need a password to run the program.

Q5. Why is MODFIT read-only?
A5: MODFIT is read-only so the macros and formulas will not be changed by accident. You may save the file as a different file name when the analysis is finished.

Q6. I have more than 7 response categories in my instrument. Will MODFIT support for more than 7 response options?
A6:  In many IRT programs, response 9 is usually the default code for missing value, and some people use response 8 for other purposes. Therefore, you may want to consider collapsing the options down to 4 or 5 categories.

Q7. Where can I find the output file of MODFIT?
A7: If you use Windows XP to view a ZIP file and open "MODFIT for Distribution 6-06-01 (web).xls" directly, the program may not give you the right output, because MODFIT needs other Excel files for computation and display purposes. After you have downloaded the ZIP file, you should extract contents into a folder you designated, such as C:\MODFIT. Then, you should be able to find the output file in that folder. If you did not designate a folder when you extract it using WinZip, the default path is under C:\My Documents\CTT and IRT Programs\MODFIT for Distribution 6-06-01 (web).

Q8. What do "singlets, doublets, triplets" mean?
A8: Please refer to: Drasgow, F., Levine M.V., Tsien, S., Williams B.A., & Mead, A.D. (1995). Fitting polytomous item response theory models to multiple-choice tests. Applied Psychological Measurement, 19, 143-165. Essentially, "the chi-square approach began by taking cross-validation sample of N examinees. There are n the chi-square statistics that can be computed for the n items individually. However, there are the chi-square statistics that can be computed for item pairs, and possible the chi-square statistics for item triples. To limit the number of the chi-square statistics to a manageable and comprehensive number, the n test items were divided into n/3 sets of three items. For each set, a the chi-square was computed for each item, for all three sets of item pairs, and for the triple of items. The sets were selected so that they each contained a relative easy item, an item of moderate difficulty, and a relatively difficult item."

Q9. What do "ORF, IIF, EMP" mean?
A9: ORF stands for Option Response Function. IIF stands for Item Information Function. EMP stands for Empirical item response function from a cross validation sample.

Q10. How can I obtain a full copy of MODFIT?
A10: For academic and research proposes, you may e-mail the webmaster to request a full copy of MODFIT.

Back to Top

F. DIF/DTF questions

PARTO3PL, PAR_3PL, PAR3PLMG, ParConversion questions

Q1. What do programs like PARTO3PL and PAR_3PL do?
A1: PARTO3PL and PAR_3PL will convert only original BILOG for DOS/Windows generated parameter files (*.PAR) by 1) delete the first four lines and 2) subsequent alternating lines, 3) create output file using the *.3PL extension by following the 32x,2f12.6,12x,f12.6 format (which can be used for running ITERLINK), and 4) create output file using the a,b,c (free-field) format (e.g., *.TXT).

Q2. What do programs like PAR3PLMG do?
A2: Similar to PARTO3PL and PAR_3PL, PAR3PLMG convert only original BILOG-MG generated parameter files (*.PAR) by 1) delete the first four lines, 2) remove standard error columns, and 3) create output file using the *.3PL extension by following the 32x,2f12.6,12x,f12.6 format (which can be used for running ITERLINK), and 4) create output file using the a,b,c (free-field) format (e.g., *.TXT).

Q3. What do programs like ParConversion do?
A3: ParConversion will convert only original BILOG for DOS/Windows generated parameter files (*.PAR) by 1) delete the first four lines and 2) subsequent alternating lines, and 3) create output file using the a, b, c (free-field) format (e.g., *.TXT). That is, the program will only retain 3 columns of parameters a, b, c only in a 3PL model (from columns 4, 5, and 7 of original BILOG's '*.PAR' file) after removing the first 4 rows of title/comment information and alternative rows thereafter that contains standard errors.

Q4. I have trouble running either PARTO3PL or PAR_3PL. What should I do?
A4: If both programs crash, you may want to use Excel to open your *.PAR file, record a macro such that it will delete the first four lines (title/comment information) and subsequent alternating rows (standard errors). Alternatively, you may try running ParConversion. Please notice that the file formats (i.e., *.3PL) generated by PARTO3PL and PAR_3PL are different from files generated by ParConversion program (i.e., *.TXT).

Q5. I have trouble running ParConversion. What should I do?
A5: ParConversion requires Microsoft .NET Framework update. You may obtain the update through Windows Update. If for some reason ParConversion does seem to be working (Conversion will be completed automatically once the '*.PAR' file selection is done and will be saved in the '*.TXT' format with parameters only. The program runs very fast!),

ITERLINK questions

Q1. Is there an easy way to feed required information into ITERLINK?
A1: If you are familiar with DOS, you may compose a DOS batch file (.BAT) and a run file (.RUN) to feed required information into ITERLINK. The batch file (a text file) may look like this:

ITERLINK < DIF.RUN

The run file (called DIF.RUN, a text file) may look like this:

Example of Lord's chi-square analysis for 3PL model
3
3plfoc.3pl
0
32
3plref.3pl
0
32
foctran.out
N
3plfoc.cov
3plref.cov
1

You may review the meaning for each line at http://io.psych.uiuc.edu/irt/dif_iterlink.asp.

Q2. I have to perform a series of simulations. What is the best way to let the computer run one simulation after another?
A2: To make sure that the computer has processed various simulations, you may want to use a DOS batch file, such as run.bat, to make sure that you have executed every simulations in your study. In addition, utilizing "run files" mentioned in the previous question may help you identify where went wrong.

Q3. I heard there are different versions of ITERLINK?
A3: For different analysis purposes, there are different versions of ITERLINK. ITERLINK (2002) may be used for analysis using BILOG. ITERLINK (2000) may be used for analysis using MULTILOG. MULTILOG's estimates for a_i contain a scaling factor of 1.7. The only difference between the two versions is the EQUATED item parameter file output (i.e., foctran.out). The linking constants and DIF analysis are the same. In addition, the new ITERLINK06 (2006) is designed to work with BILOG-MG.

Q4. How can I tell which version of ITERLINK I have?
A4: On the downloads page, all executable files are now saved in .ZIP format. Thus, you should be able to tell which version of ITERLINK you have.

Q5. I am getting confused with how to match ITERLINK with BILOG and  versions. Can you help?
A5: The most current BILOG is called BILOG-MG 3. ITERLINK06(2006) is compatible with both BILOG-MG output format. The old BILOG for DOS/Windows is compatible with ITERLINK (2002).

Q6. Can you provide a sample syntax and output files from BILOG-MG?
A6: BILOG-MG sample syntax and output files are available for download here. A working batch file with .RUN file is attached to facilitate running ITERLINK06 with BILOG-MG.

Q7. I am getting confused with this 1.7 scaling factor. Can you explain more?
A7: When you select a dichotomous model, say using BILOG for 2PL model, the mathematical form can be written as

Back to Top