• SAS:
  • You can obtain a free educational versions from SAS.com. Look for "SAS Software for Learning". There is a University Edition and an OnDemand version. The latter requires an internet connection but it also includes more procedures (e.g., GRAPH, ETS and OR). Alternatively, you can obtain a license from web-store. You can either borrow, download or purchase the program (media) from webstore. With the 2nd option (i.e., purchase), you get more procedures and packages.
  • Introduction to SAS
    • Introduction to SAS notes..
    • hsb1.sas. Creates SAS data set of level 1 data for the High School and Beyond data.
    • Slick online SAS training from SAS.com
• Resources for R users: :
  • You can download (for free) R and Rstudio from the internet.
  • Introduction to Multilevel Modeling in R by Grover, Guillermono and Hudson (2015). This document uses the high school and beyond data set to illustrate a lot about how to do data management, HLM models and analyses, and graphics. If you are using R, I highly recommend this.
  • hsb1data.txt
  • hsb2data.txt
  • Doing scatter-plots with lattice.
  • A Brief Introduction to R, the multilevel package and the nlme package by Paul Bliese (2016). This includes some introductory material on R and how to do multilevel modeling.
  • Lecture notes by Kyle Roberts. I included these because there is a nice R code section
  • Lot of R scripts and Markdown documents below!
  
  
• Draft chapters on GLM, GLMM, and LLM (i.e., HLM).   This link has been disabled for now but I will make some changes and re-post (8/23/2023).  

Lecture Notes :

Jumps to specific lectures and associated code:

• Introduction
  
  • SAS:
    • Juinor School Project..
    • Hypothetical longitudinal example
    • Two schools from NELS88
  • R:
    • Juinor School Project data in csv format..
    • Juinor School Project example..
    • Nels Example
      • NELS data in csv format..
      • Rmarkdown nels.html
      • Rmarkdown nels.rmd
      • Rmarkdown.r (R script from rmarkdown )




• Models for clustered data: Fixed and random effects ANOVA and multiple regression.
  
  • SAS
    • NELS data for 10 schools. Run this program before the next one.
    • Generates statistics, fits models & produces graphics.
  • R
    • NELS data for 10 schools in txt format.
    • Rmarkdown nels10.html
    • Rmarkdown nels10.rmd
    • Rmarkdown nels10.r




• Random Intercept Models.
  
  • SAS
    • ANCOVA.sas. Fits ANCOVA model to NELS88, N=10 data (includes centering a variable, model fitting using GLM, and SAS/GRAPH of model).
    • hsb1.sas. Creates SAS data set of level 1 data for the High School and Beyond data.
    • hsb2.sas. Creates SAS data set of level 2 data for the High School and Beyond data.
    • hsball.sas. Merges level 1 and level 2 high school and beyond sas datasets.
    • betwithin.sas. SAS/GRAPHS for looking between and within variability of SES in the high school and beyond data.
    • randomintercepts.sas. SAS PROC MIXED and fitting random intercept models (includes centering SES)...and some graphics.
  • R:
    • For the NELS example, use the R script (html or rmd) from previous notes (models for clustered data).
    • HSB Example
      • hsb1data.txt. Student level data
      • hsb2data.txt. School level data
      • Random intercept HSB.html
      • Random intercept  HSB.rmd
      • Random intercept HSB.r
    • Little R function that computes the ICC.




• SAS and R for HLM
  
  • SAS:
    • HSB1 data. (level 1 data -- students)
    • HSB2 data (level 2 data -- schools)
    • SAS code to be discussed in lecture.
    • Using SAS assist for graphics.
  • R:
    • HSB1data.txt. Student level data
    • HSB2data.txt. School level data
    • Basic model fitting.html
    • Basic model fitting.rmd
      
    • Basic model fitting.r
      

  
  

• Random Intercept and Slopes Models.
  
  • SAS
    • text file of data. NELS88 data for N=23 school.
    • school23.sas. SAS program that creates SAS data set for NEL88 data for N=23 school.
    • NELS23.sas. SAS program that fits various random intercept and slope models to the NEL88 data for N=23 school.
    • Centering & NELS data. Illustrates effects of different kinds of centering -- NEL88 data for N=23 school.
  • R
    • NELS23 data in txt format.
    • R code for NELS for 23 schools. Including among other things, code for data manipulation, creating variables, doing some graphics, model fitting using lmer.
    • HSB Example
      • Student level HSB data in txt format.
      • School level HSB data in txt format.
      • Random_intercepts_slopes.html
      • Random_intercepts_slopes.rmd
      • Random_intercepts_slopes.r

       

• Estimation of Marginal Model.
  • SAS:
    • Graphs showling likeihood of normal distribution.
    • How to simulate HLM (in SAS).
    • SAS: Simulation study on MLE vs REML and different N.
    • SAS: For those who might want to use Bayesian estimation, I worked up a small example for the empty model,random intercept with one predictor, and a random intercept and slope model. I also included SAS PROC MIXED code if you want to compare results. You can change the seed to "-1" and run the mcmc code a few times and compare results. The algorthm for sampling is Metropolis-Hastings.
      • The data for the program below.
      • SAS/PROC MCMC for HLM.
  • R
    • R code for graphs illustrating MLE
    • Simulating HLM data:
      • simulate hlm data.html
      • simulate hlm data.rmd
      • simulate hlm data.r
    • Bayesian Estimation:  The brms package is to fit multilevel model (i.e., Bayesian estimation). This example uses the nels23 data. The specification of the model is very similar to lmer.   You need to build the stan program and install the rstan and the brms package. Check internet of instructions.
      
      • Nels with N=23 schools.    
      • brms.html
      • brms.rmd
      • brms.r
      • Some references for Bayesian analysis that are all available as from UIUC library: 
        • Kruschike, J (2014). Doing Bayesian Data Analysis: A Tutorial with R, JAGS and Stan, 2rd Edition.
        • Gelman, Carlin, Stern, Dunson, Vehtari, & Rubin (2013).  Bayesian Data Analysis. Chapman & Hall/CRC
          
        • Huang, M., & Anderson, C.J. (2020).  A Bayesian Solution to Non-convergence of crossed random effects models. In Wiberg, Marie. et al.    Quantitative Psychology 84th annual meeting of the Psychometric Society.  Springer International Publishing : Imprint: Springer ; 2020.






• Statistical Inference: Marginal Model.
  • SAS
    • SAS commands for ddfm (Simulation to show what different choices for df yield).
    • HSB1 data. (level 1 data -- students)
    • HSB2 data (level 2 data -- schools)
    • hlmrsq.sas SAS macro that computes R_1^2 and R_2^2.
    • Recchia JSS paper describing the hlmrsq.sas macro and how to use it.
    • SAS for fixed effects & Rsq. Illustrates various things.
  • R:
    • hsb1.txt hsb level 1 data.
    • hsb2.txt hsb level 2 data.
    • All_functions.txt.    Use this as a single source file of functions icc, contrast, robust (sandwich) standard errors, R squares for hlm, and BIC for hlm.
      
    • Illustrates some of things done in lecture (html):  Shows  computing robust /sandwich se, computing R squares, finding mixture p-values, confidence intervals, corrected BIC & AIC, and more.
      
    • Rmd file for examples in Lecture  
      
    • R script for lecture examples        
      
  
  
• Random Effects.
  • Power and sample size:
    • Xiaofeng Steven Liu (2014). Statistical Power Analysis for the Social and Behavioral Sciences: Basic and Advanced Techniques. Routledge: NY. This book contains explanation of procedures and code for R, SAS and SPSS. If you search google scholar
    • Optimal Design Software Program and documentation from Raudenbush group. (PC only)
    • PINT Program and documentation from Snijders. (PC only)
    • Google "Xiaofeng Liu power hlm", you will find some of his papers on power and HLM (I think he was a student of Raudenbush).
  • SAS:
    • HSB1 data. (level 1 data -- students)
    • HSB2 data (level 2 data -- schools)
    • SAS program and EB estimates of U's.
    • SAS commands for mini-study on micro sample size on U's.
    • SAS commands for mini-study on macro sample size on estimates.
    • SAS commands for mini-study on effects of non-normality on the distribution of EB estimates.
    • Using SAS/ASSIST to produce graphics. (Alternatives given in next lecture).
  • R:
    • hsball.txt  (needed for inference for random effects)
    • Inference for Random Effects: html   |  Rmd  |  R
    • Function and Example for Simulation study:   html  |   Rmd  |  R
  
  
• Model Building.
  • SAS:
    • NELS:
      • Goes with (most of) the lecture notes
        • modeling nels 23.sas
        • Exploratory data analysis of fixed effects
        • Exploratory data analysis of random effects
      • Location scale models
        • school23_data_2023.sas
        • school23_Location_scale.sas    (uses proc NLMIXED)
        • school23_Location_scale_results.rtf 
    • HSB:     Location scale Models including random intercepts and slops and covariance between level 1 and 2 random effect.   This uses PROC NLMIXED.
      
      • modeling_hsb_level1_variance_2023.sas
      • modeling_hsb_level1_variance_2023.rtf
  • R:
    • This reproduces most of what's in the notes:   NELS example   html    |   Rmd
      
    • school23_data.txt 
      
    • Location scale models. Should download the paper and program (mixreglsb.exe) from web-site https://www.jstatsoft.org/article/view/v052i12 . The paper is Hedeker, D, & Norgren, R (2013). A program for mixed-effects location scale analysis. Journal of Statistical Software, 52(12)
      • R_mixregls.R This is a function that I wrote that sets up the data and calls the executable program mixreglsb.exe.
      • NELS
        • Nels 23 data
        • Example location scale analysis using R_mixregls.R:  html  |    Rmd  |   R
      • HSB
        • hsball.txt (the data)
        • More extended example of running :   html  |     Rmd  |     R
          
• Longitudinal .
  • SAS
    • Riesby time invariant predictors
    • Riesby time varying predictors
  • R:
    • Longitudinal Depression data example with time invariant predictors  (Riesby data)
      
      • Depression Data    
      • Anaylses:  html  |   Rmd  |    R
    • Longitudinal Depression data example with time varying predictors
      
      • Data    with time varying predictors
        
      • R script for modeling data
        

  
  

  • Serial Correlation
    • SAS:
      • Simulation of HLM without serial correlation
        
      • Simulation of HLM with AR(1)
      • Simulation of HLM with MA(1)
      • Simulation of HLM with ARMA(1,1)
      • Revisit Risby data
    • R:
      • Revisit Riesby data with serial correlation
        
  
  

   

  • Logistic Regression
    • SAS:
      
      • Respiratory infection  (data and analyses) -- longitudinal data
        
      • Depression example -- longitduinal data
        
      • Cool kid example (data is not public)
        
      • IRT models:   Code for both Rasch and 2 parameter logistic and covariates for the GSS (general social survey) vocabulary data
        • LSAT sas
        • GSS vocabulary example with 10 items
          
    • R:
      • Respiratory infection example (longitudinal)
        
        • Data
        • Analyses:   html  |   Rmd  |   R
      • Longitudinal Depression data example (from Agresti)
        
        • Data in long format
        • R for this example: html  | Rmd  |  R
      • Cool kid R script  (data is not public)
        
      • Longitudinal Depression data example with time varying predictors
        
        • Data    with time varying predictors
          
        • R script for modeling data
          
      • IRT models (Rasch or 1 parameter logistic)
        • LSAT data
        • LSAT model fitting:   html  |   Rmd  |    R
          
        • GSS vocabulary data
        • Model fit to vocabulary data  (txt)
          

   

  Computer Labs

  • Computer lab 1 : Creating variables and fitting random intercept models
    • Lab instructions
    • Lab1 data in csv format
    • Lab2 data in txt format
    • Template for R:   html  |   Rmd  |   R
      
    • Complete R for the lab (needed to do homework 3:   html  |  Rmd  |   R
  
  
  • Computer lab 2 : Fitting random intercept and slope models
    • Lab instructions
    • Data (lab2.txt)
    • Template for R:   html  |   Rmd  |   R.
    • Complete R for the lab (needed to do homework 4:   html  |    Rmd  |   R
  • Computer lab 3: Inference for marginal model (fixed and random effects)
    • Lab instructions
    • Data (lab3.txt)
    • All functions, which has functions that I wrote to compute ICC, robust (sandwich ses), HLM R2, and contrasts
    • Template for R, which includes models fit from labs 1 and 2:   html  |    Rmd  |   R
      
    • Complete R for the lab (needed to do homework 5) : html  |   Rmd  |     R
  • Computer lab 4:  Graphics (not done this year)
    
    • Lab instructions   The R below is nicer
      
    • Data (lab4.txt)
    • Template for R   (txt file)
      
    • R for the lab (and most of what's needed to do homework 6):    html  |  Rmd  |  R
      
  
  

  Homework

  • Homework 1
    • Assignment 1
  • Homework 2
    • Assignment 2
    • Answer to homework 2
  • Homework 3
    • Assignment 3
    • Answer Key
    • R for answers:   html  |   Rmd   |    R
      
  • Homework 4
    • Assignment 4
    • Answer Key
    • R for answers:  html  |  Rmd  |  R
      
  • Homework 5
    • Assignment 5
    • Answer Key
    • R for answers:  html   |  Rmd  |  R
  • Homework 6 (skipped this year)
    • Assignment 6
    • Answer Key
    • R for answers:  html  |  Rmd  |  R     (See computer lab 4)
      
  
  
  • Hardcopy of projects/finals are due Friday May 5 in the box folder for projects and finals; Be sure that you receive e-mail from me to confirm that I have received your work. Penalty for late finals or projects is 10 points (out of 100 points) per weekday.
    • Final Exam and data
      • Natonal Youth Longitudinal Data.
      • PIRLs reading assessment Data.
      • NELS data.   (correct data posted April 18)
        
      • Espelage Bully Data.
      • Fatal police shooting data.
    • What to include in a Final Project. (Due Friday May 5, 2023)
    
    

  
  Example Papers using Multilevel Models or About How to use them

  • Payne, B.R., Gao, X., Noh, S.R., Anderson, C.J., Stine-Morrow, E.A.L. (2011). The effects of print exposure on sentence processing and memory in older adults: Evidence for efficiency and reserve. Aging, Neuropsychology, and Cognition.
    Some examples of crossed random effects, skewed responses (i.e., reaction times), and discrete response (i.e., Poisson).
    
    
  • Segerstrom, S.C. & Sephton, S.E. (2010). Optimistic expectanices and cell-mediated immunity: The role of positive affect. Psychological Science, 21, 448-455.
    Example of where cluster centered level one variable is substantive (theoretical) interest. The response variable is numerical/continuous.
    
    
  • Allen, N.E., Todd, N.R., Anderson, C.J., Davis, S.M., Javdani, Bruehler, V., & Dorsey, H. (2013). Council-Based approaches to intimate partner violence: Evidence for distal change in system response. American Journal of Community Psychology, 52, 1-12.
    Example of a longitudinal study with creative centering of time. The response variable was a rate (probability).
    
    
  • Poteat, V.P. & Anderson, C.J. (2012). Developmental changes in sexual prejudice from early to late adolescence: The effects of gender, race, and ideology on different patterns of change. Developmental Psychology, 48, 1403-1415.
    Example of an accelerated longitudinal design.
    
    
  • Steen-Baker, A.A., Ng, Shukhan, Payne, B.R., Anderson, C.J., Federmeier, K.D., & Stine-Morrow, E.A.L (2017). The effects of context on processing words during sentence reading among adults varying in age and literacy skills. Psychology and Aging, 32, 460-472.  Example cross-random effects in an study using eye-tracking data. Data were skewed so first log-transformed and then used HLM (i.e., a log-normal model)
  • Anderson, C.J., Kim, J.S., & Keller, B. (2014).   Multilevel Modeling of Categorical Response Variables.  In L. Rutkowski, M. von Davier, & D. Rutkowski (eds)A Handbook of International Large-Scale Assessment: Background, Technical Issues, and Methods of Data Analysis (pp. 481-519).   This deals with models for response if categorical (binary, ordered multicateory or unorderd multicategory), missing data, use of survey weights due to unequal probability sampling.
    
  • Kim, J.S., Anderson, C.J. & Keller, B (2014).  Multilevel Modeling in Large-Scale Assessments.  In L. Rutkowski, M. von Davier, & D. Rutkowski (eds) A Handbook of International Large-Scale Assessment: Background, Technical Issues, and Methods of Data Analysis. (pp 389-424).  This described models and methods for multilevel modeling of a continuous outcome variables, including missing data and use of survey weights. 
    
  • Poulsen, S., West, M., & Anderson, C.J.  (2020).   The relationship between course scheduling and student performance.  Educational Data Mining in Computer Science Education.  Proceedings of 4^th annual CSEDM workshop.   This is a paper that grew out of a course projects.
    
  • Huang M. & Anderson CJ (2021) A Bayesian Solution to Non-convergence of Crossed Random Effects Models.  In: Wiberg M, Molenaar D, Gonzalez J. Bockenholt U., Kim JS. (eds) Quantitative Psychology.  Spring Proceedings in Mathematrcs & Statistics, vol 363. Springer ,  Cham.  https://doi.org/10.1007/978-3-030-74772-5_27   If your model doesn't converge using MLE, then try Bayesian estimation method.
  • Coming soonfrom Marian Pan, Mingya Huang, and Carolyn Anderson: tutorial on fitting crossed random effects models with mediators in R with an emphasis on categorical mediators (i.e., normal, binomial, poisson, negative binomial, and zero inflated poisson mediators) .  This will also have R scripts for drawing nice graphs using the diagram package.   Planned completion Fall 2023.
    
  • Examples from Tom Snijders course web-page where multilevel models have been used. (click on "info course multilevel" on left and go to bottom of page. These papers cover a range of topics (e.g., political science, sociology, school psychology, criminology, medicine, and others).
  
  

  Examples from Snijders & Bosker (1st edition) using SAS

  • MLbook.sas. Create SAS data for examples in Chapters 4 and 5.
  • Ch4_examples.sas. Example 2-level analyses from Chapter 4 (random intercept models).
  • Ch5_examples.sas. Example 2-level analyses from Chapter 5 (random intercept and slopes).
  • Ch12_examples.sas. Examples analyses from Chapter 12 (longitudinal data analysis), including creating sas  dataset using the data
  
  
  Handy Links:

  Ones specific to multilevel modeling:

  • Centre for Multilevel Modelling. This site includes trainting materials, publications, reviews of multilevel software, data sets, a BBC audio program featuring Harvey Goldstein, and more..
  • Tom Snijders' multilevel web-site. This has data & various material that's used in Snijders & Bosker An Introduction to Basic and Advanced Multilevel Modeling as well as other multilevel stuff.
  • Scientific software international home page. Student version of HLM program can be downloaded from this site.
  • NELS88 data: various data sets used in Kreft & de Leeuw (Introducing heirarchical linear modeling)
  • Using SAS PROC MIXED to fit multilevel, heirarchical models, and individual growth models.Downloadable paper by Judith Singer. Other papers also downloadable.