Correlation structure selection for longitudinal data with diverging cluster size

Correlation structure selection for non-normal longitudinal data is very challenging for diverging cluster size because of the high-dimensional correlation parameters involved and the complexity of the likelihood function for non-normal longitudinal data. However identifying the correct correlation structure is important because it can improve estimation efficiency and testing power for longitudinal data. We propose to approximate the inverse of the empirical correlation matrix using a linear combination of candidate basis matrices, and select the correlation structure by identifying non-zero coefficients of the basis matrices. This is carried out by minimizing penalized estimating functions, which balance the complexity and informativeness of modelling for the correlation matrix. The new approach does not require estimating each entry of the correlation matrix (except for an initial empirical estimate from the residuals), nor specifying the likelihood function, and can effectively handle non-normal longitudinal data. The derivation of asymptotic theory for model selection consistency and oracle properties is challenging in the framework where the cluster size and the number of basis matrices are both diverging. Our numerical studies show that the proposed method performs satisfactorily for both normal and binary responses in this diverging framework.