TY - JOUR
T1 - Assessing the self-reported honesty threshold in adolescent epidemiological research
T2 - comparing supervised machine learning and inferential statistical techniques
AU - Kosgolla, Janaka V.
AU - Smith, Douglas C.
AU - Begum, Shahana
AU - Reinhart, Crystal A.
N1 - The authors received support from the Illinois Department of Human Services’ Division of Substance Use Prevention and Recovery (SUPR; Contract # 43CAZ03292, PI: Smith) and the National Institute of Alcohol Abuse and Alcoholism (Kelly, K24AA022136-08)/Bergman (K23AA025707). The views, however, are those of the authors and do not reflect official government positions.
PY - 2023/12
Y1 - 2023/12
N2 - Background: Epidemiological surveys offer essential data on adolescent substance use. Nevertheless, the precision of these self-report-based surveys often faces mistrust from researchers and the public. We evaluate the efficacy of a direct method to assess data quality by asking adolescents if they were honest. The main goal of our study was to assess the accuracy of a self-report honesty item and designate an optimal threshold for it, allowing us to better account for its impact on point estimates. Methods: The participants were from the 2020 Illinois Youth Survey, a self-report school-based survey. We divided the primary dataset into subsets based on responses to an honesty item. Then, for each dataset, we examined two distinct data analysis methodologies: supervised machine learning, using the random forest algorithm, and a conventional inferential statistical method, logistic regression. We evaluated item thresholds from both analyses, investigating probable relationships with reported fake drug use, social desirability biases, and missingness in the datasets. Results: The study results corroborate the appropriateness and reliability of the honesty item and its corresponding threshold. These contain the agreeing honesty thresholds determined in both data analyses, the identified association between reported fake drug use and lower honesty scores, increased missingness and lower honesty, and the determined link between the social desirability bias and honesty threshold. Conclusions: Confirming the honesty threshold via missing data analysis also strengthens these collective findings, emphasizing our methodology’s and findings’ robustness. Researchers are encouraged to use self-report honesty items in epidemiological research. This will permit the modeling of accurate point estimates by addressing questionable reporting.
AB - Background: Epidemiological surveys offer essential data on adolescent substance use. Nevertheless, the precision of these self-report-based surveys often faces mistrust from researchers and the public. We evaluate the efficacy of a direct method to assess data quality by asking adolescents if they were honest. The main goal of our study was to assess the accuracy of a self-report honesty item and designate an optimal threshold for it, allowing us to better account for its impact on point estimates. Methods: The participants were from the 2020 Illinois Youth Survey, a self-report school-based survey. We divided the primary dataset into subsets based on responses to an honesty item. Then, for each dataset, we examined two distinct data analysis methodologies: supervised machine learning, using the random forest algorithm, and a conventional inferential statistical method, logistic regression. We evaluated item thresholds from both analyses, investigating probable relationships with reported fake drug use, social desirability biases, and missingness in the datasets. Results: The study results corroborate the appropriateness and reliability of the honesty item and its corresponding threshold. These contain the agreeing honesty thresholds determined in both data analyses, the identified association between reported fake drug use and lower honesty scores, increased missingness and lower honesty, and the determined link between the social desirability bias and honesty threshold. Conclusions: Confirming the honesty threshold via missing data analysis also strengthens these collective findings, emphasizing our methodology’s and findings’ robustness. Researchers are encouraged to use self-report honesty items in epidemiological research. This will permit the modeling of accurate point estimates by addressing questionable reporting.
KW - Adolescents
KW - Epidemiological surveys
KW - Machine learning
KW - Response validity
KW - Self-reported honesty
KW - Substance use
UR - http://www.scopus.com/inward/record.url?scp=85171854156&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85171854156&partnerID=8YFLogxK
U2 - 10.1186/s12874-023-02035-y
DO - 10.1186/s12874-023-02035-y
M3 - Article
C2 - 37735353
AN - SCOPUS:85171854156
SN - 1471-2288
VL - 23
JO - BMC Medical Research Methodology
JF - BMC Medical Research Methodology
IS - 1
M1 - 210
ER -