TY - GEN
T1 - Beyond theory and data in preference modeling
T2 - 4th International Conference on Algorithmic Decision Theory, ADT 2015
AU - Allen, Thomas E.
AU - Chen, Muye
AU - Goldsmith, Judy
AU - Mattei, Nicholas
AU - Popova Anna, Anna
AU - Regenwetter, Michel
AU - Rossi, Francesca
AU - Zwilling, Christopher
N1 - Publisher Copyright:
© Springer International Publishing Switzerland 2015.
PY - 2015
Y1 - 2015
N2 - Many mathematical frameworks aim at modeling human preferences, employing a number of methods including utility functions, qualitative preference statements, constraint optimization, and logic formalisms. The choice of one model over another is usually based on the assumption that it can accurately describe the preferences of humans or other subjects/processes in the considered setting and is computationally tractable. Verification of these preference models often leverages some form of real life or domain specific data; demonstrating the models can predict the series of choices observed in the past. We argue that this is not enough: to evaluate a preference model, humans must be brought into the loop. Human experiments in controlled environments are needed to avoid common pitfalls associated with exclusively using prior data including introducing bias in the attempt to clean the data, mistaking correlation for causality, or testing data in a context that is different from the one where the data were produced. Human experiments need to be done carefully and we advocate a multi-disciplinary research environment that includes experimental psychologists and AI researchers. We argue that experiments should be used to validate models. We detail the design of an experiment in order to highlight some of the significant computational, conceptual, ethical, mathematical, psychological, and statistical hurdles to testing whether decision makers’ preferences are consistent with a particular mathematical model of preferences.
AB - Many mathematical frameworks aim at modeling human preferences, employing a number of methods including utility functions, qualitative preference statements, constraint optimization, and logic formalisms. The choice of one model over another is usually based on the assumption that it can accurately describe the preferences of humans or other subjects/processes in the considered setting and is computationally tractable. Verification of these preference models often leverages some form of real life or domain specific data; demonstrating the models can predict the series of choices observed in the past. We argue that this is not enough: to evaluate a preference model, humans must be brought into the loop. Human experiments in controlled environments are needed to avoid common pitfalls associated with exclusively using prior data including introducing bias in the attempt to clean the data, mistaking correlation for causality, or testing data in a context that is different from the one where the data were produced. Human experiments need to be done carefully and we advocate a multi-disciplinary research environment that includes experimental psychologists and AI researchers. We argue that experiments should be used to validate models. We detail the design of an experiment in order to highlight some of the significant computational, conceptual, ethical, mathematical, psychological, and statistical hurdles to testing whether decision makers’ preferences are consistent with a particular mathematical model of preferences.
UR - http://www.scopus.com/inward/record.url?scp=84945980096&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=84945980096&partnerID=8YFLogxK
U2 - 10.1007/978-3-319-23114-3_1
DO - 10.1007/978-3-319-23114-3_1
M3 - Conference contribution
AN - SCOPUS:84945980096
SN - 9783319231136
T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
SP - 3
EP - 18
BT - Algorithmic Decision Theory - 4th International Conference, ADT 2015, Proceedings
A2 - Walsh, Toby
PB - Springer
Y2 - 27 September 2015 through 30 September 2015
ER -