Eliciting Judgments from Experts and Non-experts

Sunday, December 4, 2022 with Frank Hearl, Aylin Sertkaya and Cristina McLaughlin (8AM-12PM)

Decision makers must frequently rely on data or information that is incomplete or inadequate in one way or another. Judgment, often from experts and occasionally from nonexperts, then plays a critical role in the interpretation and characterization of those data as well as in the completion of information gaps. But how experts or non-experts are selected, and their judgments elicited matters – they can also strongly influence the opinions obtained and the analysis on which they rely. Several approaches to eliciting judgments have evolved. The workshop will cover topics ranging from recruitment, elicitation protocol design, different elicitation techniques (e.g., individual elicitations, Delphi method, nominal group technique, and focus groups) to aggregation methods for combining opinions of multiple individuals. The role of judgment elicitation and its limitations, problems, and risks in policy analysis will also be addressed. The workshop will include presentation of two case studies that will include a discussion of the selection process; elicitation protocol development, elicitation technique utilized, and the various issues that arose before, during, and after the elicitation process and the way they were resolved. The class will also include two hands-on exercises where participants will 1) learn about calibration of experts using a mobile application and 2) apply the Delphi and nominal group techniques to examine risk management issues associated with a popular topic.