Workshops at the Annual Meeting
On Sunday, December 4, 2011, the EBASG will sponsor two continuing education workshops - one on research synthesis methods and one on expert elicitation. The first workshop is intended for those who are interested in exploring the advantages and limitations of alternative approaches for synthesizing evidence, while the second is intended for those interested in exploring expert elicitation techniques in more detail.
To register for these workshops and to obtain information on other SRA workshops, please visit the SRA Annual Meeting website.
Synthesizing Evidence: An Introduction to Systematic Reviews, Meta-Analysis, and Expert Elicitation
Risk analysis often requires making inferences or estimating parameter values from studies that contain inconsistent or conflicting results or address dissimilar contexts. Such inferences or estimates should be consistent with the weight of evidence. Deciding whether and how to combine information from multiple studies requires thinking carefully about the nature of the problem to be addressed and the characteristics of the available evidence. In this workshop, we will investigate the advantages and limitations of alternative approaches to research synthesis from a cross-disciplinary perspective.
We will introduce the range of methods for evaluating and combining evidence and explore three prominent approaches in detail: systematic review, meta-analysis, and expert elicitation. These methods are used widely in the social sciences and medicine as well as in risk assessment. Each begins with a careful review of the research literature, but then the approaches diverge. Systematic review involves a largely qualitative evaluation of available studies against established criteria to identify those that are most appropriate for use in a particular context. Meta-analysis involves selecting studies from the available literature using formal criteria and then using statistical models to calculate summary estimates and explore sources of variation across studies. Expert elicitation uses a structured process to select experts who provide subjective probability distributions that characterize their knowledge about a quantity. The workshop will conclude with a panel discussion focused on questions raised by attendees, including the appropriate application of each method to the problems they face to support evidence-based decisionmaking.
Instructors
Lisa A. Robinson (Independent Consultant) specializes in assessing the costs, benefits, and other impacts of environmental, health, and safety regulations.
Lorenz Rhomberg (Gradient) is an expert in quantitative risk assessment and weight-of-evidence evaluation.
Julie E. Goodman (Gradient and Harvard University) is a toxicologist and epidemiologist who teaches research synthesis methods at the Harvard School of Public Health.
James K. Hammitt (Harvard University and Toulouse School of Economics) is Professor of Economics and Decision Sciences with expertise in quantitative methods and expert elicitation.
Tentative Agenda (subject to change)
8:00 - 8:15 Introduction and Overview (Lisa Robinson)
8:15 - 9:00 Overview of Approaches for Evidence-Based Decisionmaking (Lorenz Rhomberg)
9:00 - 10:00 Systematic Reviews and Meta-Analysis (Julie Goodman)
10:00 - 10:15 Break (with refreshments)
10:15 - 11:00 Expert Elicitation (James Hammitt)
11:00 - 12:00 Panel Discussion
Eliciting Judgments to Inform Decisionmaking
Decision makers must frequently rely on data or information that is incomplete or inadequate in one way or another. Judgment, often from experts and occasionally from non-experts, then plays a critical role in the interpretation and characterization of those data as well as in the completion of information gaps. But how experts or other stakeholders are selected and their judgments elicited matters - they can also strongly influence the opinions obtained and the analysis on which they rely. Several approaches to eliciting judgments have evolved.
The workshop will cover topics ranging from recruitment, elicitation protocol design, different elicitation techniques (e.g., individual elicitations, Delphi method, nominal group technique, etc.) to aggregation methods for combining opinions of multiple individuals. The role of judgment elicitation and its limitations, problems, and risks in policy analysis will also be addressed. The workshop will include presentation of two case studies that will include a discussion of the selection process; elicitation protocol development, elicitation technique utilized, and the various issues that arose before, during, and after the elicitation process and the manner in which they were resolved. The class will conclude with a hands-on exercise where participants will apply different elicitation techniques to a given problem and compare/contrast elicitation results across the techniques.
Instructors
Aylin Sertkaya (Eastern Research Group)
Cristina McLaughlin (US Food and Drug Administration)
Frank Hearl (National Institute for Occupational Safety and Health - NIOSH)
J. Michael Davis (US Environmental Protection Agency)
Christine Hendron (Research Triangle Institute)
Ron White (Johns Hopkins University)
Katherine Walker (Health Effects Institute)
Tentative Agenda (subject to change)
8.30 - 8:45 Introduction (Cristina McLaughlin)
8:45 - 9:30 A Quick Guide to Eliciting Opinions (Aylin Sertkaya)
9:30 - 10:15 Collective Judgment - Methods and Procedures (Christine Hendron)
10:15 - 10:30 Break (with refreshments)
10:30 - 11:15 Value of Diversity - Expert vs. Non-Expert Judgment (Ron White)
11:15 - 12:00 Aggregation of Judgments - Pros and Cons
12:00 - 1:00 Lunch
1:00 - 1:45 Case Study 1 (J. Michael Davis)
1:45 - 2:30 Case Study 2 (Katherine Walker)
2:30 - 4:00 Hands-on Exercise (Frank Hearl)
4:00 - 4:30 Questions and Discussion |
