John L. Malanchuk, Ph.D.

International Technology Corp.

1. Introduction

A peer review was convened December 18-20, 1995 to evaluate the NAPAP Tracking and Analysis Framework (TAF). The list of peer review members, their respective affiliations and responsibilities is given in Attachment 1. Substantial support for TAF was expressed. In fact, given the totality of constraints, time, availability of substitutes etc., it is difficult to imagine how the NAPAP 1996 Integrated Assessment could be completed without reliance upon TAF and the TAF development team. Over the past eighteen months, substantial progress has been made in developing TAF and bringing the modules and framework to a usable point. In fact, reviewers commented that TAF represented a major advancement in our ability to perform integrated assessments. Significant work remains to be completed. This was part of the subject of this review. However, given the progress to date, the prognosis is excellent that TAF will be an important part of the 1996 Assessment.

Logically, there were differences among reviewers and the degree of reviewer support for TAF and its various modules. However, there is no doubt that TAF provides a vital organizing framework for scenario evaluation and that the TAF development team should play an important role in the preparation, execution and interpretation of the scenarios chosen for evaluation. Recommendations also were made to improve TAF in the near term for use in the 1996 Assessment, and beyond, and these enhancements will only serve to strengthen TAF's credibility and flexibility as an assessment tool. Reviewers cautioned, however, that the TAF not be applied or extrapolated to regions for which it has not been developed or intended. Finally, reviewers voiced strong sentiment that there can be no substitute for proper scientific and policy analysis in any assessment. Neither TAF nor any other assessment tool can be employed in a stand alone fashion. Neither science nor public policy are well served if decision makers are provided information based upon poorly conceived or poorly executed technical analyses.

In this vein, adequate funding is imperative if NAPAP is to execute the mission given to it by Congress to perform the 1996 and 2000 Integrated Assessments. Major funding is not required; adequate funding is and it is the unanimous recommendation of these peer reviewers that NAPAP member agencies provide sufficient funding to conduct these assessments to evaluate the multi-billion dollar investment already made on behalf of the American public.

2. Organization of Peer Review

Primary reviewers were designated for each of the areas to be reviewed. Originally there were eight categories to be reviewed; the seven modules and the TAF framework itself. However, at the beginning of the review, it was decided that the health effects and the benefits valuation modules would be combined into one area and presented by Resources For the Future (RFF). Two primary reviewers were designated for each module and three primary reviewers were designated for TAF overall. Remaining reviewers provided input to the primary reviewers who employed this input as they deemed appropriate. Each primary reviewer was requested to produce a separate, independent report (Attachment 2). No reviewers for the health section were available during the review. The presentation was videotaped and sent to two reviewers. A subsequent conference call was held among presenters, reviewers and NAPAP representatives after which reviewers submitted their written reports. Reviewers were given a series of standardized questions for each module and two standardized questions for TAF (Attachment 3). Module questions pertained to:

• parent model credibility
• reduced form model credibility
• suitability for use in NAPAP Assessments
• readiness for use in NAPAP 1996 Assessment and,
• specific recommendations for short-term improvement.

• achievement of concept design goals and,
• overall model credibility.

This executive summary of reviewer's evaluations and recommendations consists of a compilation of primary reviewers reports presented in the order of the agenda (Attachment 4), the presenter's vu-graphs from the stakeholder's briefing and attempts to capture associated key discussion. The summary is prepared as a table (Table 1) that quickly draws attention to areas in need of discussion and executive action and has been reviewed by the peer review team. The table is not meant to substitute for the full write-ups prepared by the individual reviewers. Also, the table contains footnotes integral to proper understanding of the table. Table columns correspond to the five module questions and two TAF questions discussed above.

Table 1. Summary of reviewer responses to questions asked by NAPAP for each module and the Tracking and Analysis Framework (TAF). Short-term recommendations are summarized as footnotes at the end of the Table and are an integral part of the discussion. Long-term recommendations can be found in the individual reviewer reports.

Module/Model	Parent Model Credibility	Reduced Form Model Credibility	NAPAP 1996	NAPAP 2000
Atmospheric Pathways SO2/NOx	R	R	R	R
Atmospheric Base Cations	NA	U	U	U
Magic Improvements	R	NA	R	R
Magic Aquatics Module	R	R	R	R
Magic Soils Module	NR	NR	NR	NR
Visibility	NA	R	R	U
Emissions/Costs	R	R	R	R
Health/Benefits	NA	R	R	R
TAF	R	NA	R	R

R = is presently ready or can be made ready for use in the 1996 Assessment

NR = not ready for use in 1996 Assessment

U = unsatisfactory for use in assessment. See complete text for explanation

NA = not applicable

MAGIC Improved Parent Structure

• improve Al vs pH relationship
• extrapolate to other regions only with great caution

MAGIC Modules - soils/aquatic

Aquatics Module

• make explicit in TAF, the relationship between the aquatic and soil modules
• improve Al vs pH relationship (aquatic module)
• make explicit the uncertainty characteristics in the ASI (acidic stress index) in the aquatic module

Soils Module

• incorporate lag-time dependency between effective deposition and soil base saturation (soil module)
• develop relationships based upon "base saturation" instead of "changes to base saturation" in soil module
• examine significance of organic matter and carbon dioxide to base saturation predictions in soil module

Visibility Module

• examine significance of covariance between relative humidity and aerosol species
• use additional aerosol data to calibrate visibility module

Emissions/Costs Module

• include in TAF, if desired
• evaluate significance of simplifications in cost model
• improve Canadian, Mexican and NOx scenarios
• include scenarios with different geographic distributions of emissions

Benefits/Health Module

• integrate how people actually view changes: thresholds, peaks, functional form
• quantitative coverage could be expanded to include: forests, passive N fertilization, damage to car finish
• add treatment in the assessment of cultural materials and general existence/non use values
• provide clearer treatment of health and incorporate into uncertainty analysis the application of unit values to exposed and susceptible populations
• framework should be adapted to reflect important endpoints not quantifiable in monetary terms
• assessment should include discussion of limitations including whether or not endpoints can be valued properly and whether or not concentration-response functions are sufficiently complete
• the lack of formal treatment of ozone is a serious drawback in the 1996 Assessment. Should NAPAP policy prevent inclusion of ozone then its omission must be accomplished in a technically credible manner, fully caveated and thoroughly discussed
• ambient temperature is an important consideration especially in evaluating the impact of PM10. To the extent possible, temperature effects should be included in the Assessment
• whether or not formally included in TAF, the information contained in the Benefits/Health section is integral to the Assessment. Estimates should be accompanied by uncertainty analyses and methods of determining exposed and susceptible populations must be critically reviewed. Total health benefits should be distributed over different health impacts e.g. chronic respiration, coughs, etc.

• stakeholders must be integrated into the TAF and Assessment process
• there should be an overall master plan and someone must have lead and authority to make necessary assessment decisions
• in the TAF causal chain, receptor modules (especially cost/benefits and effects) need to express input data requirements more specifically
• TAF must be fully in the public domain preferably DOS or Windows based
• TAF must be made more transparent relative to underlying assumptions