Comments on the Accessibility, Use of Analytica, and Treatment of Uncertainty in TAF

Max Henrion (henrion@lumina.com)
Lumina Decision Systems, Inc
Los Gatos, CA
415-354-1841

The following comments were prepared by Lumina for the TAF team in response to a request from NAPAP for clarification of three issues pertaining to the design and development of TAF, specifically:

1. Accessibility of TAF
2. The choice of Analytica to implement TAF
3. The treatment of uncertainty

1. Accessibility of TAF

A major design goal of TAF has been to make the model widely and conveniently accessible to those interested in understanding and reviewing its structure, assumptions, and results. TAF is also designed to support users in exploring alternative assumptions, modifications, and extensions of the model. TAF's accessibility is based on features provided by Analytica, including hierarchical influence diagrams, the Analytica modeling language, integrated graphing of results, and integrated hypertext model documentation.

TAF, having been developed by public funds, is entirely in the public domain in the same way that a Fortran model or a spreadsheet model might be public domain. The TAF code is freely available to anyone in the form of ASCII files. The TAF user needs a copy of Analytica to run TAF, in the same way that another model might need a Fortran compiler or spreadsheet application (such as Excel or Lotus) to be executed.

Unlike FORTRAN compilers and spreadsheet applications, an Evaluation version of Analytica is available at no cost and allows any TAF user to explore, modify, and run TAF--that is, do anything with TAF except save modifications. When an Analytica runtime version becomes available, in the next few months, Lumina could also provide a runtime version of TAF -- that would also enable users to save modifications to model input variables. Lumina is committed to making public domain Analytica models, such as TAF, available and accessible to users without requiring them to buy Analytica.

We have developed a TAF site on the World Wide Web that makes key information about TAF available to Web browsers. This information includes the rationale, TAF participants, and model documentation, as well as the TAF model itself. The TAF model is available on the Internet, and can be downloaded using the Web site or separately via ftp (the standard file transfer protocol). Lumina has provided the Evaluation version of Demos available to be downloaded in similar fashion, and plans to provide an Evaluation version of Analytica available on the Web as soon as the beta release is available (expected in February 1996).

Lumina has proposed to further extend the accessibility of TAF to support wider review and discussion by developing a fully Web-browsable version of TAF during 1996. This would be a version of TAF structured as a linked series of Web pages, including influence diagrams, model and variable descriptions, model documentation, result tables, graphs, and maps, along with discussion of results. This version of TAF would look much like the current Analytica version, but it would be directly accessible from any computer platform supporting a World Wide Web browser, including MS Windows, Macintosh, and UNIX workstations. It would not require the user to download or run Analytica. It would provide precomputed tables and graphs for a broad set of model scenarios. Additional scenarios and results could be computed and linked in as desired. The development of this Web-browsable version of TAF is contingent on future funding.

2. The Choice of Analytica to Implement TAF

In the overview article to TAF and our presentations and demonstrations, we emphasized the following features of TAF. Each of these features is based directly on a corresponding feature of the Analytica modeling software used to implement TAF:

• The model is created, displayed, and documented as a set of influence diagrams -- a simple, visual representation of the variables, modules, and the influences that connect them.
• The model is organized as a hierarchy of modules, each defined by its inputs, outputs, and contents, in some cases containing its own submodules. Modules may be saved as separate files that can be tested and modified separately for simplicity.
• Any uncertain variable can be represented as a probability distribution. Probabilistic values are automatically propagated through the model using Monte Carlo simulation to estimate the probability distributions induced on any output variable. Probability distributions on inputs and outputs can be displayed in a wide variety of ways, including probability ranges, cumulative distributions, and probability density functions. Sensitivity and uncertainty analysis can be provided using rank-order correlations, and related methods, to compare the contribution of each input uncertainty to the uncertainty in outputs.
• The value of each variable can be a multidimensional array. The definitions or formulas relating one variable to another use intelligent arrays -- that is, they refer to and operate on other variables without regard to the number and identity of their dimensions, except when it matters to the definition. This feature of the Analytica modeling language makes it easy and flexible to modify the size and numbers of dimensions for each variable.
• Model documentation, including title, units, description, and definition for each variable, is integrated within the computer display of the model. The influence arrows in the influence diagrams are automatically kept consistent with the variable definitions in showing how the variables and modules depend on each other.
• We believe that it would have been quite impossible to provide these and other key features of TAF with the limited time and resources available had we tried to implement them directly in a conventional programming language, such as Pascal or C++, rather than employing Analytica. Analytica is itself implemented in Pascal and C, and so TAF is indirectly implemented in these standard computer languages. However, the effort to implement Analytica, and its predecessor, Demos, was many times the effort to implement TAF. It has taken over 12 years to develop Analytica and Demos in comparison to 1 year to develop TAF.
• Each of the ten modules in TAF was implemented in Analytica by a different team with at least one member who has become expert in Analytica. Early in 1995, Lumina conducted a training workshop to train TAF team members in the effective use of Analytica (then Demos) and related modeling methods. Typically, experienced modellers have been able to learn to use Analytica effectively in a week or two -- times comparable to learning how to use a new spreadsheet. Few of the TAF team members are fluent in C++. The time to learn to use C++ effectively is typically 3 to 6 months, even for experienced programmers.
• The flexibility and extensibility that Analytica provides to TAF can be illustrated by a recent extension to cover receptor sites over all of North America. At the review meeting on December 18 to 20th, 1995, some reviewers questioned the focus of TAF on the ten selected receptor sites for visibility and aquatics, and pointed out the desirability of analyzing changes in pollutant levels over the entire area of North America. Accordingly, we have expanded the dimensions of the receptors and corresponding transfer matrices, and can now generate ambient concentration and deposition levels for all contiguous states and Canadian provinces. This effort took about 2 days.

Most uncertain input values in TAF are represented by probability distributions. These distributions represent the range of values and relative likelihood of different values for these quantities. In most cases, these distributions represent expert judgment about the uncertainty, based on empirical evaluations of model accuracy against observed data wherever possible. These distributions are propagated automatically through TAF using conventional Monte Carlo or Latin hypercube sampling methods to generate corresponding probability distributions for each output variable. TAF also uses Analytica's facilities for uncertainty analysis to compare the relative contributions of uncertainty in the various inputs to each output, and hence to identify which input uncertainties have most effect on the results.

One reviewer commented that treating uncertainty explicitly 'cannot make a silk purse out of a sow's ear'. We agree completely. Explicit treatment of uncertainty in a model cannot, in itself, make a more model credible. We would add that the explicit treatment of uncertainty provides a clear and widely understood way for scientists to communicate to what degree they regard their findings as a silk purse or as a sow's ear. Given a credible model, calibrated and verified using empirical data, explicit assessment of uncertainty can serve to bound that model's credibility. This technique is very useful when combining models in an integrated assessment, where different submodels may represent their results with different degrees of uncertainty, based on each modeller's understanding of the state of the science and data in each model's field. Even judgmental characterizations of uncertainty can help us better interpret model results and provide guidance for priorities for future research that can be most cost-effective in reducing uncertainties.