Appendix B:
Consultation with Experts and Document Review
Monterey Bay National Marine Sanctuary
The process for preparing condition reports (and similarly, this update) involves a combination of accepted techniques for collecting and interpreting information gathered from subject matter experts. The approach varies somewhat from sanctuary to sanctuary, in order to accommodate different styles for work with partners. The Monterey Bay National Marine Sanctuary approach was closely related to the Delphi Method, a technique designed to organize group communication among a panel of geographically dispersed experts by using questionnaires, ultimately facilitating the formation of a group judgment. This method can be applied when it is necessary for decisionmakers to combine the testimony of a group of experts, whether in the form of facts or informed opinion, or both, into a single useful statement.
The Delphi Method relies on repeated interactions with experts who respond to questions with a limited number of choices to arrive at the best supported answers. Feedback to the experts allows them to refine their views, gradually moving the group toward the most agreeable judgment. For condition reports, the Office of National Marine Sanctuaries uses standardized questions related to the status and trends of sanctuary resources, with accompanying descriptions and five possible choices that describe resource condition (Appendix A).
In order to address the standardized questions, sanctuary staff selected and consulted outside experts familiar with water quality, living resources, habitat and maritime archaeological resources in the estuarine, nearshore, offshore and seamount environments. A few different approaches (e.g., small group meetings, conference calls, email and individual meetings) were used to get expert input on the questions, depending on the availability of experts (a list of experts who provided input is available in the Acknowledgement section of this report).
In these meetings and calls, experts were introduced to the questions and then asked to provide recommendations and supporting arguments. In small group settings and conference calls, the group converged in their opinion of the rating that most accurately described the current resource condition. In individual meetings and email correspondence, the sanctuary staff considered all input and decided on status and trend ratings. In all cases, draft status and trend ratings along with supporting narratives were made available to experts for individual comment.
Experts were also consulted to assign a level of confidence in status and trend ratings by: (1.) characterizing the sources of information they used to make judgments and (2.) their agreement that the available evidence supports the selected status and trend ratings. The evidence and agreement ratings were then combined to determine the overall confidence ratings, as described in the table here.
Step 1: Rate Evidence
Consider three categories of evidence typically used to make status or trend ratings: (1.) data,(2.) published information and(3.) personal experience.
| Evidence Scores |
| Limited |
Medium |
Robust |
| Limited data or published information, and little or no substantive personal experience. | Data available, some peer reviewed published information, or direct personal experience. |
Considerable data, extensive record of publication, or extensive personal experience. |
|
Step 2: Rate Agreement
Rate agreement among those participating in determining the status and trend rating, or if possible, within the broader scientific community. Levels of agreement can be characterized as "low," "medium" or "high." |
Step 3: Rate Confidence
Using the matrix below, combine ratings for both evidence and agreement to identify a level of confidence. Levels of confidence can be characterized as "very low," "low," "medium," "high" or "very high."
|
An initial draft of the update, which was written by sanctuary staff, summarized the new information, expert opinions and level of confidence expressed by the experts (who based their input on knowledge and perceptions of local conditions). Comments, data and citations received from the experts were included, as appropriate, in text supporting the ratings. This initial draft of the update was made available to contributing experts and data providers which allowed them to review the content and determine if the report accurately reflected their input, identify information gaps, provide comments or suggest revisions to the ratings and text. Upon receiving those comments, the writing team revised the text and ratings as they deemed appropriate. In some cases, additional review of certain sections, by those with specific expertise, was requested after revision. Sometimes, additional input on confidence scores was requested if the status and trend changed after those ratings had first been established in a small group setting.
In July 2015, a draft final report was sent to regional scientists for final review (listed in the Acknowledgemen’ section of this report). In December 2004, External Peer Review became a requirement when the White House Office of Management and Budget (OMB) issued a Final Information Quality Bulletin for Peer Review (OMB Bulletin) established peer review standards that would enhance the quality and credibility of the federal government’s scientific information. Along with other information, these standards apply to Influential Scientific Information, which is information that can reasonably be determined to have a "clear and substantial impact on important public policies or private sector decisions." The condition reports are considered Influential Scientific Information. For this reason, these reports are subject to the review requirements of both the Information Quality Act and the OMB Bulletin guidelines. Therefore, following the completion of every condition report, they are reviewed by a minimum of three individuals who are considered to be experts in their field, were not involved in the development of the report and are not ONMS employees. Comments from these peer reviews were incorporated into the final text of this report. Furthermore, OMB Bulletin guidelines require that reviewer comments, names and affiliations be posted on the agency website, http://www.cio.noaa.gov/. Reviewer comments, however, are not attributed to specific individuals. Comments by the External Peer Reviewers are posted at the same time as the formatted final document.
The reviewers were asked to review the technical merits of resource ratings and accompanying text, as well as to point out any omissions or factual errors. Following the External Peer Review, the comments and recommendations of the reviewers were considered by sanctuary staff and incorporated, as appropriate, into a final draft document. The final interpretation, ratings and text in the draft condition report were the responsibility of sanctuary staff, with final approval by the sanctuary superintendent. To emphasize this important point, authorship of the report is attributed to the sanctuary alone. Subject experts were not authors, though their efforts and affiliations are acknowledged in the report.
Estuarine Environment - Confidence Scoring Table
|
Question |
2015 Rating |
Evidence
(Limited, Medium or Robust) |
Agreement
(Low, Medium or High) |
Confidence
(Very Low, Low, Medium, High or Very High) |
Water Quality |
Question 1: Multiple Stressors |
Status: Fair/Poor |
Not updated |
Not updated |
Not updated |
Trend: Declining |
Not updated |
Not updated |
Not updated |
Question 2: Eutrophic Condition |
Status: Fair/Poor |
Robust |
High |
Very High |
Trend: Declining |
Robust |
High |
Very High |
Question 3: Risks to Human Health |
Status: Fair/Poor |
Not updated |
Not updated |
Not updated |
Trend: Undetermined |
Not updated |
Not updated |
Not updated |
Question 4: Human activities and Water Quality |
Status: Fair |
Medium |
High |
High |
Trend: Improving |
Medium |
High |
High |
Habitat |
Question 5: Major Habitat |
Status: Fair/Poor |
Robust |
High |
Very High |
Trend: Not changing |
Medium |
Low |
Low |
Question 6: Biologically-Structured |
Status: Poor |
Robust |
High |
Very High |
Trend: Improving |
Robust |
Medium |
High |
Question 7: Contaminants |
Status: Fair/Poor |
Low |
Low |
Very Low |
Trend: Declining |
Low |
Low |
Very Low |
Question 8: Human Activities and Habitat |
Status: Poor |
Medium |
Medium |
Medium |
Trend: Improving |
Medium |
Low |
Low |
Living Resources |
Question 9: Biodiversity |
Status: Fair |
Medium |
Medium |
Medium |
Trend: Not changing |
Medium |
Low |
Low |
Question 11: Non-Indigenous Species |
Status: Poor |
Medium |
Medium |
Medium |
Trend: Not changing |
Medium |
Medium |
Medium |
Question 12: Status Key Species |
Status: Fair/Poor |
Robust |
High |
Very High |
Trend: Improving |
Robust |
High |
Very High |
Question 13: Condition Key Species |
Status: Good/Fair |
Limited |
Medium |
Low |
Trend: Undetermined |
Limited |
Medium |
Low |
Question 14: Human Activities and Living Resources |
Status: Fair/Poor |
Medium |
Medium |
Medium |
Trend: Undetermined |
Limited |
Medium |
Low |
Maritime Archaeological Resources |
Question 15: Integrity |
Status: Undetermined |
Not updated |
Not updated |
Not updated |
Trend: Undetermined |
Not updated |
Not updated |
Not updated |
Question 16: Threat to Environment |
Status: Good |
Not updated |
Not updated |
Not updated |
Trend: Not changing |
Not updated |
Not updated |
Not updated |
Question 17: Human Activities |
Status: Good |
Not updated |
Not updated |
Not updated |
Trend: Not changing |
Not updated |
Not updated |
Not updated |
Nearshore Environment - Confidence Scoring Table
|
Question |
2015 Rating |
Evidence
(Limited, Medium or Robust) |
Agreement
(Low, Medium or High) |
Confidence
(Very Low, Low, Medium, High or Very High) |
Water Quality |
Question 1: Multiple Stressors |
Status: Fair |
Not updated |
Not updated |
Not updated |
Trend: Declining |
Not updated |
Not updated |
Not updated |
Question 2: Eutrophic Condition |
Status: Fair |
Robust |
Medium |
High |
Trend: Declining |
Robust |
Medium |
High |
Question 3: Risks to Human Health |
Status: Fair |
Robust |
High |
Very High |
Trend: Undetermined |
Robust |
High |
Very High |
Question 4: Human Activities and Water Quality |
Status: Fair |
Limited |
High |
Medium |
Trend: Improving |
Limited |
High |
Medium |
Habitat |
Question 5: Major Habitat |
Status: Fair |
Robust |
High |
Very High |
Trend: Declining |
Robust |
High |
Very High |
Question 6: Biologically-Structured |
Status: Good |
Robust |
High |
Very High |
Trend: Not changing |
Robust |
High |
Very High |
Question 7: Contaminants |
Status: Fair/Poor |
Medium |
High |
High |
Trend: Declining |
Medium |
High |
High |
Question 8: Human Activities and Habitat |
Status: Fair |
Robust |
Low |
Medium |
Trend: Undetermined |
Robust |
Low |
Medium |
Living Resources |
Question 9: Biodiversity |
Status: Fair |
Robust |
High |
Very High |
Trend: Not changing |
Robust |
High |
Very High |
Question 11: Non-Indigenous Species |
Status: Good |
Robust |
High |
Very High |
Trend: Declining |
Robust |
High |
Very High |
Question 12: Status Key Species |
Status: Fair |
Robust |
High |
Very High |
Trend: Declining |
Robust |
High |
Very High |
Question 13: Condition Key Species |
Status: Fair |
Robust |
High |
Very High |
Trend: Declining |
Robust |
High |
Very High |
Question 14: Human Activities and Living Resources |
Status: Fair |
Robust |
High |
Very High |
Trend: Declining |
Robust |
High |
Very High |
Maritime Archaeological Resources |
Question 15: Integrity |
Status: Fair |
Not updated |
Not updated |
Not updated |
Trend: Undetermined |
Not updated |
Not updated |
Not updated |
Question 16: Threat to Environment |
Status: Fair |
Medium |
Medium |
Medium |
Trend: Declining |
Medium |
Medium |
Medium |
Question 17: Human Activities |
Status: Good/Fair |
Not updated |
Not updated |
Not updated |
Trend: Undetermined |
Not updated |
Not updated |
Not updated |
Offshore Environment - Confidence Scoring Table
|
Question |
2015 Rating |
Evidence
(Limited, Medium or Robust) |
Agreement
(Low, Medium or High) |
Confidence
(Very Low, Low, Medium, High or Very High) |
Water Quality |
Question 1: Multiple Stressors |
Status: Fair |
Medium |
High |
High |
Trend: Declining |
Robust |
High |
Very High |
Question 2: Eutrophic Condition |
Status: Good/Fair |
Robust |
High |
Very High |
Trend: Declining |
Medium |
Medium |
Medium |
Question 3: Risks to Human Health |
Status: Good/Fair |
Not updated |
Not updated |
Not updated |
Trend: Undetermined |
Not updated |
Not updated |
Not updated |
Question 4: Human Activities and Water Quality |
Status: Fair |
Not updated |
Not updated |
Not updated |
Trend: Improving |
Not updated |
Not updated |
Not updated |
Habitat |
Question 5: Major Habitat |
Status: Fair |
Medium |
High |
High |
Trend: Improving |
Low |
High |
Medium |
Question 6: Biologically-Structured |
Status: Fair/Poor |
Medium |
High |
High |
Trend: Undetermined |
Low |
High |
Medium |
Question 7: Contaminants |
Status: Fair |
Medium |
High |
High |
Trend: Declining |
Medium |
High |
High |
Question 8: Human Activities and Habitat |
Status: Fair |
Medium |
High |
High |
Trend: Improving |
Medium |
High |
High |
Living Resources |
Question 9: Biodiversity |
Status: Fair |
Medium |
Medium |
Medium |
Trend: Not changing |
Low |
Medium |
Low |
Question 11: Non-Indigenous Species |
Status: Good |
Not updated |
Not updated |
Not updated |
Trend: Not changing |
Not updated |
Not updated |
Not updated |
Question 12: Status Key Species |
Status: Good/Fair |
Medium |
Medium |
Medium |
Trend: Not changing |
Medium |
Low |
Low |
Question 13: Condition Key Species |
Status: Good/Fair |
Medium |
Medium |
Medium |
Trend: Declining |
Medium |
Low |
Low |
Question 14: Human Activities and Living Resources |
Status: Fair |
Medium |
Medium |
Medium |
Trend: Not changing |
Medium |
Medium |
Medium |
Maritime Archaeological Resources |
Question 15: Integrity |
Status: Undetermined |
Not updated |
Not updated |
Not updated |
Trend: Undetermined |
Not updated |
Not updated |
Not updated |
Question 16: Threat to Environment |
Status: Fair |
Medium |
Medium |
Medium |
Trend: Declining |
Medium |
Medium |
Medium |
Question 17: Human Activities |
Status: Good/Fair |
Not updated |
Not updated |
Not updated |
Trend: Undetermined |
Not updated |
Not updated |
Not updated |
Seamount Environment - Confidence Scoring Table
|
Question |
2015 Rating |
Evidence
(Limited, Medium or Robust) |
Agreement
(Low, Medium or High) |
Confidence
(Very Low, Low, Medium, High or Very High) |
Water Quality |
Question 1: Multiple Stressors |
Status: Undetermined |
N/A |
N/A |
N/A |
Trend: Undetermined |
N/A |
N/A |
N/A |
Question 2: Eutrophic Condition |
Status: Undetermined |
N/A |
N/A |
N/A |
Trend: Undetermined |
N/A |
N/A |
N/A |
Question 3: Risks to Human Health |
Status: Undetermined |
N/A |
N/A |
N/A |
Trend: Undetermined |
N/A |
N/A |
N/A |
Question 4: Human Activities and Water Quality |
Status: Good/Fair |
Limited |
High |
Medium |
Trend: Undetermined |
Limited |
High |
Medium |
Habitat |
Question 5: Major Habitat |
Status: Good |
Robust |
High |
Very High |
Trend: Stable |
Medium |
High |
High |
Question 6: Biologically-Structured |
Status: Good |
Robust |
High |
Very High |
Trend: Undetermined |
Limited |
High |
Medium |
Question 7: Contaminants |
Status: Undetermined |
N/A |
N/A |
N/A |
Trend: Undetermined |
N/A |
N/A |
N/A |
Question 8: Human Activities and Habitat |
Status: Good/Fair |
Medium |
High |
High |
Trend: Undetermined |
Limited |
High |
Medium |
Living Resources |
Question 9: Biodiversity |
Status: Good |
Robust |
High |
Very High |
Trend: Undetermined |
Medium |
High |
High |
Question 11: Non-Indigenous Species |
Status: Good |
Limited |
High |
Medium |
Trend: Not changing |
Limited |
High |
Medium |
Question 12: Status Key Species |
Status: Good/Fair |
Robust |
Medium |
High |
Trend: Increasing |
Medium |
High |
High |
Question 13: Condition Key Species |
Status: Good |
Medium |
High |
High |
Trend: Not changing |
Limited |
High |
Medium |
Question 14: Human Activities and Living Resources |
Status: Good/Fair |
Medium |
High |
High |
Trend: Undetermined |
Limited |
High |
Medium |
Maritime Archaeological Resources |
Question 15: Integrity |
Status: N/A |
N/A |
N/A |
N/A |
Trend: N/A |
N/A |
N/A |
N/A |
Question 16: Threat to Environment |
Status: N/A |
N/A |
N/A |
N/A |
Trend: N/A |
N/A |
N/A |
N/A |
Question 17: Human Activities |
Status: N/A |
N/A |
N/A |
N/A |
Trend: N/A |
N/A |
N/A |
N/A |