Skip to main content
Effective Health Care Program

Suicide Prevention

Research Protocol

Background and Objectives for the Systematic Review

Despite advances in the diagnosis and treatment of major mental disorders, and increases in funding for suicide prevention, annual all-ages suicide rates in the United States have been stable for the past 60 years at around 10 to 12 per 100,000.1,2 In 2013 suicide was the second leading cause of death among 15-19 and 20-29 year olds.1 According to estimates from the Centers for Disease Control and Prevention (CDC) national Youth Risk Behavior Surveillance System, well over one million high school students are treated by a nurse or doctor annually for a suicide attempt.3

In 2014, the National Action Alliance for Suicide Prevention (NAASP) Research Prioritization Task Force developed a suicide prevention research agenda targeting interventions with the potential to reduce morbidity (attempts) and mortality (deaths), by at least 20% in 5 years, and 40% or greater in 10 years. Specifically, the Task Force prioritized the prevention of "the emergence of suicidal behavior by developing and delivering the most effective prevention programs to build resilience and reduce risk in broad-based populations."3

Several unanswered questions remain regarding the effectiveness of youth suicide prevention efforts; leveraging existing data through a creative, rigorous, and efficient systematic review will help to address these questions. Suicide prevention scientists encounter challenges in determining intervention impact due to the following issues: (1) suicide is a rare outcome, requiring large studies to demonstrate an intervention effect; (2) misclassification and under-reporting of suicide and suicide attempts occur due to stigma and other issues, (3) there is no single, comprehensive national system to document the scope of non-fatal suicide events; (4) interventions are often complex or "bundled" making it difficult to know which components are responsible for outcomes; and (5) the nature of populations at risk and interventions available require using quasi-experimental designs and "natural experiments" to evaluate prevention efforts, making it difficult to weigh the strength of available data.

Definitions and common terms

As the definitions of suicidal behavior vary greatly, it is challenging to combine or compare the results of studies.4 Until 1996, there was no standard nomenclature (common vocabulary) for suicide-related behaviors. Multiple terms are used in the literature to describe suicide-related behaviors (e.g., self-destructive behavior, suicide, suicide attempt, completed suicide, suicide gesture, suicide acts, self-directed violence, deliberate self-harm, parasuicide (suicide attempt that does not result in death), self-inflicted injury, intrapersonal violence, self-mutilation). These terms vary by level of lethality and suicide intent. To facilitate communication and comparison of data, definitions were proposed by O'Carroll et al. (1996)5 with a revision proposed in 2007 by Silverman and colleagues.6 The extent to which these recommended definitions are followed is unclear. The current definition advocated by the CDC is self-directed violence (SDV), or behavior that is self-directed and deliberately results in injury or the potential for injury to oneself. Our systematic review will include only those outcomes where suicide intent is present.

Lack of data on moderators

As pointed out by the Data and Surveillance Task Force of the NAASP (2014),7 the data systems used to estimate trends in suicidal behavior were not designed solely to address this subject. In these data systems, questions specific to suicidal behavior are often limited, and the collected data rarely provide the depth of information desired to inform effective prevention and intervention efforts. For instance, key moderators such as sexual orientation and gender identity have been identified as risk factors for suicidal behavior in multiple studies but are not routinely collected in national systems.

Time lag for available data

There is a significant lag (~2-3 years) from when suicides happen and when the CDC data are available, making it difficult to use these data to inform prevention approaches and accurately estimate the suicide rate in a timely manner.8 The most recent national mortality data from the CDC Web-based Injury Statistics Query and Reporting System (WISQARS) database are from 2013.

Lack of long-term follow-up data

Many suicide prevention programs have limited ability to study long-term outcomes under the current funding structure. National, state and community data systems could be linked to existing data from suicide prevention efforts in order to study the longer term and broader intervention impact. However, linking existing data has not been done possibly due to associated costs, feasibility of accessing data systems, applicability, and issues of sharing protected health information (PHI). Direct data linkage on an individual level may require personally identifying information such as name(s), Social Security Numbers, addresses, dates of birth, and insurance carriers which are mainly protected by HIPAA (i.e., Data Ethics and Governance issues). If this information is available from prevention efforts the following sources of outcome data can be utilized to study the impact of interventions on suicidal behaviors: National Death Index (NDI), Administrative Health Insurance Claims data, and Electronic Medical Record data. However, each of these data systems has limitations that affect their usefulness in research and/or real world applications. The NDI has been linked to prevention studies but there are associated data linkage costs, and informed consent is needed for sharing PHI.

Opportunities for using aggregate/ecologic data

If individual-level identifiers are not available from intervention studies, data in aggregate can be used to study whether interventions had an impact on local suicide rates and other intermediate or proxy outcomes of interest. In this case, the following data systems can be utilized to examine suicide mortality: CDC WISQARS data, National Violent Death reporting System, DoD (Department of Defense) Suicide Event Report (DoDSER). The following data systems can be used to study suicide morbidity: hospital discharges for self-inflicted injuries, AHRQ's Healthcare Cost and Utilization Project - Nationwide Inpatient Sample (HCUP-NIS), emergency department self-inflicted injury admissions (Source: CDC's National Electronic Injury Surveillance System – All Injury Program), self-reported suicidal thoughts and behaviors (CDC Youth Risk Behavior Survey YRBS), social media (Google, Facebook), and data on suicide hotline calls (National Suicide Prevention Lifeline). If the national, state and local datasets assessing suicide and suicide attempts could be organized through a central data broker, data linkage would be facilitated.

Future possibilities for combining data across studies

Standardized initiatives to improve quality of care, the rapid expansion of electronic health record (EHR) systems and patient registries, and the recent passage of the Patient Protection and Affordable Care Act (ACA) have all offered an exceptional opportunity to make large-scale system changes to prevent suicide.9 Passage of the ACA will lower the uninsured rate by expanding public and private insurance coverage. As mentioned in NOT-MH-14-015 (Data Sharing Expectations for NIMH-funded Clinical Trials), the widespread data sharing by research communities adds significant value to research and accelerates the pace of discovery. The NIMH issued the National Database for Clinical Trials Related to Mental Illness which intends to establish a common informatics platform for exchanging data from clinical trials. This effort will involve the use of consent forms that allow broad data sharing within the research community, creating global unique identifiers (GUIDS) for all of their research participants, creating data dictionaries that are relevant to their research and sharing results, positive and negative, specific to the cohorts and outcome measures studied. The standard collection of data on type of intervention (e.g., mental health promotion, education and training, treatment, protocols and guidelines), intervention dose, fidelity, and population targeted (universal, indicated, selective) would help facilitate the study of intervention impact.

Proposed review

Given the above considerations, the proposed project aims to provide an objective description of the state of the science on data linkage strategies and analytic approaches in suicide prevention research, as well as a systematic summary of ongoing research and research needs to serve as the foundation for an NIH Pathway to Prevention workshop. The review will focus on children, youth, and young adults ages 6-25. This spans the age range from when suicide is relatively rare, but when primary preventive efforts may be effective, through adolescence and young adulthood, when suicide rates precipitously increase. Interventions will be examined at all socio-ecologic levels (individual, community, and policy-level) and will include primary promotion of mental wellness as well as interventions targeting suicidal ideation, attempts, and completions.

Key Questions

This project topic was nominated by the NIH Pathways to Prevention program. The Key Questions (KQ) were derived from the Pathways to Prevention topic submission form and did not undergo external review by key informants. Additionally the KQs were not released for public comment.

KQ1

What national, state and community data systems can be linked to existing data from suicide prevention interventions in order to add possible value for stakeholders, and what methods are available to link the data systems?

KQ2

Which statistical methods are reliable and valid for analyzing linked national, state, and community data systems and suicide prevention data to avoid misleading conclusions?

  1. What are potential sources of bias for these statistical methods?
  2. What are the advantages and disadvantages of these different methods?

KQ3

Which statistical methods are reliable and valid for understanding possible moderators in suicide prevention programs to improve targeting interventions to populations?

KQ4

Given the current state of research, what types of methodological/analytic advances would promote further evaluation of youth suicide prevention efforts (e.g., new approaches to data linkage; increased use of common data elements; approaches to intervention harmonization) and facilitate intervention selection and implementation decisions by local community and state level policy makers?

PICOTS

Table 1 describes the parameters of the inclusion criteria for studies that may provide or identify data systems for suicide prevention interventions in KQs 1 through 3.

Table 1: PICOTS description of inclusion criteria for all key questions
PICOTS Criteria
* Suicide ideation reported over the last 12 months reduces the potential for recall bias and provides more relevant information for current prevention and intervention.10,11
Population(s) Received intervention: ages 0-25; (longitudinal follow-up past age 25 is acceptable)
Intervention(s) Behavioral, community, clinical/medical, policy level including studies promoting wellness targeting suicide ideation, suicide attempt, and suicide completion, or any combination of these interventions and outcomes.
Comparison(s) Any intervention (including usual care).
Outcome(s)

(the primary and intermediate outcomes)

Primary outcome of interest:
  • Suicide ideation, reported within 12 months of data collection
  • Suicide attempt, any time point post intervention
  • Suicide completion, any time point post intervention

Intermediate outcomes, at any time-point post intervention:

  • Psychiatric and substance abuse disorders
  • Service use (e.g. ER visit) for psychiatric/substance abuse disorders
  • Graduation rates
  • Incarceration rates
  • Violence (both perpetrator and victim)
  • Social support and connectedness
  • Access to lethal means

All outcomes will be limited to standardized measures.

Timing Publication date: 1990 or later.
Setting Studies taking place in the United States only: Schools, home, primary care, emergency department, military bases, Indian reservations, juvenile justice systems, child welfare systems, suicide hotlines, other community settings.

Analytic Framework

The following figure depicts the key questions within the context of the PICOTS described in the previous section (see Table 1 and Figure 1). The figure illustrates how data, in populations 6 to 25 years old, from suicide prevention interventions (including behavioral, community, clinical/medical, policy level) and national, community, or state data systems can be linked. Through valid and reliable statistical methods, these linked data can be analyzed for sources of bias or advantages and disadvantages as well measure the impact of moderators (gender, race and ethnicity, LGBTQ status, psychopathology status, trauma history, social connectedness, and social support).

Figure 1: Preliminary analytic framework for suicide prevention

The Analytic framework graphically describes how, in a population of youth aged 6-25 years old, national community, and state data systems, or suicide prevention programs could be linked (Key Question 1). Linkage of this data will require valid and reliable statistical methods that can be used to analyze the data (Key Question 2), or to identify moderators (Key Question 3). The analysis of the linked data and the moderators will allow for identification of specified intermediate (psychiatric and substance abuse disorders, graduation rates, incarceration rates, violence, social support and connectedness, access to lethal means, and service use), and final health outcomes (suicidal ideation, suicide attempt, or suicide completion).

ER = Emergency room; KQ = Key Question; LGBTQ: Lesbian, Gay, Bisexual, Transgender, and Queer
*See a preliminary list of prevention intervention programs in Appendix A. The list of suicide prevention programs will be supplemented by information identified by the literature search, the environmental scan, and the targeted state search as well as by input provided by external experts.

Methods

Due to the complexity and uniqueness of this systematic review, which focuses on data systems and statistical methods used in suicide prevention studies, the overall review process includes a number of parallel search methodologies and additional data abstraction and data synthesis steps for each key question (KQ). The search methodologies outlined below include details such as: the proposed search criteria, search engines, and primary abstraction considerations. Additional/secondary data abstraction and data synthesis methods for individual KQs include details such as: coding schema, abstraction considerations, and data aggregation and summarization methods.

Search Methodologies

The review process will be accomplished in three separate but parallel phases (see Table 2). In the first phase, a systematic review of literature will be completed to identify suicide prevention studies and suicide outcome data sources to which data from studies can be linked. The results of this phase will inform KQs 1, 2 and 3. In the second phase, an environmental scan will be performed to review and include suicide prevention data systems that are not covered by academic papers. This phase will further inform KQ 1. In the last phase as described in detail on page 10, a targeted search will be performed to identify data systems used in selected states, cities or communities. This phase will expand the results of KQ 1 and provide practical examples of data systems that can be leveraged by suicide prevention stakeholders including researchers. This phase will be limited to certain geographic regions because executing a data system search on all U.S. states, cities, and communities is beyond the scope of this review and is not the primary goal (which is to identify exemplar possibilities for data analysis and linkage rather than developing an exhaustive list of all possibly-available datasets). All phases will use the same PICOTS.

Table 2: Overall Search Methodology to Address Key Question 1, 2 and 3*
Key Question Search Methodology Search Criteria Search Engines or Data Sources Number of Raters/ Coders Data System Identification Statistical Method Identification Moderator Identifications
*KQ4 is not included in this table as it is considered a commentary on ‘opportunities and challenges’ and a search methodology does not apply to it.
General 'Inclusion and Exclusion Criteria' will be the same for all search methods, and therefore are not repeated in this table. Please refer to the text for further information. (c) A total of six states, two cities and one local community will be selected based on CDC report on suicide rates. Terms representing these states, cities or local communities will be added to the PICOTS' keywords.
See Table 1 for the full PICOTS description
KQ1 Systematic Literature Review PICOTS
  • PubMed
  • Cochrane
  • Campbell
  • CINAHL
  • PsycINFO
  • ERIC
2 Yes - -
KQ2 Systematic Literature Review PICOTS
  • PubMed
  • Cochrane
  • Campbell
  • CINAHL
  • PsycINFO
  • ERIC
2 - Yes -
KQ3 Systematic Literature Review PICOTS
  • PubMed
  • Cochrane
  • Campbell
  • CINAHL
  • PsycINFO
  • ERIC
2 - - Yes
KQ1 Environmental Scan (Web) PICOTS
  • Google
  • Yahoo
  • Bing
1-2 Yes - -
KQ1 Targeted Search PICOTS
+ Location
Selected state, city and local government websites 1-2 Yes - -

Note that the following section does not include the additional data abstraction process for individual KQs, as each question follows a different data aggregation and synthesis methodology. The primary data abstraction processes are described in detail for each phase of the review. To avoid repetition, items that are similar across all search methodologies are only listed once.

Systematic Literature Review

Search Criteria (Inclusion and Exclusion)

The principal inclusion and exclusion criteria for the systematic literature review will be derived from the identified PICOTS (see Table 1). All studies on prevention programs targeting suicide prevention in people 6 to 25 will be used to identify potential data sources. We will include programs taking place in the U.S. only and administered in schools, home, primary care, emergency departments, juvenile justice system, child welfare system, the military, through suicide hotlines, and other community settings. We will not limit study inclusion by study population size or study design. Studies will not be included if published prior to 1990 because, according to the National Action Alliance for Suicide Prevention, suicide became a central issue in the United States in the mid-1990s with the Surgeon General's Call to Action to Prevent Suicide published in 1999. The amount of abstractable data will be more limited prior to 1990.12

Search Engines (Literature Search Strategies)

We will search PubMed, Cochrane Collaboration, Campbell Collaboration, CINAHL, PsycINFO and ERIC. A search strategy has been developed for PubMed and will be used as a guide to develop search strategies in the other search engines (see Appendix B for the preliminary search strategy). We will search for articles authored in English (we are focusing on interventions used in the United States) and published between 1990 to present. Articles will be selected through independent screening by two screeners based on the inclusion and exclusion criteria.

Data Abstraction and Data Management

We will use Distiller SR (Evidence Partners, Ottawa, Canada) to manage the screening process. Distiller SR is a web-based data management program that manages all levels of the review process. Data from applicable articles will be abstracted and uploaded to the Systematic Review Data RepositoryTM (SRDR), a web-based data repository. This source serves as both an archive and a data abstraction tool. Data will be exported from SRDR into a project-specific database to serve as archived or backup copies and to create detailed evidence and summary tables.

We will abstract data about the PICOTS (see Table 1) including study and participant characteristics, as well as details about data systems and statistical methods. The final coding schema used for the data abstraction is not finalized yet. A preliminary list of "Intermediate Outcome Measures" is included in Appendix C. This list will be used as a starting point to develop the coding schema for the systematic literature review.

Coding Dispute Resolution

After data have been abstracted by two trained research assistants, an independent expert will review a random sample for quality assurance. If a data abstraction disagreement arises between the research assistants and the expert, another expert will review, and the issue will be resolved by a consensus approach. If consensus is not attainable for a specific case, it will be discussed among the rest of the review team and resolved by the majority of vote. The same process will be used for all data abstraction activities throughout this review project.

Assessment of Bias

This systematic review is primarily concerned with data systems used in suicide prevention programs and not the effectiveness of interventions in changing certain outcomes of those studies. A formal assessment of bias, which focuses on the quality of the study in evaluating its intervention or outcomes, will not be performed on the included papers.

Relevant Key Questions

The results of the systematic literature review will inform KQs 1, 2 and 3. For example, data systems identified during the data abstraction will be further reviewed and evaluated in the data synthesis phase of KQ 1. Likewise, statistical methods identified by the data abstraction process will be further evaluated in KQ 2, while moderators will be analyzed in KQ 3.

Scientific information packets will not be sought for this project, as they are not applicable.

Environmental Scan

Suicide prevention interventions are also conducted in operational settings where results may not be reported in scientific publications. The data systems (Appendix D) utilized by these operational suicide prevention programs, however, are of interest to the research community and need to be represented in the data synthesis of KQ 1. To represent such interventions, the review team will complement the systematic literature review with an environmental scan of grey literature (e.g., online reports) on suicide prevention programs among youth. The grey literature can include reports (pre-prints, preliminary progress and advanced reports, technical reports, statistical reports, memoranda, state-of-the art reports, market research reports, etc.), theses, conference proceedings, technical specifications and standards, non-commercial translations, bibliographies, technical and commercial documentation, and official documents not published commercially (primarily government reports and documents).13 The list of data systems (Appendix D) will be supplemented by information identified by external experts.

Search Criteria (Inclusion and Exclusion)

PICOTS will be used to develop the inclusion and exclusion criteria for the environmental scan (see Table 2). The same keywords, which were developed based on PICOTS and used in the systematic literature review, will be used for the environmental scan. Inclusion and exclusion criteria will also remain intact (e.g., only include English-language documents published between 1990 and present). We will not limit the environmental scan based on the type of document retrieved from the web (e.g., the document can be an HTML webpage or a PDF/DOC report).

Search Engines (Environmental Scan Strategies)

The environmental scan will complement the literature review by identifying additional data systems that may be helpful in evaluating suicide prevention interventions. For the environmental scan, the project team will conduct general searches to identify examples of interventions that meet the inclusion criteria. We will conduct the environmental scan by using three search engines: Google, Yahoo and Bing (Microsoft). These search engines collectively represent 97% of the U.S. online search market.14 The advanced search functions of these search engines will be used to execute the search. Due to the exhaustive list of results returned by these search engines (e.g., which may result in millions of documents), only the first 100 retrieved documents from each database will be considered in the environmental scan. This limitation is also partly due to the fact that precision of these search engines often reduces considerably after the first few dozens of results.15 However, to assure a high recall rate of the environmental scan, the review team will examine every other 10th of the results ranked between the 100th to the 300th results of these search engines. If at least a third of these results are determined to be relevant to this review, the environmental scan will be expanded to the first 300 results. Duplicate results across the search engines will be excluded from the review. Results that are already covered by the systematic review of the literature will be excluded as well.

We will also conduct a detailed exploration of specific government, foundation and professional association websites (national/federal, state-level, and local/city/community-level) such as the American Foundation for Suicide Prevention (AFSP), the American Association of Suicidology (AAS) and the Suicide Prevention Resource Center (SPRC).

Data Abstraction and Data Management

Data abstraction for the environmental scan is the same as for the systematic literature review; the goal of the scan is to find reports and other sources that are similar to those that have been published but that have not been disseminated in scientific journals. We will abstract data about the PICOTS (see Table 1), such as study and participant characteristics, as well as details about the data systems and statistical methods. The final coding schema will be similar to the schema used for the systematic review phase.

Coding Dispute Resolution

Coding disputes will be handled similar to the systematic review phase.

Relevant Key Questions

The results of the environmental scan will only inform KQ 1. Data systems identified during the data abstraction of the environmental scan will be further reviewed and evaluated in the data synthesis phase of KQ 1. However, KQ 2 and 3 will not use the results of the environmental scan due to the fact that grey literature is often not peer reviewed and thus may not provide reliable and valuable information on effective statistical methods and potential moderators to start with.

Targeted Search

State, city or community-level governments/organizations administer data systems that may hold valuable data for suicide prevention researchers (e.g., primary and intermediate outcomes). These data systems are often not represented in publications or reports. In order to provide researchers with a practical sample set of state, city, and community data systems, we will also include a targeted search of data systems providing information on suicide prevention in six states along with two cities and one local community in each state. The list of states will include CA, DE, OR, IL, MD and WI (see below for selection criteria). Cities and local communities will be selected after the state-level search is completed. This phase will be limited to these pre-selected geographies as executing a data system search on all U.S. states, cities, and communities is neither the primary goal of this review nor within its scope. Note that in contrast to the systematic review and the environmental scan, this targeted search is not retrieving suicide prevention studies and instead is focusing on data systems that provide primary or intermediate outcomes of suicide prevention interventions (see Figure 1), and are maintained by a state, city or the community-level entity. Indeed, the results of this targeted search may provide a list of data systems that are not necessarily originated from a suicide prevention program, but could be a valuable source of data for suicide prevention stakeholders and researchers.

Search Criteria (Inclusion and Exclusion)

PICOTS will be used to develop the inclusion and exclusion criteria for the targeted search (see Table 1). The same keywords developed based on PICOTS and used in the systematic literature review will be used for the targeted search; keywords representing geographical locations (e.g., California) will be added to the search criteria. We will not limit the targeted search based on the type of document retrieved from the web (e.g., the document can be an HTML webpage or a PDF/DOC report). Only documents relevant to the selected list of states, cities and communities will be included. Additional specific inclusion criteria include: data system is still in existence with underlying data available and accessible in digital format; data system is sharable and can be acquired by others for research purposes; data system collects/contains at least one of the primary or intermediary outcomes; and, the data system is not a duplicate of another data system already included in the review. We assume there will be some overlap between the data systems (Appendix D) discovered in this approach and those identified by the systematic review and the environmental scan. Duplicates will be excluded.

Search Engines (Environmental Scan Strategies)

The targeted search will provide practical examples of state, city or community-level data systems that can be utilized by suicide prevention researchers. We assume there will be some overlap between the data systems (Appendix D) discovered in this approach and those identified by the systematic review and the environmental scan. Duplicates will be excluded. The data systems identified through this search method should not be assumed as an exhaustive list of data systems and rather will be considered a sample list from which similar ones can be searched for other geographical settings. The methodology for this targeted search will follow the same methodology of the environmental scan, however, only documents relevant to the selected list of states, cities and communities will be included. Specific inclusion criteria include:

  1. Data system is still in existence, and underlying data is available and accessible in digital format (e.g., datasets are downloadable from a current website).
  2. Data system is sharable and can be acquired by others for research purposes (e.g., it has a public or transferable license that allows the data to be used for research purposes).
  3. Data system collects/contains at least one of the primary or intermediary outcomes.
  4. Data system is not a duplicate of another data system already included in the review.

We have selected six states for this targeted search: CA, DE, OR, IL, MD and WI. The selection process has been based on the following criteria: (1) the state has an active SAMHSA state Garrett Lee Smith youth suicide prevention grant which will facilitate the acquisition of information about suicide prevention strategies being implemented; (2) geographic proximity of matched pairs of low suicide rate state bordering a high suicide rate state; and to some extent, (3) familiarity of the research team experts with the data systems of those states. All of the six states match criteria #1 and #2 (see Figure 2). In regards to criteria #1, Oregon has a higher suicide rate in the Pacific region (16.12/100k), while California has a lower suicide rate in the same region (9.83/100k). Likewise, Wisconsin has a higher suicide rate in the Midwest region (12.65/100k), while Illinois has a lower suicide rate in the Midwest region (8.76/100k); and finally Delaware has a higher suicide rate in the Mid-Atlantic region (11.25/100k), while Maryland has a lower suicide rate (8.95/100k). Note that the national crude average rate of suicide in 2013 is 12.6 per 100k population.

Data Abstraction and Data Management

Similar to the systematic review, we will use Distiller SR to manage the screening process. All applicable documents identified by the search process are uploaded to the system. Data from applicable articles will be abstracted and uploaded to SRDR. This source serves as both an archive and a data abstraction tool. Data will be exported from SRDR into a project-specific database to serve as archived or backup copies and to create detailed evidence and summary tables. The coding schema for data abstraction has not been finalized yet.

Coding Dispute Resolution

Coding disputes will be handled similar to the systematic review phase.

Relevant Key Questions

The results of this targeted search will only inform KQ 1. Data systems identified during the data abstraction of this phase will be further reviewed and evaluated in the data synthesis phase of KQ 1.

Figure 2: United States death rates due to suicide per 100,000 population (2013 data)

A map of the United States identifying suicide rates by state. Map generated by CDC WISQARS.

All races, ethnicities, sexes, and ages; Annualized crude rate for U.S is 11.59; Map generated by CDC WISQARS.

Secondary Data Abstraction

The results of the primary data abstraction of the aforementioned search methods will only code the suicide prevention study characteristics and identify the data systems, statistical methods, or moderators included in them. The primary data abstraction will not code the details of the data systems, methods or moderators; thus, each of the KQs will entail additional/secondary data abstractions. In addition to Distiller SR and SRDR, Microsoft Excel will also be used to organize the secondary abstraction process.

Key Question #1 – Additional Data Abstraction

Based on the information gathered in the primary data abstraction phase, and after removing duplicate data systems, the review team will proceed with the secondary data abstraction of the data systems. As each data system will be unique, there will be various search methods that may provide the team with the additional data required to complete the secondary data abstraction. For example, one data system might be a national data repository for suicide prevention and all of its specification might be accessible online, while another proprietary or copyrighted data system may require contacting the data custodian for additional information. Note that the search for additional information about these data systems is not intended, and is not required, to be a systematic approach as the primary list of data systems is already generated using a systematic approach. Thus, the activities involved with the secondary data abstraction should be considered attempts of the review team to gather additional information about the individual data systems and not the suicide prevention studies. We will use the following methods to acquire additional information about each of the data systems:

  1. Locate additional information about these data systems by means of: finding the data dictionaries associated with the data system; downloading a sample data set of the data system; browsing the data sources used in the data system; or, searching for additional reports that may have described the data system in more details.
  2. Contacting the authors of the publications/reports that have used these data systems or contacting the custodians of the data systems to receive additional information about the data systems. We will reach out to these individuals by email once, and if no response then repeat the email approach again in 7 and 14 days. If still no response, we will call/leave a voice message on day 30 as the last attempt to reach out to the authors and/or data custodians.

After locating the additional information about the data systems, two independent reviewers will decide if the identified data systems fulfill the minimum requirements of a data system that can be useful for suicide prevention studies. Data systems that do not meet all of these requirements will be excluded from the secondary data abstraction phase. These minimum requirements are:

  1. Data system is still in existence, and underlying data is available and accessible in digital format (e.g., datasets are downloadable from a current website).
  2. Data system is sharable and can be acquired by others for research purposes (e.g., it has a public or transferable license that allows the data to be used for research purposes).
  3. Data system collects/contains at least one of the primary or intermediary outcomes.
  4. Data system is not a duplicate of another data system already included in the review.

The final list of data systems that meet the minimum requirements will be added to a 'master list' of data systems, and then coded/abstracted using an extended coding schema. Note that there is no unified, generalizable, validated, and commonly used framework for assessing the quality of a set of heterogeneous data systems. In order to complete the coding/classification schema, we will adapt a framework that has been developed in prior work to evaluate the quality of the community-based data sources.16 The coding schema for the secondary data abstraction is not finalized yet; the initial data system classification and coding schema is listed in Appendix E. The coding schema will include various perspectives of data quality issues within the data systems. Some of the data quality categories include: data relatedness and availability; data granularity; denominator coverage; data sampling; data scalability; data interoperability; data governance; data uses and functions; and data linkage mechanisms. The coding process will be limited to the information that could be extracted from the relevant data dictionaries of these data systems. Downloading and analyzing the data sources within each of these data systems will be out of the scope of this review (i.e., we will not run statistical calculations on these data systems to compute data quality indices and complete the data abstraction process).

Key Question #2 – Additional Data Abstraction

The systematic review will provide us with a list of statistical methods that are used in suicide prevention studies. The results of the environmental scan and targeted search methods will not be used for KQ 2. The primary data abstraction of the suicide prevention publications will be limited to the type of statistical methods and variables used to evaluate the interventions. More details about these statistical models will be encoded in the secondary data abstraction for KQ 2.

The secondary data abstraction will involve using current knowledge on biostatistical methodologies. This is mainly due to the fact that scientific publications often do not list all of the advantages and disadvantages of their statistical models unless their approach is uncommon. The secondary data abstraction process will include the following steps: (1) identify the primary analytic method used by the study based on the results of the primary data abstraction; (2) identify potential sources of biases as well as the benefits and risk of these methods using the current biostatistics literature as well as several methodology working groups including the Cochrane Group's handbook1; and, (3) code these statistical approaches using a coding schema that will identify the strengths and limitations of each analytic method. When available, we will also tabulate the actual statistical packages and procedures used.

The coding schema for the secondary data abstraction of KQ 2 is not finalized yet. The coding schema will include three main categories: (a) the source of bias for each of these methods and their potential effect on the validity and reliability of the suicide prevention studies; (b) the type of bias; and, (c) the advantages or disadvantages of each statistical method used in these studies.

1www.cochrane-handbook.org (accessed Jun 24 2015)

Key Question #3 – Additional Data Abstraction

Similar to KQ 2, the results of the systematic review will provide us with a list of moderators that are used in the suicide prevention studies as well statistical methods that have been used to explore the impact of potential moderators across studies. The results of the environmental scan and targeted search methods will not be used for KQ 3. Again, similar to KQ 2, the primary data abstraction will be limited to the list of moderators mentioned in these studies. We will abstract the following moderator variables: variables encountered, their general nature, specific limitations (i.e., missingness, skew) in the setting of suicide prevention research.

The coding schema for the secondary data abstraction of KQ 3 is not yet developed. The schema will involve coding fields to assess the general reliability and validity challenges associated with each of the identified statistical methods for understanding possible moderators in suicide prevention programs to improve targeting interventions to populations.

Data Synthesis

Key Question #1 – Data Synthesis

The collection of the primary and secondary data abstractions will include detailed information on data quality issues as well as data linkage challenges among the reviewed data systems. The information gathered within these steps will provide us with the information required to draw conclusions on which of the national, state, and community data systems can be linked to existing data from suicide prevention efforts in order to add value for stakeholders. In addition, the results of the data synthesis will provide us with information on what methods are being currently used to link these data systems (e.g., details of data linkage is part of the coding schema that will be used for the secondary data abstraction).

Potential incompleteness of the data abstraction may introduce errors and consequently biases in these findings. For example, not all data systems provide enough details about their data specifications and thus may cause missing information in the secondary data abstraction for KQ 1. This may result in incomplete coding for a number of data systems. Consequently, the data aggregation may not be accomplished across all coding attributes and all data systems. Most probably the data synthesis will involve various denominators of data systems thus limiting the generalizability of some of the findings.

Data will also be aggregated and summarized across various coding schema attributes (e.g., see coding schema attributes for the secondary data abstraction in Appendix E). The summary report will include ratios of various data system's attributes across the entire list of data systems. The report will be accompanied with suggestions on which models have been found to be effective and which ones have faced challenges.

Key Question #2 – Data Synthesis

We recognize that the ultimate goal of this key question is to identify methods that would provide lower bias and higher validity in utilizing multiple data systems in evaluating suicide prevention interventions. Meanwhile we do expect that various aspects of low data quality (such as missing data, varying labeling and specification of variables across systems) are inherent in certain data systems (e.g., clinical EHRs), and, thus such issues can undermine the reliability and generalizability of results of this key question.17 To flag potential data quality issues of the data systems and how they would affect the statistical methods used to link and aggregate them, we will use the data abstraction results of KQ 1. For example, mismatching data scalability between two different data systems (e.g., mixing county-wide aggregated data with personal-level data) can be challenging but specific geo-distribution algorithms and data matching processes can help reduce underlying noise. Another common instance is using specific statistical methods to deal with missing data,18 which is also considered a data quality issue.19 Harmonizing data extracted from data system originating from different levels (national, state, or local) also requires the extra effort by the statisticians and data scientists to develop common indicators for merging them. Typical analyses with these data systems include multi-level modeling if the linking variable is not at the individual level.20 If a linking variable is at the individual level, traditional causal methods can also be used. For example, Hlatky and colleagues21 linked Medicare data with their study participants using individual social security numbers and then compared hazard ratios for active versus placebo hormone therapy on a number of outcomes.

Key Question #3 – Data Synthesis

Leveraging existing data systems can aid in specifying for whom certain programs work and under what conditions. As described in KQ 2 synthesis section, there are a number of validated and experimental statistical methods to improve the accuracy of conclusions and to understand possible moderation effects that are made by aggregating and merging various community/state-wide data systems with suicide prevention research datasets. Similar to KQ 2, we will review the results of the data abstractions and propose a series of statistical and informatics methods to reduce the error generated by these types of analytical pitfalls based on the methods used in published studies.

Exploring moderation effects in analyses of pooled datasets is still a new area of research, with some researchers suggesting that meta-regression is the most accurate way to accomplish this.22,23 Brown and colleagues24 have developed methods for synthesizing data across multiple prevention trials where specific individual-level variables are consistent across trials. For example, Fernandes and colleagues25 used meta-regression to explore the effect of age, length of illness, and severity of mood disorder episode on BDNF protein levels across20 studies. If you have a common measure across studies, the measure can be used to explore moderation of the intervention impact, for example, if affiliation with peer smokers is measured across multiple trials, this individual-level covariate can be explored for its role as a potential moderator of intervention impact. These methods use multi-level latent variable modeling to explore moderator effects across trials and growth mixture models to address different times of measurement across trials.26 Thus, we anticipate that within the suicide prevention literature we will encounter studies using similar methods.

Additional Notes on Key Question 4 (Commentary)

Commentary on future research needs (KQ 4) will be based on the findings of KQ 1, 2 and 3. We will elaborate on other statistical approaches that can be used for suicide prevention studies given the underlying data systems. We will provide details on methodological and analytic advances that could promote further evaluation of youth suicide prevention efforts such as: new approaches to data linkage; increased use of common data elements; and, approaches to intervention harmonization. The review team will also expand on various strategies that could facilitate intervention selection and implementation decisions by local community and state level policy makers.

Grading the Strength of Evidence for Major Comparisons and Outcomes

We will not evaluate the strength of evidence for a particular comparison or outcome as we are not addressing such questions in this review.

Assessing Applicability

We will not assess the applicability of the evidence for addressing a particular comparison or outcome as we are not addressing such questions in this review.

References

  1. Statistical Abstracts of the United States. Washington, D.C: US Census Bureau; 2015. https://www.census.gov/prod/www/statistical_abstract.html. Accessed on July 14, 2015.
  2. Web-based Injury Statistics Query and Reporting System (WISQARS). Centers for Disease Control and Prevention; 2015. http://www.cdc.gov/injury/wisqars/. Accessed on July 14, 2015.
  3. National Action Alliance for Suicide Prevention: Research Prioritization Task Force. A prioritized research agenda for suicide prevention: An action plan to save lives. National Institute of Mental Health and the Research Prioritization Task Force. Rockville, MD: 2014.
  4. Nock MK, Borges G, Bromet EJ, et al. Suicide and suicidal behavior. Epidemiol Rev. 2008;30:133-54. PMID: 18653727.
  5. O'Carroll PW, Berman AL, Maris RW, et al. Beyond the Tower of Babel: a nomenclature for suicidology. Suicide Life Threat Behav. 1996 Fall;26(3):237-52. PMID: 8897663.
  6. Silverman MM, Berman AL, Sanddal ND, et al. Rebuilding the tower of Babel: a revised nomenclature for the study of suicide and suicidal behaviors. Part 2: Suicide-related ideations, communications, and behaviors. Suicide Life Threat Behav. 2007 Jun;37(3):264-77. PMID: 17579539.
  7. Crosby A. Improving National Data Systems for Surveillance of Suicide-related Events. Centers for Disease Control and Prevention; 2013. http://www.sprc.org/sites/sprc.org/files/event_materials/State%20Tribal%201E _Crosby_Action%20Alliance_Surveillance%20 Data1.pdf. Accessed on March, 31 2015.
  8. 2012 National Strategy for Suicide Prevention: Goals and Objectives for Action: A Report of the U.S. Surgeon General and of the National Action Alliance for Suicide Prevention. Washington DC: U.S. Department of Health and Human Services (HHS) Office of the Surgeon General and National Action Alliance for Suicide Prevention; 2012.
  9. A prioritized research agenda for suicide prevention: An action plan to save lives. Rockville, MD: National Institute of Mental Health and the Research Prioritization Task Force; 2014.
  10. Kruijshaar ME, Barendregt J, Vos T, et al. Lifetime prevalence estimates of major depression: an indirect estimation method and a quantification of recall bias. Eur J Epidemiol. 2005;20(1):103-11. PMID: 15756910.
  11. Mo?cicki EK, O'Carroll P, Locke BZ, et al. Suicidal ideation and attempts: the epidemiologic catchment area. Report of the Secretary's Task Force on Youth Suicide. Washington, DC: US Government Printing Office; 2008.
  12. Report o/the Secretary's Task Force on Youth Suicide. Volume 1: Overview and Recommendations. Alcohol, Drug Abuse, and Mental Health Administration. Washington DC: Supt. of Does. U.S. Govt. Print. Off.; 1989.
  13. Alberani V, De Castro Pietrangeli P, Mazza AM. The use of grey literature in health sciences: a preliminary survey. Bull Med Libr Assoc. 1990 Oct;78(4):358-63. PMID: 2224298.
  14. Share of search queries handled by leading U.S. search engine providers as of April 2015. Statista; 2015. http://www.statista.com/statistics/267161/market-share-of-search-engines-in-the-united-states/. Accessed on July 14, 2015.
  15. Walters W. Comparative recall and precision of simple and expert searches in Google Scholar and eight other databases. Portal: Libraries and the Academy. 2011;11(4):971-1006.
  16. Kharrazi H, Weiner J. IT-enabled Community Health Interventions: Challenges, Opportunities, and Future Directions. eGEMs (Generating Evidence & Methods to improve patient outcomes). 2014;2(3):1-8.
  17. Kopcke F, Trinczek B, Majeed RW, et al. Evaluation of data completeness in the electronic health record for the purpose of patient recruitment into clinical trials: a retrospective analysis of element presence. BMC Med Inform Decis Mak. 2013;13:37. PMID: 23514203.
  18. Little RJ, D'Agostino R, Cohen ML, et al. The prevention and treatment of missing data in clinical trials. N Engl J Med. 2012 Oct 4;367(14):1355-60. PMID: 23034025.
  19. Chan KS, Fowles JB, Weiner JP. Review: electronic health records and the reliability and validity of quality measures: a review of the literature. Med Care Res Rev. 2010 Oct;67(5):503-27. PMID: 20150441.
  20. Raudenbush S, Bryk A. Hierarchical linear models: Applications and data analysis methods. Thousand Oaks, CA: SAGE Publications, Inc; 2002.
  21. Hlatky MA, Ray RM, Burwen DR, et al. Use of Medicare data to identify coronary heart disease outcomes in the Women's Health Initiative. Circ Cardiovasc Qual Outcomes. 2014 Jan;7(1):157-62. PMID: 24399330.
  22. White I. Multivariate random-effects meta-regression: Updates to mvmeta. The STATA Journal. 2011;11(2):255-70.
  23. Curran PJ, Hussong AM. Integrative data analysis: the simultaneous analysis of multiple data sets. Psychol Methods. 2009 Jun;14(2):81-100. PMID: 19485623.
  24. Brown CH, Sloboda Z, Faggiano F, et al. Methods for synthesizing findings on moderation effects across multiple randomized trials. Prev Sci. 2013 Apr;14(2):144-56. PMID: 21360061.
  25. Fernandes BS, Gama CS, Cereser KM, et al. Brain-derived neurotrophic factor as a state-marker of mood episodes in bipolar disorders: a systematic review and meta-regression analysis. J Psychiatr Res. 2011 Aug;45(8):995-1004. PMID: 21550050.
  26. Muthen B, Muthen LK. Integrating person-centered and variable-centered analyses: growth mixture modeling with latent trajectory classes. Alcohol Clin Exp Res. 2000 Jun;24(6):882-91. PMID: 10888079.

Definition of Terms

All relevant terms are defined in the above text.

Summary of Protocol Amendments

If we need to amend this protocol, we will give the date of each amendment, describe the change and give the rationale in this section. Changes will not be incorporated into the protocol. See the example table below (Table 3):

Table 3: Template for the Summary of Protocol Amendments
Date Section Original Protocol Revised Protocol Rationale
2 September 2015 (Change of draft) III. Analytic Framework Footnote: * See a preliminary list of prevention intervention programs in Appendix A. Footnote: * See a preliminary list of prevention intervention programs in Appendix A. The list of suicide prevention programs will be supplemented by information identified by the literature search, the environmental scan, and the targeted state search as well as by input provided by external experts. To indicate that the Suicide Prevention Programs are identified by literature search, environmental scan, targeted state search and expert input.
2 September 2015 (Change of Draft) III. Analytic Framework KQ2 Analyzing linked data Added “Types of Bias” This is important to add since it can impact the way the data is linked.
2 September 2015 (Change of draft) IV. Methods: Search Methodologies Description of the environmental scan to identify data systems Added: The list of data systems (Appendix D) will be supplemented by information identified by external experts. To indicate that the data systems are identified by literature search, environmental scan, targeted state search and expert input.
2 September 2015 (Change of draft) IV. Methods: Data synthesis Addition of references to supplement the explanation of statistical methods may be approached Added reference to Curran et al. To be more thorough in identifying statistical methods.

Review of Key Questions

There was no public review of the Key Questions.

Key Informants

There were no key informants involved in the topic refinement.

Technical Experts

Technical experts constitute a multi-disciplinary group of clinical, content, and methodological experts who provide input in defining populations, interventions, comparisons, or outcomes and identify particular studies or databases to search. They are selected to provide broad expertise and perspectives specific to the topic under development. Divergent and conflicting opinions are common and perceived as health scientific discourse that results in a thoughtful, relevant systematic review. Therefore study questions, design, and methodological approaches do not necessarily represent the views of individual technical and content experts. Technical experts provide information to the EPC to identify literature search strategies and recommend approaches to specific issues as requested by the EPC. Technical experts do not do analysis of any kind nor do they contribute to the writing of the report. They have not reviewed the report, except as given the opportunity to do so through the peer or public review mechanism.

Technical experts must disclose any financial conflicts of interest greater than $10,000 and any other relevant business or professional conflicts of interest. Because of their unique clinical or content expertise, individuals are invited to serve as technical experts and those who present with potential conflicts may be retained. The TOO and the EPC work to balance, manage, or mitigate any potential conflicts of interest identified.

Peer Reviewers

Peer reviewers are invited to provide written comments on the draft report based on their clinical, content, or methodological expertise. The EPC considers all peer review comments on the draft report in preparation of the final report. Peer reviewers do not participate in writing or editing of the final report or other products. The final report does not necessarily represent the views of individual reviewers. The EPC will complete a disposition of all peer review comments. The disposition of comments for systematic reviews and technical briefs will be published three months after the publication of the evidence report.

Potential peer reviewers must disclose any financial conflicts of interest greater than $10,000 and any other relevant business or professional conflicts of interest. Invited peer reviewers may not have any financial conflict of interest greater than $10,000. Peer reviewers who disclose potential business or professional conflicts of interest may submit comments on draft reports through the public comment mechanism.

EPC Team Disclosures

EPC core team members must disclose any financial conflicts of interest greater than $1,000 and any other relevant business or professional conflicts of interest. Related financial conflicts of interest that cumulatively total greater than $1,000 will usually disqualify EPC core team investigators.

Role of the Funder

This project was funded under Contract No. HHSA 290-2012-00007 I from the Agency for Healthcare Research and Quality, U.S. Department of Health and Human Services. The Task Order Officer reviewed contract deliverables for adherence to contract requirements and quality. The authors of this report are responsible for its content. Statements in the report should not be construed as endorsement by the Agency for Healthcare Research and Quality or the U.S. Department of Health and Human Services.

Appendixes

Appendix A: NREPP Database of Suicide Prevention Programs

Table A1. Simple Search, Intervention search term: “Suicide prevention” (16)
Intervention Title Description
American Indian Life Skills Development/Zuni Life Skills Development Suicide is the second leading cause of death among American Indians 15 to 24 years old, according to Centers for Disease Control and Prevention data. The estimated rate of completed suicides among American Indians in this age group is about three times higher than among comparably aged U.S. youth overall (37.4 vs. 11.4 per 100,000, respectively).
CAST (Coping And Support Training) CAST (Coping And Support Training) is a high school-based suicide prevention program targeting youth 14 to 19 years old. CAST delivers life-skills training and social support in a small-group format (6-8 students per group).
Emergency Department Means Restriction Education Emergency Department Means Restriction Education is an intervention for the adult caregivers of youth (aged 6 to 19 years) who are seen in an emergency department (ED) and determined through a mental health assessment to be at risk for committing suicide.
Family Intervention for Suicide Prevention (FISP) The Family Intervention for Suicide Prevention (FISP) is a cognitive behavioral family intervention for youth ages 10-18 who are presenting to an emergency department (ED) with suicidal ideation or after a suicide attempt.
Kognito At-Risk for College Students Kognito At-Risk for College Students is a 30-minute, online, interactive training simulation that prepares college students and student leaders, including resident assistants, to provide support to peers who are exhibiting signs of psychological distress such as depression, anxiety, substance abuse, and suicidal ideation.
Kognito At-Risk for High School Educators Kognito At-Risk for High School Educators is a 1-hour, online, interactive gatekeeper training program that prepares high school teachers and other school personnel to identify, approach, and refer students who are exhibiting signs of psychological distress such as depression, anxiety, substance abuse, and suicidal ideation.
Kognito Family of Heroes Kognito Family of Heroes is a 1-hour, online role-playing training simulation for military families of service members recently returned from deployment (within the past 4 years). The training is designed to: (1) increase awareness of signs of post-deployment stress, including posttraumatic stress disorder (PTSD), traumatic brain injury (TBI), depression, and suicidal ideation, and (2) motivate family members to access mental health services when they show signs of post-deployment stress.
LEADS: For Youth (Linking Education and Awareness of Depression and Suicide) LEADS: For Youth (Linking Education and Awareness of Depression and Suicide) is a curriculum for high school students in grades 9-12 that is designed to increase knowledge of depression and suicide, modify perceptions of depression and suicide, increase knowledge of suicide prevention resources, and improve intentions to engage in help-seeking behaviors.
Lifelines Curriculum “Lifelines” is a comprehensive, school-wide suicide prevention program for middle and high school students. The goal of Lifelines is to promote a caring, competent school community in which help seeking is encouraged and modeled and suicidal behavior is recognized as an issue that cannot be kept secret.
Model Adolescent Suicide Prevention Program (MASPP) The Model Adolescent Suicide Prevention Program (MASPP) is a public health-oriented suicidal-behavior prevention and intervention program originally developed for a small American Indian tribe in rural New Mexico to target high rates of suicide among its adolescents and young adults.
QPR Gatekeeper Training for Suicide Prevention The QPR (Question, Persuade, and Refer) Gatekeeper Training for Suicide Prevention is a brief educational program designed to teach "gatekeepers"--those who are strategically positioned to recognize and refer someone at risk of suicide (e.g., parents, friends, neighbors, teachers, coaches, caseworkers, police officers)--the warning signs of a suicide crisis and how to respond by following three steps: (1) Question the individual's desire or intent regarding suicide; (2) Persuade the person to seek and accept help; and, (3) Refer the person to appropriate resources
Reconnecting Youth: A Peer Group Approach to Building Life Skills Reconnecting Youth: A Peer Group Approach to Building Life Skills (RY) is a school-based prevention program for students ages 14-19 years that teaches skills to build resiliency against risk factors and control early signs of substance abuse and emotional distress.
SOS Signs of Suicide SOS Signs of Suicide is a secondary school-based suicide prevention program that includes screening and education. Students are screened for depression and suicide risk and referred for professional help as indicated.
Sources of Strength “Sources of Strength”, a universal suicide prevention program, is designed to build socioecological protective influences among youth to reduce the likelihood that vulnerable high school students will become suicidal.
United States Air Force Suicide Prevention Program The United States Air Force Suicide Prevention Program (AFSPP) is a population-oriented approach to reducing the risk of suicide. The Air Force has implemented 11 initiatives aimed at strengthening social support, promoting development of social skills, and changing policies and norms to encourage effective help-seeking behaviors.
Wellness Recovery Action Plan (WRAP) Wellness Recovery Action Plan (WRAP) is a manualized group intervention for adults with mental illness. WRAP guides participants through the process of identifying and understanding their personal wellness resources ("wellness tools") and then helps them develop an individualized plan to use these resources on a daily basis to manage their mental illness.
Table A2. Advanced Search, Search criteria: 6-12 (Childhood), 13-17 (Adolescent), 18-25 (Young adult), Mental health promotion, Mental health treatment, Substance abuse prevention, Substance abuse treatment, Co-occurring disorders, Inpatient, Residential, Outpatient, Correctional, Home, School, Workplace, Other community settings, Mental health, Quality of life, Suicide, Trauma/injuries, suicide (9)
Intervention Title Description
Adolescent Coping With Depression (CWD-A) The Adolescent Coping With Depression (CWD-A) course is a cognitive behavioral group intervention that targets specific problems typically experienced by depressed adolescents. These problems include discomfort and anxiety, irrational/negative thoughts, poor social skills, and limited experiences of pleasant activities.
Attachment-Based Family Therapy (ABFT) Attachment-Based Family Therapy (ABFT) is a treatment for adolescents ages 12-18 that is designed to treat clinically diagnosed major depressive disorder, eliminate suicidal ideation, and reduce dispositional anxiety.
Depression Prevention (Managing Your Mood) The Depression Prevention (Managing Your Mood) program is a computer-tailored intervention for adults who are experiencing at least mild symptoms of depression. The program is based on the Transtheoretical Model of Behavior Change (TTM), which conceptualizes change as a process that occurs over time and in five stages: pre-contemplation, contemplation, preparation, action, and maintenance.
Dialectical Behavior Therapy Dialectical Behavior Therapy (DBT) is a cognitive-behavioral treatment approach with two key characteristics: a behavioral, problem-solving focus blended with acceptance-based strategies, and an emphasis on dialectical processes
Dynamic Deconstructive Psychotherapy Dynamic Deconstructive Psychotherapy (DDP) is a 12- to 18-month, manual-driven treatment for adults with borderline personality disorder and other complex behavior problems, such as alcohol or drug dependence, self-harm, eating disorders, and recurrent suicide attempts.
Mental Health First Aid Mental Health First Aid is an adult public education program designed to improve participants' knowledge and modify their attitudes and perceptions about mental health and related issues, including how to respond to individuals who are experiencing one or more acute mental health crises (i.e., suicidal thoughts and/or behavior, acute stress reaction, panic attacks, and/or acute psychotic behavior) or are in the early stages of one or more chronic mental health problems (i.e., depressive, anxiety, and/or psychotic disorders, which may occur with substance abuse).
Multisystemic Therapy With Psychiatric Supports (MST-Psychiatric) Multisystemic Therapy With Psychiatric Supports (MST-Psychiatric) is designed to treat youth who are at risk for out-of-home placement (in some cases, psychiatric hospitalization) due to serious behavioral problems and co-occurring mental health symptoms such as thought disorder, bipolar affective disorder, depression, anxiety, and impulsivity.
Seeking Safety Seeking Safety is a present-focused treatment for clients with a history of trauma and substance abuse. The treatment was designed for flexible use: group or individual format, male and female clients, and a variety of settings (e.g., outpatient, inpatient, residential).
Trauma Focused Coping (Multimodality Trauma Treatment) Trauma Focused Coping (TFC), sometimes called Multimodality Trauma Treatment, is a school-based group intervention for children and adolescents in grades 4-12 who have been exposed to a traumatic stressor (e.g., disaster, violence, murder, suicide, fire, accident).
Table A3. Additional Search, Intervention search term: “Suicide” (7) – additional studies not identified in the above searches
Intervention Title Description
Cognitive Behavioral Therapy for Late-Life Depression Cognitive Behavioral Therapy (CBT) for Late-Life Depression is an active, directive, time-limited, and structured problem-solving approach program that follows the conceptual model and treatment program developed by Aaron Beck and his colleagues.
Community Trials Intervention To Reduce High-Risk Drinking Community Trials Intervention To Reduce High-Risk Drinking is a multicomponent, community-based program developed to alter the alcohol use patterns and related problems of people of all ages. The program incorporates a set of environmental interventions that assist communities in (1) using zoning and municipal regulations to restrict alcohol access through alcohol outlet density control; (2) enhancing responsible beverage service by training, testing, and assisting beverage servers and retailers in the development of policies and procedures to reduce intoxication and driving after drinking; (3) increasing law enforcement and sobriety checkpoints to raise actual and perceived risk of arrest for driving after drinking; (4) reducing youth access to alcohol by training alcohol retailers to avoid selling to minors and those who provide alcohol to minors; and (5) forming the coalitions needed to implement and support the interventions that address each of these prevention components.
Emergency Room Intervention for Adolescent Females Emergency Room Intervention for Adolescent Females is a program for teenage girls 12 to 18 years old who are admitted to the emergency room after attempting suicide. The intervention, which involves the girl and one or more family members who accompany her to the emergency room, aims to increase attendance in outpatient treatment following discharge from the emergency room and to reduce future suicide attempts.
Interpersonal Psychotherapy for Depressed Adolescent (IPT-A) Interpersonal Psychotherapy for Depressed Adolescents (IPT-A) is a short-term, manual-driven outpatient treatment intervention that focuses on the current interpersonal problems of adolescents (aged 12-18 years) with mild to moderate depression severity.
Peer Assistance and Leadership (PAL) Peer Assistance and Leadership (PAL) is a peer helping program that seeks to build resiliency in youth by pairing youth with peer helpers who receive training and support from teachers participating in the program.
Prevention of Suicide in Primary Care Elderly: Collaborative Trial (PROSPECT) Prevention of Suicide in Primary Care Elderly: Collaborative Trial (PROSPECT) aims to prevent suicide among older primary care patients by reducing suicidal ideation and depression. It also aims to reduce their risk of death.

Appendix B: Sample PubMed Search Strategy

Table B1. Sample PubMed search strategy
# Search
1 suicide/prevention[mh]
2 Suicide, Attempted/prevention[mh]
3 suicid*[tiab] AND (prevent[tiab] OR prevention[tiab])
4 1 OR 2 OR 3
5 clinical trial[pt]
6 “Non-randomized”[tiab]
7 Nonrandomized[tiab]
8 cohort[tiab]
9 “next study”[tiab]
10 observational[tiab]
11 “Case-control”[tiab]
12 “cohort studies”[mh]
13 cross-over studies[mh]
14 prospectiv*[tiab]
15 registr*[tiab]
16 restrospectiv*[tiab]
17 "Comparative Study" [pt]
18 “propensity score”[tiab]
19 “propensity Score”[mh]
20 5 OR 6 OR 7 OR 8 OR 9 OR 10 OR 11 OR 12 OR 13 OR 14 OR 15 OR 16 OR 17 OR 18 OR 19
21 4 AND 20
22 1990:2016[dp]
23 Eng[la]
24 4 AND 20 AND 22 AND 23
  N=2404

Appendix C: Intermediate Outcome Measures

Table C1. Graduation rate measures
High school graduation rate % of HS students per year
Source: http://nces.ed.gov/pubs2014/2014391.pdf
4 year adjusted cohort graduation rate State education agencies (SEAs) report ACGR data for each school, LEA, and for the state total cohort rate. For freshman entering HS 2008-09 and graduating 2011-12: (Number of cohort members who earned a regular high school diploma by the end of SY 2011–12) divided by (Number of first-time 9th-graders in fall 2008 (starting cohort) plus (students who transferred in) minus (students who transferred out, emigrated, or died during school years 2008–09, 2009–10, 2010–11, and 2011–12)
Averaged HS freshman graduation rate Estimate of the percentage of high school students who graduate within 4 years of first starting 9th grade. For graduating class 2011: (Number of regular high school diplomas awarded in SY 2010–11) divided by (The number of 8th-graders enrolled in the fall 2006) plus (the number of 9th-graders enrolled in the fall 2007) plus (the number of 10th-graders enrolled in the fall of 2008) divided by 3
Public high school dropout rates (Number of dropouts) divided by (the number of students enrolled in grades 9–12 at the beginning of that school year). Includes:
  • enrolled in school at some time during the school year
  • expected to be in membership the following school year
  • not enrolled in grades 9–12 in by October 1 of the following year

Does not include:

  • reported as a dropout in the year before
  • among students who graduated high school by completing the state graduation requirements, receiving a high school equivalency credential without dropping out of school, or completing a state or district-approved educational program
  • confirmed as having transferred to another public school district, private school, or state or district-approved educational program
  • temporarily absent due to suspension or illness
  • deceased
Table C2. Incarceration rates
Concept Definition
Sources: http://www.bjs.gov/index.cfm?ty=tdtp&tid=13; http://www.bjs.gov/content/pub/pdf/fjs1112.pdf
Total incarceration rate The number of inmates held in the custody of state or federal prisons or in local jails, per 100,000 U.S. residents.
Total correctional population Total correctional population is the population of persons incarcerated, either in a prison or a jail, and persons supervised in the community, either on probation or parole.
Sentenced prisoners Prisoners under the jurisdiction of state and federal correctional authorities who have been given a sentence of more than one year.
Incarcerated population Incarcerated population is the population of inmates confined in a prison or a jail. This may also include halfway-houses, bootcamps, weekend programs, and other facilities in which individuals are locked up overnight.
Imprisonment rate The number of prisoners under state or federal jurisdiction sentenced to more than one year, per 100,000 U.S. residents.
Imprisoned population The population of inmates confined in prison or other facilities under the jurisdiction of the state or Federal Bureau of Prisons.
Custody count The number of offenders in custody. To have custody of a prisoner, a state or the Federal Bureau of Prisons (BOP) must physically hold that person in one of its facilities. A locality, state, or the BOP may have custody of a prisoner over whom a different government maintains jurisdiction.
 

Types of offense: Violent; Property: Fraud/Other; Drug; Public Order: Regulatory/Other; Sex Offense; Weapons; Immigration; Material witness; Supervision violations; and, Misdemeanor

Disposition/Sentences: Convicted: Plea/Bench/jury trial; Not convicted: Dismissed/Bench/Jury Trial; Prison; Probation only; Fine only; and, Suspended sentence

Table C3. DSM Diagnosis Codes
Disorder Code Diagnosis
Drug abuse 292.0 Amphetamine Withdrawal
292 Cocaine Withdrawal
292 Nicotine Withdrawal
292.0 Opioid Withdrawal
292.0 Other (or Unknown) Substance Withdrawal
292.0 Sedative, Hypnotic, or Anxiolytic Withdrawal
292.11 Amphetamine-Induced Psychotic Disorder, With Delusions
292.11 Cannabis-Induced Psychotic Disorder, With Delusions
292.11 Cocaine-Induced Psychotic Disorder, With Delusions
292.11 Hallucinogen-InducedPsychotic Disorder, With Delusions
292.11 Inhalant-Induced Psychotic Disorder, With Delusions
292.11 Opioid-Induced Psychotic Disorder, With Delusions
292.11 Other (or Unknown) Substance Induced Psychotic Disorder, With Delusions
292.11 Phencyclidine-Induced Psychotic Disorder, With Delusions
292.11 Sedative, Hypnotic, or Anxiolytic-Induced Psychotic Disorder, With Delusions
292.12 Amphetamine-Induced Psychotic Disorder, With Hallucinations
292.12 Cannabis-Induced Psychotic Disorder, With Hallucinations
292.12 Cocaine-Induced Psychotic Disorder, With Hallucinations
292.12 Hallucinogen-Induced Psychotic Disorder, With Hallucinations
292.12 Inhalant-Induced Psychotic Disorder, With Hallucinations
292.12 Opioid-Induced Psychotic Disorder, With Hallucinations
292.12 Other (or Unknown) Substance-Induced Psychotic Disorder, With Hallucinations
292.12 Phencyclidine-Induced Psychotic Disorder, With Hallucinations
292.12 Sedative, Hypnotic, orAnxiolytic-Induced Psychotic Disorder, With Hallucinations
292.81 Amphetamine Intoxication Delirium
292.81 Cannabis Intoxication Delirium
292.81 Cocaine Intoxication Delirium
292.81 Hallucinogen Intoxication Delirium
292.81 Inhalant Intoxication Delirium
292.81 Opioid Intoxication Delirium
292.81 Other (or Unknown) Intoxication Delirium
292.81 Phencyclidine Intoxication Delirium
292.81 Sedative, Hypnotic, or Anxiolytic Intoxication Delirium
292.81 Sedative,Hypnotic, or Anxiolytic Withdrawal Delirium
292.82 Inhalant-Induced Persisting Dementia
292.82 Other (or Unknown) Substance-Induced Persisting Dementia
292.82 Sedative, Hypnotic, orAnxiolytic-Induced Persisting Dementia
292.83 Other (or Unknown) Substance-Induced Persisting Amnestic Disorder
292.83 Sedative, Hypnotic, or Anxiolytic-Induced Persisting Amnestic Disorder
292.84 Amphetamine-Induced Mood Disorder
292.84 Cocaine-Induced Mood Disorder
292.84 Hallucinogen-Induced Mood Disorder
292.84 Inhalant-Induced Mood Disorder
292.84 Opioid-Induced Mood Disorder
292.84 Other (or Unknown) Substance-Induced Mood Disorder
292.84 Phencyclidine-Induced Mood Disorder
292.84 Sedative, Hypnotic, orAnxiolytic-Induced Mood Disorder
292.85 Amphetamine-Induced Sleep Disorder (new code as of 10.01.2005)
292.85 Caffeine-Induced Sleep Disorder (new code as of 10.01.2005)
292.85 Cocaine-Induced Sleep Disorder (new code as of 10.01.2005)
292.85 Opioid-Induced Sleep Disorder (new code as of 10.01.2005)
292.85 Other (or Unknown) Substance-Induced Sleep Disorder (new code as of 10.01.2005)
292.85 Sedative, Hypnotic, or Anxiolytic-Induced Sleep Disorder (new code as of 10.01.2005)
292.89 Amphetamine-Induced Anxiety Disorder
292.89 Amphetamine-Induced Sexual Dysfunction
292.89 Amphetamine-Induced Sleep Disorder (before 10.01.2005)
292.89 Amphetamine Intoxication
292.89 Caffeine-Induced Anxiety Disorder
292.89 Caffeine-Induced Sleep Disorder (before 10.01.2005)
292.89 Cannabis-Induced Anxiety Disorder
292.89 Cannabis Intoxication
292.89 Cocaine-Induced Anxiety Disorder
292.89 Cocaine-Induced Sexual Dysfunction
292.89 Cocaine-Induced Sleep Disorder (before 10.01.2005)
292.89 Cocaine Intoxication
292.89 Hallucinogen-Induced Anxiety Disorder
292.89 Hallucinogen Intoxication
292.89 Hallucinogen Persisting Perception Disorder
292.89 Inhalant-Induced Anxiety Disorder
292.89 Inhalant Intoxication
292.89 Opioid-Induced Sexual Dysfunction
292.89 Opioid-Induced Sleep Disorder (before 10.01.2005)
292.89 Opioid Intoxication
292.89 Other (or Unknown) Substance-Induced Anxiety Disorder
292.89 Other (or Unknown) Substance-Induced Sexual Dysfunction
292.89 Other (or Unknown) Substance-Induced Sleep Disorder (before 10.01.2005)
292.89 Other (or Unknown) Substance Intoxication
292.89 Phencyclidine-Induced Anxiety Disorder
292.89 Phencyclidine Intoxication
292.89 Sedative, Hypnotic, or Anxiolytic-Induced Anxiety Disorder
292.89 Sedative, Hypnotic, or Anxiolytic-Induced Sexual Dysfunction
292.89 Sedative,Hypnotic, or Anxiolytic-Induced Sleep Disorder (before 10.01.2005)
292.89 Sedative, Hypnotic, or Anxiolytic Intoxication
292.9 Amphetamine-Related Disorder NOS
292.9 Caffeine-Related Disorder NOS
292.9 Cannabis-Related Disorder NOS
292.9 Cocaine-Related Disorder NOS
292.9 Hallucinogen-Related Disorder NOS
292.9 Inhalant-Related Disorder NOS
292.9 Nicotine-Related Disorder NOS
292.9 Opioid-Related Disorder NOS
292.9 Other (or Unknown) Substance-Related Disorder NOS
292.9 Phencyclidine-Related Disorder NOS
292.9 Sedative, Hypnotic, orAnxiolytic-Related Disorder NOS
304.00 Opioid Dependence
304.10 Sedative, Hypnotic, orAnxiolytic Dependence
304.20 Cocaine Dependence
304.30 Cannabis Dependence
304.40 Amphetamine Dependence
304.50 Hallucinogen Dependence
304.60 Inhalant Dependence
304.60 Phencyclidine Dependence (new code as of 10/01/96)
304.80 Polysubstance Dependence
304.90 Other (or Unknown) Substance Dependence
305.10 Nicotine Dependence
305.20 Cannabis Abuse
305.30 Hallucinogen Abuse
305.40 Sedative, Hypnotic, or Anxiolytic Abuse
305.50 Opioid Abuse
305.60 Cocaine Abuse
305.70 Amphetamine Abuse
305.90 Caffeine Intoxication
305.90 Inhalant Abuse
305.90 Other (or Unknown) Substance Abuse
305.90 Phencyclidine Abuse
Alcohol abuse 291.0 Alcohol Intoxication Delirium
291.0 Alcohol Withdrawal Delirium
291.1 Alcohol-Induced Persisting Amnestic Disorder
291.2 Alcohol-Induced Persisting Dementia
291.3 Alcohol-Induced Induced Psychotic Disorder, With Hallucinations
291.5 Alcohol-Induced Psychotic Disorder, With Delusions
291.81 Alcohol Withdrawal (new code as of 10/01/96)
291.82 Alcohol-Induced Sleep Disorder (new code as of 10.01.2005)
291.89 Alcohol-Induced Anxiety Disorder (new code as of 10/01/96)
291.89 Alcohol-Induced Mood Disorder (new code as of 10/01/96)
291.89 Alcohol-Induced Sexual Dysfunction (new code as of 10/01/96)
291.89 Alcohol-Induced Sleep Disorder (10.01.1996-10.01.2005)
291.9 Alcohol-Related Disorder NOS
303.00 Alcohol Intoxication
303.90 Alcohol Dependence
305.00 Alcohol Abuse
Violence V61.10 Partner Relational Problem (new code as of 10/01/96)
V61.12 Physical Abuse of Adult (if by partner) (new code as of 10/01/96)
V61.12 Sexual Abuse of Adult (if by partner) (new code as of 10/01/96)
V61.20 Parent-Child Relational Problem
V61.21 Neglect of Child
V61.21 Physical Abuse of Child
V61.21 Sexual Abuse of Child
V62.83 Physical Abuse of Adult (if by person other than partner) (new code as of 10/01/96)
V62.83 Sexual Abuse of Adult (if by person other than partner) (new code as of 10/01/96)
V61.10 Partner Relational Problem (new code as of 10/01/96)
V61.12 Physical Abuse of Adult (if by partner) (new code as of 10/01/96)
V61.12 Sexual Abuse of Adult (if by partner) (new code as of 10/01/96)
Psychiatric 90.4 Vascular Dementia, Uncomplicated
290.41 Vascular Dementia, With Delirium
290.42 Vascular Dementia, With Delusions
290.43 Vascular Dementia, With Depressed Mood
293.0 Delirium Due to...[Indicate the General Medical Condition]
293.81 Psychotic Disorder Due to...[Indicate the General Medical Condition], With Delusions
293.82 Psychotic Disorder Due to...[Indicate the General Medical Condition] , With Hallucinations
293.83 Mood Disorder Due to...[Indicate the General Medical Condition]
293.84 Anxiety Disorder Due to... (new code as of 10/01/96) [Indicate the General Medical Condition]
293.89 Catatonic Disorder Due to...[Indicate the General Medical Condition]
293.9 Mental Disorder NOS Due to...[Indicate the General Medical Condition]
294.0 Amnestic Disorder Due to...[Indicate the General Medical Condition], Without Behavioral Disturbance
294.1 Dementia Due to...[Indicate the General Medical Condition], Without Behavioral Disturbance
294.1 Dementia of the Alzheimer's Type, With Early Onset, Without Behavioral Disturbance
294.1 Dementia of the Alzheimer's Type, With Late Onset, Without Behavioral Distrubance
294.11 Dementia Due to...[Indicate the General Medical Condition], With Behavioral Disturbance
294.11 Dementia of the Alzheimer's Type, With Early Onset, With Behavioral Disturbance
294.11 Dementia of the Alzheimer's Type, With Late Onset, With Behavioral Disturbance
294.1x Dementia Due to Creutzfeldt-Jakob Disease
294.1x Dementia Due to Head Trauma
294.1x Dementia Due to HIV Disease
294.1x Dementia Due to Huntington's Disease
294.1x Dementia Due to Parkinson's Disease
294.1x Dementia Due to Pick's Disease
294.1x Dementia Due to...[Indicate the General Medical Condition not listed above] (also code the general medical condition on Axis III)
294.8 Amnestic Disorder NOS
294.8 Dementia NOS
294.9 Cognitive Disorder NOS
294.9 Cognitive Disorder (new code as of 10/01/96)
295.10 Schizophrenia, Disorganized Type
295.20 Schizophrenia, Catatonic Type
295.30 Schizophrenia, Paranoid Type
295.40 Schizophreniform Disorder
295.60 Schizophrenia, Residual Type
295.70 Schizoaffective Disorder
295.90 Schizophrenia, Undifferentiated Type
296.00 Bipolar I Disorder, Single Manic Episode, Unspecified
296.01 Bipolar I Disorder, Single Manic Episode, Mild
296.02 Bipolar I Disorder, Single Manic Episode, Moderate
296.03 Bipolar I Disorder, Single Manic Episode, Severe Without Psychotic Features
296.04 Bipolar I Disorder, Single Manic Episode, Severe With Psychotic Features
296.05 Bipolar I Disorder, Single Manic Episode, In Partial Remission
296.06 Bipolar I Disorder, Single Manic Episode, In Full Remission
296.20 Major Depressive Disorder, Single Episode, Unspecified
296.21 Major Depressive Disorder, Single Episode, Mild
296.22 Major Depressive Disorder, Single Episode, Moderate
296.23 Major Depressive Disorder, Single Episode, Severe Without Psychotic Features
296.24 Major Depressive Disorder, Single Episode, Severe With Psychotic Features
296.25 Major Depressive Disorder, Single Episode, In Partial Remission
296.26 Major Depressive Disorder, Single Episode, In Full Remission
296.30 Major Depressive Disorder, Recurrent, Unspecified
296.31 Major Depressive Disorder, Recurrent, Mild
296.32 Major Depressive Disorder, Recurrent, Moderate
296.33 Major Depressive Disorder, Recurrent, Severe Without Psychotic Features
296.34 Major Depressive Disorder, Recurrent, Severe With Psychotic Features
296.35 Major Depressive Disorder, Recurrent, In Partial Remission
296.36 Major Depressive Disorder, Recurrent, In Full Remission
296.40 Bipolar I Disorder, Most Recent Episode Hypomanic
296.40 Bipolar I Disorder, Most Recent Episode Manic, Unspecified
296.41 Bipolar I Disorder, Most Recent Episode Manic, Mild
296.42 Bipolar I Disorder, Most Recent Episode Manic, Moderate
296.43 Bipolar I Disorder, Most Recent Episode Manic, Severe Without Psychotic Features
296.44 Bipolar I Disorder, Most Recent Episode Manic, Severe With Psychotic Features
296.45 Bipolar I Disorder, Most Recent Episode Manic, In Partial Remission
296.46 Bipolar I Disorder, Most Recent Episode Manic, In Full Remission
296.50 Bipolar I Disorder, Most Recent Episode Depressed, Unspecified
296.51 Bipolar I Disorder, Most Recent Episode Depressed, Mild
296.52 Bipolar I Disorder, Most Recent Episode Depressed, Moderate
296.53 Bipolar I Disorder, Most Recent Episode Depressed, Severe Without Psychotic Features
296.54 Bipolar I Disorder, Most Recent Episode Depressed, Severe With Psychotic Features
296.55 Bipolar I Disorder, Most Recent Episode Depressed, In Partial Remission
296.56 Bipolar I Disorder, Most Recent Episode Depressed, In Full Remission
296.6 Bipolar I Disorder, Most Recent Episode Mixed, Unspecified
296.61 Bipolar I Disorder, Most Recent Episode Mixed, Mild
296.62 Bipolar I Disorder, Most Recent Episode Mixed, Moderate
296.63 Bipolar I Disorder, Most Recent Episode Mixed, Severe Without Psychotic Features
296.64 Bipolar I Disorder, Most Recent Episode Mixed, Severe With Psychotic Features
296.65 Bipolar I Disorder, Most Recent Episode Mixed, In Partial Remission
296.66 Bipolar I Disorder, Most Recent Episode Mixed, In Full Remission
296.7 Bipolar I Disorder, Most Recent Episode Unspecified
296.80 Bipolar Disorder NOS
296.89 Bipolar II Disorder
296.90 Mood Disorder NOS
297.1 Delusional Disorder
297.3 Shared Psychotic Disorder
298.8 Brief Psychotic Disorder
298.9 Psychotic Disorder NOS
299.00 Autistic Disorder
299.10 Childhood Disintegrative Disorder
299.80 Asperger's Disorder
299.80 Pervasive Developmental Disorder NOS
299.80 Rett's Disorder
300.00 Anxiety Disorder NOS
300.01 Panic Disorder Without Agoraphobia
300.02 Generalized Anxiety Disorder
300.11 Conversion Disorder
300.12 Dissociative Amnesia
300.13 Dissociative Fugue
300.14 Dissociative Identity Disorder
300.15 Dissociative Disorder NOS
300.16 Factitious Disorder With Predominantly Psychological Signs and Symptoms
300.19 Factitious Disorder NOS
300.19 Factitious Disorder With Combined Psychological and Physical Signs and Symptoms
300.19 Factitious Disorder With Predominantly Physical Signs and Symptoms
300.21 Panic Disorder With Agoraphobia
300.22 Agoraphobia Without History of Panic Disorder
300.23 Social Phobia
300.29 Specific Phobia
300.3 Obsessive-Compulsive Disorder
300.4 Dysthymic Disorder
300.6 Depersonalization Disorder
300.7 Body Dysmorphic Disorder
300.7 Hypochondriasis
300.81 Somatization Disorder
300.82 Somatoform Disorder NOS (new code as of 10/01/96)
300.82 Undifferentiated Somatoform Disorder (new code as of 10/01/96)
300.9 Unspecified Mental Disorder (nonpsychotic)
301.0 Paranoid Personality Disorder
301.13 Cyclothymic Disorder
301.20 Schizoid Personality Disorder
301.22 Schizotypal Personality Disorder
301.4 Obsessive-Compulsive Personality Disorder
301.50 Histrionic Personality Disorder
301.6 Dependent Personality Disorder
301.7 Antisocial Personality Disorder
301.81 Narcissistic Personality Disorder
301.82 Avoidant Personality Disorder
301.83 Borderline Personality Disorder
301.9 Personality Disorder NOS
307.80 Pain Disorder Associated With Psychological Factors
307.89 Pain Disorder Associated With Both Psychological Factors and a General Medical Condition
307.9 Communication Disorder NOS
308.3 Acute Stress Disorder
309.0 Adjustment Disorder With Depressed Mood
309.21 Separation Anxiety Disorder
309.24 Adjustment Disorder With Anxiety
309.28 Adjustment Disorder With Mixed Anxiety and Depressed Mood
309.3 Adjustment Disorder With Disturbance of Conduct
309.4 Adjustment Disorder With Mixed Disturbance of Emotions and Conduct
309.81 Posttraumatic Stress Disorder
309.9 Adjustment Disorder Unspecified
310.1 Personality Change Due to...[Indicate the General Medical Condition]
311 Depressive Disorder NOS
312.3 Impulse-Control Disorder NOS

Appendix D: Potential Data Systems

Table D1. Potential data systems
Data Type Data System Web Site Organization
Suicide Web-based Injury Statistics Query and Reporting System http://www.cdc.gov/injury/wisqars/ USDHHS, CDC
Army Study to Assess Risk and Resilience in Servicemembers http://www.armystarrs.org/ US Army
Safe Supportive Learning Survey http://safesupportivelearning.ed.gov/ topic-research/school-climate-measurement/school-climate-survey-compendium US Department of Education, Office of Safe and Healthy Students
Deaths Arrest-Related Death Survey http://bjs.ojp.usdoj.gov/index.cfm?ty=tp&tid=82 Department of Justice, BJS
National Violent Death Reporting System http://cdc.gov/ViolencePrevention/NVDRS/index.htm USDHHS, CDC
Department of Defense Suicide Event Report http://dodser.t2.health.mil/welcome Department of Defense
Death Certificates from National Vital Statistics System http://cdc.gov/nchs/nvss.htm CDC WISCARS
National Death Index http://www.cdc.gov/nchs/ndi.htm USDHHS, CDC
Deaths-in-Custody Reporting Program http://bjs.ojp.usdoj.gov/index.cfm?ty=tp&tid=19 Department of Justice, BJS
Healthcare Provider Records Adolescent Suicide Attempt data System (Oregon) http://public.health.oregon.gov/PreventionWellness/ SafeLiving/SuicidePrevention/Pages/ASADS2.aspx Oregon Health Authority Public Health Division
Biosense http://cdc.gov/Biosense USDHHS, CDC
Department of Defense Suicide Event Report (nonfatal section) http://dodser.t2.health.mil/welcome Department of Defense
Drug Abuse Warning Network (no longer operational) http://samhsa.gov/data/DAWN.aspx USDHHS, SAMHSA
HCUP-NIS http://www.hcup-us.ahrq.gov/nisoverview.jsp Agency for Healthcare Research and Quality
HCUP http://hcup-us.ahrq.gov/overview.jsp Agency for Healthcare Research and Quality
National Ambulatory Medical Survey http://cdc.gov/nchs/ahcd.htm USDHHS, CDC
National Corrections Reporting Program http://ncrp.info/SitePages/Home.aspx Department of Justice
National Electronic Injury Surveillance System - All Injury Program http://www.cpsc.gov/en/Research--Statistics/NEISS-Injury-Data/ Consumer Product Safety Commission
National Emergency Medical Services Information System http://www.nemsis.org/ National Association of State Emergency Medical Services Directors, National Highway Traffic Safety Administration, Health Resources and Services Administration
National Hospital Ambulatory Medical Care Survey http://cdc.gov/nchs/ahcd/about_ahcd.htm USDHHS, CDC
National Hospital Care Survey http://cdc.gov/nchs/nhcs.htm USDHHS, CDC
National Hospital Discharge Survey http://cdc.gov/nchs/nhds.htm USDHHS, CDC
National Suicide Prevention Lifeline http://suicidepreventionlifeline.org USDHHS, SAMHSA
National Prison Health Care Website not available, Report using data: http://static.nicic.gov/Library/015999.pdf Department of Justice
National Trauma Data Bank https://www.ntdbdatacenter.com/ American College of Surgeons
Resource and Patient Management System http://ihs.gov/RPMS/index.cfm?module=home&option=index&CFID=14067134& CFTOKEN=48279019 USDHHS, Indian Health Service
Suicide Prevention Coordinator Reports Website not available Report describing data: www.va.gov/opa/docs/Suicide-Data-Report-2012-final.pdf U.S. Department of Veterans Affairs
Population-based Surveys Behavioral Risk Factor Survey System http://cdc.gov/brfss/ USDHHS, CDC
National Co-morbidity Survey and Replication http://hcp.med.harvard.edu/ncs/instruments.php USDHHS, National Institute of Mental Health
National Survey and Drug Use and Health http://icpsr.umich.edu/icpsrweb/SAMHDA/index.jsp USDHHS, SAMHSA
Youth Risk Behavior Surveillance System http://www.cdc.gov/HealthyYouth/yrbs/index.htm USDHHS, CDC
National Epidemiologic Survey on Alcohol and Related Conditions http://niaaa.census.gov USDHHS, NIH
Health Insurance Claims Medicare/Medicaid http://cms.gov/Research-Statistics-Data-and-Systems/Research/ResearchGenInfo Centers for Medicare and Medicaid Services

Appendix E: Preliminary Draft Data System Classification/Coding Schema (KQ 1)

Data Relatedness / Availability

  • Directly available
    • Dependent variables
      • Primary (e.g., suicide, suicide attempt, suicide ideation)
      • Secondary / Intermediate (e.g., incarceration rate, graduation rate, emergency department utilization)
    • Independent variables
      • Genomic data
      • Demographic data
      • Social data
      • Environmental/Geographic data
      • Health/Medical/Clinical data
  • Indirectly available
    • Dependent variable
      • Primary
      • Secondary
    • Independent variable
    • Current form of the variable
    • Method to make it available
      • Simple statistics/arithmetic conversions (e.g., metric to imperial)
      • Inference models (e.g., logical rules)
      • Imputation method (e.g., regression methods)
      • Spatial-triangulation (e.g., time-trend analysis)
      • Geo-triangulation
      • Other methods?
  • Data definitions
    • Has clear definitions for data fields (e.g., data dictionary) – if yes, include them
    • Follows a standard definition for these data fields – if yes, mention which ones

Data Granularity

  • Patient-level
    • Cross-sectional
    • Repeated
      • Retrospective/Historical
      • Time-series/Ongoing
  • Aggregate on certain dimension
    • Demographics (e.g., age, gender, SES)
    • Geographical (e.g., zip code, census block)
    • Other patient specs (e.g., Dx, Tx, Rx, Lx and other attributes)
    • Entity (e.g., payer, provider, center)
  • Total sums / aggregates with no levels

Data Denominator Coverage

  • Geographic coverage
    • National
    • Regional (one or more)
    • State (one or more)
    • Locality (e.g., one or more)
    • Specific geographical boundary smaller than State
      • County
      • Zip code
      • Census Block
      • Tribal
  • Demographic coverage
  • Data Source / Entity coverage
    • Payer
    • Provider
    • Department of Health

Data Sampling

  • Entire Population
    • Absolute (e.g., Census)
    • Relative to the data source (e.g., all population of a provider)
  • A sample of a larger population denominator
    • Includes more than 30% of the population
    • Includes less than 30% of the population
    • Samples are not enough to make statistical inference

Data Scalability

  • Data Architecture
  • Data Types
    • Structured data
    • Unstructured data
      • Free text
      • Images
      • Other?

Data Interoperability

  • Standard Terminologies
    • Diagnosis / Problem List
      • ICD9, ICD9CM, ICD10 or a variant
      • SNOMED
      • DRG
      • Other?
    • Procedures
      • CPT
      • Others
    • Lab
      • LOINC
    • Medication
      • RxNorm
      • NDC
      • Other?
    • Sign/Symptom
      • ICD variant
      • Other?
    • Family history
  • Standard Data Exchanges
    • HL7
    • DICOM

Data Quality

  • Completeness
  • Accuracy
    • Missingness (e.g., random and non-random)
  • Timeliness
    • Frequency
      • Real-time
        • Frequency of updates
      • Not Real-time
        • Regular Interval
          • Daily
          • Weekly
          • Monthly
          • Quarterly
          • Semi-annual
          • Annual
        • Irregular interval
    • Time-factor
      • Ongoing / Current
      • Stopped
        • Year stopped
      • Planned
        • Year planned
    • Data collection's [anticipated] starting date/year
  • Validity (i.e., potential sources of bias)
    • Internal
    • Criterion
    • Face
    • Construct
  • Reliability (i.e., potential sources of bias)
    • Inter-rater (i.e., same data captured by two people/sites)
    • Non-redundant (i.e., same data captured next time)
    • Coherence (i.e., same data captured in another data source – mix of inter-rater and non-redundant versions)
  • Main sources of bias
    • Numerator quality issues
    • Denominator quality issues

Data Governance

  • Data Access
    • No restrictions (e.g., publicly available)
    • Restrictions apply
      • Available for research (e.g., IRB required; approval required)
      • Available for QI only
    • Limited to a certain entity
      • Federal / State / City
      • National / Local
  • Data Security
    • HIPAA exempt
    • HIPAA non-exempt
  • Data Commodity
    • Free to the public
      • Type of free license
    • Free for research (i.e., needs an approval process)
    • Commercial
  • Data Owner / Steward / Custodian
    • Patients
    • Providers
    • Payers
    • Public health departments
    • Federal body
    • National association
      • For-profit
      • Non-profit
  • Data Sponsor
  • Data Capture / Generation
    • Humans-Manual
      • Researcher
      • Patient
      • Provider / Clinician
      • Payer
      • Public health officials
      • Policy makers
    • Computers-Automated
  • Data Curators
    • Humans
      • Researcher
      • Patient
      • Provider / Clinician
      • Payer
      • Public health officials
      • Policy makers
    • Computers
  • Data Users
    • Humans
    • Computers

Data Ethics

  • Anonymization Method
    • Records are anonymized by removing data pieces
    • Records are anonymized by removing individual records
    • Records are anonymized by aggregation
  • Anonymizer
    • Human-Manual
    • Computer-Automated
      • Statistical modeling
      • Pattern detection
      • NLP techniques
  • HIPAA status
    • All HIPAA information are removed (HIPAA-compliant)
    • Limited-HIPAA data set (includes some of the HIPAA elements)
    • HIPAA information are included

Data Feasibility

  • Update Process
    • Manual
    • Semi-automatic
    • Fully automatic
  • Data Sustainability
    • Federal operations
    • State operations
    • City Municipal/Community operations
    • Based on a grant
  • Data Cost
    • Cost to capture data (very high, high, medium, low, no cost)
    • Cost to maintain data system (very high, high, medium, low, no cost)
    • Cost to analyze the data (very high, high, medium, low, no cost)

Data Uses / Functions

  • Primary function of the data
    • Research
    • Clinical care
    • Social services
    • Public health (e.g., Surveillance)
  • Secondary functions of the data