Inside the quest to prevent suicides through better data

Something as impersonal and mundane as incomplete datasets could be exacerbating a national tragedy: the suicides of thousands of veterans and hundreds of active-duty service members every year.

Preventing such suicides depends in part on the quality of the government's data on potential contributing factors such as mental health and disciplinary history. Officials at the departments of Defense and Veterans Affairs have underlined that point by making improved data management one of the bedrocks of their suicide-prevention strategies in recent years.

Interviews with DOD and VA officials reveal a joint data policy to track suicides that is gradually getting off the ground and overcoming bureaucratic inertia. At the same time, however, a recent report by DOD's inspector general revealed that the information in the department's main collection system for suicide data -- recent improvements notwithstanding -- is often incomplete.

The IG investigation, published last month, made clear the potential consequences of flawed reporting of suicides and called inadequate suicide-prevention programs "a substantial and specific danger to public health and safety."

The technology at issue is a DOD-wide database created in 2008 for reporting service-member suicides and suicide attempts: the DOD Suicide Event Report (DODSER). Information on those deaths is gathered via a report and used by military officials to try to make the next suicide less likely. The data includes medical history, military history (such as demotions, disciplinary cases and deployments) and demographic data. The report also covers contextual details such as where and in what environment the suicide took place.

The IG analyzed the 287 suicide cases reported through the DODSER database in 2011, the most recent annual data available at the time, and found that nearly a quarter of them had answers marked "don't know" or "data unavailable" in 50 percent to 100 percent of the data fields.

The most common fields marked unknown or unavailable were whether the deceased was a victim of emotional abuse (missing 61 percent of the time), whether he or she had visited chaplain services (57 percent) and whether the deceased had a family history of mental illness (57 percent).

It is important to note, however, that the IG study did not include the 2012 DODSER annual report, which was released on April 25. An appendix to the IG report notes "several areas of marked improvement" in DODSER data collection in 2012. The rate of "unknown" or "unavailable" answers to question of emotional abuse, for example, dropped from 61 percent to 35 percent.

Nonetheless, the 2012 report shows a significant degree of incomplete DODSER data. The IG's report notes that one reason for the missing information in 2011 is that many of the DODSER questions had medical jargon that only an expert could answer. If DODSER are incomplete or inaccessible to some, officials are working with an incomplete picture in devising mental health policies.

Picking up missing data pieces

The database is managed by the National Center for Telehealth and Technology (known as T2), a DOD organization charged with applying technology to mental health problems. T2's fiscal 2015 budget for managing the DODSER database and generating an annual report from it is $500,000, which does not include the military services' budgets for using the database.

There are two parts to the training an officer receives to learn how to fill out a DODSER report. The first is an online evaluation, and the second is the specific DODSER training administered by each of the military services, said T2 Deputy Director Mark Reger, who leads the DODSER program.

He said user training for DODSER is adequate, but no one is going to be an expert in every aspect of what DODSER requires. For example, a military commander might know a soldier's deployment history but not his medical diagnosis.

The IG report recommends that DOD take a more multidisciplinary approach to reporting suicides, with each suicide triggering a local review board made up of unit leaders, medical professionals and military investigators. That approach would help deliver more complete and accurate data to the DODSER database, the IG said.

Rajeev Ramchand, a senior behavioral and social scientist at Rand Corp. who studies suicide-prevention strategies for service members, offered an additional remedy to the problem of inexpert DODSER users. He called for a team of trained data experts at DOD to handle the data input for every case of suicide rather than relying on a wide range of officers who might have very little experience with the database.

"Are we looking for somebody who's an expert in every single field or are we looking for somebody who's an expert in collecting the data?" Ramchand asked. "I would always go with someone who is [an expert in] collecting the data."

Ramchand said the DODSER database is an effective and comprehensive means of tracking military suicides and suicide attempts but pointed out what he said is a blemish: The input fields do not leave much room to explain the source of the information. For example, if an entry notes that a soldier had trouble in a romantic relationship, it might simply cite the commander as the source of that information.

Ramchand said limited information in that case might misidentify a contributing factor to suicide. "The point of surveillance is to identify trends so that we can intervene, and we need to make sure that we're intervening on the right things," he said.

Although DODSER matches up closely with many of the Centers for Disease Control and Prevention's recommendations for compiling suicide data, one area where it does not is in detailing the source of the information, Ramchand added.

The hunt for better data is not likely to ever be fully satisfied, but the previous approach puts things in perspective. Before the DODSER database began to offer a standard method of suicide surveillance in 2008, the military services "each had their own [systems] for analyzing and understanding the characteristics and nature of suicide in their service," Reger said.

DODSER reports offer a more complete picture of suicide across the services, one that DOD is trying to combine with data provided by VA.

The promise of data sharing

DODSER reports from military bases around the world are one piece of suicide-prevention policy. Another is sharing that data with VA as soldiers retire and become veterans. To that end, the two agencies set up a joint Suicide Data Repository (SDR) in fiscal 2013 by acquiring mortality data on veterans and service members from the CDC.

VA had been working with the CDC since 2006 to collect suicide data on users of Veterans Health Administration services, said Robert Bossarte, co-director of the SDR and director of the Epidemiology Program in VA's Office of Public Health. But that gave a startlingly incomplete picture of the problem, considering that many veterans do not use VHA services.

When you run a veteran's name and Social Security number through the nascent repository, it matches that information up with the CDC's National Death Index and then returns a probabilistic score of matching records. A 99 percent score would indicate a near-certain match.

Estimated suicides among veterans and service members
CY 2008-2012*

- Veteran Suicides

- Service member suicides**

* Sources: 1) The 2012 Suicide Data Report from the Department of Veterans Affairs, which includes daily estimates that FCW extrapolated to yearly estimates. 2) The annual reports of DOD's National Center for Telehealth and Technology

** Includes active-duty, reserve and National Guard.
*** Veteran suicide data for these years is currently unavailable.

"The clinical and public health importance [of data from the repository] can't be understated," Bossarte said. "Understanding…increased risk following separation from service, understanding the impact of VA and DOD prevention and transition programs, and understanding new opportunities for intervention [are] only [possible] if we understand the epidemiology of risk in this population."

"One of the things that [the repository] did was allow for VA and DOD to agree upon a matching algorithm to identify when a case is a case," he added.

Bossarte said the SDR currently holds about 25 million personal records, including some duplicates. Of those, the system has identified 2 million veteran deaths since 1979, which is how far back the data goes. Bossarte said he and his colleagues are looking for other sources of data to track veterans who died before 1979. That is another chapter in the endless hunt for data to help prevent future suicides.

DOD and VA will open the SDR to public viewing when they release the first annual report on the database in the coming months, Bossarte said, adding that he hopes the report will help researchers understand what percentage of the total veteran population has been included in suicide data over the years.

For independent researchers like Ramchand who want to review the data, that annual report can't come soon enough. He said he would like to see those in charge of the SDR be more specific and public about their goals for the database and a timeline for achieving them.

SDR data has been made available to researchers for 51 studies, but all those researchers are affiliated with VA or DOD.

Key piece of DOD/VA repository untapped

DOD and VA have given the cause of suicide prevention significant attention and resources in recent years. "Increasing data fidelity" was one of the nine priorities identified by the Suicide Prevention General Officer Steering Committee, a group of flag officers, Senior Executive Service members and other officials in charge of implementing DOD's suicide prevention strategy.

At times, the challenges of harnessing the two vast bureaucracies of DOD and VA toward a common goal have been plain. In August 2010, a DOD task force recommended a revised, standardized suicide reporting methodology. But it was another three and half years before the department formally implemented that methodology in the form of a March 2014 memo.

When FCW asked Jackie Garrick, acting director of the Defense Suicide Prevention Office, why it took so long to implement the new methodology, she said officials needed time to include new data for the National Guard and Reserves and to test large datasets.

With that revised methodology, DOD can now compare factors that potentially increase the risk of suicide for active-duty and reserve service members and, in turn, "do more to target our policies and our programs…specific to those different populations," she said.

Perhaps an even more important hurdle for improved data policy will be getting the joint VA/DOD repository to actually share records, as it is intended to do. Right now, although data from the DODSER database and the VA equivalent is in the joint SDR, Bossarte said it is not being shared back and forth.

The repository's current focus is on mortality, whereas its mandate includes sharing data on possible suicides. If one wants to track a recently retired service member, now a veteran, using the SDR, "you would have to piece the data together, but all of the elements are included that would make that possible," Bossarte said.

DOD and VA officials say they will work in the coming months to reap more from predictive analytics for their suicide-prevention programs. Garrick said the Defense Suicide Prevention Office is tapping into the resources of the Defense Manpower Data Center, the Pentagon's data hub for military personnel, to turn data into a "predictive lens" to spot suicide risk among service members.

As for DODSER, Reger said T2 will seek to implement the IG recommendations for improving the database and its management. He added that many of the recommendations were not news to T2 and the agency has been already addressing them.

For example, the IG report noted that the software used by DODSER automatically archives a record 180 days after it has been entered and prevents it from being updated. That rigidity prompted some DODSER users to submit a report before the medical examiner's investigation was finished, resulting in incomplete information. That deadline stems from a regulatory mandate to protect service members' privacy, Reger said.

"We have already begun coordination of documents to change those regulatory requirements," he said, adding that he expects the change to be made in 2015.

The quality of DODSER submissions has improved since September 2013, when T2 began providing feedback to the services on their submissions, according to the IG report. For example, in the third quarter of 2012, "the average DODSER submission for the Army and Navy was less than 70 percent complete," the report states, but by the first quarter of 2014, "both services had increased to an average of more than 90 percent."

An invisible finish line

Improving the quality of data on suicide attempts among service members and veterans will likely be an endless task. Given advances in big-data analytics, improvements will always be possible. As Reger said, "I think the challenge of ensuring the highest quality data that you can will be an ongoing process."

Congress is also playing a part in suicide data policy. Lawmakers have included a provision to improve suicide data collection in the $585 billion defense authorization bill for fiscal 2015. The bill tasks DOD with developing "a standard method for collecting, reporting and assessing information regarding" suicide and suicide attempts, something the department has already committed to doing.

A critical challenge for DOD, VA and Capitol Hill will be to ensure that too many cooks don't spoil the broth. All the working groups, task forces and memos in the world will matter little if they do not lead to a drop in suicides. And, of course, improving data is but a piece of policy efforts to prevent military and veteran suicides. More important to the cause is the quality of mental health care that service members and veterans receive.

But data is a window into the problem, and the sharper that view comes into focus, the better off America's uniformed and retired military men and women will be.