Validating Model Software

Some interesting thoughts from Forrest Shull's article "Assuring the Future? A Look at ValidatingClimate Model Software"

Validating Model Results

[The main question is,] how the field assesses the quality and trustability of results coming out of the software models—in other words, how does anybody know that the software is working correctly? I’ve been particularly fascinated by this issue for some time. Unlike many other domains in which software is created, in climate studies, the correct answer or behavior is not known in advance; that’s why we need the software in the first place.

[There are] a number of ways the community validates climate code. Some of these are strategies that any developer should recognize, while others are uniquely fitted to this domain.

Testing the Code

Climate code is typically very well tested. ...Good testing practices, such as regression testing and tracking issues to completion, are typically adopted. Nightly builds and automated test suites catch a lot of inadvertent changes.

Divide-and-Conquer StrategiesAt a high level, climate models tend to have four major components that model the atmosphere, the ocean, sea ice, and land. For purposes of validation, the components can be separated and their output judged independently by experts in the relevant domain. Each of these comparisons is a large undertaking that can occupy a research scientist for years. But, as Robert suggests, this approach represents a "step-by-step approach to building confidence in the pieces."

Catastrophic SW Dev Failure

In an amazing and refreshingly transparent article, ("Why the FBI Can’t Build a Case Management System", IEEE Computer Magazine, June 2012), Jerome Israel (one of the leading technical directors in the intelligence community and CTO of the FBI) describes some recent, high-value SW development projects that went horribly awry.

Early on, he cringes as he describes some early projects he was involved in:  

"These were NSA’s proud turn-of-the-century transformation programs, but they all failed, including our flagship program Trailblazer, which cost more than $1 billion."

He proceeds to describe at length the problematic pursuit of a boondoggle of a project to digitize the FBI's massive Case Management System.

He was chagrined by the lackluster response to the original RFP. Not only did they get only two proposals, but, 

"Both proposals were generic and lacked creativity; they definitely weren’t equal to the effort the FBI had put into the RFP.... We were surprised that neither proposal contained a coherent plan that addressed the FBI’s most challenging technical problems."

Ultimately the project struggled on for nearly eight years and cost over $400 million.

And it failed. It was never used.

One of the fatal flaws in the project was the Agency's attempts at program management: 

"How we managed IT programs was a subject of constant debate, which centered on the following question: How could an engineer who spent four to five years in college earning a difficult degree be assigned to the number-two spot on a project, next to a person who only had an eight-week certification? .... [In one case,] the senior PM had all the skills OMB demands, but he was a linguist by training and over his head on a software development project. He admitted that he didn’t know what software development standards and processes the company had used, nor how the code was tested, the results of those tests, what kinds of bugs emerged, or how fast the company had closed them."

More quotes:

In October, Sentinel was beset with hundreds of software bugs, including many priority ones and twos.... The team concluded that Lockheed Martin had significantly deviated from accepted systems engineering practices, didn’t follow its own published documentation requirements, and hadn’t adequately followed testing procedures. According to the team’s report, these deficiencies resulted in over 10,000 inefficiencies in Sentinel’s software code”.

Around September 2010, the FBI decided to remove Lockheed and complete Sentinel in-house using FBI employees and agile development methodologies. The bureau is now hoping to deliver the system sometime in summer 2012.

History is still repeating itself. In 2010, OMB had 26 projects on its high-risk watch list. They earned this dubious honor by incurring significant cost increases and schedule delays, failing to meet mission objectives, frequently revising the baseline, or lacking clear executive sponsorship or leadership. These projects span 15 departments and are estimated to cost $30 billion to complete.