Catastrophic SW Dev Failure

In an amazing and refreshingly transparent article, ("Why the FBI Can’t Build a Case Management System", IEEE Computer Magazine, June 2012), Jerome Israel (one of the leading technical directors in the intelligence community and CTO of the FBI) describes some recent, high-value SW development projects that went horribly awry.

Early on, he cringes as he describes some early projects he was involved in:  

"These were NSA’s proud turn-of-the-century transformation programs, but they all failed, including our flagship program Trailblazer, which cost more than $1 billion."

He proceeds to describe at length the problematic pursuit of a boondoggle of a project to digitize the FBI's massive Case Management System.

He was chagrined by the lackluster response to the original RFP. Not only did they get only two proposals, but, 

"Both proposals were generic and lacked creativity; they definitely weren’t equal to the effort the FBI had put into the RFP.... We were surprised that neither proposal contained a coherent plan that addressed the FBI’s most challenging technical problems."

Ultimately the project struggled on for nearly eight years and cost over $400 million.

And it failed. It was never used.

One of the fatal flaws in the project was the Agency's attempts at program management: 

"How we managed IT programs was a subject of constant debate, which centered on the following question: How could an engineer who spent four to five years in college earning a difficult degree be assigned to the number-two spot on a project, next to a person who only had an eight-week certification? .... [In one case,] the senior PM had all the skills OMB demands, but he was a linguist by training and over his head on a software development project. He admitted that he didn’t know what software development standards and processes the company had used, nor how the code was tested, the results of those tests, what kinds of bugs emerged, or how fast the company had closed them."

More quotes:

In October, Sentinel was beset with hundreds of software bugs, including many priority ones and twos.... The team concluded that Lockheed Martin had significantly deviated from accepted systems engineering practices, didn’t follow its own published documentation requirements, and hadn’t adequately followed testing procedures. According to the team’s report, these deficiencies resulted in over 10,000 inefficiencies in Sentinel’s software code”.

Around September 2010, the FBI decided to remove Lockheed and complete Sentinel in-house using FBI employees and agile development methodologies. The bureau is now hoping to deliver the system sometime in summer 2012.

History is still repeating itself. In 2010, OMB had 26 projects on its high-risk watch list. They earned this dubious honor by incurring significant cost increases and schedule delays, failing to meet mission objectives, frequently revising the baseline, or lacking clear executive sponsorship or leadership. These projects span 15 departments and are estimated to cost $30 billion to complete.