A decade of missteps on a digital case-management system stands as a case study in project management
For over a decade, the FBI had been immersed in a vast, costly, but indisputably critical effort to build a modern digital case-management system. The bureau well understood the importance of information: gathering, organizing, sharing, assessing, and putting it to use to fight crime and protect national security. Yet what followed were countless missteps and misjudgments, more than $1 billion of public money squandered, and the worst terrorist attack on American soil since Pearl Harbor. And it still took more than 10 years to get it right.
In this two-part series, public interest technologist Mark Lerner examines the process, from the project’s outset in 2000 to its completion in 2012. In Part 1, Lerner reviews the history of the FBI’s efforts to develop and deploy a leading-edge case-management system, and then analyzes where and how the agency and its vendors stumbled and failed; the colossal waste of public money; and how it finally succeeded by using the Product Operating Model to focus on desired outcomes instead of project deadlines. The approach brought success at a jaw-dropping fraction of the costs already incurred.
Although a specific case study of one federal agency’s digital process-improvement program gone awry, the FBI’s experience has lasting relevance and broad application. It offers lessons for organizations across the public and private sectors in such areas as oversight, compliance, overcoming operational barriers, containing costs, and most important, in delivering the promised product for the public’s benefit.
By the end of the 20th century, digital tools for capturing, sorting, sharing, analyzing, and storing information were already ubiquitous, vastly expanding the ability of teams and organizations to collaborate and put knowledge and data to practical use. Finance, medicine, manufacturing, research — the digital revolution was changing them all.
One surprising exception was a pillar of the U.S. national security apparatus: the FBI.
Arguably the world’s premier domestic law enforcement and intelligence agency, the Federal Bureau of Investigation had long promoted itself as the forward edge of organizing, synthesizing, assessing, and deploying information and knowledge. And yet, as criminals and their crimes grew more sophisticated and global along with digital technology, the FBI lagged, firmly stuck in the past.
To their credit, the bureau’s leaders recognized the need to modernize its systems, and Congress backed the effort by committing the resources. And in 1995, the bureau rolled out a new digital system for managing its cases — everything from bank, mail, wire and corporate fraud, organized crime, racketeering, and public corruption, to interstate gang activity, international human trafficking, and cybercrime.
A case management system that could help organize countless pieces of potential evidence — surveillance, financial transactions, witness interviews and other sources within and across active and closed cases — would put the bureau on an equal footing with the criminals.
And the day the new system launched …
… it was already antiquated: obviously, hopelessly, exasperatingly out of date.
Known as the Automated Case Support (ACS) system, the system was a “greenscreen” application: Navigable only via keyboard instructions, it took dozens of complex commands to do simple tasks. This system was so difficult to use that agents would regularly delegate to administrative and junior staff any work that required using ACS, or they would just avoid using it entirely. Even if staff or agents did use it, they still needed to print out paper records and store them in their offices to serve as the official source of truth.
In subsequent fits and starts, the bureau had tried to improve over ACS by building new digital systems. Two programs were conceived — but failed to ever leave the ground. By the end of 2000, preliminary work had begun on a third attempt. Meanwhile, the lack of a capable digital system meant that the thousands of agents spread across the bureau’s dozens of field and support offices throughout the country hit obstacles in trying to combine data points from different files. For an agent in one field office to access records from another office meant filing and routing paperwork requests, and waiting for those records to be transported by ground courier.
This inability to easily and rapidly “connect the dots” among important pieces of information would have catastrophic results in the terrorist attacks of September 11, 2001. The intelligence community’s failure to preempt the attacks exposed this deep vulnerability in U.S. national security. Among the 9/11 Commission’s findings was that the FBI had data on people related to the attacks entering the country and on their engagements with flight schools, but that the information was sequestered across different FBI offices, unknown and thus unavailable to those who could have made use of it. At the FBI, investigators couldn’t effectively combine intel because the processes and systems the bureau used to maintain its records had no capabilities for sharing information easily.
The technical shortcomings were so severe and disabling that immediately after the attacks, FBI agents in Florida had to send records of the suspected terrorists to Washington by overnight shipping due to a lack of software capabilities for sending data and attachments electronically. At a subsequent Senate oversight hearing, Sen. Charles E. Schumer (D–N.Y.), then-chair of the Judiciary Committee’s Subcommittee on Administrative Oversight and the Courts, noted that an FBI official “had to personally deliver [testimony] on a disk, making it impossible to circulate and store it online.”
As Schumer put it, “For a long time, the FBI’s database warehouse was like Medusa, with over 40 databases with separate functions operating out of the same body but totally disconnected from one another.”
How did an agency widely seen as among the most sophisticated and effective domestic intelligence and law enforcement organizations in the world, one that boasted of its skill at organizing and synthesizing information, stumble so badly over organizing and synthesizing information?
The FBI spent over a billion dollars on its failed attempts at building a software system to enable investigators to do the most fundamental aspect of the job: make information useful. What happened inside the agency that led to these failures? Outside the agency, where were the key oversight bodies tasked by Congress with keeping the effort to build a working digital case management system in check — the Department of Justice’s Office of the Inspector General, the Government Accountability Office, and Congress itself? How did their oversight efforts go awry, and what could they have done better?
After a decade of failing to equip the bureau with state-of-the-art tools to fight crime, thwart terrorism, and make cases, the FBI finally found its way through a combination of bringing in new people to take fresh approaches to an old and vexing problem, and freeing in-house talent to take a crack at the case.
A theme running through this saga is that modernizing government infrastructure is not about executing a series of isolated, one-time software upgrades. It requires shifting the foundational elements of how the government funds, builds, and buys software. This case study serves as a textbook example of how the legacy-project mentality erodes state capacity, and how adopting a product-focused digital infrastructure restores it.
Trilogy and the Virtual Case File
In June of 2001, the FBI had just launched an effort to modernize the hardware and software used across its operations via a three-part program called Trilogy. Parts one and two of Trilogy focused on hardware infrastructure and providing field offices with new equipment and network connections. The third part of Trilogy aimed to produce a new set of software applications for managing cases, sharing data, and reducing paperwork. This third part housed the FBI’s latest attempt at updating its modern case management system — the Virtual Case File (VCF).
The FBI awarded the contract to develop VCF to a single vendor: Science Applications International Corporation, or SAIC, a major defense and technology contractor. In a press release in June 2001, the company said the system would enable agents “to move data between applications and store all data for a case in one location, including audio and video,” through a bureau-wide intranet. “Most importantly, our solution is designed to meet the day-to-day, real-world requirements of FBI agents,” a company executive boasted.
After 9/11, Congress expanded the Trilogy budget from its original $380 million to $458 million (just over $850 million in today’s dollars) and shortened the project timeline from mid-2004 to the end of 2003.
The FBI and SAIC took a fairly traditional approach to planning and building the Virtual Case File: define the requirements up front, finalize them, and build against those finalized requirements. While the scope of the initial June 2001 contract was limited, the FBI and SAIC renegotiated and expanded the project requirements after 9/11 to encompass a full overhaul of the FBI’s case management system. The requirements were so sprawling that it took six months of nonstop work to gather them into a document that ran over 800 pages.
That requirements rework was just one of many setbacks for the Trilogy program. The VCF portion, in particular, endured the most delay, raising red flags throughout its development. In one instance, a whistleblower at SAIC tried to raise concerns about the quality of the company’s work, claiming that SAIC lacked modern cybersecurity practices, maintained excessive staffing levels on the project, produced extraneous and unnecessary documentation, and generally produced subpar quality work. As he put it, “The company’s attitude was that it’s other people’s money, so they’ll burn it every which-way they want to […] Would the product actually work? Would it help agents do their jobs? I don’t think anyone on the SAIC side cared about that.”
It was a revealing — and potentially costly — account. In response, the FBI took him off the project.
The slow pace of change, and the paperwork involved
After SAIC and the FBI finalized and locked in the 800-page requirements document, they instituted a restrictive change control process for any new changes to go through. That meant that anytime they wanted to update the requirements to change a workflow, screen layout or button text, they had to file paperwork, hold meetings and come to a group consensus about the changes. Change-control processes are generally designed to disincentivize changing the requirements, but that didn’t stop the FBI from making over 400 change requests between December 2002 and December 2003.
Congress, for reasons known only to Congress or perhaps for no reason at all, approved another $123 million for Trilogy, bringing the total program budget up to $581 million, or just over $1 billion today, from the original $380 million. It might have been worth the extra expense had it produced a high-quality product. But it didn’t. In fact, SAIC’s work product continued to degrade in quality. And the FBI noticed.
When SAIC attempted to deliver its VCF product at the end of 2003, the FBI rejected the delivery. It deemed the software unacceptable for agents’ use, with issues ranging from severe (nonfunctional search features) to cosmetic (mislabeled buttons). The FBI and SAIC blamed each other, with one arguing poor development and the other arguing too many requirement changes. After engaging with a third-party mediator, they agreed to a last-ditch effort to launch a smaller set of features … for another $16.4 million and another year for the timeline.
When SAIC delivered this narrowed-down version of VCF, the FBI tested it with real users. And came to the same conclusion: It was unusable. In early 2005, the FBI terminated the tests, and fully cancelled the Virtual Case File project. The software developed by SAIC was of such poor quality that the bureau decided it was not worth reusing, and scrapped it all. Four years of work and over $100 million in VCF-specific funds had gone to waste with nothing to show for it, all while the country’s already slow and outdated national security systems continued to age and degrade.
After the cancellation of VCF, the FBI started working immediately on a new system to fill the need. The bureau was eager to prevent the mistakes it had made in VCF, and implemented far more project management procedures in an attempt to exert further control over the development process. The goal was to deliver a complete working system by the end of 2009, just under four years after the failure of VCF.
With a new project came a new name: Sentinel.
Sentinel
Sentinel’s scope was broader than VCF’s, with more features and functionality, such as workload management tools and advanced search features. And with a broader scope came a bigger budget: In 2005 the FBI outlined the plan for developing Sentinel, breaking it down into a four-phase project with a budget of $425 million, or just about $700 million in today’s dollars. The FBI set a December 2009 launch date.
The difficulties arose at the outset, during the procurement process for securing a contractor. The FBI put out a $300 million+ RFP for Sentinel, seeking a single contractor to build the entire system. It drew just two proposals. After considering whether it should continue at all given the lack of competition, on March 16, 2006, the bureau decided to move forward and awarded the contract to Lockheed Martin, another giant federal contractor.
One key difference between VCF and Sentinel was that Sentinel followed a much more intensive set of project management practices. VCF’s project management practices were clearly flawed and relied heavily on trust in SAIC’s behavior and expertise. This led to a welter of issues, including table-stakes contract management mistakes such as approving nonexistent invoices and paying for first-class travel for contractors — rookie errors that are particularly problematic practices for a law enforcement agency charged with investigating exactly those kinds of practices in other organizations.
Sentinel followed much more strict and rigid project management practices, both in the management of contracts and in planning the software development. The FBI’s chief information officer, for instance, noted that “processes were put in place — program management discipline, lifecycle development methodology, enterprise architecture, earned value management, IT governance process, security certification and accreditation — to put us and keep us on track.” These ostensibly corrective measures led to the production of over 20,000 pages of documentation on merely the first of Sentinel’s four phases, representing a massive increase in administrative burdens on the Sentinel staff.
Phase 1 was intended to provide agents with a stop-gap solution to address the immediate case-management problem. Rather than creating a new system from scratch, Phase 1 entailed building a more modern front-end interface that FBI staff could use to make older systems easier to use. The remainder of the Sentinel program was more complex, and involved building a completely new digital case management system with such features as linked digital records, digital signatures, and file routing.
The FBI and Lockheed Martin deployed Phase 1 to users in June 2007. Though the system was not without issues, it was functional enough that it was being used across the FBI. With lessons learned from Phase 1, the FBI and Lockheed Martin revisited the Sentinel development plan, breaking the remaining phases into segments and the segments into increments. The aim was to define more granularly the capabilities to be delivered in each new, smaller element.
One aspect of the new plan that wasn’t smaller, of course, was the budget, which Congress expanded from $425 million to, ultimately, $451 million – $728 million today..
It wasn’t long before issues began to arise. Significant staff turnover disrupted the FBI’s Sentinel program management office, project timelines began to slip, and tests of the new software Lockheed Martin had built for Phase 2 were going exceedingly poorly: it was missing certain necessary forms, lacked autosave, and would delete hours of users’ work. After Lockheed Martin delivered an unusable version of Phase 2 with massive usability and performance issues, the FBI issued a stop work order that prohibited the contractor from working on anything other than mending the unacceptable software. An independent audit of the Sentinel program performed by the MITRE Corporation confirmed Lockheed Martin’s failures — not just of the products they built but of the way they were working. It estimated that if Lockheed continued to work with its existing practices, the program would cost an additional $351 million on top of the $405 million already spent and that it would not be finished until 2016.
A quick tally of the costs up to this point: The FBI estimated the total lost value from the VCF failure at about $105 million. When Lockheed delivered its unusable version of Sentinel, the FBI had spent $405 million on Sentinel alone. Adjusting for inflation, the FBI had spent $884 million on unusable software and no significant improvements to its capabilities.
Put another way, the government had spent close to a billion dollars for nothing.
Lavender
As Lockheed Martin’s work quality worsened, an internal and unofficial group of FBI staff with technical skills decided to take matters into their own hands. They worked together to build their own version of Sentinel. They nicknamed the small internal effort “Lavender,” and over time they built a functioning prototype of a case management system using more modern technologies and, critically, more modern software development practices: working on a small team, building up the product iteratively, and engaging directly with their users while building the software.
After the group demonstrated its work to FBI executives, the bureau decided that it had had its fill of working with — and relying on — giant contractors to do a job that could be done much more quickly, cheaply, and effectively in-house. And so, in September 2010, the FBI fully took over development of Sentinel from Lockheed Martin. It brought the project in-house, using bureau employees to develop the software. They reduced the number of people working on Sentinel from hundreds to around 40, with fewer than half doing hands-on software development.
Furthermore, the bureau abandoned the process-heavy management practices that had generated thousands of pages of documentation but hadn’t prevented yet another failure, let alone deliver success. In its place, it adopted agile software development methods, at the time a relatively new management practice that deprioritized documentation, process and rigid plans, and instead prioritized engaging with customers, building functioning software, working in small iterations, and putting people first.
This new effort operated in line with the Product Operating Model, which emphasizes working iteratively and engaging users directly. The team was small and cross-functional, and regularly included users and stakeholders directly in the software development process. Team members used modern product management practices and iterative development to focus on valuable added features and functionality.
They added new features and functionality to the product incredibly frequently, and delivered those new additions directly to the set of users already using the product rather than having them wait for an official release. They set a cadence of providing executives real demonstrations of those new functions every two weeks rather than just giving status reports. And they focused their efforts on delivering working software products over all else, including over following any particular rigid procedures.
The FBI was extremely confident in this new approach, believing that it could deliver all of the originally promised functionality rapidly and for a fraction of the cost: a “mere” $20 million, or some 2 percent of what had already been spent. For context, Lockheed Martin said it could deliver the product in three years for $194 million — after it had already delivered a failed version of the product. In what turned out to be one of the wiser of its decisions regarding a state-of-the-art case management system, the FBI rejected Lockheed’s proposal, and instead dove into the development themselves.
The new Sentinel team regularly released working software to users across the agency, and while initial releases had their own issues — such as inadvertently causing a bureau-wide network outage — the speed at which it resolved those issues dramatically increased trust in the system. In one of the final system tests, users gave the in-house version of Sentinel a satisfaction rating of 8.5 out of 10, far higher than any prior version of Sentinel.
The FBI launched Sentinel across the entire enterprise on July 1, 2012, more than a decade after embarking on a project to equip agents with a contemporary analog of a notepad and pen. Users across the bureau immediately adopted the software. After the initial rollout of the full system, the small in-house team – by now augmented with new hires from outside — continued to develop and improve the product, delivering valuable updates and enhancements at a regular pace to users across the bureau.
Root causes of the FBI’s failures and ultimate success
A cluster of factors contributed to the failures of the VCF and Sentinel programs, including rigid and misaligned management processes, counterincentives from bureaucratic oversight, and a dearth of in-house technical talent.
Rigid and misaligned management processes meet the von Moltke rule
The management practices that the FBI employed during the development of both VCF and Sentinel were designed to prescribe and predict the entire path of the program, from start to finish, before a single line of code was written. In this way, the FBI treated its digital case management system as a rigid and time-bound project driven by budgets and arbitrary deadlines, rather than by the goal of delivering an enduring product that addresses a real need.
In both instances, the FBI wrote hundreds of pages of requirements for the systems based on predefined knowledge. These requirements documents were formalized and finalized into the agreed-upon standard for the program. All future progress would be judged against these requirements — regardless of what program developers learned along the way, what users said they needed, or as other needs and circumstances evolved. Any changes to the requirements had to move through the arduous change control processes mentioned earlier.
As part of formalizing and finalizing the requirements, the FBI mapped them against a rigid schedule far in advance, further entrenching how difficult and costly it would be to change. This traditional waterfall-style program management practice defined success by whether the requirements were met on schedule — rather than whether users’ experiences were improved.
The purpose of these upfront requirements and schedules is to provide predictability in the development process by explicitly determining what features will be delivered when and, by extension, how much it will cost. It’s an appealing approach, particularly for oversight and management bodies steeped in budgets and timelines. However, to paraphrase Prussian Field Marshal Helmuth von Moltke the Elder, no software development plan survives its first encounter with reality.
No software development plan survives its first encounter with reality.
Particularly in scenarios of developing an evolving product for a large user base, as was the case with VCF and Sentinel for tens of thousands of FBI staff, rigid upfront requirements and schedules create inflexibility and punish adaptability at the expense of meeting users’ needs. The FBI prioritized adherence to their predefined plans regardless of the value being delivered — or not delivered — to users.
Bureaucratic oversight
This was only made worse by oversight bodies’ singular focus on whether or not the project met the pre-approved plans, and whether they used specific, procedurally intensive project management bureaucracies to meet those plans. The oversight bodies measured success via documentation, reports, statistics, and other artifacts that supposedly demonstrated program activities and progress. But these documents were merely caricatures of the realities of the program, which unfortunately is common in software development. Reports documenting adherence to a feature roadmap are inherently disconnected from the actual implementation of those features, which itself is disconnected from value delivered to users. Assessments of procedural compliance are at best a proxy for understanding the value of a program, and at worst a false legitimization of faulty and ineffective technology and management practices.
That’s not to say that all oversight reviews are useless. In some cases, particularly in the event of retrospective analyses, these audits are helpful. The Government Accountability Office’s postfailure assessments of the VCF program revealed the previously discussed contracts mismanagement of overpaying for contractor travel and paying for nonexistent invoices. But when an audit body requires creating and maintaining a bevy of bureaucratic management documents — earned value management reports, bills of materials, enterprise integration strategies, contract risk registers, program management staffing plans — it is further incentivizing the expenditure of time, energy, and money on paperwork over the development of successful products. This is particularly true when adherence to irrelevant requirements is the only thing that oversight bodies measure.
Inflexible funding
Both VCF and the initial Sentinel programs were funded using traditional federal budgeting structures that reinforced the strictly defined requirements of expected functionality. This creates further inflexibility, and is often accompanied by a restrictive change-control process that attempts to minimize unexpected costs. When the FBI pulled Sentinel in-house with a slim $20 million budget, though, it adopted a capability-based budgeting structure by necessity. It stopped treating Sentinel as a project to be finished, and instead treated it as an enduring capability requiring a steady and accountable product team. This change in funding structure gave the internal team the flexibility to rapidly shift priorities midstream based on real use feedback.
Lack of in-house technical talent
Ultimately, though, the FBI’s ability to deliver a functional product hinged on one critical factor: the presence of in-house technical talent. The FBI turned Sentinel around by building and empowering an in-house team of federal employees with technical skills who employed modern and agile methods of engaging directly with users, building working software products in small increments, and focusing on delivering valuable products and tools. The leaders who enabled the transformation of Sentinel — the FBI’s chief information and chief technology officers — were both recent hires from the private sector with hands-on technical product development experience. The oversight that was most effective at understanding and predicting VCF’s and Sentinel’s failures came from the outside audit groups that Congress and the FBI brought in — namely the MITRE Corporation, the National Academy of Sciences, and the Aerospace Corporation, each of which had in-house technical talent.
Without in-house technical talent, program offices charged with managing technical projects rely on the same types of proxy measurements that are disconnected from reality. Possessing technical talent — not just the ability to write software, but also such skills as managing digital products and undertaking human-centered design — puts the government in the driver’s seat of major technical programs. Modern digital products and services require cross-functional teams with specialized skills, and in-house talent ensures the alignment of incentives and capabilities that are needed for successful long-term product operations.
Possessing technical talent — not just the ability to write software, but also such skills as managing digital products and undertaking human-centered design — puts the government in the driver’s seat of major technical programs.
After more than a decade and more than $1 billion, the FBI finally succeeded in building a modern digital case management system. Along the way, it learned valuable lessons in building and empowering technical teams and using modern software development practices.
What of the other groups involved? This program had been under heavy scrutiny from Congress and multiple oversight groups, none of which preempted colossal waste and failure. What could they have done differently? We’ll explore those issues in Part 2 – coming soon.
