Sentinel, or the case of the high price of technical delivery risk

This is the second of a two-part exploration of the FBI’s attempt to build a modern digital case management system between 2000 and 2012. In Part One, public interest technologist Mark Lerner tracks the wayward paths that the bureau, hitched to giant government contractors, stumbled down; its own critical missteps along the way; and how it finally corrected its course and ultimately succeeded by following the principles of modern software practices described in the Product Operating Model.

In Part Two, Lerner explores the role that oversight played — and misplayed — across the effort, how it failed to prevent successive and costly failures; what the FBI could have done differently; and what other agencies should do in their technology initiatives.

Part Two picks the case up in 2005, following the failure of the bureau’s Virtual Case File (VCF) and the subsequent launch of its next effort, known as Sentinel.

The Federal Bureau of Investigations launched Sentinel in 2005 with an expanded budget and intensified project management discipline. With the active involvement of at least 18 concurrent oversight and governance bodies — the Government Accountability Office, the Department of Justice’s Office of the Inspector General, and numerous congressional committees among them — what could go wrong?

Turns out, quite a lot.

Sentinel rapidly degraded into an unusable product that threatened to waste hundreds of millions of dollars.

Oversight was omnipresent throughout Sentinel and its predecessor, the Virtual Case File. Yet it systematically failed to predict or prevent their collapse and the accompanying loss of hundreds of millions of public dollars that had been poured into them. The silver lining from this failure is that it serves as evidence that traditional oversight frameworks are fundamentally optimized in the wrong direction with regard to IT programs: They excel at catching bureaucratic noncompliance or fraud, waste and abuse, but they’re functionally blind to technical delivery risk and the kind of waste that comes from years of ineffective digital service delivery.

Oversight bodies can take immediate, actionable steps in their management of software programs to uncover technical realities and provide true value. By shifting from rigid compliance with bureaucratic specifications to a focus on evaluating in-house technical capacity, enforcing right-sized governance metrics, rejecting bloated single-vendor contracts, and demanding live product demonstrations within months of funding, oversight bodies can transform from passive chroniclers of failure into active enablers of state capacity.

The context: From VCF to Sentinel

It’s important to recall the operational crisis underway at the time. In the wake of the September 11 terrorist attacks, the 9/11 Commission revealed that the FBI had possessed critical intelligence that could have prevented the strikes, but that it didn’t have the tools to “connect the dots”: The bureau’s antiquated IT systems isolated files in regional offices and made data consolidation — and thus organization-wide access — a tremendously challenging task.

Determined to prevent such a calamity from happening again, Congress flooded the bureau with the resources to build a modern digital case management system.

The bureau outsourced the initial effort, the Virtual Case File, to a single vendor using a rigid waterfall project management structure that promised predictability and linear progress, but delivered inflexibility and lack of customer engagement. After four years and over $170 million spent, VCF was completely scrapped in 2005 as unusable. The FBI immediately launched Sentinel, a larger, more expensive iteration intended to fix VCF’s mistakes through hyper-intensive project discipline and strict contract compliance.

Yet despite omnipresent administrative monitoring, Sentinel repeated the exact same downward trajectory — until an internal pivot fundamentally changed the Bureau’s operational model. I share the full story, including reflections on the FBI’s actions, in Part 1.

The Role of oversight and governance

There were numerous levels of oversight and governance throughout the 12 years of the FBI’s journey to implement its case management software. Between VCF and Sentinel, at least 18 government oversight and governance bodies across both the executive and legislative branches were involved, as well as from multiple outside expert auditors and reviewers.

Table of entities engaged in oversight of the VCF and Sentinel programs

* The National Academy of Sciences, the MITRE Corporation, and the Aerospace Corporation are quasi-governmental organizations that were consulted to provide audit reports and oversight on the FBI’s technology programs.

The vast majority of these oversight bodies were made up of investigators with expertise in generalized project management, financial management, or other forms of business operations — and not in software development.

Though these groups were nominally conducting oversight, their engagement essentially diffused decision-making authority and accountability, which is precisely what oversight aims to protect. Across both the VCF and Sentinel programs, multiple oversight groups had to approve any large-scale decisions, supposedly to incorporate their expert insights but in reality to slow the rate of change from the pre-defined plan. In turn, this distributed the accountability of who had made each decision, further diluting oversight effectiveness.

Other oversight bodies pushed the program to adhere as closely as possible to both the pre-defined plan and rigid waterfall project management processes. Some requested and reviewed project management documentation that, in theory, should have provided transparency into the progress of the program, but in practice only demonstrated that the program office had merely written the documentation and provided no real evidence of the underlying progress.

Only a small set of these oversight groups had the purpose, if not the means, of providing true technical insights on the underlying software.

Combined, these factors constituted systemic breakdowns in the oversight process. They underscore the fact that in modern public service delivery, the health of a program depends heavily on the speed and accuracy of its feedback loops. High-capacity product teams build tight loops in which real-time user data and working software directly inform management decisions.

Conversely, traditional waterfall-based oversight architecture relies on voluminous paper reports and compliance checklists rather than working software, creating an administrative echo chamber. They’re optimized to catch fraud and abuse after the fact, but they completely miss the compounding waste of nonfunctional delivery because their feedback loops measure process compliance rather than actual utility.

As Niskanen’s research on oversight notes, federal watchdogs excel at measuring raw fiscal metrics, yielding an immense return on investment by identifying potential cost savings. However, the impact of oversight extends far beyond dollars and cents. It shapes an agency’s behavioral incentives. Traditional oversight suffers by focusing exclusively on avoiding failure, and penalizes experimentation and agility.

In the case of Sentinel, this dynamic created a compliance trap: The FBI focused on checking boxes to appease its oversight bodies, pushing the bureau into a state of prioritizing proceduralism over delivering value. By punishing deviation from the original and flawed plan, oversight inadvertently choked out the agency’s ability to take the calculated risks necessary to save the project.

These dynamics arguably are best seen in the work of the Government Accountability Office (GAO), the Department of Justice’s Office of the Inspector General (DOJ OIG), and the congressional committees with jurisdiction over VCF and Sentinel. 

Table of GAO reports, OIG reports, and congressional hearings in relation to program milestones. Despite significant oversight activity, VCF and Sentinel still resulted in failure.

Government Accountability Office

By and large, the GAO’s reports relating to VCF heralded the FBI’s practices. They lauded the IT human capital policies, integrated planning, enterprise architecture documentation, acquisition practices, and financial controls employed by the FBI over the course of the Trilogy project. Where they did find fault, they focused on the need for more detailed plans, more voluminous documentation, and more bureaucratic governance.

None of the GAO’s published reports accurately predicted the VCF program’s failure, nor identified the root causes.

When the FBI began the Sentinel project, GAO reports followed the same pattern as its reports on VCF. They promoted the FBI’s acquisition practices and recommended the bureau implement additional policies and procedures such as future workforce forecasts, inventories of existing workforce capabilities, and human capital gap analyses. These reports did not call for additional technical talent, merely the existence of plans for generalized human capital management.

To give them credit, the GAO did retrospectively find that the FBI had incurred “questionable costs” in developing VCF, including “first class air travel by contractors, incorrect billing for overtime, over-charging of labor rates, charges for training never provided, and in some cases payment of insufficient or nonexistent invoices.” These findings exposed program mismanagement after the fact, but did little to predict or prevent the massive failures of the system. In fact, the FBI successfully implemented and closed the vast majority of GAO recommendations, and yet still failed to deliver the program. 

Overall, the questionable costs that the GAO found amounted to peanuts compared with the $100 million-plus that the VCF program failure wasted.

Department of Justice Office of the Inspector General

The DOJ’s Office of the Inspector General (OIG) was, for all intents and purposes, the authoritative oversight body for the Sentinel program. It released 10 reports over the span of Sentinel’s development, roughly one every year, totalling over 750 pages. These reports were referenced regularly in congressional hearings, and the IG was usually present in hearings. The OIG also provided audits of the Trilogy program and VCF, though fewer than with Sentinel. OIG staff were fully embedded with the Sentinel program management office, enabling them to peer over the shoulder of the program during its development.

As with the GAO, the Inspector General’s reports tended toward compliance with procedures and up-front plans. Based on its reports, the FBI needed more detailed planning, contractual penalties for missing deadlines, higher fidelity requirements documents, and a more normalized schedule. In the OIG’s audits of the Sentinel program, the most common flags it raised were calling for more rigorous project management practices, requesting more documentation, noting risks in the program management office’s staffing levels, and excessive changes to the program’s requirements.

The table below shows a subset of the types of concerns that the OIG raised and how often they appeared in its reports, and categorizes the recommendations based on their alignment with either more traditional and inflexible project management practices, or with the flexible and adaptable Product Operating Model structures.

Table of categorized recommendations from Inspector General reports on the Sentinel program.

While the majority of the OIG’s concerns followed the faulty patterns of rigid management processes and misaligned bureaucratic incentives, one report stands out as an exception: Sentinel Audit V: Status of the Federal Bureau of Investigation’s Case Management System, published in November 2009, just one month before the original deadline for completing Sentinel. In it, the OIG explicitly recommends “increasing user involvement in the development of Sentinel,” noting that “user involvement is vital to the successful development” of the program. 

Moreover, the report recommended placing a federal employee engineer on the project because “that engineer would have a much more defined technical knowledge of the project and access to channels of communication to both receive and deliver project information.” The report also recommended staffing a designer on the project as a federal employee with responsibility “for ensuring that Sentinel is user-friendly and that the final product effectively satisfies user requirements.”

These recommendations closely follow the Product Operating Model, and address the roots of the Sentinel program’s challenges more than any of OIG’s other recommendations. They spoke to the benefits to the agency of building its own capacity to develop its own digital services and manage its own products. One could argue that the FBI eventually followed these recommendations when it brought Sentinel development in-house, though it seems clear that the recommendations weren’t the impetus for doing so — the FBI brought the development in-house due to advocacy from internal staff and new leadership from outside of government. 

Regardless, this OIG report supported the major change of the Sentinel program toward the structure that eventually saw its success. Each of the OIG’s other reports not only reinforced the same faulty practices that had been keeping the program in disarray, but pushed the FBI further toward stifflyingly bureaucratic measures in what had become an epic slog to deploy a functioning case management system.

Congressional committees: The final frontier

For close to 10 years, Congress had pushed, prodded, and dutifully funded the bureau’s keystone effort to field a functioning case management system that would enable it not only to connect dots but to see them all in the first place. By 2010, it had showered some $800 million on the bureau with near-zero return on their investment, despite its own substantial oversight resources. So what were all those congressional oversight committees doing that whole time?

Both the VCF and Sentinel programs were frequent topics at congressional hearings, largely within both chambers’ Appropriations and Judiciary committees. In some hearings, the programs were merely one part of a larger discussion. But in many, particularly those held in the wake of the big public failures, the panels focused exclusively and explicitly on the FBI effort. Robert Mueller III, the FBI director throughout the projects’ lifetime, was the most frequent witness at these hearings, providing insights and explanations himself.

As a general matter, Congress has a challenging task in these types of hearings. Operating against time constraints and a narrow window through which to view a program, committee members attempt to understand a program’s progress and determine the appropriate level of power and influence to apply to effect course corrections to avoid impending errors or failures. They call witnesses to testify at hearings, ask specific questions to attempt to uncover or clarify challenges, and request updates from or specific actions for the agency to take. The tenor of these types of hearings can be tense, and the information they produce is inherently limited due to the duration, scope, and nature of the conversations.

Over the course of these FBI programs, Congress asked questions about specific features and capabilities, strategic planning documents, cost and schedule estimates, and other matters relating to bureaucratic documentation and process. As we discussed in Part One, these inquiries neither revealed the impending failures nor prevented them.

In some instances, however, Congress wisely interrogated FBI officials about the bureau’s foundational capacity to build and deliver digital products and services as complex as those envisioned in the Virtual Case File and Sentinel. After the failure of VCF, for example, Congress asked the FBI whether or not it had the necessary technical talent in-house to manage the development of a program such as VCF, and whether or not that technical talent had the necessary decision-making authority.

A later hearing on Sentinel would similarly ask about technical skillsets, capabilities, and leadership. But although these lines of questions were much more relevant to the success or failure of the program, they were eclipsed by other inquiries.

The appropriations process is another tool at Congress’s disposal for diverting programs away from the brink of failure. In both appropriations language and hearings, Congress has the power to direct agencies to take specific actions. Throughout the 12 years of VCF and Sentinel, Congress included language either strongly advising or explicitly requiring the FBI to expedite deployment of the software, produce quarterly reports, create an advisory panel, commission independent reviews, submit detailed reports on cost and schedule estimates, use specific management practices, and produce a detailed list of all functionality it expected from the finished product.

These requests and requirements were divorced from the factors that led to the program’s root challenges, and didn’t prevent the program’s failure.

Other language included in appropriations packages only added to the program’s challenges. In 2007, Congress included a requirement that the FBI break the remaining work for Sentinel into “work packages,” declare the start and end dates of these packages (which must last no longer than four months per package), and outline the expected costs of each. This is what led the FBI to break the plan for Sentinel from phases into segments and increments. Undertaking such a fine-grained level of specificity made it all but inevitable that the plan would be moot as soon as it was finalized.

Additionally, the budget allocations that Congress provided to the FBI for this work followed traditional budgeting practices: Congress provides funding for a program, with no specific tie to the capabilities to be built or the experience to be delivered to users. This essay from Solitaire Carroll explores in depth the challenges with program-based funding and extols the values of capabilities-based funding, particularly with regard to IT and software capabilities. It’s a style of funding that could have significantly benefited Sentinel’s underlying goals.

A new paradigm for oversight bodies

This case study highlights a common pattern: Oversight from the GAO, inspectors general, and Congress frequently yields insights that don’t relate to the underlying challenges of software development, or that incentivize failure by rewarding compliance with bureaucratic checklists over delivery of results.

But effective oversight is possible. Oversight bodies should maintain a steadfast focus on three factors:

• capacity of the government to deliver on the program;
• progress made on addressing the underlying problem; and,
• reviewing the product itself.

Build In-house capacity

First and foremost, oversight should ensure that agencies have sufficient in-house technical talent to support agencies’ goals. Oversight bodies should encourage investments in in-house technical talent by evaluating technology programs on whether the program teams have the in-house technical talent necessary for the program’s particular technology needs. The Justice Department’s Office of the Inspector General and Congress both asked about the FBI’s in-house technical talent, but neither prioritized it nearly as heavily as the traditional metric of adherence to project plans. For any large-scale and critical technology development effort, the presence of federally employed technical talent — ideally situated in cross-functional teams — is a necessary ingredient for success.

During the Sentinel hearings, Congress, after requesting the work histories of those in charge of the program, discovered that the program lead at the time had a background not as a technologist but as an accountant, but didn’t pursue the topic or ask whether that was suitable for a complex IT effort. Congress and other oversight groups should insist on being informed of the expertise of federal staff and assess their technical experience.

Congress can support this in the appropriations process as well by providing funding for technical talent, encouraging and enabling the use of flexible hiring and recruitment solutions, or even withholding funds until project leadership roles are filled with technical talent. Agency leadership and inspectors general can immediately use their oversight powers to flag programs that lack cross-functional and federally-employed technical teams, treating a deficit of in-house talent as an immediate risk.

Additionally, governance and oversight should provide information about the ability for the program office to grow and retain the technical capacity necessary for the program’s success, such as recruitment of technical talent, time-to-hire for technical talent, staffing plans that include technical talent needs, and funding and authorities to hire necessary talent.

Track the underlying problem

Second, oversight and governance processes should closely track the underlying problem that the program is seeking to address. Every technology program is meant to solve a problem, such as increasing efficiency, reducing wait times, connecting disparate data, delivering a public benefit, or enabling analytical insights. Rather than focus on strict roadmaps and timelines of what specific software features would be available when, oversight focus should be on metrics and measurements that indicate whether the underlying problem is seeing improvements.

In the case of VCF and Sentinel, the underlying problem was the inability for bureau staff to coordinate easily and rapidly on cases spread across different offices. This could be tracked via metrics and measurements on the speed with which cases were processed, the amount of work each agent could take on, the number of staff dedicated to manual paperwork processes, or speed of connecting different points of information.

Ask to see the product itself

Lastly, Congress and other oversight bodies should ask to see the working product themselves at every opportunity. When the FBI took over production of Sentinel, its small tech team was able to develop working software rapidly and iterated on improvements every two weeks. In that same cadence, the Sentinel team provided demonstrations of their work product to stakeholders and executives across the bureau and the Justice Department. There’s no reason why Congress itself, let alone other oversight bodies, shouldn’t also see working products rapidly.

To enforce this, Congress can include provisions attached to appropriations requiring a live demonstration of a working version of the product within the first three months of its development. When SAIC, the contractor on the Virtual Case File, was developing the program, it didn’t hand the FBI a working version of the product for years — an obvious red flag. If a program office is unable to provide a working and live demonstration within the first few months of development, either extraneous blockers are preventing them from doing so — a contract stuck in protests, for instance — or, more likely, the program is structured incorrectly and needs to be adjusted.

Final thoughts

A personal note from the author: This case study of the FBI fascinated me for many reasons. Chief among which were the amount of money spent on failed technologies, the number of years spent repeating the same mistakes, and the underdog story of the in-house Lavender team that finally drove the project forward.

But it’s also a compelling story because of its timing and context. The FBI adopted agile practices and the Product Operating Model in 2010 — three years before the failure of Healthcare.gov, when the federal government’s technical incompetence made for worldwide news. If our government could get it right in 2010, then we can certainly get it right today. The public sector especially needs to move away from dated project management practices that pose real peril to the public interest, and oversight practices such as change control boards, near-infinite lists of requirements, documentation on documentation, excessively large teams, and multiyear upfront rigid plans.

We know what works, and we’ve seen it done it before. We can do it again. 

Appendices

List of oversight reports from GAO and DOJ OIG covering Trilogy/VCF and Sentinel

List of relevant Congressional hearings

Table of Department of Justice Office of Inspector General reports, by year, and which factors they surfaced in relation to the development of Sentinel.

Examples of lines of questioning from Members of Congress to the FBI that reinforced the faulty project-oriented operations.

Examples of lines of questioning from Members of Congress to the FBI that adopted the modern product operating model.