In the early to mid-twentieth century American security required information about the capabilities, intentions, and actions of closed societies. For such adversaries the information that is most valuable, such as strategic intent, sources, and methods, is almost always the most closely guarded. So the “very best of men” who created the U.S. intelligence community from the 1930s to the 1950s placed, quite correctly, a strong emphasis on obtaining and analyzing highly protected information.
This approach worked well in the United States’ competition with closed societies like Nazi Germany and Soviet Russia. Furthermore, despite the myriad revolutions of the early 1990s, a number of closed societies remained and others have resurged. Some of these retain or are developing sufficient power to threaten American interests. In approaching these societies, the current organization remains effective. However, the predominance of competitors with closes societies in the area of overall American security faded along with the appeal of their models of government. Today’s range of threat is much broader and requires a much broader approach to intelligence gathering and analysis.
The National Security Act of 1947 that organized the U.S. intelligence community along its present lines turns 60 this year. In human terms, it is old enough to receive Social Security checks. The time has come to make plans for its honorable retirement and to have a serious discussion about what its successor Act should look like.
The analytical side of the U.S. intelligence community as we know it began as a group of academic experts working from carrels, sorting through the open source material in the Library of Congress. Over time these scholars largely opted out due to objection over the way the intelligence community conducted its business or were pushed out by the need for security clearances. The result was a divide between many of the experts who could do the work of open source analysis and the community whose need for it has mushroomed.
The information age pushes open source material into unprecedented prominence. Those who would threaten the United States leave an ever-larger open source footprint – personal histories, manifestos, internal and external communications, and their entire online presence are increasingly available. The consensus among intelligence professionals is that open source material now comprises the overwhelming majority of the data needed for analysis, and that volume of material available is growing at an unprecedented rate. Enormous amounts of information are easily obtained, but proving difficult to digest. Turning this information into solid intelligence requires a community structured around the concept of high data volume, rather than a few choice pieces of difficult-to-obtain data.
Penetrative capability will always be an important aspect of intelligence. Highly guarded secrets tend to be highly valuable. It is an indispensable area of intelligence gathering, but it is also insufficient and increasingly so in the intelligence environment of the twenty-first century. The intelligence community must restructure to reflect the realities of today’s intelligence challenges and prepare for those of the future. This means recognizing open source intelligence (OSINT) as the preponderant source of data and embracing those technologies and process that are most efficient in dealing with it.
Unfortunately, the approach taken by the intelligence community toward information technology is reminiscent of import substitution industrialization. According to this economic policy, governments attempt to erect domestic industries that produce what it imports from foreign competitors. The result is an expensive project that creates plenty of jobs, but produces inferior or obsolete products at a much higher expense.
Private sector needs drive the creation of new information technology, while intelligence organizations attempt to replicate existing products internally or purchase and adapt their limited versions. Watching a demonstration of Intelink and Intellipedia after using software like Silobreaker gives one the sense of watching a demonstration of V-2 rocket technology after a shuttle launch. This intelligence community’s approach to information technology is foolhardy, for it ensures the private sector stays one or two generations ahead, and also that the development of new information technology is driven almost entirely by the needs of the open market. The result is that the intelligence community creates a costly, obsolete, and inferior information technology product and becomes an ever-smaller and less sophisticated consumer. It is time to abandon this effort.
The next National Security Act should not create organizations that replicate, compete with, or substitute for the private sector, which has been shown to be far more capable of collecting and analyzing vast amounts of open source data. It should instead make collaboration the primary model for OSINT. It is interesting that few have noted the similarities between the private sector and media’s contributions to open source intelligence and the open source software movement. The private sector can and will do much of the work of OSINT for its own purposes. Unable to fight or control this wave, the intelligence community has little choice but to ride it. Collaborating with the private sector for those parts of open source intelligence gathering and analysis that can be done securely and efficiently is the central part of any model of future intelligence competitiveness.