Developed in 1994 for the Naval Surface Warfare Center, SHADOW IDS remains a popular open source intrusion detection solution for enterprises and ISPs alike.

by Jeff Goldman [May 30 , 2002]

Frederick J. Kerby is the Information Systems Security Manager for the Naval Surface Warfare Center, Dahlgren Division. In 1994, he recalls, programmer Stephen Northcutt observed that while the Department of Defense was supposed to track all contact with foreign nationals, Internet connections weren't being monitored in the same way that postal mail and telephone calls were.

Northcutt developed a software solution initially named the Cooperative Intrusion Detection Evaluation and Response (CIDER) project, which was eventually redubbed SHADOW, short for Secondary Heuristic Analysis for Defensive Online Warfare. "Steve figured out what we needed to have from a technical perspective—he came up with it, and it's continued to evolve from there," Kerby said.

SHADOW 17320 Dahlgren Road Dahlgren, Virginia 22448 E-mail: shadow@nswc.navy.mil

At the time, while a few commercial products were already available from Internet Security Systems, Inc. and others, the software was still very rudimentary. "You've got to remember, this was back in 1994," Kerby said. "Firewall was not a word in most people's vocabulary at that point and intrusion detection really wasn't a mature technology."

The software itself was based on freeware, so it was relatively logical to release it to the public. "We're using tcpdump, which is the packet-sniffing capability that's built on libpcap," Kerby said. "We're also using OpenSSH, Apache Web Server, and the academic version of Tripwire. So we bundled a lot of freely available code, used that with some scripts that we'd written, and made those available."

Network neighborhood watch
Kerby says what made SHADOW unique when it was first released was the way it looked at traffic. "It's based on the idea of traffic analysis, which is to say, if I can stand outside for a month and just see the size of packages that are coming in and where they're coming from, without actually looking at the content, I can tell a lot about what's normal behavior in your neighborhood," he said.

That means that SHADOW looks for probes preceding an attack rather than an outright attack by network intruders (left). "We noticed that hackers tended to case the joint before they tried to break in—you would see a probe before they mounted the attack," Kerby said. "So we could give advance warning, whereas most intrusion detection systems at that point were based on a signature or an indication that something was happening in real time."

The problem with real time solutions, Kerby says, is that the warning usually comes too late. "What happens with most of the signature-based intrusion detection systems is that you get the radar detector going off right after you see the flashing blue light in your rear view mirror," Kerby said. "It's real time, meaning that you know about it right now."

Traffic analysis also allows SHADOW to handle encryption in various forms. "If you've got a secure Web server, or a VPN, or you're encrypting e-mail with either PGP or a hierarchical PKI implementation, SHADOW's going to be largely unaffected by that, simply because we're not looking at the content in the packet itself," Kerby said.

Shadowy view
In the years since SHADOW was first developed, a number of additions have been made to the software, including a statistics page to provide an overview of activity. "It's the commanding officer or executive director's view," Kerby said. "When I click on the statistics page for today, I can see a summary of all the services or protocols we used, the number of packets, and bytes of information transferred."

The software can also show which machines at a site are responsible for the bulk of the traffic (right). If an unexpected PC appears in that list, it's easy to take action. "You can click on that link and see the addresses of all the sites that PC connected with yesterday," Kerby said. "We also see the top machines on the outside, on the Internet, that we're connecting to, which gives us a really interesting view of the world."

Kerby admits, however, that there are some ways in which SHADOW is a little behind the times. "Most of the stuff is text-based," he said. "The statistics page, right now, is all numbers. One of the things we're working on is a graphical display: we would show the traffic in terms of the number of packets, or megabytes or kilobytes of data, as a bar graph."

Though he has no way of knowing how many people are actually using SHADOW, Kerby says the site is currently registering 400 downloads a month. There's no fixed schedule for updates, but they're usually released about once a year-and users regularly contribute comments and suggestions to the project by dropping a note to shadow@nswc.navy.mil.

Human engineering
Whether you use SHADOW, another open source solution like Snort, or a commercial IDS product, Kerby stresses the fact that the most important element to keep in mind is the person sitting in front of the machine, not the machine itself. "All intrusion detection systems have a common trait, and that is that you have to have a knowledgeable individual sitting there using it," he said.

"Anyone who's running an intrusion detection system needs to know about services, ports, and protocols, what's typical or normal behavior, and how to spot something out of the ordinary," Kerby said. "Just being able to throw a CD in a drive and click on 'Next' several times to do an install won't make you capable of running an intrusion detection system, whether it's SHADOW, Snort, RealSecure, or something else."

The software itself is free, which means that aside from hardware, your only costs will come from acquiring the personnel to SHADOW your network. Kerby's dream is to eliminate that last requirement. "The big challenge is to take somebody that knows nothing about security, networks or intrusion detection, put them in front of a box with a mouse and turn them into an expert," he said. "That's a problem everybody has."

In the meantime, though, Kerby's very happy with SHADOW as it currently stands. "It's a great product," he said. "We use it quite a bit here—every day, as a matter of fact—and it's a key piece of our information assurance program."

— End

Online Resources:
   Intrusion Detection Systems Directory
   IDS Quick Reference Chart

Related articles:
	[Dec. 24, 2001]	White Paper: Reducing Network Security Risk
	[Sept. 25, 2001]	Physical Security Augments Logical Security
	[July 11, 2001]	ISP-Planet Survey: MSSPs