TSA Lessons for Security Analysts

In the past I've run several security teams, such as the Air Force CERT's detection crew and the MSSP division of a publicly traded company. In those positions I was always interested in assessing the performance of my security analysts. The CNN article TSA tester slips mock bomb past airport security contains several lessons which apply to this domain.

Jason, a covert tester for the Transportation Security Administration, has been probing airport weaknesses for five years, beginning with big mock bombs before switching to ever smaller devices as the TSA adapts to evolving terrorist threats...

Even before the September 11, 2001, terror attacks, government agencies deployed "red teams" such as this one to look for holes in airport security...

But instead of running from tests, the agency has embraced the idea that testing has a value that goes beyond measuring the performance of individual screeners.

Tests, the TSA says, can show systemwide security vulnerabilities...

[S]creeners who fail to detect contraband are "pulled off the line" and retrained before being allowed back.

The test CNN witnessed was conducted by the TSA's Office of Inspection, which the agency calls the most sophisticated of its covert tests. But there are others.

For starters, every TSA X-ray machine has a Threat Image Projection system, which digitally inserts images of guns, knives and bombs into the X-rays of luggage, to keep screeners alert...

If screeners observe a suspicious object, they can check with the simple click of a computer mouse. If they detect a threat object, the computer congratulates them. Successes and failures are recorded for use in a screener's performance evaluation and are factors in determining pay.

Some 69,929 threat image tests are conducted on an average day, or more than 25 million tests per year. An array of other tests also are conducted to assess screeners, including the red team ones.

I've described elsewhere why I support red teams. I certainly recognize that one of my Three Wise Men savages red teams, but I've never seen anything else -- short of an actual incident -- make a dent in the attitudes of management. Furthermore, red teaming, as a real-life test, tends to discover and link vulnerabilities in ways not anticipated by some vulnerability assessors (blue teams) and general security architects. There's no ground truth like saying "I accomplished the mission using this method" when someone is claiming their network is "secure."

I also like the method to test analysts by inserting false images. Fighting analyst boredom is a big problem in some operational teams.