The Value of Branding and Simplicity to Certifications

At the risk of stirring the cyber pot (item 3, specifically) I wanted to post a response to a great mailing list thread I've been following. A reader asked about the value of the CISSP certification. Within the context of the mailing list, several responders cited their thoughts on SANS certifications. Many mentioned why the CISSP tends to be so popular. I'd like to share my thoughts here.

In my opinion, the primary reason the CISSP is so successful is that it is easy to understand it, which facilitates marketing it. It is exceptionally easy for a recruiter to search LinkedIn profiles, other databases, or resumes for the term "CISSP." If you encounter a person with the CISSP, you basically know what the person had to do to get the certification.

Before continuing, answer this quick question: what are the following? 1) SSCP, 2) CAP, 3) CSSLP?

Let me guess -- you didn't recognize any of them, just like I did?

Now, let me see if you recognize any of the following? 1) GGSC-0400, 2) GNET, 3) GAWN-C, 4) GBLC, 5) GCIM?

I believe you didn't recognize any of those either.

How about? 1) GISP, 2) GLEG, 3) GCIH, 4) GAWN?

I'm guessing some of you might recognize GCIH as the SANS "GIAC Certified Incident Handler," which actually doesn't have much to do with "incident handling." That's a topic for another day, but it does show GCIH benefits from decent branding.

You've probably figured out that the last two lists of acronyms were SANS certifications. The first list was a selection of a few of the retired SANS certifications. There's 26 of those.

The second list was a selection from the list of 24 active SANS certifications.

What about the first list, starting with "SSCP?" Those are other certifications offered by ISC2. They're utterly forgettable. Had I not visited the ISC2 Web site, I would never have known they existed.

Now, one could argue that the brand "SANS" is as recognizable, or even more recognizable, than the brand "CISSP."

The problem is that a person's resume could list "SANS" as a course he or she attended, without noting if a certain achievement (i.e., certification) was achieved. "SANS" is also a poor search term because the diversity of the SANS ecosystem means you could be dealing with a legal person, or a reverse engineer, or a UNIX system administrator.

What is the answer for SANS, if the CISSP will likely continue to out-market it? I recommend adopting the model used by Cisco. If you hear a person has a CCIE, that means something -- you immediately think of deep knowledge, several levels of work, and grueling hands-on testing over two days in a controlled environment.

The genius of Cisco's approach is that they have "tracks" for the CCIE, e.g. Data Center, Routing and Switching, etc. Those aren't the brands though; that stays with CCIE.

The Cisco approach isn't perfect, because you can't simply search resumes for "CCIE" intending to get a CCIE in security. You might find a CCIE in routing and switching, or wireless. However, if one finds a CCIE, you get a sense of the level of seniority and ability to operate in a stressful environment (at least as far as a test can simulate).

SANS has tried something like the CCIE with their "GIAC Security Expert (GSE)." The GSE is similar to the CCIE in many respects, including horribly tough hands-on labs, but unfortunately hardly anyone knows about it. It is really difficult to reach that level in SANS certification. However, because only 63 people hold it, there's no real market for them.

By the way, I smell a branding failure when SANS certifications like GSE, GCIH, and so on all have a "G," which references another acronym -- "GIAC," for "Global Information Assurance Certification." That doesn't even include the term "SANS," which is the stronger brand. GIAC originally meant "Global Incident Analysis Center," but that's another story.

In brief, I think SANS could increase the branding value of their certifications if they retired the existing acronyms and names, incorporated "SANS" into a new naming scheme, and concentrated on a "level" approach seen with Cisco. Focus on Entry-Level, Associate, Professional, and Expert as Cisco does, and develop programs to accelerate the adoption of the Expert level among its constituency as Cisco did with CCIEs.

Rebranding would cause lots of SANS folk plenty of heartache, but I think integrating "SANS" into the new level-oriented structure would more than compensate for the initial transition costs. Ultimately the system would be stronger for everyone.

What do you think?

Tweet

Read More

Cisco's Secure Routers Are Winning

Infonetics Research published a synopsis of a recent report they authored. I'd like to highlight a few points.

Growth in the network security appliance and software market is expected to slow to the single-digits after 2007 as content security gateways and NAC products begin to infringe on network security product budgets.

“The most important appliance category to watch over the next year is secure routers. Sales were up 25% in 2006 and this year will pass $1 billion in worldwide sales, representing a significant portion of the overall network security market..."

Secure routers account for 29% of the total integrated security appliance market in 2006 and will continue to increase their share of the market through at least 2010...

Cisco continues to lead the overall network security market, with 38% worldwide revenue share in 2006, posting growth in all network security market segments tracked by Infonetics

Juniper and Check Point are tied for second, each with 9% worldwide revenue in 2006...

Infonetics’ network security report provides worldwide and regional market size and forecasts and worldwide market share for integrated security appliances in 6 price categories, secure routers, SSL VPN gateways, VPN and firewall software, and host- and network-based IDS/IPS products. Companies tracked include 3Com/TippingPoint, AEP, Alcatel-Lucent, Array, Aventail, Check Point, Cisco, Citrix, CA, D-Link, Enterasys, F5, Fortinet, Juniper, McAfee, NETASQ, Nokia, Nortel, Secure Computing, SonicWALL, Symantec, WatchGuard, ZyXEL, and others.

I wrote last year that all network security functions will end up in the switch. This Infonetics story is talking about "secure routers," which I assume are devices like Cisco's Integrated Services Routers. That idea is consistent with my vision for the "security switch."

The revenue share numbers are interesting; Cisco dominates, and when you add in Juniper (where the wheels are apparently coming off) you've got 56% of the market accounted for.

It's important for technical folks to understand that the people with budgets who ultimately procure equipment are not looking for the best technical solution. They are looking for something that is good enough. This is called satisficing. Decision-makers want to spend the least amount of money necessary to get them to a level they believe is as good as their peer group. Given these conditions, buying a Cisco ISR that advertises "Routing, Security and VPN, Voice, Wireless [and] Optimization of network bandwidth and applications" makes perfect sense.

If you're a vendor that makes a network traffic inspection and/or manipulation product, you're going to either end up on someone's switch or be a niche player. I see two functions that will not end up on the switch:

1. Network forensic appliances: ISRs are not going to incorporate a multi-TB storage array and the associated analysis software required for network forensics.


2. SIM/SEM/SIEM/Network Security Management suites: Just because all of your gear will end up belonging to Cisco doesn't mean you'll want to use something like Cisco MARS to make sense of it. It's likely you'll buy into the Self-Defeating Network but I see room for vendors like Q1 Labs to survive and maybe thrive.

Please note I am not talking about all security here. I am talking about devices that inspect and act upon network traffic.

Read More

Earth to MARS

Disclaimer: I'm going to single out a book by Cisco employees that talks about a Cisco product. I have no personal feelings about Cisco. I have friends there. I've done work for Cisco. Since I think Cisco is eventually going to own all network security functions in their switches, I may even work for Cisco one day.

This post is for all product vendors who approach understanding and defending the network in the ways described here. Wherever you read "Cisco" feel free to add products that share the characteristics I outline below.

Once again I found myself hanging in the sky last week. Trips to and from the West Coast gave me the opportunity to read Security Threat Mitigation and Response: Understanding Cisco Security MARS by Dale Tesch and Greg Abelar. This is mainly another Cisco marketing book, like Self-Defending Networks: The Next Generation of Network Security by Duane DeCapite. While I have a few thoughts on the book, I would much rather address the underlying philosophy presented by the authors. I'm fairly sure they're only repeating Cisco market-speak, but I hear the same message from many vendors, consultants, and individuals.

In this post I'd like to take issue with that message. In short, almost nothing about the approach taken by Cisco MARS (and similar products) is new or will "solve" your security problems. Unless you augment tactics and technologies like MARS, you will find yourself wondering why you spent time and effort to end up as frustrated and confused as you were pre-MARS.

I'll refer to this book as STM, short for the Security Threat Mitigation term in the title. STM is supposed to be "beyond Security Information Management" (SIM). According to STM, the five core SIM functions (pp 6-7) are:

• Collect event data


• Store data


• Correlate to show relationships (what the book calls "true power")


• Present data


• Report on, alarm on, and/or notify about data

STM starts to knock the value of SIMs by introducing the idea of "garbage in, garbage out," saying "information or events from several different sources can be 'garbage' unless they are put together in a useful way." I disagree -- garbage in always produces garbage out. The idea that a ton of garbage can be turned into something valuable (like gold from coal) is the fallacy of SIM/SEM technology.

The book tries to position "STM" as "beyond SIM" by granting STM the following attributes:

• Data reduction


• Timely attack mitigation


• End-to-end network awareness


• Integrated vulnerability assessment


• Session correlation

You are probably think what I am thinking; how is that really different from a SIM/SEM? I bet all the SIM/SEM vendors are saying "we do that already."

Things start to get really weird when the authors talk about the "advantages of a proactive security framework" (pp 14-15) compared to what everyone else must be using. STM says:

The key to this framework is the network's capability to behave actively. This does not necessarily mean to take action itself, but to automatically collect data from numerous sources and come to a decision..." (emphasis added)

... so that a person can react like we've always done. It is intellectual dishonesty to claim this product (or any other) is acting "actively" when the end result is still waiting for a person to react. Mind you, I'm not knocking reaction. Too many people seems to think proactivity is king and reactivity is evil, but sometimes there's no other option. It bugs me that Cisco is packaging manual "shunning" in a new wrapping and calling it "self-defending" and "active" when the end result is a person having to make a reactive decision.

The real problem is far more insidious, however. As has always been the case with alert-centric products, there is not enough data available to make a decision. In other words, Cisco MARS, like other products, still cannot tell me if an attack was successful.

More absurdity appears in the section (p 15) on "false-positive reduction," where three methods are given for MARS to "determine the validity of event data." They are, basically:

1. Network topology


2. Vulnerability assessment via limited network scanning


3. User determination

So, we have 1) could the attack reach the target? Guess what -- if it's a TCP connection it probably did. Next we have 2) scan to see if the IIS attack hit an Apache server; limited at best, worthless at other times. Finally -- and this kills me -- 3) let the user decide. How is this better than anything else again?

The section "Enhancing the Self-Defending Network" adds insult to injury by naming three "missing links" in the SDN addressed by STM:

1. Automated log correlation


2. Automated threat response


3. Automated mitigration

We know SIM/SEM does #1. A product that accomplished 2 and 3 would be impressive, but guess again -- MARS does neither automatically:

Automated mitigation is not yet achievable because many security devices still put out false-positive alerts, but CS-MARS makes a recommendation for mitigation and offers security responders a single click to deploy commands on devices that will stop offending traffic after the responder has analyzed the attack data. (emphasis added)

Again, it all comes back to having the data necessary to make a decision, and then letting an analyst make the decision. There's nothing proactive or special here.

STM emphasizes the speed with which one can respond, but how can an analyst even know to start the escalation process? The end of chapter 3 presents a case that aims to show "ROI" for MARS based on shutting down a spreading virus faster than a non-MARS solution. Here's the core of the problem, however:

The virus starts trying to infect other hosts... [MARS sends] an email page... to a security responder 24 hours a day, 7 days a week... The security responder gets the page and logs into CS-MARS.

And that's faster or better how?

STM demonstrates real ignorance of the sorts of data an analyst needs to make decisions regarding security events. On MARS, "forensic analysis" is "visual tools for attack-path analysis and attack propagation" (p 82) or mapping NAT'd IPs to public IPs or IPs to MAC addresses. "Attack validation" (i.e., should I worry?) is knowing "if an attack actually reached the intended destination." Furthermore, p 89 says:

[U]pon detecting an anomalous behavior, CS-MARS starts to dynamically store the full NetFlow records for the anomalous activity. This intelligent collection system provides all the information that a security analyst needs. (emphasis added)

Wrong. What else does MARS offer? Page 127 says "retrieval of raw messages" is available as a download of a zipped text file! I bet that's easy to manipulate when trying to understand alerts. On pp 229-230 we see examples of the sort of "incident ID" data available to analysts -- almost all of which is unactionable and worthless. On MARS, the analysis process consists of looking at a rule MARS used to generate an alert and then guessing at the relationship between the rule and events listed in the "detailed data" section.

I think the real problem with this approach is demonstrated by the case studies in chapter 9, which I guess Cisco sees as a validation of their approach. We read about a Trojan found at a state agency via an IDS alert (no different from non-MARS). We see a .edu respond using MARS because phone calls prompted an investigation of traffic on port 25! We read about a hospital that used a spike in traffic to a Web server as a reason to investigate the Web server (wow). A financial firm investigates high ICMP traffic, and a small business sees odd DNS names in its weekly MARS reports to find another company using its wireless access point. Honestly, is this supposed to make me a believer?

If you want to know how I recommend dealing with this and similar situations, I've already written about it. In brief, analyst must have high-fidelity, original, content-neutral data sources to investigate. That is the key term here. If you do not have such data (as provided by NSM) to investigate, you are doing alert management. And you will lose.

So what is MARS (and similar products) good for? In short, there is value in centralizing and presenting events from disparate security products. There is value in having a single window into that world, with some sort of accountability, escalation, and case management. There is value in being able to contain network-centric exploitation via centralized device control. Just remember that making the decisions to protect the enterprise requires having the proper data. NSM is one way to get that data.

Read More

Monitor Your Routers

Today I read this new Cisco advisory containing these words:

Cisco routers and switches running Cisco IOS® or Cisco IOS XR software may be vulnerable to a remotely exploitable crafted IP option Denial of Service (DoS) attack. Exploitation of the vulnerability may potentially allow for arbitrary code execution. The vulnerability may be exploited after processing an Internet Control Message Protocol (ICMP) packet, Protocol Independent Multicast version 2 (PIMv2) packet, Pragmatic General Multicast (PGM) packet, or URL Rendezvous Directory (URD) packet containing a specific crafted IP option in the packet's IP header...

A crafted packet addressed directly to a vulnerable device running Cisco IOS software may result in the device reloading or may allow execution of arbitrary code.

This is the sort of "magic packet" that's an attacker's silver bullet. Send an ICMP echo with the right IP option to a router interface and whammo -- you could 0wn the router. Who would notice? Most people consider their border router an appliance that connects to an ISP, and only begin watching traffic behind the router. I've been worried about this scenario since I posted Ethernet to your ISP in 2005 and read Hacking Exposed: Cisco Networks in 2006.

A router, like a printer, is just a computer in a different box. We should design architectures such that all nodes within our control can be independently and passively observed by a trusted platform. We should not rely on a potential target to reliably report its security status without some form of external validation, such as traffic collected and analyzed by network security monitoring processes. This Cisco advisory reminds me of a case I encountered at Foundstone. Shortly after joining the company in early 2002 the Apache chunked encoding vulnerability became known in certain circles. As the new forensics guy on the incident response team, I was asked how I could tell if Foundstone's Apache Web server was hacked. I replied that I would first check session data collected by a trusted sensor looking for unusual patterns to and from the Web server, as far back as the data and my time would allow.

Foundstone, being an assessment company, didn't share my opinions on NSM. No such data was available, so the host integrity assessment descended into the untrustworthy and often fruitless attempt to find artifacts of compromise on the Web server itself. Of course the Web server could not be taken down for forensic imaging, so we performed a live response and looked for anything unusual.

This case was one of the easier ones in the sense that all that was at stake was a single server. We didn't have a farm of Web servers, any of which could have been hacked.

Whenever I am trying to scope an incident I always turn to network data first and host data second. The network data helps decide which hosts merit additional individual scrutiny at the host-centric level. It makes no sense and is not time-efficient to deeply or even quickly analyze a decent-sized group of potential victims.

Because most security shops do not bother to collect NSM data, their first response activities usually involve a system administrator perusing the victim file system, manually checking potentially altered or erased logs on the target, and inspecting potentially obfuscated processes in memory controlled by a root kit. While I believe it can be important to collect live response data and even image and inspect a hard drive, I never start an incident response with those activities.

Remember: network first. I don't care if the intruder uses an encrypted channel. As long as he has utilized the network in a mildly suspicious manner, I will have some validation that the system is not behaving as expected. This presumes system owners know what is normal, preferably by comparing a history of network data against more recent, potentially suspect, activity.

---

Copyright 2007 Richard Bejtlich

Read More

Thoughts on Cisco Self-Defending Network Book

I didn't exactly "read" Self-Defending Networks: The Next Generation of Network Security by Duane DeCapite. Therefore, I won't review the book at Amazon.com. I definitely didn't read a majority of the text, which is a personal requirement for a book review. However, I'd like to discuss the title here.

The book has a ton of screen shots and is essentially a big marketing piece for Cisco's Self-Defending Network gear, which includes:

• Cisco Traffic Anomaly Detector for DDoS identification


• Cisco Guard for DDoS mitigation


• Adaptive Security Appliance for firewalling (including IPS)


• Incident Control System for malware containment with Trend Micro


• 802.1X for port-based security; note to Cisco: it's not "802.1x"


• Network Admission Control (NAC) with NAC Appliance or NAC Framework


• Cisco Security Agent (CSA) for host protection


• Cisco Security Manager


• Cisco Monitoring, Analysis and Response System (MARS) for alert management

Why do I mention this, especially with product listings? Well, I realized the Self-Defending Network (SDN) is a security integrator's dream. I'm working with a client who has sold essentially this entire setup to a customer, and they want me to help get the most value from the deployment. I'm also going to assist with incident response planning.

The point is a security integrator can pitch this entire SDN suite as a coherent, one-brand "solution," and cover pretty much all the bases. That's impressive and I'm interested in knowing what sort of traction Cisco is getting with this approach. My sense is that it will sell well to non-technology companies who are really late in the security game. Yes, there are many companies who have no real protection, even in 2007. I severely doubt the readers of this blog are in that category, but what are you seeing?

Read More

Notes from Cisco TV on CCNP

I'm listening to the first episode of CCNP TV. It's not really TV, more like slides plus audio. I'm listening because I wanted to know about the CCNP changes mentioned earlier. The new material will all use Cisco IOS 12.4. I found this change interesting: assume I earn my CCNP; in the future, I can recertify using any Professional level exam, like one from CCSP.

Mixing and matching of old and new exams will be allowed within the guidelines explained in these two press releases.

Parts of this webcast were hilarious. Despite Cisco's drive into voice, none of the callers could communicate with the hosts. The caller who got closest to speaking should have been told "turn your radio down!" because the broadcast was on a 30 second-or-so delay. The host also asked two questions twice each, and the other panel members pretended like it didn't happen. I heard the terms "billion dollar marketplace" and "revenue opportunity" repeated several times. This was probably the funniest exchange:

Host (paraphrasing): "So what's the difference between the old and the new?"

Bob: "These new networks go up to layer 7."

Host: "Oooohhh."

If you have questions the best place to look for answers is www.cisco.com/go/certifications or www.cisco.com/go/certsupport. For specifics on preparing for CCNP, try www.cisco.com/go/prep-ccnp.

Read More

CCNP Changes

If you're interested in earning the Cisco Certified Network Professional (CCNP) certification, you should read about changes in the cert. This TCPmag.com article is a nice summary. Basically Cisco removed some old material and added a lot more on security, wireless, and other modern topics. I would like to test for the CCNP before my CCNA expires in March 2008, but I'll wait until updated study books are available.

Read More

Brief Thoughts on Cisco AON

I received my copy of Cisco's Packet Magazine, Fourth Quarter 2005 recently. The new digital format for the magazine makes linking to anything impossible, but I found the relevant article as a .pdf.

It describes the company's Application-Oriented Networking (AON) initiative. According to this story that quotes Cisco personnel, AON "is a network-embedded intelligent message routing system that integrates application message-level communication, visibility, and security into the fabric of the network." According to this document:

Cisco AON is currently available in two products that integrate into Cisco switches and routers:

• Cisco Catalyst® 6500 Series AON module, which is primarily deployed in enterprise core or data centers


• Cisco 2600/2800/3700/3800 series AON module, which is primarily deployed at branch offices

AON is part of Cisco's Intelligent Information Network project. From the article:

"The Cisco AON module in the branch puts intelligent decision-making at the network edge. It can intercept and analyze traffic in various message formats and protocols and bridge between them, provide security, and validate messages, creating a transparent interface between trading partners and, in effect, a good business-to-business gateway. It can manage remote devices that send messages to the Cisco Integrated Services Router in the branch. It can also filter messages from multiple sources that come into the branch router for duplicates or by other criteria, aggregate them, make decisions according to instructions, and transmit selected messages to a sister AON module deployed in the data center." (emphasis added)

I find this aspect very interesting. It sounds like AON could be used to enforce protocol and security policies. I wonder if this might eventually happen on a per-port basis? Security on a per-port basis would allow validation of network traffic itself, not just whether a host should be accessing the network. Per-port security would move the job of enforcing security away from choke-point products like firewalls (which include IPSs, application firewalls, whatever) and into switches.

This is not necessarily a great idea, as this Register article confirms. One of the strengths of the Internet has been the fact that it inverted the telecom model, where the network was smart and the end device (the phone) was dumb. The traditional Internet featured a relatively dumb network whose main job was to get traffic from point A to point B. The intelligence was found in those end points. This Internet model simplified troubleshooting and allowed a plethora of protocols to be carried from point A to point B.

With so-called "intelligent networking," points A and B have to be sure that the network will transmit their conversation, and not block, modify, or otherwise interfere with that exchange to the detriment of the end hosts. As a security person I am obviously in favor of efforts to enforce security policies, but I am not in favor of adding another layer of complexity on top of existing infrastructures if it can be avoided.

Read More

Measuring Bandwidth Utilization on Cisco Switch Ports

Yesterday I spoke at the third Net Optics Think Tank in Santa Clara, CA. During the event one of the Net Optics product managers asked me about measuring bandwidth utilization on switch ports. I did not have an answer for him... until I took a look at the latest Packet magazine. The Q305 (.pdf) edition features a tip from Aurelio DeSimone on p. 13 mentioning the show controllers utilization command.

If anyone knows of a similar set of information via SNMP, please let me know via a comment here.

Here is sample output:

Switch> show controllers utilization
Port       Receive Utilization  Transmit Utilization
Fa0/1              0                    0
Fa0/2              0                    0
...truncated...
Total Ports : 12
Switch Receive Bandwidth Percentage Utilization  : 0
Switch Transmit Bandwidth Percentage Utilization : 0
Switch Fabric Percentage Utilization : 0

This is just the sort of data I would like to see for SPAN ports. You can specify the SPAN port in your syntax (e.g., show controllers fastethernet0/1 utilization) to see how much traffic it is carrying to your sensor.

The current Packet issue also features excellent articles on new modularity features of Cisco IOS and an overview of 10 GB Ethernet and its seven (yes, seven) variants. (There appear to be more, actually.) That sort of information reminds me of my "second law of information technology," which is "complexity increases." The second law of IT is constantly fighting the second law of thermodynamics, which is "entropy increases."

Read More

SecurityFocus SNMP Article

Thanks to Simon Howard for pointing me toward a new article by Mati Aharoni and William M. Hidalgo titled Cisco SNMP configuration attack with a GRE tunnel. The article shows the dangers of not denying packets from the Internet using spoofed internal addresses. The article builds on Mark Wolfgang's Exploiting Cisco Routers: Part 1, where an intruder uses an SNMP SET command to retrieve a router configuration file via TFTP. As Simon wrote in his email to me: "Applying an inbound ACL on the Ethernet0/0 interface denying any traffic from the 192.168.1.0 network would resolve this issue [in the article]."

On a related note, I am looking forward to the second edition of Essential SNMP, pictured at left.

Read More

Thoughts on Cisco Packet Magazine

I like to read Cisco's quarterly Packet magazine. It's free, and it provides insight into developments by the world's networking (and one day, security) juggernaut. While waiting for car maintenance this morning, I managed to read much of the Quarter 2 2005 issue, devoted to Self-Defending Networks.

According to Cisco, they have been releasing Self-Defending Network components every few years. In 2000 they offered integrated security, followed by collaborative security in 2003. Now, in 2005, we have adaptive threat defense. The first term means security is part of Cisco products, such as routers and switches. The second term means these products should work together. Let's look closer at the third term, which Cisco claims will "protect every packet and every packet flow on a network."

I was skeptical when I saw the cover text. The phrase "eliminating the source of attacks" and the sentence "network security grows adaptive, reaching inside Web applications and excising attacks at their source" also worried me. When I read words like that, I imagine Cisco police forces banging down doors of attacker apartments in Bucharest or Beijing. That's how one really "excises" a threat!

As I read more about Cisco's plans, however, I realized they refer to containing malicious systems that connect to protected networks. For example, one article described how the CS-MARS [Security Monitoring, Analysis and Response System], developed by the former Protego Networks, "will send the network administrator the appropriate command to execute an action to excise the problem from the network at its source." In other words, Cisco gear will help identify and disable misbehaving network assets (or rogue visitors).

I found this article on wireless defenses interesting. It describes Cisco's approach to handling rogue wireless access points:

"With rogue access point suppression, the sensors detect wireless-device information, aggregate it, and pass it up to elements in the network that can correlate it and act upon it. When a wireless access point is detected on the network, the WLAN intrusion prevention system sends RF management frames that disassociate any clients that connect to it and attempt to trace and shut down the switch port to which the rogue is connected."

That seems cool. Only a few years ago I remember Mike Schiffman demonstrating libradiate at Black Hat by disassociating wireless clients. Now he works at Cisco!

I also learned a little about virtual routing and forwarding, which acts like "multiple routers within a single chassis."

Overall, I found Cisco's magazine very useful. I also subscribe to the free IP Journal, which I recommend. Whenever I read articles by Cisco, it reminds me I am not currently a networking engineer. That would require far more protocol and algorithm knowledge than I currently possess!

Speaking of Cisco, I will be speaking on 10 October 2005 to the Cisco Fall 2005 System Engineering Security Virtual Team Meeting in San Jose, CA. I will probably discuss network security monitoring and give a Sguil demo.

Read More

Cisco Connection Online Compromised?

Thank you to Simon Howard for the tip. He and I were both unable to log into our Cisco Connection Online (CCO) accounts today. We received the warning above when authentication failed. The Register is now reporting Cisco portal password security compromised. The Cisco security page has this information:

"Cisco Systems, Inc. was made aware of a vulnerability of a search tool on Cisco.com that could expose passwords for registered users.

Registered users of Cisco.com consist of employees, customers, partners, and other third-party users.

Cisco has since researched this issue and has taken the necessary steps to correct it.

Cisco is taking precautionary measures to protect our registered Cisco.com users, including resetting registered user passwords."

I wonder if Cisco has suffered a SQL injection breach?

When I sent an email to have my account reset, I got this reply:

"This is an automated reply ONLY to have your CCO p/w changed.

DO NOT reply directly to this email!

Your CCO User ID is - [myuserID],[somethingthatlookslikeatemporarypassword]

Sorry, your attempt to change your p/w on CCO has not been successful.

Reason: Your account may not be active.

1Any further inquiries should be directed to cco-team@cisco.com"

I guessed that the email was wrong, so I used my user ID and the part that looked like a temporary password. I was able to log in and change my password from the Cisco supplied version.

Are we going to see more of this guerilla warfare against Cisco? I imagine so.

Update: Check out the video from Black Hat showing Mike Lynn's presentation being ripped from briefing books. All they need is a bonfire!

Read More

More Resources on Ciscogate

The media is now calling the Lynn affair "Ciscogate." I'm using this term now because it's so much easier to Google. Multiple new articles provide good reading, including Exploit writers team up to target Cisco routers by Rob Lemos and Router Flaw Is a Ticking Bomb by Kim Zetter. The latter is an interview with Mike Lynn that has several choice quotes:

"Cisco said, 'You guys are lying. It is impossible to execute shell code on Cisco IOS.' At that point (ISS) management was annoyed.... They were like, 'Mike, your new research project is Cisco IOS. Go find out how to exploit bugs on Cisco IOS so we can prove these people wrong...'

[Cisco] also sent out an engineer ... who described himself as an IOS architect.... I was told he helped design parts of the source code.... And his jaw hit the ground. He was very impressed, he was just (saying), 'Wow, that's cool.' That was June 14th.

WN [Wired News]: Cisco saw your Black Hat presentation long before they decided to pull it. When did they see it?

Lynn: Probably June 14th, the day that they came out (to Atlanta). We told them about the vulnerabilities well before (that)...

[ISS] actually wanted to distribute the full working exploit very widely inside the company.... I was told ... 'Give this to all the sales engineers and to all the pen testers...'

I told them, 'You do realize if you do that, it's going to leak?' And (one of the ISS guys) says, 'That's Cisco's problem.' And then (another ISS guy) turns to me and says that they need to understand this could be their Witty worm...

At that point, I told them all no, and they fought it and I resigned right there on the spot. And this was about a month ago...

[Wired News]: (ISS talked him out of the resignation by agreeing to give him control over who could see or have the exploit.)

(Then) it was two weeks ago, I was first told that Cisco might want to come onto (the) stage with me and say a couple words. And I said, provided the words aren't something to the effect that 'he's a liar,' I'm OK with it....

[Wired News]: (However, the plan changed even more and Lynn was told to remove any mention of reverse engineering from his talk or cancel the presentation. If he did neither, he would be fired.)

Mind you this is a complete reversal. Like a week or so prior, the night of the close of the fiscal quarter, and they were all celebrating that they hit the numbers, the [ISS] CEO invited me out for a beer, and he just couldn't say enough awesome things about this talk.

WN: Was Cisco threatening them?

Lynn: I asked point-blank, 'Are you being threatened by Cisco?' They said no.... To be perfectly honest, I don't think there was any legal threat. I think that it was more of a 'scratch our back and we'll scratch yours.'

(Cisco asked him to wait a year until it could release a new version of its operating system. When he didn't back down, Cisco threatened a lawsuit against Lynn and Black Hat. Then with Black Hat's cooperation, Cisco arranged to tear out pages with images of Lynn's slides from the conference book.)"

There you go. Now you see the operation of the respective "market leaders" in their fields: Cisco in networking and ISS in security. Notice how the CEOs of both companies (ISS' Noonan and Cisco's Chambers) are members of the National Infrastructure Advisory Council. This is how their companies practice "responsibility" in matters affecting national security. One company (ISS) is ready to distribute devastating exploit code to make the competition look bad, while the other embarks on futile attempts to hide that fact that its products are broken.

This disturbs me greatly. Why would anyone wonder why my first post was Free Michael Lynn?

At least Mike will be involved in improving the country's security. From the same Wired interview:

"Lynn: US-CERT (Computer Emergency Response Team) asked me if I would come up to D.C. in a week or two and help them formulate the nation's strategy for cybersecurity."

Presentations from Black Hat (besides Mike Lynn's) are now online. Check you favorite BitTorrent search engine for Mike's slides.

PS: Here's an article on debugging memory on a Cisco 1600.

Read More

New Cisco Advisory and Statements

I guess we can wrap up the Cisco and ISS vs. Mike Lynn and Black Hat saga by mentioning the new Cisco security advisory released today: IPv6 Crafted Packet Vulnerability, which states:

"(IOS®) Software is vulnerable to a Denial of Service (DoS) and potentially an arbitrary code execution attack from a specifically crafted IPv6 packet. The packet must be sent from a local network segment. Only devices that have been explicitly configured to process IPv6 traffic are affected. Upon successful exploitation, the device may reload or be open to further exploitation."

Assuming these details are correct -- and who knows now? -- this is not an earth-shattering discovery. However, this may have been a sample vulnerability Mike demonstrated to explain his technique. He may have picked this vulnerability because he thought it would not affect much of the Internet, but he needed to let people know that his technique was already in use by malicious parties.

Cisco's main security page addresses the Lynn affair directly as well.

This Reuters article quotes Jeff Moss:

" Jeff Moss, president of Black Hat, predicted the ruling would have a dampening effect on security enthusiasts.

People will say, Why would we tell the public about this if we're going to be sued? We're just going to post this anonymously,' he said. 'Who is going to tell Cisco about a problem now?'"

Who indeed. Good work, Cisco. You've just alienated anyone who would consider quietly approaching you with vulnerability details. You've probably also stirred up an army of independent researchers who will look for new holes in IOS.

The real tragedy is the vulnerability of all the enterprises running Cisco gear, to include all of my clients. It's time for me to figure out better ways to monitor Cisco equipment for signs of compromise. The protected domain or boundary does not start inside your border router -- it must now include that router, as it remains at risk of direct attack. How long before the first router-based worm, I wonder?

Read More

Cisco Releases IOS 12.4

I can't find any real press releases on this, but I noticed Cisco released IOS 12.4 this month. This is a Major Release that incorporates features developed in the IOS 12.3T line. Two product bulletins explain what's new in IOS 12.4, and an 84 slide .pdf presents similar information in PowerPoint format. I was surprised that the IOS Feature Navigator shows there is no support for the Secure Shell SSH Version 2 Server or Client for my Cisco 2651XM router in 12.4(1). Hopefully this will be added soon.

Read More

Researching Cisco Switch Backplane Statistics

While teaching at USENIX last week, I discussed SPAN ports. I mentioned that copying traffic to the SPAN port was less important than moving packets through the switch. One of the students asked if measuring the utilization of the switch backplane would reveal how well the switch was performing the SPAN function. Another student said there was a Simple Network Management Protocol Management Information Base (SNMP MIB) from which backplane statistics could be retrieved. I decided to research this issue as it affects using switches to collect traffic for network security monitoring. (Incidentally, Talisker offers SPAN port configuration advice for all sorts of switches.)

One answer appears in the Cisco document How to Get Catalyst Switch Backplane Utilization Using SNMP. This sounds promising until we read "the information in this document is valid for Cisco Catalyst switches that run Catalyst code only." Since modern Cisco switches run IOS, we seem out of luck.

That document produced several leads. First, it mentioned the CISCO-STACK-MIB. Finding this MIB clued me in to the multitude of MIBs offered by Cisco. They are available via FTP from ftp://ftp.cisco.com/pub/mibs/.

The link to the CISCO-STACK-MIB brought me to the SNMP Object Navigator. This is a really helpful tool. You can search object names and descriptions to receive a list of matching objects and MIBs for terms like backplane.

Another excellent resource is the MIBs Supported by Product tool. For example, here are all of the MIBs supported by the Cisco 2950 switch.

Cisco offers a few other helpful sites. These include the Cisco IOS MIB Tools page. That site has a link labelled MIB Locator. Follow it, select your IOS release, platform family (device), and IOS feature set, and you will learn what MIBs are present. Also useful are SNMP: Frequently Asked Questions About MIBs and the IP Application Services page for SNMP.

Getting back to the original question -- the original Cat OS discussion of the CISCO-STACK-MIB mentioned the sysTraffic Object Name as a place to find backplane information. Specifically, the description reads "Traffic meter value, i.e. the percentage of bandwidth utilization for the previous polling interval." The question now is to find out what Cisco devices support providing this information via SNMP. The View Supporting Images link on the sysTraffic page shows the Cisco IOS images which offer this SNMP value.

According to the results, the Cisco 3550 appears to be the cheapest switch which provides backplane statistics. I guess I won't be able to test this on my 2950! If anyone else managed to try this out, perhaps using snmpwalk from Net-SNMP, please post a comment.

Read More

Cisco Routers Run Tcl

This morning I was reading The State of the Scripting Universe by Lynn Greiner. That article features interviews with leaders in the development communities for Perl, PHP, Python, Ruby, and Tcl. The article pointed me towards a reference titled Dynamic Languages — ready for the next challenges, by design by David Ascher. While reading this second article I was surprised by this statement:

"Tcl is part of Cisco's IOS router operating system, and as such is included in all high-end Cisco routers and switches."

What's that? Tcl on my router? A quick Google for "tcl cisco ios" revealed two helpful resources: Cisco IOS Scripting with Tcl by Cisco and TCL'ing Your Cisco Router by Peter J. Welcher. Cisco's document revealed that Tcl 8.3.4 was introduced in Cisco IOS 12.3(2)T. Sure enough, it's on my Cisco router:

gill#sh ver
Cisco IOS Software, C2600 Software (C2600-ADVSECURITYK9-M),
 Version 12.3(11)T3, RELEASE SOFTWARE (fc4)
...trucated...
gill#tclsh
gill(tcl)#puts "Hello world"
Hello world

gill(tcl)#exit

Peter Welcher's article expands on Cisco's document to show some of what can be done with Tcl on the Cisco router.

This absolutely blows my mind, and I can't believe I haven't heard of this before. I found a 2001 email mentioning the Tcl shell as "undocumented," although there is a reference in the Cisco IOS Configuration Guide Master Index, Release 12.1. Imagine what might be possible with this; you might be able to run arbitrary code -- Sguil even? -- on Cisco routers.

Speaking of Tcl, the newest versions of Salvatore Sanfilippo's Hping rely on the language. I reported on this last year. Salvatore has been working on a Tcl interpreter called The Jim Project, which he says "will likely merge with Hping." Salvatore is also writing a book on Tcl, some of which is online as TCLWISE: An introduction to the Tcl programming language.

Read More

Passed My CCNA Test

I just finished testing for my Cisco Certified Network Associate certification. I passed with a 973 out of 1000. The test was 90 minutes long and I finished with only 8 minutes to spare. I think I missed one question, maybe two. The exam was as tough as I expected, meaning it was not easy. I know it was difficult since I usually breeze through majority multiple-choice exams. (For example, I answered all 250 questions on the CISSP exam in 90 minutes, and walked out the door.)

cannot say enough about the CCNA class I took with Todd Lammle at GlobalNet Training. He was not kidding when he said students need to know everything he writes in his slides and says while teaching. I was amazed how much of my knowledge Cisco managed to test with a 55 question exam. Also, if I did not know Todd's block size method of IP subnetting, I do not think I would have finished the test in time.

If you want to pass the CCNA, I recommend the following:

1. Take GlobalNet Training's CCNA class. I do not think you will find another training source who provides as much gear for each student to configure. The labs are first-rate.

2. Read Todd's book.

3. Practice using Todd's CertSim software. This was invaluable because it taught me to slow down and be more careful. I learned this lesson even though I only had a chance to use the software once, 2 hours before my test. I blazed through the simulation and scored only a 749 -- 100 points too low to pass! I vowed to be more careful on the real exam, and it payed off.

4. Practice configuring real Cisco gear, and use Todd's RouterSim Network Visualized software. I did not get a chance to play with this much outside of class, but it's a great way to build, configure, and test Cisco gear in a virtual environment.

5. For deeper knowledge, check out Alex Zinin's book.

Later this year I plan to start working towards the CCNP.

Read More

Use FTP Instead of TFTP to Transfer IOS Images

Michael Lucas' book Cisco Routers for the Desperate saved me this evening. I was trying to update the flash image on my Cisco 2950T-24 switch via TFTP, and had this problem (twice, actually):

gruden#copy tftp flash                 
Address or name of remote host [192.168.2.7]? 
Source filename [c2950-i6k2l2q4-mz.121-22.EA3.bin]? 
Destination filename [c2950-i6k2l2q4-mz.121-22.EA3.bin]? 
Accessing tftp://192.168.2.7/c2950-i6k2l2q4-mz.121-22.EA3.bin...
Loading c2950-i6k2l2q4-mz.121-22.EA3.bin from 192.168.2.7 (via Vlan1):
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
...edited...
!!!!!!!!!!!O!O!O!O!OO!OO!OO!OOOO
%Error reading tftp://192.168.2.7/c2950-i6k2l2q4-mz.121-22.EA3.bin (Transfer aborted)

Luckily the switch was not hosed at this point. I still had my command prompt and normal functionality. For some reason TFTP kept failing. The image stored on the TFTP server appeared good, since its MD5 hash matched that provided by Cisco. What to do?

I remember Michael Lucas describing how to use FTP to transfer IOS images, so I tried that:

gruden#copy ftp://richard:pasword@192.168.2.7/c2950-i6k2l2q4-mz.121-22.EA3.bin flash
Destination filename [c2950-i6k2l2q4-mz.121-22.EA3.bin]? 
Accessing ftp://richard:password@192.168.2.7/c2950-i6k2l2q4-mz.121-22.EA3.bin...
Loading c2950-i6k2l2q4-mz.121-22.EA3.bin !!!!!!!!!!!!!!!!!!!!!!!!!!!!!
...edited...
2867200 bytes copied in 256.536 secs (11177 bytes/sec)

I then reloaded the switch and it came up without any problems. Thanks Michael!

Read More

Switch to Router-on-a-Stick Communication

In January I described how I configured my Cisco 2651XM router to pass traffic between two VLANs on my Cisco 2950T-24 switch. I never assigned an IP for management purposes to the switch, since I always reached it via console cable. Today I decided to try upgrading the switch IOS, but that required applying a management IP to the switch.

My router had this configuration on the interface facing the switch:

interface FastEthernet0/1
 description Connection to gruden, Cisco switch
 no ip address
 duplex auto
 speed auto
!
interface FastEthernet0/1.1
 encapsulation dot1Q 10
 ip address 10.10.10.1 255.255.255.0
!
interface FastEthernet0/1.2
 encapsulation dot1Q 20
 ip address 172.27.20.1 255.255.255.0

I assumed that if I assigned a management IP to my switch with either a 10.10.10.0/24 or 172.27.20.0/24 address, the switch would be able to speak to the router. I assigned 10.10.10.100/24 to the switch. Because switches do not receive IPs on individual ports, I applied the IP to VLAN 1:

gruden(config)#int vlan1
gruden(config-if)#ip address 10.10.10.100 255.255.255.0
gruden(config-if)#no shutdown
gruden(config-if)#exit
gruden(config)#ip default-gateway 10.10.10.1
gruden(config-if)#end

Unfortunately, this did not work. I could not reach the router from the switch and vice-versa.

I decided to try assigned a new IP address directly to router interface fa0/1, and give the switch an IP in the same netblock:

gill(config)#int fa0/1
gill(config-if)#ip address 192.168.168.1 255.255.255.0
gill(config-if)#no shutdown 
gill(config-if)#end

Here's how I configured the switch:

gruden(config)#int vlan1
gruden(config-if)#ip address 192.168.168.2 255.255.255.0
gruden(config-if)#no shutdown
gruden(config-if)#exit
gruden(config)#ip default-gateway 192.168.168.1
gruden(config-if)#end

That did it. Now I can reach both devices. Apparently the switch can only communicate with the router when the address on the switch is outside of the VLANs in use. I believe Todd Lammle refers to this sort of setup as a management overlay network, where certain IPs are used solely for device management.

If anyone can comment on this design or suggest an alternative, I welcome feedback. I think the wrinkle in my setup involves the router having to pass traffic between VLANs 10 and 20.

Read More