The team is a Cisco Systems Inc. internal design squad, and its product is an in-house processor that started development in 1997.

"The words 'network processor' didn't appear until around 1998. That's when we found out what we had been working on," said Scott Nellenbach, manager of silicon development for Cisco's WAN edge business unit.

Indeed, internal development is a tactic still being pursued by the large networking OEMs, many of which started their chips before the network processor concept had hit the merchant market. What's remarkable about Cisco's particular chip is the way it came to life: as a rogue idea, embraced by an engineering team looking for new projects and acting outside the normal corporate boundaries. The end result is proof that skunkworks ingenuity is still possible inside a company as large as Cisco.

Network processors — a catch-all term usually referring to microprocessors performing Internet Protocol (IP) packet functions — are intended to replace the ASICs at the heart of switches and routers, primarily as a way to speed development. Often, software developers had to explain the necessary algorithms for the ASIC team, and by the time the ASIC was designed and fabricated, the standards and algorithms would change.

"What you want to do instead is create an infrastructure that allows you to adapt," said Bill Jennings, the Cisco manager who led the network processor team. As a result, most network processors sport some form of programmability.

But the lead time for routers and switches can be years. While the merchant market continues to crank out new network-processor companies — such as XStream, EZchip and Silicon Access — OEMs are designing their next generation of hardware, and in many cases they're continuing to use their own ASICs.

Cisco rival Juniper Networks Inc. did just that with its Internet Processor II. The chip is at the heart of Juniper's most sophisticated router, the M160, but its design was started before the company had released its first system, the M40. Juniper knew what it wanted — and knew such a device couldn't be bought yet.

"This technology, at the system level, is not at the maturity where you can buy something off the shelf and just put it in a router," said John W. Stewart, manager of marketing engineering for Juniper. "Ours is the second generation of a processor that nobody else has the first generation of."

Likewise, Network Peripherals Inc. spins its own packet-processing chips. "We have our own ASICs that do everything the network processor does and more," said Ramana Kattula, systems engineering manager for NPI.

Cisco is not alone, then, in hanging back from the network processor bandwagon. "There are external vendors who get frustrated," Jennings said. "They think they have a right to get a design win at Cisco."

Free agents

Cisco has two in-house network processors. One was created specifically for the GSR 12000, Cisco's top-of-the-line router; the other, based on what is now called the Parallel Express Forwarding (PXF) architecture, is being shopped to divisions in Cisco and has found a home in some half-dozen boxes.

The PXF architecture got its start in Research Triangle Park, N.C., in 1997 as Bill Jennings' team of eight engineers had just wrapped up its seventh token ring project for Cisco.

Normally, that's the cue to start a bigger/faster/stronger design, but this was token ring: It didn't take a tarot reading to figure out that the future was elsewhere. Jennings' group ruled out doing an eighth project, and that left it adrift.

"Like it or not, Cisco's a big company, and [people] have their little fiefdoms," Jennings said. "We had to build a market."

Along came Darren Kerr, a Cisco distinguished engineer whose job is to think up radical concepts. "Darren was kind of shopping for a hardware team . . . to take on this flaky idea," Nellenbach said.

Jennings assigned engineers Ken Key and Mike Wright to vet the concept with simulation models, and when the results came back positive — finding that Kerr's network processor could theoretically work — Jennings' team had found a new calling.

For the next year, Jennings' group would operate as a self-made skunkworks, working unfunded to build an unofficial new product. The group became a startup microcosm inside Cisco, and Jennings relished the relative freedom from accountability that such status afforded him.

"I was willing to fail . . . the only way I could displace an internal design was to be much, much better," Jennings said. "So we took a lot of risks." And since Jennings' team wasn't beholden to any particular group inside Cisco, "we were more a central resource than a dedicated resource."

The point, of course, was to find a better way to do packet processing. Kerr's solution was bombardment: "Throw as many processors as you could on an ASIC and bypass the limitations of off-the-shelf processors," as Nellenbach described it. The first design called for a 4-by-4 array, so the team picked up the most dense processor on the market — the ARM 7 core — and crammed 16 of them into a prototype chip unofficially named Toaster.

The team also drafted some processor expertise into the fold, specifically engineer John Marshall. "While we were developing this prototype chip, we had him go out and look at what was out there," Nellenbach said.

Toaster 2

Under Marshall's guidance, the team created a VLIW engine that became the heart of Toaster 2, which emerged in mid 1998. (Although the general architecture was named PXF, no official name has ever been assigned to the chips themselves, Jennings noted.) It's Toaster 2 that lies at the heart of the Cisco 10000 edge-services router, in which complex packet forwarding is handled by two processors in serial, creating four eight-stage pipelines. The PXF architecture has found its way into the Cisco 7200 router, the Catalyst 6500 switch and other designs.

The ARM 7 used in the first Toaster was just a stopgap; Toaster 2 was built entirely with Cisco intellectual property, down to Marshall's VLIW engine. Comparable technology just can't be had on the open market, Nellenbach said, and that fact has helped spread the Toaster gospel to other parts of the company.

"We've always had kind of a dual role. We ended up in this advanced-technology mission," Nellenbach said. "We had a product need, and we built this chip. Word gets around: 'Yeah, that solves a problem.' "

Jennings describes the PXF architecture itself as a combination of network processing and concurrent RISC processing. The RISC side is used for stack management or to terminate protocols such as BGP4, while the network processing side handles algorithms and features related to packet forwarding.

Like most network processor designs, the Toaster is parallel-pipelined. "Each column you can think of as doing a system function with separate memories," allowing for better I/O bandwidth in any column, Nellenbach said. Toasters also can be chained together for handling more complex problems.

Kerr's main concern was in maximizing the possible number of 32-bit lookups per second, which meant getting the memory interface right, Jennings said. "Many memory designs are very good at bursting data. But if you're trying to walk a tree of addresses, bursting doesn't work very well."

Likewise, the content-addressable memory isn't suitable, because it carries only 16,000 entries, Jennings said. "Every VPN [virtual private network] has its own set of routing tables. You're talking about 30,000 tables," which rapidly can translate into millions of table entries — a scope that renders CAMs inadequate.

Kerr's plan called for larger memory, at little cost either in power or money. So Jennings' team went with eight memory interfaces, all connecting to synchronous DRAM.

In all, the PXF processors are "pretty substantial," Jennings said. Toaster 2 weighs in at 30 million transistors, and the next-generation part is due to be 50 percent larger.

Just as the optical networking world keeps moving, changes have shifted the PXF team, although most of the original members have stayed close by. Jennings himself is taking a temporary leave from the company, and original team members Key and Wright are now managers in Nellenbach's group.

Architecture experts Jeff Scott and Kip Potter, who Nellenbach said "turned this into a product," are moving on as well; Potter is fronting Cisco's coprocessor efforts, and Scott is Nellenbach's first-line manager in charge of Toaster.