Black Hat’s network ops center brings rivals together for a common cause – Source: go.theregister.com

Black Hat Neil “Grifter” Wyler is spending the week “looking for a needle in a needle stack,” a task he’ll perform from the network operations center (NOC) that powers the Black Hat security conference in Las Vegas.

The Register popped in to check it out.

We found the NOC in an unremarkable meeting room deep inside the Mandalay Bay hotel. Conversations inside are quiet, because the volunteers play a mixture of techno and metal, and beam muted hacking movies onto a screen in the background – Catch Me if You Can when this hack visited.

We’re looking for a needle in a needle stack

Sofas, the odd beanbag, a couple of plush mascots (a monkey and an inflatable sheep) provide places to crash. The volunteers work five or six hour shifts – with time off to go to training sessions and briefings or rest up in the ensuite napping area.

Volunteers needed three days to build the network operations center (NOC), and this year streamed it on Twitch so people can watch and send in questions. The stream continues throughout the show.

The Black Hat NOC is entirely separate from the hotel’s own NOC, not because it’s subpar, but because the demands of a conference like Black Hat are so large and attendees are (usually) so security conscious that organizers feel they need specialists with access to the latest kit who can react at high speed to spot and stop problems before they grow.

“It allows us to do mitigation if we see an attack, if there’s something that’s going on that could be detrimental to the stability of the network or the security of the attendees,” said Grifter, whose day job is VP of defensive services at security vendor Coalfire but has worked the Black Hat NOC for nearly 20 years.

“We can’t open up a ticket and wait 90 minutes for somebody from the hotel to come by and ask us what’s wrong. We have to handle it immediately.”

That’s not to say the hotel, and others, don’t help – indeed the week known as “Hacker Summer Camp” sees rival hotel chains (even those not hosting one of the three infosec conventions that come to town in August), local police, and the FBI hold regular briefing sessions to work out a response to any malicious activity. Thankfully, organizers only needed the FBI once, when the NOC picked up worrying indications that an attendee was in physical danger.

“There were some documents that were going across the network in the clear that were a guy’s license plates, his car, his pictures of his house, his wife and kids,” Grifter said. “It set off enough alarm bells that we made the call.”

The subsequent investigation found the source of the data was a somewhat inept private detective who was trailing an attendee who had recently left a pharmaceutical firm and was suspected of having stolen corporate secrets.

There’s plenty of malicious activity on the network during show week – a lot it of generated by attendees, particularly those attending training sessions in which they are taught offensive and defensive hacking techniques and how to counter them. Often, the temptation to exercise those skills is irresistible.

Grifter recalled a major alert in the very early days of the show when the entire network went down. The culprit turned out to be one of the trainers, who had discovered a zero-day flaw in the Cisco networking kit used by the show, and had demonstrated it to their class.

In response to this and other cases the team sandboxed the networks used for the training sessions.

Problems persist despite that precaution. One student decided to use an exploit they had just learned to try and hack his local police department.

“If you’re on your network and you’re looking for a malicious actor, you’re looking for a needle in a haystack,” Grifter opined. “We’re looking for a needle in a needle stack.”

When the NOC volunteers find whichever class is doing horrid things, they pop in to remind students “doing illegal things at Black Hat is still illegal”. In more serious cases a NOC representative will inform the class exactly what is being done and tell the person to knock it off.

You’d think that at a security conference the delegates would be savvy to risk and wouldn’t engage in unsafe behavior. Not so, sad to say – many folks turn up at the show with malware already preinstalled on their devices. The NOC team often finds it.

Grifter said finding new malware strains is a fun part of the job. When something new pops up the team “gets excited about it,” and immediately begins pulling the code sample apart to find out how it works and how to block it.

Hardware, software, and wetware

Vendors donate all the hardware used in the NOC. Organizers pick what they consider the best tools for the job.

“About 10 years ago, we decided that the scale of the show was too much for us to do with open source scripts and small vendor boxes tucked in corners,” Grifter remembered.

“We went down to the expo floor and walked up to a vendor that we wanted to work with and said: ‘Hey, we’re the guys who run the NOC. Would you be interested in letting us use your hardware and we’ll put your logo on the website.’”

Almost every vendor of note was keen to get on board. Conference organizers then had the problem of picking the right tools for the job. They now hold testing days where vendors show off their stuff and the team “act like CISOs” and decide what they want to use.

Selection criteria are strict and vendors cannot buy their way into the NOC. They can tell others that their kit is in the center, but must also name all other chosen suppliers so no single vendor gets bragging rights.

“We have been made offers by vendors ‘Hey, we’ll cut you a check. How much will it cost?’ We say: ‘Why don’t you take that money, invest it in your product, make it better, and maybe we’ll choose it next time.'”

The vendors chosen often to volunteer their own staff to help out in the NOC. In this year’s facility we observed representatives from arch-rivals Palo Alto and Cisco working face-to-face to sort out network problems. But the bulk of the 100+ staff are techies who take paid time off to volunteer at the NOC, mainly for the fun of it but also for the learning opportunity.

Grifter said that in some cases vendors fix problems identified at the show.

• Patch now: Millions of Dell PCs with Broadcom chips vulnerable to attack
• Palo Alto Networks execs apologize for ‘hostesses’ dressed as lamps at Black Hat booth
• Software innovation just isn’t what it used to be, and Moxie Marlinspike blames Agile
• If you give Copilot the reins, don’t be surprised when it spills your secrets
• US elections have never been more secure, says CISA chief

While the NOC uses commercial software, the volunteer crew also write their own code. Grifter showed off a new network visualization screen that was developed to make it easier to spot problematic network traffic, which you can see below.

The app’s developers first used vibe coding tools, then refined code using data from past Black Hat events from around the world.

“We put a lot of work into it, it’s a lot of work,” Grifter said. “But it is fulfilling enough that we keep doing it, and so much so that all of these folks take time off just to be a part of it.” ®