Automated Collaborative Network Threat Analysis

At Galois, we are working on new methods for detecting and responding to network security threats. Our software allows a group of collaborating network systems to exchange information and analyze traffic together, at high traffic volumes. This collaborative analysis presents several advantages over current techniques:

• automatic analysis is much faster than manual inspection and ad hoc data sharing
• collaboration between sites provides mutual benefit (more information) to all participants
• we can identify attacks that isolated analysis cannot even see

 

Accelerating the Pace of Understanding

Network security analysis is currently hampered by the need for wide-scale human inspection of anomalous traffic. Most organizations use custom tools or off-the-shelf software to look for threats, but then must manually compare their observations with reports from other organizations. This process is too slow to keep up with modern, automated attacks such as newly emerging botnets.

Our software automatically detects traffic anomalies and shares the information using a distributed, peer-to-peer (P2P) infrastructure that is resilient to attacks and outages. Our software analyzes the shared information, allowing a “big picture analysis” that is not possible at individual sites. For example, our software allows cooperating sites to discover spoofed IP traffic that can be difficult for existing tools to detect.

In addition, our tools operate at both the infrastructure layer — core ISPs and network backbone providers — but also at the network edge. 3DCoP and similar tools can help operators understand patterns across their entire user base. For example, an edge ISP can more quickly discover the signatures of compromised home IoT equipment, and throttle traffic in order to provide a more consistent experience.

 

Analysis at Massive Scale

We have adopted flow-based traffic monitoring protocols (NetFlow and IPFIX) suitable for processing data at very high bandwidths. Thanks to the highly compact data representations provided by these protocols, our software can process hundreds of gigabits/sec (Gbps) at each site, allowing the total network to track and analyze terabits/sec (Tbps) of internet traffic.

These kinds of traffic volumes far exceed the processing capabilities of current packet-based analysis tools, as well as most tools used for traffic simulation. To aid in our research and experimentation, we developed ddosflowgen: a tool that models distributed attacks and generates datasets with extremely high traffic volumes.

 

Current and Future Development

Our collaborative analysis software tracks flow information in real-time tables, sifting the traffic and ranking it by characteristics. Most notably, we track how anomalous a given flow is and note the characteristics which make it anomalous. Traffic that is ranked as highly significant is shared with other network systems over the P2P infrastructure.

Automated analysis and threat detection rules are continuously applied against the real-time traffic tables. Our experiments have shown that even a few simple threat detection rules can be highly effective with real-world data. Most importantly, our collaborative analysis software can identify attacks that isolated analysis cannot even see.

For example, Distributed Denial of Service (DDoS) attacks are one type of internet threat that can benefit from automatic collaborative analysis. Under a program funded by the Department of Homeland Security, Galois built a collaborative analysis tool called DDoS Defense for a Community of Peers (3DCoP). Our software detects early signs of DDoS attacks, shares the information with other sites, and recommends actions to mitigate attacks. The distributed P2P messaging system that links the peers ensures that collaborative analysis continues even under degraded network conditions. In experiments, we demonstrated that our software can analyze and mitigate attacks exceeding 1 Tbps.

In the future, our rule sets can be expanded to enable partnership with human operators, allowing users to mix human insight with the automated information distribution and analysis in 3DCoP. In addition, we imagine integrating other detectors and rule sets to allow deeper analysis of additional threats, including botnet command and control infrastructure and internal exfiltration threats. Finally, 3DCoP can also be paired with testing and mitigation capabilities, allowing the system to verify threats through the strategic deployment of honeypots, or act on threats by adjusting routing or configuration tables.