TransWikia.com

Processing Exceptionally High Volume Singular Flows

Information Security Asked by ReedGhost on March 9, 2021

I wanted to know if anyone had experience sensoring single flows that generate 90 kpps or upward of traffic. My conundrum is that I use tools which I would like to be able to properly see an entire flow with (Zeek, Suricata), however both of these utilities are not capable of handling larger traffic volumes if a single connection exceeds a "worker thread’s" maximum processing rate. Because a single connection is able to generate this volume of traffic, I am not aware of a load balancing method and accompanying software that would produce similar resulting information from Zeek & Suricata(Understandably this matters a lot less with Zeek, as seeing the entire connection isn’t strictly necessary, however having a giant connection fill the rx ring results in losing subsequent connections).

For referencing I’m currently using the AF_Packet family sockets for both Suricata in Zeek, which are both in their own fanout. I get roughly 48,000pps running suricata with 4 capture threads with my current configuration and ~26,000pps per Zeekctl cluster.

I’m aware the most sensible thing to do would be to strictly analyze the flow, and then filter it out pre-sensor, however I’m doubtful policy in my org will allow me to do this, but I’ve been unlucky in my search for a satisfactory tool or reconfiguration.

I am hoping someone might have experience handling this type of situation or knows of software which has a less restrictive processing per "worker/thread" rate that would produce similar results(or honestly any results that aren’t massive amounts of packet loss).

One Answer

First thing I would look into is switching to use PF_RING sockets.

If that's not sufficient then it's probably a job for dedicated network packet broker hardware. You will want to look at vendors like Arista, Gigamon or Ixia. Some high-end switches may have the functionality built-in as well.

If you are really and truly CPU-bound by single flows and not total flow volume then short of buying CPUs with extreme single-core performance you are probably out of luck with Zeek and will have to switch to e.g. a commercial flow processor that supports on-NIC hardware packet processing.

Answered by matoro on March 9, 2021

Add your own answers!

Ask a Question

Get help from others!

© 2024 TransWikia.com. All rights reserved. Sites we Love: PCI Database, UKBizDB, Menu Kuliner, Sharing RPP