The business of network traffic analysis as seen an interesting evolution over the past 25 years (starting in 1990) which many believe began post thick net or 10Base5. It was the introduction of 10BaseT and repeaters and shortly after concentrators that brought about the network management market. Twisted pair networks introduced star configurations and trunk connections that grew very quickly. This growth was fuel by business managers who saw the value of having more employees with access to the same information.
Layer 2 Troubleshooting
Network engineers realized the need for insight into the occurrence of certain events on the wire. Collisions, out of window collisions, runts, giants, CRC issues, etc. were the main stay of typical network problems that were difficult to trouble shoot. To assist on site troubleshooting, manufacturers introduced LEDs that would flash for the occurrence of one of these events. As networks continued to grow, the costly nature of running out into the field to observe LEDs was determined to be not cost effective and as a result, SNMP to aid with network traffic analysis was born.
Layer 3 and Up Troubleshooting
SNMP allowed networks to scale but, as the a fore mentioned layer two issues (I.e. OSI model) became less common and problems related to the higher layers started to emerge, pressure was placed on SNMP. Vendors released thousands of Object Identifiers (OIDs) which provided access to more information about the operations within the hardware such as CPU, memory, routing tables, etc. but, it wasn't enough. Network admins needed access to end user traffic and because of this, packet analysis became a huge part of network traffic analysis.
The Death of RMON
SNMP tried to assist packet analysis by introducing RMON and later RMON2 however, adoption was weak due to the cost to manufacture, lack of consumer demand and poor engineering design. SNMP had run its course in network management and seen its share of innovations but, the industry was looking outside the box. Packet analysis was also a suffering strategy because it is very costly to deploy and maintain.
The Birth of NetFlow
Due to emerging security threats and the recognition that performance issues can come from every corner of the network, the network traffic analysis industry had to find a new technology. Cisco answered this cry and developed NetFlow . Not without its share of problems, it too needed to evolve. For example, the first version with a CPU pig. With the release of NetFlow v5 the technology started to see wide spread adoption.
It wasn't long before other vendors recognized the value of NetFlow and copied it. Some even renamed their implementations to NetStream, J-Flow, Cascade Flow, AppFlow, etc. but, it was all the same technology under the hood. NetFlow growth exploded at the end of the 1990s several reasons:
- It provided 80% of the details that packet analyzers did and answered the questions: who, what, when and where
- It provided visibility in all corners of the network far better than any other technology
- It aggregated the data down to a manageable size which was something packet analysis had failed at.
The introduction of much more stable networks and cloud applications necessitated continual evolution of NetFlow. The industry responded with Deep Packet Inspection (DPI) to identify applications sharing the same port (e.g. Facebook, Skype, BitTorrent, Webex, etc.). They have since also started focusing on performance metrics such as round trip time, retransmits, TCP window size, jitter, packet loss, caller ID, etc. With the release of greater statistics to choose some to isolated problem errors, NetFlow of course created a problem for itself "volume".
The network traffic analysis industry today is faced with solving the volume of flow data problem and the answer is being pushed back onto the user much as it was with SNMP. The answer of course is choice. What do you want your NetFlow or IPFIX capable routers, switches and servers to export? When you are ready to decide, companies like Plixer International support 100% of all choices being made with Flow technologies.
NOTE: Cisco worked with the IETF and released NetFlow as an industry standard called IPFIX.