Real-time Regex Matching with Apache Spark

Author USMA Department

Electrical Engineering and Computer Science

USMA Center/Institute Affiliation

Cyber Research Center

Document Type

Conference Proceeding

Publication Date

Fall 9-2017


Regular expressions, Parallel computing, Apache spark, Bro, Sparks, Real-time systems, Organizations, Partitioning algorithms, Anomaly detection, Monitoring, Big data, Data protection, Invasive software, Parallel algorithms, Pattern matching, Public domain software, Network monitoring systems, Enterprise networks, Custom regular expression patterns, NMS data, Network traffic, Network administrators, Regular expressions, Traffic data collection, Apache Spark, Bro HTTP log events, Real-time regex matching, Army Engineer Research & Development Center, Time 1.047 s


Network Monitoring Systems (NMS) are an important part of protecting Army and enterprise networks. As governments and corporations grow, the amount of traffic data collected by NMS grows proportionally. To protect users against emerging threats, it is common practice for organizations to maintain a series of custom regular expression (regex) patterns to run on NMS data. However, the growth of network traffic makes it increasingly difficult for network administrators to perform this process quickly. In this paper, we describe a novel algorithm that leverages Apache Spark to perform regex matching in parallel. We test our approach on a dataset of 31 million Bro HTTP log events and 569 regular expressions provided by the Army Engineer Research & Development Center (ERDC). Our results indicate that we are able to process 1, 250 events in 1.047 seconds, meeting the desired definition of real-time.


DOD High Performance Computing Modernization Program (HPCMP) and the Army Engineer Research & Development Center (ERDC)

Conference Name

High Performance Extreme Computing Conference (HPEC), 2017 IEEE

Conference Location

Waltham, MA

Conference Dates