BorderPatrol: Isolating Events for Precise Black-box Tracing

High-level causal request traces are of interest to develop- ers of large concurrent and distributed applications. These traces show how a request is processed as it passes through several modules which may be processes, threads, machines, or devices. They aid programmer understanding and are in- creasingly analyzed by tools used to detect performance and correctness errors. Precise traces are more useful than statis- tical approaches because they can detect anomalous behavior and allow decisions at run-time. Since these traces are dif- ficult to obtain without application-specific instrumentation of each module of the system, much of the recent work that analyzes request traces is limited to applications for which source code and developer expertise is available. We present BorderPatrol, which obtains precise request traces through systems built from a litany of unmodified modules, written in varied languages, with varying architec- tures. These include Apache, thttpd, PostgreSQL, Turbo- Gears, BIND and notably Zeus, a closed-source event-driven HTTP/1.1 web server, which uses helper processes. Border- Patrol obtains these traces using active observation which slightly modifies the event stream observed by system mod- ules, simplifying precise observation. Protocol processors aid active observation by leveraging knowledge about standard protocols and interfaces between concurrent modules, avoid- ing the need for implementation-specific instrumentation. BorderPatrol obtains precise traces for black-box systems that cannot be traced by any other technique. Further, it does so with limited overhead on real systems (approximately 10-15%) making it a viable option for deployment on produc- tion systems.