NetPanel: Traffic Measurement of Exchange Online Service

NetPanel: Traffic Measurement of Exchange Online Service

Global cloud applications are composed of thousands of components. These components are constantly generating large volumes of network traffic, which is a major cost of cloud applications. Identifying the traffic contributors is a critical step before reducing the traffic cost. However, this is challenging because the measurement has to be component-level, cost-effective, and under strict resource restrictions. In this paper, we introduce NetPanel, which is a traffic measurement platform for the Exchange Online service (EXO) of Microsoft. NetPanel fuses three data sources, namely IPFIX, Event Tracing for Windows (ETW), and Application Logs, to measure the service traffic at the component level, where each component is owned by a service team. NetPanel uses feature extraction and data aggregation to reduce the cost; applies top-k filters to reduce production resource consumption. NetPanel has been in operation for more than one year. It has been used to uncovered the network traffic characteristics and the network traffic cost composition of Exchange Online. Applying the insights, we have saved millions of dollars in network resources. The cost of running NetPanel is relatively small, which requires less than 0.01% of EXO computation cores to process the data in the big-data platform and less than 1% CPU and disk I/O on production servers.

Camera ready version to come soon.