Differential Regression Testing for REST APIs

MSR-TR-2019-23 |

Published by Microsoft

Revised version published in ISSTA'2020, July 2020.

Download BibTex

Cloud services are programmatically accessed through REST APIs. Since REST APIs are constantly evolving, an important problem is how to prevent breaking changes of APIs, while supporting several different versions. To find such breaking changes in an automated way, we introduce differential regression testing for REST APIs.

Our approach is based on two main observations. First, breaking changes in REST APIs involve two software components, namely the client and the service. As such, we observe that there are also two types of regressions: regressions in the API specification, i.e., in the contract between the client and the service, and regressions in the service itself, i.e., previously working requests are “broken” in later versions of the service. Finding both kinds of regressions involves testing along two dimensions: when the service changes and when the specification changes.

Second, to detect such bugs automatically, we employ differential testing. That is, we compare the behavior of different versions on the same inputs against each other, and find regressions in the observed differences. For generating inputs (sequences of HTTP requests) to services, we use RESTler, a stateful fuzzer for REST APIs. Comparing the outputs (HTTP responses) of a cloud service involves several challenges, like abstracting over trivial differences, handling out-of-order requests and non-determinism.

Differential regression testing across 17 different versions of Azure networking APIs deployed between 2016 and 2019 detected 5 regressions in the official specifications and 9 regressions in the services themselves. This shows that our approach can be used to find regressions even in mature and widely used REST APIs.

Publication Downloads

RESTler-Fuzzer

November 16, 2020

RESTler is the first stateful REST API fuzzing tool for automatically testing cloud services through their REST APIs and finding security and reliability bugs in these services.

Download Data

REST API Fuzz Testing

November 16, 2020

This self-hosted service developed for Azure, including its orchestration engine and security tools (including MSR's RESTler), enables developers to embed security tooling into their CI/CD workflows.

Download Data

Fuzzing to improve the security and reliability of cloud services with RESTler

In the past few years, cloud services have experienced tremendous growth. Most of these services are programmatically accessed through REST APIs. As the pace of development increases, both the APIs and service implementations are evolving rapidly. There is an urgent need for automated tools to test the reliability and security of cloud services that can keep up with today’s fast-paced service development and deployment—tools that provide the necessary level of automation and coverage for the growing number of APIs being deployed across the web.

In this webinar, join Marina Polishchuk, a Software Engineer at Microsoft Research, in exploring how RESTler—the first stateful REST API fuzzer—can help efficiently find security and reliability bugs in cloud services. RESTler analyzes a Swagger/OpenAPI specification and produces a fuzzing grammar that contains information about requests and their dependencies. RESTler only fuzzes a request if all its dependent resources have been successfully created—this enables RESTler to achieve deeper coverage out of the box. RESTler also offers a pluggable model for checking security properties. RESTler is open source and available at its GitHub repository.

Together, you’ll explore:

  • How to use RESTler to fuzz a cloud service
  • The types of bugs that can be discovered through the REST API and their security impact
  • How RESTler addresses the challenges of testing large, complex services through their REST APIs

Resource list:

*This on-demand webinar features a previously recorded Q&A session and open captioning.

This webinar originally aired on May 26, 2021

Explore more Microsoft Research webinars: https://aka.ms/msrwebinars (opens in new tab)