Warnings, /sdl, and improving uninitialized variable detection

Tim Burrell and Thomas Garnier of the TwC Security Science team present the sixth and last blog installment describing more /sdl functionality in Visual Studio 2012 RC. Please note that there will be an MSDN webcast discussing the security enhancements to Visual Studio 2012 RC – a wrap-up of sorts – on June 13 at 9:00AM (PST).

——————————————————————————————————————–

In previous posts  we introduced the /sdl compiler switch and described its dual role of treating security-relevant warnings as errors to find bugs during development and enabling compiler mitigation features to help defend against residual bugs.

This post highlights how the /sdl compiler switch makes prioritizing important security warnings easier and describes a new warning useful to identify potentially uninitialized pointers.

/sdl and compiler warnings

As noted in our overview post, /sdl causes a number of compiler warnings to be treated as errors – Microsoft’s SDL treats these warnings as mandatory for native code:

Introducing the C4703 warning

In reviewing available compiler warnings, we noted that C4701 had interesting results but we believed that there was value in helping developers prioritize which instances were the most likely to have security implications. The C4701 warning identifies potential usage of uninitialized local variables. In MSRC cases involving uninitialized local variables, pointers are the most common case leading to code execution.

Visual Studio 2012 RC introduces a new level 4 warning (C4703) as a subset of C4701 to detect potentially uninitialized local pointer variables:

Both warnings are generated, maintaining the existing behaviour and also highlighting uninitialized pointers.

The detection of potentially uninitialized local variables is achieved by finding a path where the local variable is used before being initialized. In this example, if size is bigger than 256 then a path exists that would lead to ‘p’ being used without being assigned a value.

The C4701 and C4703 warnings are comparatively noisy as the existence of a bug depends on whether the path to the uninitialized usage is feasible. The compiler’s analysis does not attempt to solve the constraints to determine this, leading to false positives.

Expanding C4703 with LTCG (Link-time Code Generation) support

The warning analysis described in the previous section is done locally for each function. If a reference to a variable is passed to another function, the compiler’s analysis lacks information and assumes that the function will initialize the variable correctly. Changing this assumption for all function calls would increase false positives making the warning unacceptably noisy.

In Visual Studio 2012 RC we used LTCG and optimization support to increase the scope of the analysis on specific functions. A summary is generated on target functions and the resulting information is used during local analysis.

These functions are examples of the problem we aimed to detect:

ExampleFunction delegates the local variable p initialization to Callee without verifying if this function succeeded on cleanup. From local analysis we cannot state if p would be initialized on failure or not.

Prior to Visual Studio 2012 RC the analysis would assume that Callee initialized p; in the absence of information about Callee this helped keep the number of false positives in check.

In Visual Studio 2012 RC, the new inter-procedural analysis allows a warning to be reported while reducing the potential number of false positives.

This type of analysis works only with these conditions:

• The binary is built with /LTCG
• Optimization is enabled (/O1 or /O2 for example)
• The target callee function return type is HRESULT
• The target callee function has at least one double pointer argument being initialized
• The usage happens on a failure path on the calling function

The LTCG and optimization requirements enable the elements of the compiler’s inter-procedural analysis framework that we are using. The other criteria above are targeted at finding a common class of uninitialized variable usage that spans function calls for which we had seen real historical examples. We are looking forward to improving and generalizing this kind of analysis in the future.

Conclusion

Compiler warnings, along with the CodeAnalysis warnings mentioned in an earlier post, are valuable and complementary techniques that assist secure code development by flagging bugs during the development process.

In this series of posts we have highlighted a number of elements of SDL and how Visual Studio 2012 RC makes these easier to implement: this includes a more useable and intuitive interface for CodeAnalysis and a centralized /sdl switch to treat the most security-relevant compiler warnings as errors and to enable additional compiler-based mitigation features.

We encourage you to use CodeAnalysis and /sdl and provide us with feedback.

You can also find more information about the Security Development Lifecycle (SDL), including articles, tools and more on this website.

Tim Burrell & Thomas Garnier, MSEC Security Science.