For the past several years, the Vundo family (also known as Virtumonde) of malware appeared high on AV vendors’ prevalence lists — this stuff is everywhere. To get there, the malware employs an aggressive set of tactics over the course of its distribution to evade AV and anti-spyware solutions. A close examination exposes an interesting observation that some of its user-mode rootkit tactics use the Microsoft Research Detours library in order to hide its presence from security solutions. Below is a somewhat technical description.

First off, the Detours project out of Microsoft Research focuses on “Binary Interception of Win32 Functions“. In other words, when a developer or malware writer wants to hook a function inline and insert their own code, they can intercept a win32 function with code from the Detours library.
To use this code commercially, “Detours Professional 2.1 includes a license for use in production environments and the right to distribute detour functions in products…For information on licensing Detours Professional 2.1 contact Microsoft’s IP Licensing Group at iplg@microsoft.com”. Let’s assume either that Microsoft never provided the vundo developers with a license or that the vundo developers never attempted to obtain a license for their “commercial” use.

One of Vundo’s library components currently in the wild is injected into processes as a part of its attack. This component may in turn be detected by anti-spyware scanners using the EnumProcessModules api call, which would provide an anti-malware scanner using that call with a handle to the injected module. And this is where the abuse begins.
You can see the malicious Vundo hook in this screenshot, implementing the hook functionality from the Detours library. Basically, if a process calls EnumProcessModules, the vundo appropriated code will intercept the win32 function and report that the module enumeration procedure failed. When the EnumProcessModules call fails, certain security scanners are unable to detect the vundo component’s presence:

How can Detours code be identified in this dll? Well, the source of the detours library can be placed side-by-side with the unpacked and disassembled vundo component. In many places, the same sequence and order of instructions and data is unmistakably identical. For the sake of brevity, we’ll focus on just a couple that briefly illustrates our point in this post.

Here, the deadlisting for the vundo function is on the left, and the matching Detours source code on the right. This chunk of Detours code is at the core of the hooking functionality within disasm.cpp of detours.lib. The source from the Detours library here is determining the length of the currently evaluated instruction and then copying the instruction to the trampoline buffer (this location is the place where the inlined vundo rootkit function can call back into the original function without interception). The appropiated code on the left is compiler optimized, and it is a mirror image of the Detours logic on the right:

Here, in a similar fashion, we see vundo functionality that was stolen from the Detours library calling the DetourCopyInstructionEx() function and an inlined detour_does_code_end_function() function. In this reversing illustration, the vundo function is performing checks to ensure the target function’s eligibility for interception. In other words, vundo’s appropriated Detours code is checking to see if the target function contains a select set of instructions that would prevent hooking: