AMTSO Progress

PC Tools is proud to be a participating member of The Anti-Malware Testing Standards Organization (AMTSO). The group recently met to discuss (argue) over details of proposed standards in Washington on the Microsoft Campus, and we look forward to eventual formal completion and public release of these standards.

AMTSO and CARO Workshop

The AV industry was busy this past week amongst the blooming tulips in Hoofddorp, the Netherlands. Both an AMTSO conference and a CARO workshop was held the last three days of the week.

A large group of attendees arrived for the Wednesday all-day testing standards meeting, with more journalists in attendance than before. It was encouraging to see, because one of the AMTSO's formative goals has been to invite and include representatives from all parts of the computer security industry. Progress is being made toward a set of testing standards for anti-malware products for everyone involved.

The CARO workshop followed on Thursday and Friday, with presentations focusing on malware obfuscation from the AV industry's perspective (googling "datasecurity event caro" provides a link to the home page). The opening talk by Paul Ducklin from Sophos set the tone for most of the event -- legitimate compressors/packers are acceptable and good (according to a number of individuals in the AV scanner business), while software protection solutions like Themida and SVKP are unacceptable and evil (to a number of individuals in the AV scanner business).
It was interesting that while AV vendors and Ilfak Guilfanov of IDA Pro/Hex Rays spoke and gave presentations over the two days, none of the developers or vendors from Themida or ASProtect (a couple of software protection systems that were referred to in the presentations) were invited or presented their thoughts.

Even at the workshop, it seems that there remains disagreement on how the industry should handle software obfuscation, and there remains a sense that software obfuscation is a major source of problems for the AV industry. Whether it's due to difficulties in emulation, performance issues when unpacking, the complexities of the virtualization packers (where Sophos' Boris Lau showed that a single NOP instruction can be easily and inexpensively be translated into over 50 virtual instructions) or simply disagreement over how to identify what is behind software protection, it continues to be a weakness for traditional AV scanners.
Just to give an idea of the volume of difficulties and tricks that researchers have to develop methods to deal with, Peter Ferrie's paper was presented by Mady Marinescu of Microsoft, and in it he enumerated over 50 anti-unpacking tricks commonly seen in packers and often seen in malware.
Presenters also included evaluations of the proportions of malware seen packed by specific packers and various approaches to dealing with them, including blacklisting. It seems that it is easier to include this approach in a scanner than to have to actually implement an unpacker in a scanner for all the different varieties of packers. Blacklisting is cheap and easy, but is more prone to causing fp's, and often decisions to blacklist may be debatable.
We will see what this turn away from extremely low false positive rates will do to the major advantage that the scanners had over behavioral based solutions.

From the perspective of an individual pushing a behavioral solution that solves for the difficulties that scanners have with obfuscation, it is somewhat easy to be critical of AV scanner products' inability to continue performing with such a low level of false positives and exacting matches in the face of ongoing obfuscation and "server-side polymorphism"/"rapid release" techniques currently used by malware distributors to evade the AV solutions. The complexity and difficulties are high for the guys trying to develop elegant and effective AV solutions to these problems.
We'll see more of this obfuscation topic, but from the "hackers" perspective, when defcon's "Race To Zero" contest is held this fall.

AMTSO website

The work of the AntiMalware Testing Standards Organization, or AMTSO, is moving forward. This morning, the group's website went up, thanks to the efforts of volunteers. It presents the group's charter, pro-tem committees, membership, and a brief list of resources all related to anti-malware technology testing.

The group continues to grow in its formative stage:

"AMTSO is dedicated to helping improve the objectivity, quality and relevance of anti-malware technology testing. AMTSO membership is open to industry-wide academics, reviewers, testers and vendors, subject to guidelines determined by AMTSO."


The press is catching the buzz as well with articles at SecurityFocus, Fox Business, InformationWeek, SCMagazine and the Washington Post.
We look forward to further helping open efforts to better evaluate and understand security solutions as an AMTSO member.

Improving tests and collaboration

What do you get when you put 40+ AV and software security experts together in a room with testing organizations? It sounds like a bad joke, but it happened for the first couple of days this week in Bilbao, Spain. The event itself has the potential to have a very large positive impact on the state of anti-malware testing overall and the relevance and meaning of test data for all of its consumers -- communications between vendors and testers, guidelines for tests, neutrality of the group enforced by academic members, and more.

The world's largest and smallest software security vendors and testing groups are working together to create this non-profit coalition of vendors, testers and academics. The group will be called the AMTSO, or the Anti-Malware Testing Standards Organization. The overall goal will be for the coalition to take on all challenges related to anti-malware security software testing, improving all aspects of the process. It will be a large task to set up standards, and PC Tools is pleased to take part in this effort.

The event was formative in nature, establishing temporary committees for most of the sessions before breaking off into the beginnings of some discussion and debate over technical matters and details that will come up in future meetings. Dr. Igor Muttik of McAfee's AVERT Labs posted detailed information of the proceedings, for those interested.

We will keep you updated on this ongoing effort to improve the state of anti-malware security software testing.