Across the U.S. the ThreatFire community saw huge numbers of FakeAv variants disappointingly being run on systems, the Vundo ad-popping trojan appearing all over desktops, and Koobface worming its way across social networks. In India, the Sality virus/downloader and varieties of bots attempted to infect systems — when ThreatFire’s community’s statistics are extrapolated out to the 40 million likely computers in that country, we can estimate that  millions of Indian systems were attacked by this virus. In China, we saw gaming password stealing worms continue to spread out across the country, most likely distributed through usb sticks and other removable drives. Hot topics consistently led to blackhat SEO and phony codecs. Socially engineered bulk email schemes delivered attachments that dropped password stealing Zbot and Bredolab downloaders, users were easily convinced that they received invoices from delivery services or social networks were updating their systems. The Conficker hype grew exponentially and is all too slowly whimpering away, while the Waledac threat mutated and began to dry up altogether.

Our PC Tools ThreatFire team finished the year with a bang. The award winning PC Tools’ Internet Security Suite and its ThreatFire Behavioral Intelligence component topped all other suites as champion in the lengthiest, most comprehensive, real-world dynamic-testing malware blocking competition to date. It’s exciting to see AMTSO dynamic testing best practices being adopted and used to better drive testing and scenarios that best evaluate malware attacks that most computer users really can encounter on a daily basis. Nice testing effort and results indeed.

As 2010 arrives, we hope that existing and new ThreatFire/Behavior Guard users around the world look forward to fewer of these threats being realized on their own systems and another year of confidence in their information driven world.

The group continues to pursue ways to improve testing methods, and finding and collecting malware has always been an issue for improvement. The group is attempting to ensure testing samples that are current, and providing testing matter that exercises products in ways adequate to support reviewer conclusions.

Various papers were discussed and only two of these put up for vote. The group passed the two important papers today that will be posted to the website soon — “Issues in Creating Samples for Testing”, and “Network AV Testing”.

The AMTSO website has changed a bit, but the goals and our commitment to contributing to these standards and meeting challenges around anti-malware testing methodologies has not. Our second year of active participation should witness more outbound efforts by the organization. The three papers passed in this meeting will be posted on the documents section of the web site soon:
1. Testing Sample Validation
2. A Process for Evaluating Testing and Reviews
3. In the Cloud Testing Procedures.

Now that the body has voted these standards are firmly in place, the testing groups, media, academics, advisors and vendors participating in the group will see it move forward with a more active role in applying and clarifying these standards.

It’s always great to see the words “I’m really impressed with what I’m reading in these standards.” He even goes over the “Best Practices for Dynamic Testing” document, which is relevant to properly evaluating ThreatFire and other behavioral-based anti-malware solutions — delivering malware to the system in the same way that a user would see it attacking their system. We were especially interested in the “Dynamic Testing” document details and crafting at the last Oxford meeting. He understands the issues addressed in the document, including issues with using Virtual Machines in testing, and the article finishes with a hint of the reality of the process: “Don’t expect that you’ll start seeing results compliant with these guidelines a lot. Testing like this is difficult and expensive and few labs are set up to do it. If all goes well, more will be from now on.”
It’s great to see positive interest in the testing standards already. Let’s hope that Larry and others at eWeek are interested in becoming a member as well.

We had a great accommodations on the trip. Great room number.

A large group of attendees arrived for the Wednesday all-day testing standards meeting, with more journalists in attendance than before. It was encouraging to see, because one of the AMTSO’s formative goals has been to invite and include representatives from all parts of the computer security industry. Progress is being made toward a set of testing standards for anti-malware products for everyone involved.

The CARO workshop followed on Thursday and Friday, with presentations focusing on malware obfuscation from the AV industry’s perspective (googling “datasecurity event caro” provides a link to the home page). The opening talk by Paul Ducklin from Sophos set the tone for most of the event — legitimate compressors/packers are acceptable and good (according to a number of individuals in the AV scanner business), while software protection solutions like Themida and SVKP are unacceptable and evil (to a number of individuals in the AV scanner business).
It was interesting that while AV vendors and Ilfak Guilfanov of IDA Pro/Hex Rays spoke and gave presentations over the two days, none of the developers or vendors from Themida or ASProtect (a couple of software protection systems that were referred to in the presentations) were invited or presented their thoughts.

Even at the workshop, it seems that there remains disagreement on how the industry should handle software obfuscation, and there remains a sense that software obfuscation is a major source of problems for the AV industry. Whether it’s due to difficulties in emulation, performance issues when unpacking, the complexities of the virtualization packers (where Sophos’ Boris Lau showed that a single NOP instruction can be easily and inexpensively be translated into over 50 virtual instructions) or simply disagreement over how to identify what is behind software protection, it continues to be a weakness for traditional AV scanners.
Just to give an idea of the volume of difficulties and tricks that researchers have to develop methods to deal with, Peter Ferrie’s paper was presented by Mady Marinescu of Microsoft, and in it he enumerated over 50 anti-unpacking tricks commonly seen in packers and often seen in malware.
Presenters also included evaluations of the proportions of malware seen packed by specific packers and various approaches to dealing with them, including blacklisting. It seems that it is easier to include this approach in a scanner than to have to actually implement an unpacker in a scanner for all the different varieties of packers. Blacklisting is cheap and easy, but is more prone to causing fp’s, and often decisions to blacklist may be debatable.
We will see what this turn away from extremely low false positive rates will do to the major advantage that the scanners had over behavioral based solutions.

From the perspective of an individual pushing a behavioral solution that solves for the difficulties that scanners have with obfuscation, it is somewhat easy to be critical of AV scanner products’ inability to continue performing with such a low level of false positives and exacting matches in the face of ongoing obfuscation and “server-side polymorphism”/”rapid release” techniques currently used by malware distributors to evade the AV solutions. The complexity and difficulties are high for the guys trying to develop elegant and effective AV solutions to these problems.
We’ll see more of this obfuscation topic, but from the “hackers” perspective, when defcon’s “Race To Zero” contest is held this fall.

The group continues to grow in its formative stage:

“AMTSO is dedicated to helping improve the objectivity, quality and relevance of anti-malware technology testing. AMTSO membership is open to industry-wide academics, reviewers, testers and vendors, subject to guidelines determined by AMTSO.”

The press is catching the buzz as well with articles at SecurityFocus, Fox Business, InformationWeek, SCMagazine and the Washington Post.
We look forward to further helping open efforts to better evaluate and understand security solutions as an AMTSO member.

Update 5/9/2009 — see ongoing participation and group progress here.

The world’s largest and smallest software security vendors and testing groups are working together to create this non-profit coalition of vendors, testers and academics. The group will be called the AMTSO, or the Anti-Malware Testing Standards Organization. The overall goal will be for the coalition to take on all challenges related to anti-malware security software testing, improving all aspects of the process. It will be a large task to set up standards, and PC Tools is pleased to take part in this effort.

The event was formative in nature, establishing temporary committees for most of the sessions before breaking off into the beginnings of some discussion and debate over technical matters and details that will come up in future meetings. Dr. Igor Muttik of McAfee’s AVERT Labs posted detailed information of the proceedings, for those interested.

We will keep you updated on this ongoing effort to improve the state of anti-malware security software testing.