News reports and Twitter chatter suggest thousands of Windows PCs in large organizations around the globe were thrown into fits of rebooting yesterday after antivirus giant McAfee distributed a routine update carrying an egregious error.
Now each one of those computers will have to be manually cleaned. Affected organizations can expect to expend a minimum of 30 minutes of manual labor per PC to get each one back into working order, says Steve Shillingford, CEO of tech forensics firm Solera Networks.
"There's no way to automate the process," says Amrit Williams, CTO of security management system company Big Fix. "It will take however long it takes to touch each single machine. The companies affected by this could be dealing with this for days or weeks."
In a blog posting late Wednesday, McAfee executive vice president Barry McPherson said "less than one half of one percent of our enterprise accounts globally" were affected. "McAfee teams are working with the highest priority to support impacted customers," he says.
The incident unfolded after McAfee somehow classified a well-known, legit Windows operating system file, called svchost.exe, as a malicious program. Svchost.exe has long been a crucial part of the Windows operating system. Without it, a PC cannot be networked with other PCs.
Legit files, like svchost.exe, can get intermingled with the tens of thousands of slightly different variants of malicious programs antivirus researchers cull through each day, says Immunet CEO Oliver Friedrichs. "It doesn't help that some viruses actually masquerade as svchost.exe, leading to confusion and the submission of the legitimate svchost.exe process for analysis," says Friedrichs.
But quality assurance testing processes are well developed and most of the time prevent antivirus companies from designating legit files as a "false positive" virus that ends up quarantined or scrubbed out. "As for why the false positive was not detected during quality assurance, McAfee will have to answer that," says Friedrichs. "I can definitely sympathize with McAfee. Nobody wants to have this problem while striving to protect people."
McAfee declined to answer questions, instead directing reporters to McPherson's post. At about noon Pacific time on Wednesday, McAfee sent updated virus signatures to its corporate clients around the globe. This is all part of a time-honored cat-and-mouse game in which hackers create slightly different versions of computer viruses -- thousands of new variants each day. Antivirus companies compete against each other to be the first to detect the latest variants. They then hustle to create fresh virus "signatures," then push out these protective signatures to corporate customers.
A standard test — running the update on an in-house Windows PC before distributing it to customers — should have caught the glitch, says Big Fix CTO Williams. "It's very basic testing, not something weird or intricate," says Williams, who previously worked at McAfee. "The fact that McAfee didn't see this as part of normal testing is really shocking."
Solera Networks, a supplier of network forensics technology, says it helped one large U.S. multi-national company quickly determine that the poisonous update from McAfee threw 50,000 of its PCs into a rebooting frenzy. McAfee advised the company that "remediation time is estimated to be 30 minutes per user, " says Solera CEO Shillingford.
"Estimating $100 per hour, this organization's lost time alone can be conservatively estimated to cost more than $2.5 million," says Shillingford. "And that does not factor in lost productivity while users are down."
Security experts say false positives are impossible to completely eliminate in the frenetic cat-and-mouse world of antivirus protection. But McAfee's gaffe suggests traditional antivirus signature protection may be at its limits, says Ashar Aziz founder and CEO of network security firm FireEye.
"While I'd like to say this is an anomaly this has happened to several other antivirus vendors and the problem is that antivirus is an antiquated technology that is requiring them to literally process tens of thousands of malware daily," says Aziz. "What we are seeing is that this technology framework is collapsing under the weight of maintaining a broken signature approach to security."
Source: USAToday