Things have been buzzing on our private email listserv over the past two months. Peter Horne, one of the most active members of Patty’s Pioneers*, began discussing a troubling problem he had found on a Lenovo computer he purchased in Sydney, Australia in early January, 2015. Pete quickly discovered malware on his new computer. He realized that this malware—Superfish Adware—had been pre-installed at the Lenovo factory as part of the Lenovo additions to the pre-installed version of the Windows operating system. He found that the Superfish Adware had compromised the Windows network software at a very low level, allowing it to insert its own script into every single page viewed by a browser. It was at such a low level that it did not matter which browser was used—Explorer, Chrome, or Firefox—it was the operating system that was compromised. Furthermore, it was so deep in the operating system that neither McAfee, Trend Micro, nor the Microsoft malware removal tool, found the Superfish software.

Customer Tried to Alert the Company; But Was Ignored

Peter reported the infected computer to the store, and they contacted their Lenovo sales rep. However, Lenovo had a policy of not talking directly to customers about store enquiries, and he waited. Nothing happened, and so he logged his own call with the Lenovo Help desk.

But, this was all to no avail. Repeatedly, company spokespeople told this savvy customer, who was only trying to help, that he was mistaken. Nothing like this could possibly be happening. “Lenovo doesn’t distribute Malware.” Pete offered to walk the Lenovo product manager through the process to demonstrate the existence of the Malware, but nobody ever got back to him. In the end, the store manager refunded Pete the money because he was convinced of the issue himself, and he wanted to keep a valuable customer who had purchased many items at the store in the past with no problems.

While he was getting the run around from Lenovo, Pete also did a fair amount of time-consuming due diligence. He checked computers at Lenovo stores in four cities around the world. He asked other Pioneers to check their own machines and at local stores.

If Lenovo’s management had paid attention to the customer feedback from Pete and other customers, their security team might have discovered the issue, quietly dealt with it, and avoided the ensuing uproar.

Customer Alerts the Press

Pete was troubled. He’s also a busy guy. He was tempted to move on, but was troubled by the fact that less tech-savvy consumers would be buying a spyware-infected computer. He reached out to the other members of the Pioneers’ forum, including my brothers, Jonathan and Andy Seybold, who encouraged him to get the word out, and they helped by contacting reporters they knew at The New York Times.

Luckily, a tech-savvy reporter, Nicole Perlroth, paid attention, interviewed Pete, and began doing her own investigation.

Other reporters also got wind of the story. The first article that appeared was written by Timothy Seppala for Endgadget.com. New Lenovo PCs shipped with Factory-Installed Adware appeared at 1:25 am on February 19^th. Timothy based his story on the user discussions about this adware he found on the Lenovo Forums. It was also discovered that Superfish used a product from Komodia that corrupted the machine’s trust store—the store of certificates that vendors include that certify that SSL connections can be trusted.  The Komodia certificate opened all infected computers to “man-in-the-middle” attacks—an attack that allows bad guys to impersonate the sites you trust and capture your traffic.

Nicole Perlroth’s first New York Times article appeared online at 7:44 pm on February 19, 2015, Researcher Discovers Superfish Spyware Installed on Lenovo PCs, and in the print edition the next day. Essentially the same story was published as “Spyware Is Found Installed on PCs Made by Lenovo,” as well as in newspapers around the world, since it was submitted to, and distributed by, the Associated Press. It was Peter Horne who revealed to Nicole the darker truth—it wasn’t just that adware was being pre-installed inside the machine's operating system—it was tracking every single page and image a user was looking at, and sending all the metadata to the Superfish servers! And it could not be turned off.

Once the story was out, a feeding frenzy quickly spawned lots of follow-on articles, among them:

• Ars Technica: Lenovo PCs Ship with Man-in-the-Middle Adware that Breaks HTTPS Connections, Feb. 19^th and then updated

• C/Net: Lenovo's Superfish Security snafu Blows up in its Face, Feb. 20^th

• Tripwire.com: Superfish-Lenovo Adware FAQ, Feb. 19^th and then updated

• Wired.com: Lenovo’s Response to Its Dangerous Adware Is Astonishingly Clueless, Feb. 19th

• NBC News: Lenovo Made Laptops Vulnerable to Hacking, Feb. 19^th, and Government Urges Lenovo Computer Owners to Remove Superfish Software on Feb. 20^th.

• Mashable.com: Department of Homeland Security urges Lenovo Users to remove Superfish

• Endgadget.com: How could Lenovo miss its Superfish security hole? February 20, 2015

• And many more….

As part of her due diligence, Nicole Perlroth of The New York Times interviewed Lenovo CTO, Peter Hortensius, and asked him why the company had ignored the issue when it was reported by an obviously concerned and knowledgeable customer. Here’s his reply:

“Q.

Peter Horne, the technologist that first alerted me to this issue, said he alerted Lenovo about the security issue through your customer service channels in mid-January, and nothing was done until now. When did you first learn that this practice was unacceptable to your customers? When did you take action?

A.

We first got complaints in December, but they were more about web compatibility. Customers were saying “Hey, I did this and I got that back, what’s going on?” In January, we concluded [Superfish] was not going to deliver the experience we had wanted. At that point, we had Superfish shut down, and shut down the servers on their end.

Unfortunately that’s not what the security exposure was motivated by. That was motivated by the certificate that was created. That we really did not know until last Thursday, midday.

Q.

I have to press you on that. Mr. Horne brought the security issue to Lenovo’s attention in mid-January, more than six weeks earlier.

A.

At that time, we were responding to this issue from a web compatibility perspective, not a security perspective. You can argue whether that was right or wrong, but that’s how it was looked at. We thought turning off the servers at that point would address that problem and that was what was done. At that point, we concluded [Superfish] was not very useful and that is why we started to remove it from the preloads.

…….

Q.

How did you miss the fact that Superfish was hijacking the certificates?

A.

We did not do a thorough enough job understanding how Superfish would find and provide their info. That’s on us. That’s a mistake that we made.

Q.

By simply unplugging the Superfish servers, you did not address that issue.

A.

That’s exactly correct. In January, we turned off the servers to respond to the compatibility concern. But unfortunately that did nothing to solve the security problem, which is that someone could hijack the certificate. The actions we took on Thursday and Friday to remove the certificate, and remove all traces of the application, that is what solved the security problem.

Q.

Were you aware that Superfish was using Komodia to serve its certificates?

A.

We were told by Superfish that they were using Komodia but we never looked into it. In December, there was no reason for us to be suspicious. Superfish had a good reputation. But we should have dug in more. I won’t debate that."

Hong Kong-listed Lenovo’s stock price dropped 2% on Tuesday, Feb. 24^th after this interview was published.

The Damage Continues

Lenovo’s stock price has been hit. The company is now facing lawsuits. The Lenovo websites are under siege. Many customers have decided they won’t ever trust the brand again, for either consumer or business computers.

And there’s more troubling information about to come to light (stay tuned).

Peter Horne is raising some additional questions:

• What’s happening to the massive amounts of personal data that has already been siphoned off by these services for anyone who is using one of the affected models of a recently purchased Lenovo consumer PC?

• Why is it so easy to spoof the supposedly secure Certificate Authority on which our global e-commerce infrastructure is built? Look at how corrupted the Certificate Authority process is. This incident highlights its incredible flimsiness and vulnerability.

• What is visual search, where did it come from and how is it being used? If Superfish is collecting the photo DNA of all the photos your mouse touches and combining that information your internet session data, and mining that data, this is a huge invasion of privacy.

• Who are these companies, Superfish and Komodia, and who are the people behind them? Executives at both companies are open about their backgrounds in intelligence work in Israel, their work for intelligence specialist companies, their work on intelligence contracts, and the decision to move their operations to the U.S. Why haven't they said anything about their products and services?

The Moral of the Story: Listen to What Your Customers Are Trying to Tell You!

Don’t ignore your customers’ attempts to warn you about a product or a process flaw that will damage your reputation! To their credit, Lenovo executives have finally reached out to Peter Horne (and probably other smart customers) and asked for their help in keeping similar problems from happening in the future. After all, if you have smart customers, why not harness their intelligence to keep you out of trouble?

*Patty’s Pioneers is a group of our customers—tech-savvy IT architects—who have been hanging out electronically and meeting twice a year for over two decades. I learn incredible amounts from participating in these wonderful, rich, conversations whose topics range broadly from organizational issues, to tech industry personalities, to trends in IT architecture, implementation, and adoption, to financial markets and philosophy.