Recommendation Evaluation Framework, Version 2

Evaluating Solutions for Personalization and Recommendations

September 22, 2011

Recommendation solutions are hot among online marketers and customer service executives, Recommendation engines increase order size, improve search results and increase personalization. We identify the key criteria for selecting a recommendation solution. Use this framework as you evaluate recommendation solutions for your organization.

NETTING IT OUT

Recommendations are hitting their stride. A decade ago merchants and publishers saw the recommendations on Amazon’s site and wanted it. A decade ago they had to spend a fortune to get it. Today, recommendation engines can be had for some thousands of dollars a month and can be implemented in a few weeks. My guess is that recommendations will be ubiquitous within the next two to three years. I’m always optimistic on these guesses, but since recommendations are widely available as a service, rollout can be very swift.

SaaS offerings greatly simplify the technology aspect of implementation and virtually eliminate up-front costs, making it relatively easy for customers to sign up with a vendor. Nevertheless, choosing a vendor wisely is always better than choosing often. To aid in that choice, I offer a set of requirements and evaluation criteria set forth in this evaluation framework. This framework updates and replaces the recommendation evaluation framework published in January, 2010. I will be using the framework to evaluate a number of the leading products using the framework during 2011-12, leading to a detailed comparison.

WHAT’S INTERESTING ABOUT RECOMMENDATIONS

What Are Targeting and Personalization

Targeted marketing selects content (including products and offers) for consumers based on traits such as consumer context or behavior. The consumer is likely anonymous. Personalization requires a consumer profile that includes traits such as demographics, purchase history, and behavior, and this profile is used by the algorithms that select the content to be presented. While registering and logging in increases the customer profile data, the consumer is most often identified by web browser cookies and an associated identification number. Without registration, it is likely the profile does not contain user information such as name and address, so consumer anonymity is retained.

What Is Recommendation Technology

Web site owners worldwide have yearned for recommendations ever since Amazon started telling us that “people who bought this also bought that” and today tell us “52 percent of people who looked at this bought that; 26 percent bought this other thing.” A decade ago marketers had to spend a great deal to implement recommendations on their sites. Today, recommendation engines can be had for some hundreds of dollars a month and can be implemented in a few days. Don’t you just love SaaS?

For the user, recommendations on a site can mean pretty darn good search results, much of the time; emails with interesting offers; banner ads that catch your eye; useful guidance when you are selecting content, whether it’s a digital camera, a news article, a problem resolution, or a research document; clicks that take you from Google right to the content you need.

For the content owner—whether merchant selling products, publisher presenting articles, or marketer presenting offers—recommendation technology means delivering the most attractive item in those few seconds before you lose your audience’s attention.

A recommendation engine can double or quadruple the click-through rate as compared with the recommendations selected by the experts—the merchandisers or researchers or support specialists. It typically has significant impact on revenue, time on site, employee productivity, and customer satisfaction.

Where Are Recommendations Used

Because recommendations had their most visible debut in ecommerce, people tend to associate recommendations with shopping. In the ecommerce arena, recommendations are used most often on product pages, shopping cart pages, category pages, and order confirmation pages. Recommendations are also used to tailor the content of those emails that entice you to stop work for a moment and shop. As a marketing team becomes more familiar and more confident with using recommendations, they expand recommendations to cover more of the interactions across the customer lifecycle.

But I think product recommendations are the tip of the iceberg for recommendations. They belong everywhere content must be winnowed for a user, or everywhere that personalization improves a user’s productivity or experience. For example:

  • A news site that knows I love football and don’t care about rugby, and always shows me the most interesting world news
  • A personalized view of the corporate intranet that highlights my department, my division, and my projects
  • My view of corporate research knowledge bases, weighted to what I’m working on
  • Web sites that deliver coaching, e.g., for runners, dieters, investors, tailored to the my style and goals
  • My personalized support portal to the corporate help desk
  • My company’s portal to a key supplier, e.g., Cisco, personalized by role or person
  • My dashboard with my KPIs and corporate reports, with the hottest items on the first page
  • A personalized investment site, e.g., Fidelity, tailored to the kind of investor I am
  • A personalized commerce site tailored to my relationship, e.g., the car I drive, the sports I play
  • Billing inquiry that always shows the disputed bills first
  • Order history inquiry that always shows the most referenced order first

My guess is that recommendations will be ubiquitous within the next two to three years. I’m always optimistic on these guesses, but since recommendations are widely available as a service, rollout can be very swift.

The technology effort, and the up-front costs, for deploying recommendations are relatively small, which makes it reasonably easy for customers to sign up for a recommendation service. The greatest effort for recommendation customers seems to be in learning how to use them, where to use them, and how to use them effectively—skills which will be portable to any vendor’s solution. Nevertheless, choosing a vendor wisely is always better than choosing often. To aid in that choice, I have compiled a set of requirements and evaluation criteria set forth in this evaluation framework. And I will be evaluating a number of the leading products using the framework during 2011 and 2012, leading to a detailed comparison.

REQUIREMENTS

The requirements for recommendation services are derived from customer requirements. “Customers” for recommendation services cover several roles, including the end-consumers of recommendations, the business people managing recommendations, and technical staff. Their requirements generate the evaluation criteria which are listed in the Table.

Recommendation Consumer Requirements

People consuming recommendations—the visitors, shoppers, readers, researchers to whom content is being recommended—of course, need relevant and enticing recommendations, that’s the whole point. But they also need privacy and may wish they had some control over what personal information is used and how it is used. As recommendations are increasingly used to personalize experiences, consumers may also want a mechanism to indicate the persona they represent. “Don’t recommend for me, recommend for my [boss, niece, colleague].”

Recommendation Manager Requirements

The marketers, merchandisers, editors, business analysts and other business people who are using recommendations as a tool to improve user experience have a broad range of requirements, starting from the initial deployment.

The business people who are responsible for recommendations need help making great ones, including:

  • Guidance on how to deploy recommendations effectively
  • Advice on how to increase recommendation effectiveness
  • Training and tools to track and analyze recommendation effectiveness

Business people need tools to deploy, test, analyze, and optimize recommendations...


Sign in to download the full article

0 comments


Be the first one to comment.

You must be a member to comment. Sign in or create a free account.