Customer portals make it easier for your customers to do business with you. They’re portals that deliver your online customer experience. They support all of your customers’ Customer Scenarios®, providing access to all the resources that your customers need in one convenient, custom-tailored environment.

We’ve developed and refined a framework to help you evaluate and select the portal technology platform that’s best for your customer portal. We’ve also evaluated seven portal platforms against the framework and published reports documenting those evaluations.

In this report, we describe Version 2 of our framework for evaluating portal platforms for customer portals and explain how and why we’ve refined it.

Briefly, Version 2 of our customer portal platforms evaluation framework contains the following four major criteria:

* Technologies to support Customer Scenarios (portlets, UI content management, document content management, search, personalization, and collaboration)

* Analytic Functionality

* Architecture

* Product Viability

WHAT ARE CUSTOMER PORTALS? AND WHY ARE THEY HOT?

Customer Portals Make It Easier for Your Customers to Do Business with You

Customer portals make it easier for your customers to do business with you. Instead of forcing customers to traverse different and separate business unit-specific, product-specific, process-specific, or task-specific Web sites and Web applications to explore, buy, use, maintain, upgrade, renew, and replace products and services, savvy companies now can offer their customers a single Web “place” that can support all of their activities throughout their lifecycles--a customer portal. The advantages of customer portals are convenience and relevance--they contain everything customers need in one online place (no need to navigate across multiple Web sites and logins). And, in a well-designed customer portal, your customers “see” only those products and services that are relevant to their contexts.

Customer Portals Defined

In our previous research, we’ve defined portals as user experiences targeted at specific constituencies--customers, partners, and your internal users--that perform the following tasks:

* Portals aggregate and deliver information from a variety of internal and external resources.

* Portals wrap up and present application functionality from existing internal IT systems and from external applications.

* Portals enable single sign-on and authentication for the applications and resources that end users access via the portal.

* Portals offer customization capabilities to portal owners and basic personalization capabilities to end users[1].

Customer portals extend this definition. They deliver all the portal services described above, but they are specialized for the customer constituency, as follows:

* The audience for customer portals is a specific group of customers (e.g., small business customers or individuals in a specific client account).

* Customer portals are an integral component of the cross-channel, cross-lifecycle customer experience that you deliver. They support the Web channel, and they support activities in every phase of the customer lifecycle.

* The activities supported by customer portals are your customers’ key Customer Scenarios®, the sequences of activities that customers perform to achieve their objectives. (See “Customer Portals and Customer Scenarios,” below.) We say that customer portals contain your customers’ key Customer Scenarios.

* Role-based portal access (via single sign-on) determines the particular Customer Scenarios that are featured on the portal.

Portals for your partner and internal user constituencies remain very important. We recommend that you align your partner portals with the same scenarios that your customer portals support and add partner scenarios[2]. You can use the same principles to design your employee portals: Focus on what different groups of employees need to support customers and partners, then add the scenarios employees need to do their jobs and manage their work life. Since customers drive your business, it makes the most sense to start with customer portals. And customer portals will have the most significant impact on your business. They make it easier for your customers to do business with you.

CUSTOMER PORTALS AND CUSTOMER SCENARIOS

Customer Portals Contain Customer Scenarios

When we say that customer portals should contain the most common Customer Scenarios used by the customers for whom you design them, we mean that customer portals help your customers accomplish what they want to do and how they want to do it. For example, at a customer portal designed to support small business customers of a technology provider, customers could accomplish the following:

* Manage their accounts, updating contact names and email addresses

* Find and buy new products and services

* Resolve a problem or track an issue

* Get self-service support for upgrading existing products and services

* Get tips or join a peer group

* Find a local expert or partner

* Renew subscriptions or licenses

* Replenish supplies

We can’t list the specific Customer Scenarios for all of the customer segments in all companies, and we can’t evaluate how well portal platforms support those Customer Scenarios. But we can identify the general types of Customer Scenarios that customer portals should contain, and we can evaluate how well portal platforms support them, because it’s a much shorter list. In fact, support for Customer Scenarios is the key customer portal evaluation criterion.

In Table A, we list what customers want to accomplish in these general Customer Scenarios in the left column. Then we use our Customer Scenario Mapping® methodology and derive at least one “We should…” for each “I want…” We put these--the portal capabilities necessary to support what your customers want to do--in the right column of Table A.

Containing and Supporting Customer Scenarios
Please download the PDF version to see the table.
Table A. The capabilities that you should provide in your customer portal for each of your customers ’ Customer Scenarios are shown in this table.

How did we determine that the types of Customer Scenarios listed in Table A are those that should be contained in your customer portals? Easy--we spoke with your customers. Many of our consulting engagements (actually most of our consulting engagements) deal with the customers of companies like yours. Through our group interviews and Customer Scenario Mapping techniques, we discover how they want to do business with you. Using customer portals to accomplish the Customer Scenarios listed in Table A is the way that customers want to accomplish their online, self-service business. And you should support those Customer Scenarios by implementing the capabilities shown in the right-hand column of Table A.

BUILDING CUSTOMER PORTALS

Do Customer Portals Need to Be Built Using Portal Platform Technology?

A customer portal, as we’ve just defined it, can be built and delivered using many types of Web infrastructure technology. You don’t have to build your customer portals with portal platform technology. In fact, many companies have delivered account-specific, customer-specific, or role-specific customer self-service scenarios without using portal platforms. In the late 1990s, Dell built its original account-specific Premier Pages without the benefit of portal technology. Microsoft and National Semiconductor rolled out hundreds of account-specific Web sites without the benefit of portal technology--each site was customized by an account manager for her specific customers. Most banks and financial services institutions have been offering secure personal account management Web sites for years without the benefit of portal platform technology.

Portal Technology Platforms Make It Faster and Easier to Design and Deploy Customer Portals

But portal technology platforms make it easier to design, develop, deploy, and maintain customer portals. These platforms package facilities and tools that give you a big boost in building portals. Today’s portal platform technologies now make it easier than ever before to design, develop, deploy, and manage hundreds or even thousands of role-based, customizable, customer portals. Packaged in portal technology platform products are key components of customer portals, including:

* Content management facilities with lots of templates for the pages in your customer portal

* Search and navigation facilities to help your customers find the information that they need

* Collaboration technology that enables self-service to escalate to assisted service

* Portlets (mechanisms that integrate data)

* Single sign-on security facilities that let customers log in once, then access through portlets all the protected resources that they need to do their jobs

And you may not need skilled developers to build your portals. Many of these facilities have been designed for business users. (Really--it’s not just marketing hype!)

Many Portal Technology Platforms

While portal technology platforms make it easier and faster for you to build customer portals, choosing the platform that’s best for you isn’t that easy. There are many platforms from which to choose. That’s why we offer this evaluation framework--to make it easier for you to make this choice.

In our 2003 research series on portal platforms, we identified 19 portal platform suppliers and published evaluations and comparisons of five of them--the market leaders and technology innovators. Since then, there have been some changes in the market for portal platforms. Most significantly, we’ve seen market consolidation through M&A activity and the emergence of open source portal platforms. For example, BEA purchased Plumtree in December 2005, turning two leading suppliers into one. And, you’ve downloaded open source portal platforms literally hundreds of thousands of times.

In our 2005-2006 research series, we’ve evaluated these seven portal platforms.

* BEA WebLogic Portal

* IBM WebSphere Portal

* Liferay Portal

* Microsoft SharePoint Services and SharePoint Portal Server

* Oracle 10g Application Server Portal

* SAP NetWeaver Portal

* Vignette Portal

Now, let’s take a closer look at how we’ve evaluated them.

A FRAMEWORK FOR EVALUATING PORTAL TECHNOLOGY PLATFORMS

PSGroup Evaluation Frameworks

We can help you select the portal technology platform that best addresses your customers’ requirements--the platform that best supports the content, data, and applications that your customers want to access to perform the activities of their Customer Scenarios.

Through our work with you, our work understanding portal software solutions for customer portals, and our ongoing customer-centric analyses, we’ve identified the requirements for customer portals. And we’ve created and then refined a framework that will help you determine how well portal technology platform products address those requirements. Like all of our product evaluation frameworks, this one delivers the following significant benefits:

* It shortens your time to market.

* It saves time and reduces your costs in product evaluation, comparison, and selection.

* It reduces your risks in product evaluation, comparison, and selection.

The framework enables you to make apples-to-apples comparisons of the most important product evaluation factors. Then our product evaluations against the framework speed and simplify your work even more.

Evaluation Framework for Customer Portals

Version 2 of our framework for evaluating portal platforms for customer portals platforms has these four top-level evaluation criteria:

* Technologies for supporting Customer Scenarios
* Analytic functionality
* Architecture
* Product viability

The portal technologies for supporting Customer Scenarios are portlets, UI content management, document content management, search, personalization, and collaboration. We show the top level criteria and subcriteria in Illustration 1.

Customer Portal Platform Evaluation Framework
Please download the PDF version to see the illustration.
Illustration 1. This illustration shows the top-level criteria and their subcriteria for our customer portal platform evaluation framework.

WHAT’S NEW IN VERSION 2 OF OUR FRAMEWORK?

Refinements to Improve and Shorten Evaluation and Selection

Support for your customers’ Customer Scenarios is the key requirement for the portal technology platforms that you use for designing, developing, and deploying your customer portals. You support these Customer Scenarios by implementing the technologies listed in the right-hand column of Table B.

* PORTLETS. We change our emphasis on architecture to focus on reusing and managing portlets rather than on their internal structure.

* CUSTOMER PROFILE MANAGEMENT. Portal platforms let your customers access the information that you manage about them. The platforms don’t manage customer profile attributes beyond those needed for security and for collaboration, and all of the platforms manage the same attributes. So, we’ve dropped this technology that supports Customer Scenarios.

* PROCESS MANAGEMENT SERVICES. Process management is an important technology for supporting Customer Scenarios, but none of the portal platforms that we evaluated packaged process management functionality except for supporting content approval workflows. You have to purchase and implement an add-on product to get process management capabilities, even if it’s a tightly integrated add-on offered by the portal platform supplier. So the requirement and, therefore, the evaluation criterion is for accessing these capabilities from within a portal, typically through a portlet, ideally a packaged or third-party portlet.

* CONTENT MANAGEMENT SERVICES. Portal platforms manage two types of content: the content that makes up its UI and the content that contains the answers to customers’ questions and the solutions to their problems. You’ve got to manage both kinds and you’ve got to manage them separately. Most portal platforms manage them separately, too.

* SEARCH. We had covered navigation within the search criterion. We’ll move that evaluation to UI content management. We’ll also add a taxonomy sub-criterion to search.

* ANALYTIC CAPABILITIES. In our original framework, we distinguished between real-time analytics and analytics against logged data. We eliminate that distinction in Version 2. Analytics of any type are virtually absent in portal platforms.

* PRODUCT VIABILITY. We made our product background evaluation criterion within product viability a little more specific. Within product background, we present the current version and its introduction date and the list of previous versions with their introduction dates. We also note any acquisitions by the supplier that contribute important technology to the portal platform and we describe the supplier’s approach to development, internal vs. outsourced, for example. This information can answer questions like how frequently to expect new versions that may entail significant migration or how much a supplier’s R&D organization can contribute to support and planning (not much if it’s outsourced).

* COMPANY VIABILITY. Four of the suppliers of the portal platforms that we evaluated are: IBM, Microsoft, Oracle, and SAP. There’s no question about their viability. Of the others, BEA and Vignette are certainly going concerns. At the other end of the viability spectrum is Liferay, a privately-owned, 16-person, supplier of services for implementing its open source portal platform. There’s little we could add in evaluating Liferay’s viability beyond stating its size and its participation in the open source market. So, we’ll drop this evaluation criterion.

Now, let’s look at the complete framework.

TECHNOLOGIES FOR SUPPORTING CUSTOMER SCENARIOS

Support for your customers’ Customer Scenarios is the key requirement for the portal technology platforms that you use for designing, developing, and deploying your customer portals. You support these Customer Scenarios by implementing the capabilities listed in Table B. Note, though, that portal technology platforms don’t package the functionality to implement these capabilities directly. Rather, they package general-purpose facilities that you customize using toolkits that they also package.

Technologies for Supporting Customer Scenarios for Customer Portals
Please download the PDF version to see the table.
Table B. The portal technology requirements necessary to support the capabilities that you should provide in your customer portal for each of your customers’ Customer Scenarios are shown in this table.

So, if the key requirement for portal technology platforms is Customer Scenario support, then the key criteria for evaluating the portal technology platforms are focused on their general-purpose facilities and toolkits.

We want to see facilities and toolkits that are geared for creating customer portals. General-purpose facilities offer the most flexibility, enabling you to create partner and employee portals as well as customer portals. But, if a portal technology platform’s facilities and toolkits are specialized in any way for customer portals, then your efforts to create and deploy them will be faster and simpler.

Building on the Customer Scenarios and capabilities listed in Table A, we’ve identified the types of technologies that you’ll need to support Customers Scenarios in your customer portals. They’re shown in the right hand column of Table B. Following Table B, we describe these technologies in a little more detail.

Also, because a customer portal is a Web-based place for customers to perform the tasks of their Customer Scenarios, UI content, and UI content management are essential technologies.

Combining the analysis offered in Table B with UI content, we identify these technologies as critical for supporting Customer Scenarios in your customer portal:

* Portlets
* UI content management
* Document content management
* Search
* Personalization
* Collaboration

PORTLETS

Portlets is the most important evaluation criterion of our framework. Portlets are the mechanisms for accessing data, content, and applications and for presenting UI content within customer portals. You’ll use portlets to facilitate your customers’ access to customer, order, incident, and contract data, their access to the services and functions of your applications, and their access to the content that contains the answers to their questions and the solutions to their problems. These are the evaluation criteria for portlets:

* Packaged portlets
* Third-party portlets
* Portlet development tools
* Development samples
* Portlet management
* Portlet-to-portlet communication
* JSR 168 support
* WSRP support

Packaged Portlets, Third-Party Portlets

Portal platforms should package portlets that integrate a wide range of applications, data, and content, including the popular RDBMSs and applications, as well as the particular database implementations of these popular applications. Secondary applications, data sources, and content sources should also be accessible through prebuilt portlets. Bundled portlets are the nicest approach to these, but availability of portlets through third parties is pretty good too--certainly better than having to build them yourself.

Portlet Development Tools, Development Samples

But, for those portlets that you’ll have to build, portals should also package toolkits and APIs that let you extend packaged and third-party portlets or let you build custom portlets. Ideally, a portal has multiple portlet development toolkits, each designed for a different level of development skill and experience. And the portal should package reusable samples of portlet components built with each of its toolkits.

Portlet Architecture

Portlet architecture describes their interfaces, organization, internal structure, and properties. You’ll be reusing and extending packaged and third-party portlets to address most of the requirements for your customer portal. With the emergence and broad adoption of the Java portlet specification, JSR 168 portlet interfaces, organization, and internal structure have become standardized across portal platforms. And the legacy portlet architectures of portal platforms are being phased out, especially as introduction of the JSR 186 follow-on, JSR2 286, nears. So, what matters in portlet architecture are portlet properties, the attributes you’ll use to manage portlets and to promote their reuse.

Portlet-to-Portlet Communication

You might like to wire multiple portlets together to perform a compound task for a customer or to help automate a series of tasks in a Customer Scenario. So portlets should be able to get their input from either your customers or other portlets, and to deliver their output to either your customers or other portlets. Ideally, you’d like to let your business users do this wiring, especially when they’re working with packaged portlets or with portlets that they’ve built.

Unfortunately, portlet-to-portlet communication is not supported within JSR 168. So, if this communication is supported, it’s implemented using portal-specific mechanisms. Note however, that JSR 268 will support portlet-to-portlet communication. For now we’ll examine the portal platform-specific mechanisms. In future versions of the framework, we’ll place more emphasis on JSR 286 compliance.

Of course, portlets in Microsoft SharePoint Services and SharePoint Portal Server aren’t going to support any Java standard. But, you’ll choose the Microsoft portal platform at a higher level than portlet-to-portlet communication capabilities.

JSR 168 for Portability

The Java Portlet Specification, commonly known as JSR 168, defines a standard for the architecture (interfaces, modes, states, context), processing, relationships, security, deployment, and user interaction of portlets for J2EE environments. Portlets developed on the JSR 168 standard are potentially portable across portal platforms. This portability delivers significant benefits:

* More productive development because code, developer skills, and development practices are reusable.

* Shorter deployment times because widely existing JSR 168 portlets can be reused in any J2EE portal implementation.

Think about it this way. With JSR 168, the packaged and third-party portlets of one portal platform could be reused in the deployment of another.

JSR 168 was released on October 7, 2003 and since that time it has been adopted by just about every J2EE portal platform. Of the portal platforms that we evaluated, only SAP NetWeaver Portal does not currently support JSR 168, but SAP promises that support is coming soon.

Unfortunately, because most portal platforms predate JSR 168, they also have legacy, portal platform-specific portlet interfaces and most of their packaged portlets are built on these legacy interfaces. It will take some time for the legacy portlets to be replaced with standard portlets but the migration is happening.

As we mentioned in portlet-to-portlet communication above, JSR 168 will soon be replaced by JSR 286. JSR 286 will specify richer portlet architectures and processing. We anticipate immediate wide acceptance but phased adoption and implementation as suppliers plan migration from both their legacy portlets and their JSR 168 portlets.

JSR 168 support is important for the benefits that it can deliver. We feel that it’s an important requirement for J2EE portal platforms. Of course, JSR 168 does not even apply to Microsoft SharePoint Services and SharePoint Portal Server portals. You’ll make a higher-level selection decision between J2EE and .NET. Given a J2EE decision, JSR 168 support will help you differentiate J2EE portal platforms.

WSRP for Integration and Distributed Processing

WSRP, Web Services for Remote Portlets, is a specification published by the Web Services for Interactive Applications (WSIA) and Web Services for Remote Portals (WSRP) OASIS Technical Committees in August 2003 that defines a “Web service interface for accessing and interacting with interactive presentation-oriented Web services.” Portlets are examples of presentation-oriented Web services, and usage scenarios for WSRP include portlets consuming Web services produced by Web services providers.

Web services are built on platform-independent technologies: SSL/TLS, URI/URL, and SOAP. As a result, WSRP is a broader specification than JSR 168. Web services producers and consumers may be implemented on different platforms. For example, a portlet in a J2EE portal could consume a Web service produced by a .NET Web service provider.

WSRP support is an important distributed processing and integration mechanism for portal platforms. It enables portlets to access external applications and data through standard, Web services interfaces, and most commercial software applications, especially those that you’ll want to access through your customer portal, are Web services producers.

UI CONTENT MANAGEMENT

Evaluation Criteria

Within UI content management, the evaluation criteria are listed below and described in the sections following.

* UI content management approach
* UI content model
* UI content management services
* Metadata for managing UI content
* Globalization/localization
* Samples and templates

UI Content Management Approach

Unfortunately, all of the portal platforms that we evaluated manage their UI content with an embedded portal-specific content management system--a content island. Some of the platforms offer tight integration between their UI content management systems and popular or richer Web content management systems. This integration can be useful if you’re using one of these systems. Conversely, they’re not so useful if you’re using something else.

When portal technology platforms are integrated with external content management systems, you rely on those systems to provide most of the design time content management capabilities--from integration with authoring tools to globalization/localization to content staging and testing--and you rely on the portal technology platform for samples and templates. The points of integration are content publishing and metadata for findability. The content management system must be able to publish to the particular portal platform, and the portal technology platform’s metadata model for findability metadata must be supported by the content management system.

Your customer portal might not have the size and scale to justify the additional purchase of a separate content management system, but, if you’ve already implemented a content management system that supports Web content, then you’d sure like to be able to leverage it in your customer portal implementation. On the other hand, if you don’t already have one, then you’d really like the portal platform that you select to package the basic content management capabilities--access control, version control, staging, testing, and publishing--so that you don’t have to custom-build these capabilities or run with the exposure of not having them.

Within this evaluation criterion, we’ll identify the approach and list and describe the integrations. You’ll be able to determine how isolated the content island of portal UI content will be in your installation.

UI Content Model

UI components, metadata attributes for their management, and the relationships between them make up the UI content model. You’re looking for a small set of coarsely grained component types, a large set of predefined properties for each type, and hierarchical relationships between types. You should find content types for look and feel content types, navigation types, portal services presentation types (usually portlets), and static and dynamic content presentation types. The UI content model should facilitate your ability to manage UI content. Component properties should be complementary with content management services and globalization/localization.

UI Content Management Services

You need a wide array of services to support UI content. Essential services are creation, replacement, update, and delete of content objects, check-out/check-in and version control, content lifecycle with multiple phases, content staging, publishing, and retirement. Some nice-to-have services include approval workflows.

Metadata for Managing UI Content

The metadata for managing UI content are a subset of the properties of UI component types. You’ll need properties to identify the content components and their versions, to identify their owners, authors, and modifiers, to specify their status and lifecycle phase, to specify their relationships to other UI content objects, and to specify their locale or their relationship to locales.

Globalization/Localization

You need to implement a customer portal that supports your global business. Customer portal content must be localizable, and customers should be able to perform their Customer Scenarios in the locale of their choice. Content management capabilities must include localizable content. Customers new to your customer portal should be able to select a locale. Personalization (see below) should include the automatic selection of locale for returning customers.

Samples and Templates

The UI content of your customer portal will be different in organization and structure than the UI content of individual Web applications. For example, your ecommerce system manages entire Web pages, but if you implement that ecommerce system within a portal, then the portal platform will take over the Web UI and will give the ecommerce system a portion of a Web page, likely a portlet. The organization and structure of portal UI content as well as the UI content model will be different. If your customer portal is your first portal implementation, then your content authors will have to learn a new approach to the Web UI. UI content samples and templates can make the portal’s UI easier to learn and, if the samples are reusable, then easier and faster to implement, too. You’d like to see lots of samples and template-based UI content development with lots of packaged templates.

DOCUMENT CONTENT MANAGEMENT

Evaluation Criteria

Within document content management, the evaluation criteria are listed below and described in the sections following.

* Document content management approach
* Document content model
* Document content management services
* Metadata for managing document content
* Metadata for finding document content
* Taxonomy
* Document Content Management Approach

Document content is very important in customer portals. Documents of various types contain the answers to customers’ questions and the solutions to their problems.

In addition to embedded, portal-specific UI content management systems, portal platforms also embed platform-specific document management systems. These document management systems might be the second content island introduced with your customer portal implementation. On the other hand, portal platforms might alternatively have unified systems for managing UI and document content or they might integrate a popular external document content system.

As with UI content, within this evaluation criterion, we’ll identify the approach, and list and describe the integrations. You’ll be able to determine how isolated the content island of portal document content will be in your installation.

Document Content Model

Documents and/or their constituent components, the types of documents supported, metadata attributes for their management and findability, and the relationships between them make up the document content model. You’re looking for a large set of supported document types (because you’d like to be able to use many of your customer service documents in your customer portal), a large set of predefined metadata attributes for the management, a metadata attribute type or two, typically, keywords, for their findability, and relationships document, perhaps within a folder hierarchy to facilitate both management and findability.

Document Content Management Services

You need a wide array of services to support document content. Essential services are check-out/check-in and version control, content lifecycle with multiple phases, content staging, publishing, and retirement. Some nice-to-have services include approval workflows. Customer portals don’t necessarily need services to create documents because you’re already creating documents effectively, probably in desktop applications. But, you need a customer portal document repository and the repository services to organize, add, delete, and copy/move documents in it. Generally, the more document content management services offered in a portal platform, the better. It’s not a problem to not use a service or two, but a lack of services will result in your spending time, money, or both to add them via purchase or customer development.

Metadata for Managing Document Content

The metadata attributes for managing document content are a subset of document properties. Like metadata for managing UI content, you’ll need metadata attributes to identify the documents and their versions, to identify their owners, authors, and modifiers, to specify their status and lifecycle phase, to specify their relationships to other documents, and to specify their language. It’s handy when the portal platform is able to populate document metadata automatically from existing documents.

Metadata for Finding Document Content

Metadata attributes for finding document content are the sets of terms or keywords that describe documents topics, subjects, or areas. They’re the terms or tags that you specify to classify and/or categorize documents. Specification and management of findability metadata are ongoing responsibilities. It’s time-consuming and tedious but it’s essential for findability. Sometimes, portal platforms can help by automatically generating these attributes as a result of the indexing process of its search engine.

Taxonomy

Taxonomies are collections of findability metadata terms that are associated with one or more document repositories or even with the entire portal. Taxonomies facilitate finding documents by browsing and searching. Terms in taxonomies correspond to document keywords.

Portal platforms should support multiple, hierarchical taxonomies. You’d like a separate taxonomy for each application, business area, or customer type supported by your customer portal. For example, the taxonomy that supports your home appliances would have different terms than the taxonomy that supports your industrial boilers. Hierarchical structure is important to support the way that many customers find information, browsing from high-level concepts to low-level detail.

Specifying and managing taxonomies are ongoing responsibilities. Like findability metadata for documents, this is time-consuming, tedious work. Sometimes portal platforms help by packaging predefined taxonomies for common business topics or by automatic generation through search indexing.

Document Content Storage

For this evaluation criterion, we describe where document content is stored. Document repositories in RDBMSs make it easier for you to manage documents.

SEARCH

Search is an important function within Customer Scenarios and, therefore, a requirement for any customer portal. Portal technology platforms should package search capabilities that allow customers to find the information that they need from within portal content and any other relevant content that you manage. Ideally, search should extend to any content anywhere on the Web that can help customers do their jobs.

Your customers continually teach us that good findability is critical to them. The more information that they can search from your customer portal, the more flexible the search strings that they can enter and the more forgiving your search engine is with handling those strings. The more relevant the search results, the better their experience.

Within search, the evaluation criteria are listed below and described in the sections following.

* Approach
* Search sources
* Source management
* Findability metadata
* Search queries
* Search results management
* Search analysis

Approach

We’ve seen two approaches to search in portal platforms. The first are embedded, portal-specific or document repository-specific search capabilities. The second are premium search engines bundled and integrated with the portal platform. We prefer the bundled and integrated premium search engines. They provide more powerful and more flexible search capabilities and make it easier for your customers to find the information that they need.

Search Sources

Search scope should be as wide as possible. The information that your customers need may be in the document repository of your portal, in some other document content system, or somewhere on the Web. You don’t want to replicate this information but your search engine should be able to index a lot of it from a wide range of search sources in addition to its document repository.

Source Management

Your portal’s search engine should let you control what it indexes and searches. It should also give your customers some control over search scope. As they become familiar with your document management approach, they’ll learn how to make it work best for them.

Findability Metadata

As we mentioned in Document Content Management, findability metadata are document metadata attributes (keywords) and the taxonomy terms. The indexing process of search engines can automatically discover these attribute terms within document content and can assign documents to taxonomy categories, and even discover document keywords. You can’t rely completely on the search engine to do these jobs, but search engines can make the work easier.

Search Queries

Customers want to search using keyword terms, sequences of keyword terms, and strings of text in their language. They want your search engine to help them hone-in on the search queries that find the content that they need by correcting spelling errors, automatically including related terms and synonyms, and by expanding search queries to include different forms of search terms. They also want to be able to search iteratively, refining the terms in search queries based on the results that they return. They want to save search queries that worked.

Search Results Management

Search engines return a list of documents and/or document links in response to search queries. Most typically, the list is ordered according to the search engine’s determination of relevance of each result to the search query. You’d like a fair degree of control over the presentation of search results--the more control the better. Some of the controls that are helpful to customers are:

* Sorting results by factors other than relevance or sub-sorting by factors within relevance

* Presentation of links to the documents

* Presentation of document summaries

* Presentation of document summaries that highlight the terms of search queries

* Identification of “important” results

Search Analysis

You should be continuously measuring, analyzing, and refining your customer portal’s search experience. The search engine should instrument its indexing and search processes, collecting information that can help you analyze the effectiveness and the efficiency of the findability services that you deliver to your customers. It should also package some reports to help you analyze the information that it collects. For example, you must know the queries that your customers are using and you must know which of those queries doesn’t product results. In practice, portal platforms do a poor job at search analysis. Premium search engines are better.

PERSONALIZATION SERVICES

The first version of our evaluation framework for customer portals focused on personalization through rules-based or analytics-based content selection and presentation. Our evaluation of portal platforms taught us that there are additional approaches to delivering a personalized experience through your customer portal. Some of these approaches are:

* Customization of look and feel by administrators for customers or by customers, themselves

* Customization of portal functionality by administrators for customers or by customers, themselves

* Creation and customization of personal pages or, even personal sites both by administrators and by customers

The wider the range of personalization approaches, the better. You need the flexibility to deliver the portal experience that makes it easiest for your customers to do business with you.

COLLABORATION SERVICES

Customer portals can deliver assisted service as well as self-service. Web chat, instant messaging, and online meetings can be appropriate points of escalation when customers have difficulty getting their questions answered or their problems resolved through the self-service mechanisms of your customer portal. Collaborative workspaces can help you help your customers in situations in which self-service may not be appropriate--negotiating the terms and conditions of a new service contract, for example. Collaboration in forums and discussions among your customers and their peers and colleagues, as well as with your support and R&D personnel, can be helpful, too.

Portals should package or integrate a broad range of collaborative capabilities. They should package portlets that let your customers use these services.

ANALYTIC FUNCTIONALITY

Customer portals help your customers perform operational activities. Through customer portals, customers execute transactions like updating their account information or diagnosing a problem they’re having when they use your products.

It’s critical to measure, monitor, and analyze how effectively and how efficiently your customers are able to do their work, as well as how well the customer portal and the applications and data that support these activities perform. Remember, also, that your customization of a portal technology platform’s packaged facilities produces just an initial customer portal implementation, and it’s critical to monitor and refine your implementation to ensure that your customers can accomplish their work efficiently and effectively. So you’ve got to measure and analyze your customers’ behavior and the performance of your customer portal. You use the results of the analysis step to refine operational processing, thereby improving the customer experience that you deliver through the customer portal. Then you go back to monitoring, measuring, and analyzing again.

Portal technology platforms should support this operations/analysis/refinement loop. For example, you should monitor and measure customer behavior for accessing account information. Sessions abandoned before data are presented may indicate performance issues in accessing data. As a result, you may do some database tuning, thus closing the loop. You continue to measure and monitor, both to ensure that you’re tuning results in fewer abandoned sessions and to be aware of other issues.

The evaluation criteria for analytics capabilities focus on the portal technology facilities for monitoring, measurement, and analysis. They are:

* Instrumentation
* Reports
* Analytics

Instrumentation

Portal technology platforms should collect the information that represents their usage and performance. A product that is instrumented prevents your having to dig into its internals to capture usage and performance information. For example, a portal technology platform should be instrumented to record your customers’ click-streams through the platform’s content, as well as the platform’s resulting internal flow, such as portlet-to-portlet communication.

Reports

Portal technology platforms should package reports that present how customers are interacting with their content and how well their facilities are performing. These reports should present instrumented information in easily-understood formats.

Analytics

You need to know how well many aspects of your customer portal are performing in real time. You’d prefer not to (or you can’t wait to) move performance data to a data warehouse and run queries and analytics against it. For example, you might like to track whether your large account customers are aware of and accept a new contract policy that you deliver through customer portal content, and to display these acceptances on the dashboards of your account management team. If specific accounts don’t accept, you might schedule a face-to-face sales call.

Sometimes reports aren’t enough, and analytic processing is required in order to understand customer behavior and customer portal performance. For example, you might use data mining analytics to refine your approach to personalization.

ARCHITECTURE

We’ve been describing architecture in our evaluation frameworks by using the following components:

* Environments
* Organization
* Infrastructure
* Structure
* Customization
* Integration

Let’s take a look at which of these are subcriteria in our evaluation of portal technology platform architecture.

Environments

The environments of a portal technology platform are the clients, server platforms, and databases it supports. Environments are an important evaluation subcriterion of architecture, and they’re easy to evaluate. Yet these factors can be showstoppers. If your installation has standardized on the IBM DB2 database, and a product supports only Oracle10g and Microsoft SQL Server 2000, then that product won’t be a good fit for you.

Organization

All portal technology platforms have essentially the same components and the same interfaces between them. Therefore, how a portal solution is organized is not an evaluation criterion.

Infrastructure

By infrastructure, we mean the deployment environment of the portal platform. A deployment environment provides runtime services like request handling, process and thread management, memory management, and security. Simply put, products should deploy on J2EE and/or Microsoft .NET infrastructures. It’s not uncommon for a product to provide some of its own deployment services. For example, J2EE doesn’t specify a data access standard. So products must provide their own data access mechanisms.

A portal technology platform will support either a range of infrastructures or support the infrastructure of its platform supplier. For example, the Vignette Portal supports Apache Jakarta Tomcat, BEA WebLogic, IBM WebSphere, and Sun AS while Oracle Application Server Portal supports, not surprisingly, Oracle Application Server. If your company has implemented IBM WebSphere, then you probably won’t consider Oracle Application Server Portal.

Structure

In structure, we examine the internals of a product or a software technology. For portal technology platforms, structure comes into play for integration. We won’t evaluate structure separately.

Customization

Customer portal implementation is a customization process. We won’t consider customization separately and additionally in architecture. Rather, we’ll cover it in the technologies that support Customer Scenarios, especially UI content management and personalization.

Integration

Integration of external applications and data is a critical customer portal requirement. But it’s a requirement that’s addressed by a portal technology platform’s portlets. We’ll evaluate portlets in our evaluation of the technologies that support Customer Scenarios. Packaged portlets, third-party portlets, portlet development tools, and support for WSRP are the integration mechanisms of portal platforms and the integration evaluation criteria of our framework.

PRODUCT VIABILITY

You want to purchase a portal technology platform that is well proven and widely used for your type of business. You also want a product that you can implement within a budget and schedule, and a product that will continue to be able to address your requirements in future versions. In other words, you want a viable product.

We consider the business aspects of portal technology platforms in the product viability section of our framework. A product’s viability is much easier to evaluate than its technologies for Customer Scenarios, analytics, or architecture, but the factors that contribute to product viability can be real deal breakers. For example, a product may be targeted for industry segments other than the ones in which you do business. A product may be brand new with no reference customers similar to your company. Or the product’s price might break your budget.

We’ve identified six subcriteria in evaluating product viability. We feel that they apply to any type of software product. The product viability subcriteria are:

* Product background
* Installed base
* Target market(s)
* Pricing
* Product plans
* Competition

Product Background

Within Product Background is a high-level evaluation criterion that we’ve broken down into these evaluation criteria:

* Current version
* Version history
* Development approach
* Key acquisitions

In product background, we present the product’s release history, from the date of its introduction to the date of its current version. Versions at regular intervals (once a year is good) demonstrate a good development plan and a supplier’s ability to execute the plan to deliver products on a timely basis. More than one new version per year or several years between versions could indicate problems. Your organization could be kept very busy doing installations and then reimplementing your customizations and integrations when a supplier introduces new versions frequently. On the other hand, there might be significant problems when a supplier is slow to introduce new versions. For example, its R&D organization might have been reduced in staff as a result of poor company performance resulting from economic conditions or mergers and acquisitions.

Sometimes, the lack of a long release history is not a negative. New products may be risky for those of you who like to stay off the leading edge, but new products can be functionally innovative, incorporating the latest technology standards and/or addressing a new business area.

The portal platform supplier’s R&D approach is another indicator of product viability. Ideally, products are designed, developed, and maintained by the supplier’s internal R&D organization. Suppliers have sought to reduce R&D costs by outsourcing and/or off-shoring product development. These approaches have been effective controls but they have the significant disadvantage of removing development personnel from the supplier’s day-to-day operations. Developers contribute much more than the program logic of software products. They play a critical role in product strategy, planning, and customer support. When developers work for another company, they can’t play those roles and, as a result, the product and its customers suffer. Many software suppliers have struck a balance between internal and outsourced development. They get the cost effectiveness of outsourced development on a project basis while keeping a good number of internal developers involved in their companies.

Another dimension of development approach is the use of open source. Software suppliers no longer have to write every line of code in their products. They can leverage open source for more efficient, more effective, and more timely development. Open source components and products are available for a wide range of functionality ranging from utility services like parsers to end-user services like search or even to entire sub-systems like content management.

We also examine the supplier’s recent merger and acquisition history from the perspective of how that activity has contributed to the portal technology platform’s capabilities. Sometimes, M&A activity can result in a product that’s not seamless, causing you extra work in customization and integration.

Installed Base

Installed base is the number of customers who have purchased and implemented a product. A large installed base is a sign that a product really does address business requirements. But large is a relative term--again, the more, the better. We like to see at least 50.

Target Market(s)

A product’s target market identifies the types of companies that a supplier feels its product best fits. Target markets typically have two dimensions: company size and industry segment. If your company doesn’t fit either or both dimensions, then stop considering the product.

A product’s target market should be supported by both the product’s technology and the supplier’s marketing initiatives. Technology support is more important. Company size-specific and/or industry-specific features and functions make your implementation of a product faster and reduce the scope of customization and integration.

Pricing

For pricing, we try to provide a product’s implementation costs--the license fees and consulting services costs necessary to get a product up and running in your installation. Suppliers are not always willing to share their pricing, usually because their fees are negotiable. When we can provide pricing details, we will. Minimally, we’ll describe the pricing model, an entry price, and a typical price, as well as give an indication of a percentage of those prices that other companies have spent on consulting services.

Product Plans

You don’t just buy the current version of a product. Your purchase is also an investment in the future versions of a product. It’s essential to know the product’s direction. Product plans present the details of the next product version and are an indication of the longer-term direction in which the supplier wants to take its product.

Competition

In competition, we identify alternative products and highlight key differentiators. Understanding a product’s competition can reduce the risk in selecting a particular product and provide ammunition for its justification.

THE BOTTOM LINE

That’s the refined framework. Its criteria are similar to those that we’ve applied to other types of products and technologies, most recently cross-channel, cross-lifecycle customer service and customer self-service search, both of which, incidentally, should be integrated with your customer portal. Going forward, we’ll be updating our evaluations of particular portal technology platforms against the revised framework as new versions of the platforms are introduced. We’ll also update our Feature Comparison Matrix to reflect the new evaluations.

***ENDNOTES***
1) See “Portal Framework: Our Framework for Evaluating and Comparing Portal Platforms, ” by David S. Marshak, December 12, 2002, http://www.psgroup.com/detail.aspx?ID=64.

2) See “Partner Portals Should Be Combined with Customer Portals: Why Not Design Your Partner Portals to Surround and Complement Your Customer Portals?” Patricia B. Seybold, February 10, 2005, http://www.psgroup.com/detail.aspx?ID=367.
***ENDNOTES***