A Conversation with ForeSee Results’ Eric Feinberg

I figured I would get into some trouble with my post on Voice of the Customer. I am still researching and updating my “technology stack” of web analytic offerings and services and expanding into social media and customer experience, so I had a feeling if I called out a couple of product offerings that someone would be left out. However I was seeing trends that I felt compelled to respond to before I got everything organized into nice boxes and flow diagrams.

1. An ongoing controversy of how to interpret unique visitors and visits among some the most important web analytic pundits – Eric T. Peterson, Avinash Kaushik, and Matt Belkin – that casts doubt on the validity of web analytic data. Sure I addressed this directly here but still it indicates a startling lack of consensus on what the data can and cannot do.
2. A growing consternation within traditional marketing on how to approach and exploit social media leading to attempts to map social measures into traditional marketing metrics [Peterson, Sterne]. Yes social media is a new way of capturing voice of the customer but how does one know that it indeed does capture that voice and not some new social behavior – somewhere between squeaky wheels and mob rule. Heck, are the comments even from real customers? Some still doubt the hype.
3. Application of traditional voice of the customer marketing surveys and studies to web experience that appear to be an alternative to web analytics, at least to the point where these are used independent of web analytics (or as this segment likes to call it – click stream data). iPerceptions being an example of this with their free 4Q offering and strong support if not explicit sanction from Avinash.
4. Development of a third way called Customer Experience Management (CEM) that purports to be a new analytics (beyond web analytics) concealed behind an opaque marketecture that seems to do everything without explaining anything. Tealeaf being an example of the offerings in this sector.

I realize a lot to mesh together but my intent is to argue that somehow these different approaches will work much better if done together. If I have small initial budget with which to demonstrate value of analytics to a business, where do I start – tactical with Google Analytics, iPerceptions or more strategic with Omniture or Coremetrics with Tealeaf, or go high-end analytic with Truviso? Now I understand why each company provides their offering as a stand-alone product. It is not good business to require other products as prerequisite. It limits one’s potential customer pool. But at some point these products will have play together or assume a shrinking customer market share as customers select initial offering other than their own.

Then suppose I am successful and have established value and committed significant resources in analytics and business intelligence with strategic ownership over the data, what do I need to add to go to the next level and can I do that with out-of-the-box offerings that do not have open APIs that integrate with my current data systems? At this point for large online companies that I have worked for – Yahoo!, eBay, and PayPal – data is so integrated into the functioning of the business that it is tempting to go ahead and build the capability in-house. Amazon is another perfect example. But in these case, large companies may need special expertise or technology that is not available in-house. How can these capabilities be integrated into workflows and platforms that are core to the business. After all, if WA, VoC or CEM are to become must-haves and not just nice-to-haves, the offerings must integrate with the core processes of the business.

Who is ForeSee Results?

So I figured something was up when a company and then its CEO started following me on twitter. To my horror, it was a company I had not heard of. (Now I see the name pops up everywhere. They are Diamond Sponsors at eMetrics, and of course there is my twitter. Funny how that happens. Poetic justice I imagine.) A quick check of the web site, (after accepting to perform the survey at the end of my visit), indicated that my work was cut out for me.

On the one hand their offerings seemed to resonate with the points I made – in particular that VoC Analytics (in their terminology: Attitudinal Analytics) and Web Analytics (i.e. Behavioral Analytics) are both necessary and work better together. Indeed they provide APIs so that web analytics, CRM, and Financial systems can integrate. Great! Furthermore their approach incorporates psychometrics and econometrics. It is about time that economist come into the field. Super Great!

On the other hand, what is being provided is concealed behind strong marketing spin that described pretty clearly what it could do but not so much how it could do it for me as an online business. Their offerings come with expert specialist and professional analyst that will do all that hard stuff. Since their methodology predates the web itself, I imagined that their customers were already aware of the approach but what about all us rascal online entrepreneurial types – how do we pickup on what is going on?

Both excited and perplexed, I was relieved to receive a kind and tactful note from Eric Feinberg saying,

Read your recent VOC blog post. I liked it but thought it was incomplete. If you want to chat more about it, shoot me a note.

— Eric Feinberg, Industry Director for ForeSee Results

I shot him a note.

UPDATE: 05/26/2010 : At the same time I interviewed Eric, he was also doing a pod cast with Beyond Web Analytics, which is now available at Voice of Customer with Eric Feinberg.  This gives a great introduction to Voice of Customer and how satisfaction plays in this space.

About ForeSee Results

The primary premise of ForeSee Results is that customer satisfaction is a prerequisite for any business performance measure and the American Customer Satisfaction Index (ACSI) is a leading (meaning predictive) indicator of business performance. Unless you understand ACSI, what is offered by ForeSee Results will not make much sense and its claims too good to be true.

Instead of providing tools for conducting surveys and collecting Voice of the Customer data, ForeSee Results focuses on a methodology for performing analysis on VoC data. In fact it uses the latest scientific findings in psychometric measurement combined with econometric statistical modeling to provide a means to predict business economic performance based upon customer satisfaction survey results.

Most all ACSI activity is centered at the University of Michigan in Ann Arbor where Claes Fornell, the founder and principal developer of the methodology is a chaired professor at the business school. American Customer Satisfaction Index LLC is a private company also in Ann Arbor that conducts and publishes ACSI survey results for both business and government sectors. As an index, the ACSI’s ability to track both macro and micro economic timelines as leading trends has been well documented in the peer-reviewed literature. It’s ability to trend even stocks, which are often assumed to move as a random walk, has also been demonstrated supporting Fornell’s original assertion that customer satisfaction is basic to all economic processes.

ForeSee Results also in Ann Arbor applies this methodology to on-line site performance with the ability to foresee business performance and visitor behavior results from VoC survey analysis. I noticed in Eric’s profile that he too was a UofM graduate, but according to him, just coincidence. He had already been in California at Vividence and Keynote Systems with an already established reputation in user experience and business development before coming on the radar and being picked up as a Regional Manager for ForSee Results.

I noted that I met my wife in Ann Arbor and with that the ice was quickly broken. As more evangelist than salesman, Eric was ready and eager to address the questions I had concerning ForeSee Results. The discussion moved quickly from interview to conversation where we quickly brought together our two different world views. To get context for what ForeSee Results offered, which he admitted somewhat difficult to categorize in the Gartner-esque categories, Eric presented the analytic ecosystem in which ForeSee Results operates.

Analytic Ecosystem

There is much presented now a days concerning analytic ecosystems [1]. It is difficult to determine whether we are at Web Analytics 2.0 or 3.0 or perhaps being superseded by something completely different such as traditional Voice of Customer (VoC) or Social CRM or Enterprise Feedback Management (EFM) now morphing into Customer Experience Management (CEM). Each company will have their spin on how and with whom they play in the analytic space.

However what Eric presence makes general good sense in that it relates various aspects of what is happening in understanding on-line customer behavior and voice and how these processes should work together. Instead of attempting to explain the entire world, the focus is on the analytics that the business owns and can control. This separates out all data from external sources that the business does not control and is difficult to attribute to actual customers such as VoC in social media, or traditional marketing research that may or may not have implications to the on-line properties. Not to say this external view is not important but that its impact requires interpretation by the business to determine how it affects business. By focusing on data from processes that the business controls, this data can potentially inform business processes and lead to data directed decision systems.

Measuring the value of social media is doable by using a smart new ForeSee Results approach to be launched in June. Maybe another blog post on that then!

— Eric Feinberg, ForeSeeResults

In this limited but strategic analytics ecosystem there are three broad categories of analytic systems that view and attempt to understand the customer of a business. These are:

1. Web Analytics – “click stream data” – that captures what the customer did on and off-line. This can be called behavior analytics and includes most all Web Analytic Providers as well as analytic data from external advertising or support networks such as Google and Yahoo as well as DART.
2. Voice of the Customer – “on-line site surveys” – that captures why the customer did what they did on and off-line. This can be called attitudinal analytics and includes most on-line V0C offerings but not VoC captured from the social media sources since these are not controlled by the business. The purpose is to not only collect why but if they were satisfied doing what they did on-line and off-line.
3. Customer Experience Management – “path analysis” – combining the behavioral and attitudinal analytic data to determine how the customer performed relative to their own objectives and diagnose how the experience can be improved to increase customer satisfaction. There are a number of offerings that claim to be CEM such as Tealeaf and Responsetek, but here would include any offering the provides session replay such as ForeSee Results or performs path analysis and visitor segmentation such as Bluenote and Interwoven Optimost, or provides test and targeting support including Google Test and Omniture Test and Target. The key is that all data including the content and treatments viewed by a customer are brought together to discover patterns and recommend actions for improving customer experience.

A key feature of this ecosystem is that behavioral and attitudinal analytics play a balanced role in observing the customer – as one author has defined it – they form the two ears to hear the customer. ForeSee Results has a two-way API with the major web analytic vendors where visitor level data can be exchanged between these two system such that the attitudinal data collected through surveys can be used to extend the web analytic reports, while behavioral informs the analysis of the survey results, which we will see how in a moment.

This ecosystem interacts with the business intelligence systems – Ops Finance, Customer Relations Management, and Business Process Management – where typically web analytic metrics support the monitoring of Key Performance Indicators that often include financial goals such as ROI, Profit, Margin, and Revenue goals as examples. In this model, the VoC likewise supports Key Success Indicators that typically includes anonymous or specific customer comments that inform Sales and Marketing of their success in reaching and engaging customers.

The problem with most product offerings in this ecosystem is that collecting data and reporting KPIs and KSIs are all that are supported. However in ForeSee Results this is only the beginning. The important work is yet to be done in analyzing this data and using it to model the business to show how customer satisfaction is integrally linked with future business performance.

Voice of the Customer Analysis

The ACSI Methodology for Voice of the Customer analysis includes using latest psychometric principles in developing survey questions that collect the right data and then relate this data in a causal manner to business objectives [2]. A distinguishing feature of the survey design is to measure each latent variable (the things we want to know about the user satisfaction) using multiple manifest variables (the things that can be asked or observed in a survey). This along with the 10 level scale for responding favorably or unfavorably to each question provides a richly varied data set that can more precisely measure latent variables such as content appropriateness, search effectiveness, ease of use, or look and feel, than attempting to directly measure each with a single yes / no response.

The survey questions are professionally designed by ForeSee Results’ analyst and specialists to elicit the precise measurements necessary for the modeling and analysis. The definition of satisfaction is not that complicated. Within each domain, we want to know overall satisfaction (site rating); whether the customer’s expectations for the visit were met (expectancy disconfirmation), and did the experience meet the customer’s ideal (performance vs the ideal).

According to Eric the literature shows that all visitors that come to a site have some expectation of what they want to achieve and will always compare their experience with an ideal – though the ideal may not be fully articulated.

What is missing from the definition is equally illuminating. Task Completion is a behavioral measure that by itself cannot be a central satisfaction metric. Many may complete a task because they have to but still can be dissatisfied with the experience. According to Eric, it is satisfaction rather than task completion that has long-term effect on business performance.

Also prominent metric that is missing is Net Promoter that is based upon if the visitor will recommend the service or product [3]. Recommendations are a predicted behavior outcome of satisfaction.

ForeSee has asked the Likely to Recommend question in nearly all of their 40+ million completed surveys and has found that their means-based approach to analyzing recommendation outperformed the NPS approach.

— Eric Feinberg, ForeSeeResults

Also bounce rates and conversion rates are not satisfaction measures but instead are observable indicators of satisfaction or predicted outcomes of satisfaction or lack thereof.

Application of ACSI to On Line Customer Experience

When one visits the ForeSee Results site, one immediately learns that the ACSI is a core concept of their offering and that it is scientifically based. One finds that as a methodology it has been around since 1993 and has been used successfully to monitor industrial sectors, the US government and its agencies, and the overall economy as a whole. Also one can see graphs that “show” how ACSI timelines “appear” to match a number of lagged (time shifted) economic indicators to illustrate its predictive power. One has the impression that the developers and practitioners of this approach are very smart, have published peer-reviewed papers, and whatever they do will forever be incomprehensible to everyone except a few hardcore statisticians and econometrician.

With this legacy of tracking entire economies, how can this approach help someone develop a better web site? Challenging Eric with this question his response:

ACSI methodology provides a financial model that is sensitive to the specific business objectives such as driving sales on-line or increasing sales in the store. The model’s financial and behavioral predictions are driven by satisfaction metrics that directly affect outcome and can be changed by actions that improve user experience such as changing content, improving navigation, shortening process paths etc.

— Eric Feinberg, ForeSee Results

To understand this in further detail, the illustration below gives an example. The methodology consists of mapping drivers of satisfaction on the left to the predicted behaviors on the right in cause and effect relationship. The drivers represent all the elements of the on-line experience that the visitor uses and can express satisfaction or not. The predicted behaviors are the customer actions that we would like the visitor to do such as return, purchase, recommend that directly affect business KPIs.Using web analytic measures combined with satisfaction measures through surveys, an analyst develops a statistical model that maximizes the causal link between drivers and predicted behaviors. This results in a set of weights that give the relative importance of the various drivers to the desired behavior outcomes based on their patented methodology. The output is a statistical econometric model, configured by ForeSee professionals, that is sensitive to the variations of customer experience and the objectives of the particular business.

Please note that “Satisfaction” is placed in the middle of this graph. The dependent variable, Satisfaction [as captured in the ACSI scoring] is crucial to the cause-and-effect system.

–Eric Feinberg, ForeSee Results

The result is a mapping of drivers of satisfaction to likelihood of predicted behaviors that would appear something like this:

where orange numbers represent ACSI Scores and purple the relative weights giving the  “importance” of the drivers and “impact” on future behaviors.  An alternative view that maps score to impact to pull out the low score high impact drivers can be seen in this priority map for the above model:

From this one can clearly see that internal search is a top priority for change that will have high impact on performance.

During eMetrics in San Jose last week, Larry Freed, the CEO of Foresee Results announced the satisfaction scores for the top 100 Online Retail sites. Scores below 60 represent serious problems in customer satisfaction. The overall score 78 gives a 5 point improvement over the previous years 73 and shows an overall concentration of these sites on user experience optimization. The highest scoring were NetFlix with 87 and Amazon with 86. This gives an indication of the relative sensitivity of the scoring to efforts to improve user experience.

The forming of maps between two regimes – drivers to results – is not that different from non-linear approaches such as gradient descent, neural networks, fuzzy sets or vector clustering. What maybe different and significant is the degree of determining causal relationships as opposed to correlated relationships that less rigorous approaches may fall prey to.

ForeSee analysts will perform the heavy lifting to collect the data and generate the model. They work with the customer to identify the business goals then work backwards to identify and generate drivers and configure weights. It may seem expensive to purchase software and then hire specialist to configure it and develop surveys, but on the other hand, the degree of sophistication and experience necessary to perform the analysis may not be available in-house. Therefore the entire package including setup, configuration and reporting services maybe the most cost-effective for this level of analysis.

Whenever one has a predictive model that is sensitive to the particular business or process, then options and their impact can be explored off-line and prioritized without expending additional and potentially large allocations of resources for prototypes or experiments. Not that these are eliminated but with the resulting model one can quickly identify low scoring high impact modifications (low-lying fruit so to speak in VoC) that can incrementally improve performance.

Customer Experience Management

In the end VoC analysis (as it is with WA analysis) only highlights problems and indicates where to look for solutions. To further diagnose the problem and devise a strategy to resolve, the output from ForeSee Results are different sets of reports to help prioritize efforts in improving usability and monitor when user experience problems affect the business bottom line.

The first set of reports is the online portal that allows access to the raw data and results, allow the user to “slice and dice” as necessary or download into spreadsheets or ETL into other databases or tools. Then there are scheduled reports and threshold reports that report continuing satisfaction metrics to web analyst, marketing, product, and user experience researchers. These are reports that monitor and trend the VoC for the site.

The third set of reports provide the deeper analysis of site usability and summary of best practices for addressing short comings. These are produced quarterly at either a strategic or tactical level. The strategic analysis covers how KSI affects the business and from the model where low scoring but high impact drivers could lead to significant improvement. The tactical studies go further into the user experience to identify more specific solutions and options for changes that benefit usability.

To aid in the customer experience management ForeSee Results provides session replay where actual user sessions can be replayed and studied. Also function analysis that helps evaluate how online functions are actually used by customers along with how they evaluate these functions.

I as a veteran of thousands of hours looking log session data and replays I can attest to the value of these tools as well as the futility of being able to extract actionable results from large sets of session paths by hand. Ultimately one will want to include path analysis metrics to the Web Analytics such as path churn, or incorporate path analysis tools the can extract patterns and segments for the combination of events, content, and satisfaction that can diagnose and recommend solutions to targeted individuals.

Maturity Model and Product Offerings

Now for the money question, when should a customer consider the type of solution provided by ForeSee Results. Let us develop context for addressing this question. Maturity models along with analytic ecosystems are topics in much discussion these days. My take on this subject is somewhat different from others in that maturity to me is not a factor of competence or experience nor in the sophistication of the tools and analysis. Maturity is dependent upon the degree that data is integrated into the business process and has proven to provide real measurable benefit to the business.

One would never think of starting a business without a means of accounting for every penny going in and out so that one could calculate exactly what is the profit or lost. One does not mine the financial data for profits but may consider mining for inefficiencies or cost reduction – perhaps starting with spreadsheets and then moving to high-powered financial analysis systems. But these movements are done only if it can be shown that the cost and effort will improve profits or some other business KPI.

It is similar for web sites. At one time businesses set them up without any idea of how they made money or affected their business. This did not seem to matter until the bubble popped in the .com bust. This would be level 0 maturity. Now a web site is built with specific marketing and direct business objectives and must provide measurable results on the business’s bottom line. Initially Web Analytics provided the metrics and links to financial performance in marketing, lead generation, and online ecommerce through multichannel attribution and conversion (maturity level 1).

Now it is becoming apparent that the experience of the visitor online can positively and too often negatively affect company brand. For online businesses negative user experience has direct impact on revenue. But for businesses that exist both online and offline the effects can be equally startling. On the success side is Apple where the Apple Store is an extension of the online experience and interaction and brand are seamlessly blended. On the other hand there is Block Buster and Barnes & Nobel still struggling to use their store properties as an online advantage over their pure online competitors – NetFlix and Amazon.

So with respect to a business wanting to take their business to the next level, clearly they have to address customer experience thru VoC or CEM or both. Eric’s take is that everyone is currently trying to ring out the last bit of performance from their web analytics. This means moving more to testing or path analysis solutions. I know how difficult it can be to get WebDev and IT to consistently get a page name on a page view, the effort it would take to extract content for every page viewed would require 10 times the planning, coordination, and discipline to implement. Indeed, solutions such as Tealeaf are major projects to get everyone coordinated.

So my view is that sooner rather than later, one should consider VoC solutions that integrate with their web analytic solution so that they get two eyes on their customers. Like the initial adoption of web analytics, VoC analysis should expose immediate problems that can have high impact on performance, the low score high impact drivers that Eric references. These initial insights do not require drilling deep into session path analysis. Then when path analysis is necessary and one is ready to call in the heavy drills to mine the data for patterns and segments, the behavioral and attitudinal analytic data streams integrated with the content / treatment stream becomes much more powerful and fruitful. Now Eric thought that this seemed pretty darn logical and a reasonable approach.

Closing Thoughts

If one is going via the VoC route, it would be good that one consider the quality and accuracy of the data they are collecting. Surveys are a marketing tool that many marketers are comfortable with to identify customer needs and build marketing and user experience personas. Using satisfaction metrics derived from psychometric surveys that drive business decisions is not in the marketers comfort zone. That is left to the PhDs who are hired and put in the back room.

What has impressed me about ForeSee Results is not so much their offering, which I am sure is good and delivers on what it promises, but on their approach to bring heavy duty analysis skills into the solution. In web analytics we have become so occupied with collecting data and building metrics, that high quality analysis necessary to make this data useful is often missing. As the analytic ecosystem advances the offerings will have to incorporate much higher powered analytical skills and tools.

The implication is that we in the analytics space including those in marketing and business analytics must become much more comfortable talking about data quality and accuracy; analysis methods and tools; and scientific research and findings. We must change the environment such that marketing for these tools and solutions is not so compelled to hide all this behind marketing spin and clever brand names. Can you imagine a doctor purchasing the latest scope without at least a tech sheet or even training? Are we any less a discipline? So ForeSee Results is a warning shot to the rest of us in the field of what is coming. If you think Web Analytics is difficult now, just wait till it becomes a discipline.

Last Challenge

If I search Voice of the Customer, Voice Of Customer Analysis, or VOC online, the results will be articles and companies that provide support for conventional VOC analysis and services. My challenge to the recently announced CMO at ForeSeeResults is within 2 years (about the time it takes to flush out obsolete results in Google (another pet peeve another story)) to make online VOC results appear in the first results page. Maybe in that way someone will be spared the embarrassment of not finding ForeSeeResults (without having to go to eMetrics).

Much thanks to Eric Feinberg for his time and for most of the material presented here. I would have quoted him more but I became too engrossed in the discussion to take verbatim notes.

White Papers (from ForeSee Results web site):

1. Eric T. Peterson of Analytics Demystified, Inc. The Voice of Customer: Web Analytics in the Public Sector, ForeSee Results White Paper, 2008
2. Russ Merz, Ph.D. The American Customer Satisfaction Index (ACSI) Methodology, ForeSee Results White Paper, February 2009
3. Larry Freed Rethinking Net Promoter: Serious Flaws Tarnish Simple Idea, ForeSee Results White Paper, October 2007