Applying Cost-Benefit Analysis for Usability Evaluation Studies on an Online Shopping Website

The guest post today is an academic paper from Chui Yin Wong, and very much worth reading. If you are looking for more articles and papers that give you cost-based research on the impact of UX please click here!

Applying Cost-Benefit Analysis for Usability Evaluation Studies on an Online Shopping Website

Given the perception that employing usability evaluation method is expensive and time- consuming, industries are sceptical to integrate usability throughout the product development lifecycle. A Cost-Benefit Analysis is claimed to be the most feasible approach to calculate the cost incurred in any employed usability technique, and the estimated return on investment (ROI) and benefits gained in the long run. This paper highlights two usability evaluation methods, namely Usability Testing and Heuristic Evaluation, which were employed to determine the most cost-effective approach based on a study for an online shopping website. The Usability Testing approach was conducted in context of the Common Industry Format for Reporting Usability Evaluation whilst the Heuristic Evaluation was conducted referring to Nielsen’s (1994) Discount Usability Engineering approach. As a result, the analysis parameter of Cost-Benefit Analysis for both usability evaluation studies were based on the elements employed in each method. For instance, the costs for Usability Testing and Heuristic Evaluation were justified on pre-evaluation, post-evaluation of the evaluator on the time spent for the evaluation, and miscellaneous charges involved. On the other hand, the estimated benefits were referred to the estimated increased sales revenue, an average profit margin of a product sold and an increased conversion rate after optimizing usability. A summary was drawn on the most cost-effective method for usability evaluation studies on an online shopping website. In view of the study conducted, this paper also reports on the effectiveness of applying Cost-Benefit Analysis for both usability evaluations.

1. Introduction

Usability has become a crucial concept these days for corporate to convince consumers or clients in choosing, reviewing or purchasing computing products. Making more usable computing products is a smart business. Even though industries are quite aware of the significance of usability, they are still sceptical on the know-how of integrating usability as part of the software or product development lifecycle. The myth of organization’s scepticism towards integrating usability as part of the process [Mayhew and Bias (1994)] such as “quality of the user interfaces doesn’t really matter”, “user interface design tasks don’t arise until the detailed design phase of a software project”, “usability is subjective and cannot be measured or engineered”, “user interface design can be done right the first time in the design phase”, “user interface design is an implicit part of software design and development and need not be

explicitly planned and budgeted”, “as long as designers are familiar with interface guidelines and principles, good user interface will be designed”. These common fallacies have become a major challenge for usability practitioners to conduct usability evaluation studies and apply User-Centred design principles throughout the development process. The jobs have become more difficult when Project Managers perceive usability work as delaying and extending the product development lifecycle, and not being able to meet the targeted deadlines for shipping. There is a rippling effect if usability evaluation studies cannot be justified and quantified as being “value-added” for the Project Managers, especially to meet and uplift the bottom line.

To help reduce such scepticism and to encourage usability as part of the product development integration, a measurable tool to quantify usability evaluation studies in financial basis is greatly required. As such, Cost-benefit Analysis is conceived as a feasible tool to quantify and justify the cost incurred and to help estimate benefit for conducting any usability evaluation techniques on a project or an application. This paper aims to apply Cost-benefit Analysis on two usability evaluation methods, Usability Testing and Heuristic Evaluation, and to study which method is more cost-effective on an online shopping website (which is noted as the “client” in this paper).

2. Background of an Online Shopping Website Project

This usability evaluation study is an extension of a PRUE (Providing Report for Usability Evaluation) project [Wong and Maguire (2003)] in the United Kingdom (UK). The EU- funded PRUE project was conducted by 4 European partners to specify, test and report usability requirements using a new Common Industry Format (CIF) for documenting usability results. The CIF has been approved by NIST (National Institute of Standard and Technology) and is accepted by the ISO (International Organization for Standardization). In this case, the said project was carried out with a key focus on studying the effectiveness of CIF with respect to the usability evaluation methods employed on a UK-based online shopping website. As a peripheral study, Cost-benefit Analysis was also considered to assist in justifying and quantifying which usability evaluation methods are more cost-effective. The method and its process will be discussed later (details of CIF can be found at Wong and Maguire (2003)).

The mentioned UK-based online shopping website retails premium and unique gifts to the online audience worldwide. According to the Marketing Director, their major customers comprised of male and female shoppers from the UK. The client prides themselves for having a special “gift search” feature in matching user’s characteristics with their gift listing preferences. The client was also curious to know the feedback from their intended and potential online customers about their overall website performance.

3. Usability Evaluation Methods

Usability Testing and Heuristic Evaluation were selected in this study to identify user feedback on the client’s online shopping website in a quantifiable and qualitative format. The main difference between these two methods is that usability testing involves real users in the user trial while heuristic evaluation is a “quick and dirty” technique to evaluate the interface with reference to a rule-of-thumb by experts [Nielsen (1994)].

A usability test was carried out to ascertain the effectiveness, efficiency and user satisfaction of the online shopping website. The findings of the usability test were then reported

conforming to the CIF format with additional user comments and evaluator’s observation. During the usability testing process, 14 subjects were recruited as a fair representation of the online customers. The age group of the 14 subjects ranged from 16 to 54, and a fair distribution for gender ratio (7 female and 7 male). 14 subjects were chosen due to time and budget constraint for larger participation. The number of subjects also conforms to the minimum requirement of 8 representative subjects as stated in the CIF format. The objective of CIF is to help communicate usability test results to consumers and supplier organizations during the product (software) procurement process, which in turn, to help reducing users’ frustration, and increasing users productivity and/or profitability. The CIF provides a standardized reporting format for usability testing on this particular study for the client to gain the overall user performance and satisfaction metrics. It does not aim to obtain a sound statistical data for inferential analysis in this instance.

Another usability method, Heuristic Evaluation, was also conducted to examine the strengths and weaknesses of the online shopping website interface. This is intended to complement and to compare the findings of the usability test where its combined findings, comments and a list of usability issues with appropriate design recommendations is fed back to the clients for further website improvement. 5 experts were recruited to go through a list of preset tasks with a reference to 10 Web Heuristic Principles [Wong (2003)]. The findings were reported with highlighted usability problems in severity rating with design recommendations to the clients.

4. Cost-Benefit Analysis for an Online Shopping Website 4.1 Benefits of Usability Investment

As mentioned earlier, Cost-Benefit Analysis is a feasible approach to quantify and calculate the cost incurred and estimated benefit for ROI on the usability aspect of a project. It has become crucial for usability practitioners to justify the worthiness of allocating budget on usability aspects, especially when industries are facing an economic downturn. Usability provides many ROI benefits to products developed either for internal development organization or external vendor companies [Bias and Mayhew (1994)]. The ROI for organizational logistics include increased user productivity, decreased user errors, decreased training costs, savings gained from making changes earlier in design lifecycle, and decreased user support. Meanwhile, the benefits gained for external vendors are increased sales, reduced customer support, savings gained from making changes earlier in the design lifecycle, and reduced cost of training provisions (if training is offered by the vending company).

There have been many well-documented examples and statistics of cost savings with the deployment of usability applied to the product development process [Marcus (2001)]. For example:

“At we monitored site traffic, observed consumers in usability studies and worked with internal business groups. This helped us make changes that made the site’s purpose clearer and increased transaction rates measurably. The change increased the traffic up 129% the week we put it up.” [Interaction design, Inc., 2001]

“The rule of thumbs in many usability-aware organizations is that the cost-benefit ratio for usability is $1:$10-$100. Once a system is in development, correcting a problem costs 10

times as much as fixing the same problem in design. If the system has been released, it costs 100 times as much relative to fixing in design.” [Gilb (1998)]

4.2 A Comparison of Cost-Benefit Analysis for Usability Evaluation Methods

A usability practitioner must first identify the general requirements of the system and issues involved on the overall usability evaluation study. Most likely the time limit and budget constraint are considerably informed at the first place by the Program Manager. The goal of the Cost-Benefit Analysis is to determine the cash value of the positive difference on a product based upon its usability. The approach is done by comparing the costs involved for any chosen usability evaluation methods (i.e. usability testing, task analysis, heuristic evaluation, etc.) with the estimated benefits (increased revenue sales, decreased training cost, or increased productivity) of the particular product or project.

In this instance, usability testing and heuristic evaluation were the two chosen usability evaluation methods for the client’s online shopping website. The breakdown of the costs and estimated benefits on the client’s website are discussed in the next section of the paper. All monetary values are illustrated in GBP.

4.2.1 Cost-benefit Analysis for Usability Testing

As mentioned earlier in section 3, a usability test was conducted with 14 subjects recruited from a wide range of users. The user trial was conducted in a usability laboratory within a 3- week timeframe. The parameters applied for calculating cost and estimating benefits of Usability Testing are described as below:

  1. 1)  The online shopping website in question;
  2. 2)  Evaluator’s fully loaded hourly wages are estimated as GBP20;
  3. 3)  Subject’s payment for their voluntary participation in the testing is GBP25 per person;
  4. 4)  Rental for usability laboratory is GBP1,000 per full day (7 hours).

Issues such as evaluator’s time preparing the test material, subject recruitment, payment for 14 subjects, laboratory rental charges, telephone bill, time for transcribing observation, user comments, data analysis and report write-up were all considered in the costing calculations (please refer to Table 1 for more details). In terms of benefit, a measure of a website’s effectiveness is the Conversion Rate – this refers to a website’s ability to convert visitors to buyers. During the user trial, task completion rate was collected to study the mean effectiveness of the website in overall. In this case, task completion rate was designed in a scenario by given the subjects an amount of budget to look for any gift to purchase from the online shopping website.

The client provides an estimated 120,000 (a conservative figure) online users visiting their website per year. The tricky part is identifying the percentage of intended buyers from the estimated online users. A few literature reviews have described the Conversion Rate of online shoppers turning to online buyers. Ray (2001) stated that 65% of shopping carts were abandoned and no purchase was made. This means 35% of successful completion rate (purchase) was conducted. Spool et. al. (1999) in a series of shopping website tests indicates that 42% of online users were made purchases from a series of renowned online sites, while 28% for the least well-known sites. Other studies provide 50% – 60% success rates [Nielsen

(2001)], 40% completion rate of purchasing an item during the transaction process [Thumlert (2001)], and 75% task completion rate [Wong (2002)].

Hence, based on the survey data and marketing response from the client’s online site, an average of 30% (36,000) out from the estimated annual 120,000 visitors would be the intended and potential buyers. During the user trial, the task completion rate of successfully making a purchase from the 14 subjects shows a 75% mean effectiveness. This also meant a 25% failure to make a “purchase”, thus showing a potential loss in numbers of sales. A summary of severe usability problems were drawn for the usability test report such as lack of focus on homepage, cluttered web page layouts, misleading “back” button, and failure to ensure ease-of-use to complete the online registration during the checkout process. Other non- usability factors such as service delivery and offline marketing campaigns which drive the total success of task completion rate are not considered in this study. Optimistically, to achieve a 100% completion rate if the mentioned usability issues are implemented, an assumption of 25% additional conversion (completion) rate is assumed after optimizing usability on the website. Thus, 25% of 36,000 intended buyers are 9,000. Once the additional number of buyers is calculated, a conservative number for an average profit margin per product sold is GBP5. Further details are shown in Table 1.


Gilb, T. (1998). “Principles of Software Engineering Management.” In Usability is good business.

Interaction Design, Inc. (2001). “Design does provide return on investment.”

Marcus, A. (2001). “Return on Investment for Usable User-Interface Design: Examples and Statistics.”

Mayhew, D. and Bias, R. (1994) “Organizational Inhibitors and Facilitators” in Bias, R and Mayhew, D. (eds) Cost-Justifying Usability. Boston: Academic Press.

Nielsen, J. (1994) “Usability Engineering.” San Diego: Morgan Kaufmann.
Nielsen, J. (2001). “Did Poor E-commerce Kill Usability?”

Ray, A. (2001). “How to Encourage Internet Shopping.” Marketing, London, 3 May 2001.

Spool, J.M., Scanlon, T., Schroeder, W., Snyder, C., DeAngelo, T. (1999). “Web Site Usability: A Designer’s Guide.” San Francisco: Morgan Kaufmann.

Thumlert, K. (2001). “Abandoned Shopping Carts: Enigma or Sloppy E-Commerce?”

Wong, C. Y. (2002) “Usability Testing Versus Heuristic Evaluation: A Case Study on An Online Shopping Website.” Loughborough University, Thesis.

Wong, C.Y. and Maguire, M. (2003) “The Effectiveness of Common Industry Format for Reporting Usability Testing: A Case Study on an Online Shopping Website” International HCI Conference, Greece, July 2003, Elsevier Science.

One Response to “Applying Cost-Benefit Analysis for Usability Evaluation Studies on an Online Shopping Website

  • Hi,

    Glad you like the article – most of the findings were presnted at the International HCI conference. If you are on LinkedIn and want to read more articles like this our group is called “User Experience Insights

    Hope you can join and take part!


Leave a Reply