Sunday, March 19, 2017

Comparing Usability of Navigation for two Alternative Website Designs

By Adam Bennett


A website is an important medium for small organizations to get information to their target audiences. However, lack of budget, technical prowess, or awareness of the importance of software usability can lead an organization to not invest in having a usable website. Unfortunately, this can result in users having difficulty finding the information or services that the organization provides; in extreme cases the user may become frustrated and give up their search altogether.

Children's Resources On Wheels (CROW) is a non-profit organization that provides childcare information and services to Lanark County. Their current website is somewhat dated and contains a number of usability issues. An alternative version of the CROW website was designed with the intent of addressing the usability issues pertaining to navigation. In terms of usability, navigation can be one of the most important design factors; it doesn't matter how usable a webpage is if the user cannot access it (Valacich et al., 2007). Other aspects of usability were not addressed in the new version of the website.

Changes in the redesigned website include adding, moving, and removing options from the main menu, making links always open in the current tab except for external links and downloads, and changing the terminology of navigation options to be more consistent. Some of these changes can be seen below:

Side by side comparison of the fully expanded main menus of the original and redesigned website versions

However, just because the new design was intended to improve upon the usability of the original website doesn't mean that it will succeed at that goal. In order to know that an alternative design is actually an improvement, an experiment may be performed to measure the usability of the redesigned website.

The Experiment

The experiment was designed to measure the navigation usability for the original and redesigned websites. A list of twenty navigation tasks was made based on suggestions from CROW staff, with care taken to ensure that the order of the tasks didn't bias the experiment's results toward either website version.

To test the website designs, participants would perform as many of the tasks on the list as they could for a given version of the website within a time limit of five minutes. After the time elapsed, the number of tasks successfully completed up to that point was tallied up and the total was used to quantify the usability of the website tested.

Testing conditions were kept as close to the same as possible for all participants and both website versions. Each participant performed one test of each version of the website. To prevent bias due to participants becoming familiar with the task list and similarities between the two website versions, half of the participants tested the original website followed by the redesigned version; the other half tested the redesigned website first and then tested the original version.

After testing both website versions, each participant was asked to fill out a short survey. The survey asked their opinions of the websites in general, and how easy they found navigating each version to be.

There are two main categories of users for the CROW website: parents of young children looking for child care information or services and child care providers looking to register with the CROW organization. To help obtain more accurate results, participants were selected who could fit into one of these categories. Six participants took part in the experiment, with four being parents of young children and two being child care providers.

Results and Discussion

The results obtained from the experiment were the objective measure of how many tasks each participant completed for each website version, and each participant's perception of the websites based on their responses to the survey questions.

Number of Tasks Completed

Ideally, the redesigned website version would have been more usable than the original website and the experiment results would have reflected that. Unfortunately, this was not the case. As it turns out, there was no significant difference in the number of tasks completed for the two website versions (results are considered significant only if it is probable that the same results would be found 19 times if the experiment was repeated 20 times).

Box plot showing the number  of tasks completed for the original and redesigned website versions

So why didn't the redesigned version perform better than the original? The most straightforward answer is that the redesigned version simply wasn't different enough in terms of navigation usability than the original. Another possibility is that while there were improvements to the usability of the redesigned website, there were also new usability issues added unintentionally.

One of the main examples of a change which caused issues with the redesigned website was moving the Programs menu to become a sub-menu of the Ontario Early Years menu. Participants often had trouble finding the Programs sub-menu in the redesigned website. The Programs menu should be restored to its original position for subsequent experiments.

It is also possible that the small sample size of six participants was too small to provide accurate results. A small sample size causes outliers (results that differ from the overall trend that we are trying to measure) to have a larger impact on results than they would in a larger sample. The only way to know if this was the case would be to run the experiment again with more participants.

Finally, allowing participants to skip tasks which they found difficult or frustrating may have had an impact on the results. Participants were allowed to skip tasks so that they would not become frustrated during the experiment, but outside the experiment a user might give up entirely on a website (and the corresponding organization) if they are unable to complete basic navigation tasks. It is possible that this skewed results in some way, especially since many changes in the redesigned version were intended to prevent certain tasks from taking as long as they did in the original website.

Perceived Performance

Most users indicated on the survey that they preferred the original version of the website over the redesigned version and found the original website easier to navigate. This is interesting, since the number of tasks completed by participants doesn't seem to reflect this. Based on the behaviour of participants, one possible explanation for this is the difference in the perceived number of options between website versions. When participants could not immediately complete their current navigation task, they often resorted to looking at the options under each top-level menu item. While in practice a new user might be overwhelmed by the sheer number of options in the original website (in fact, one of the primary design goals for the redesigned website was to not overwhelm the user with options), in this experiment participants who were actively searching for an option tended not to give up on the current task so long as there were unexplored top-level menus. Thus, while presenting the user with more options might not improve the usability of website navigation in terms of the number of tasks completed per unit time, it may encourage users to spend longer on navigation tasks that they would otherwise give up on. This effect further reinforces the idea that subsequent experiments should do more to take into account skipped tasks (such as penalizing the total number of tasks completed), since it is better that a task is completed slowly than not at all.

Conclusion and Future Work

It is clear from the results of the experiment that the redesigned website was not a substantial improvement over the original website. This does not necessarily mean that all changes introduced in the newer version should be discarded, but it does indicate that there are still improvements to be made and that some problems were introduced.

If the experiment is repeated, it should be modified slightly based on the above observations. In particular, more participants should take part and the measure of usability should address skipped tasks and the participants' satisfaction while performing tasks. The new website design should be redesigned again, to take this experiment's results into consideration and remove flaws that were not in the original website.


Carol Barnum. 2011. Usability Testing Essentials – Ready, Set...Test!. Elsevier.

Ruiili Geng and Jeff Tian. 2015. Improving Web Navigation Usability by Comparing Actual and Anticipated Usage. IEEE Transactions on Human-Machine Systems 45, 1 (Feb 2015), 84-94

Timothy Lethbridge. 2017. Software Usability Winter Course Notes.

Joseph S. Valacich, D. Veena Parboteeah, and John D. Wells. 2007. The online consumer's hierarchy of needs. Commun. ACM 50, 9 (September 2007), 84-90. DOI=

No comments:

Post a Comment