A quasi-experiment measuring how organization of content affects the usability of web sites.

Participants
29 undergraduate students at the Middle East Technical University participated in this experiment. They were chosen from the available population in the Humanities and Industrial Engineering computer laboratories.

Materials
A standard PC with a 14" monitor, a standard keyboard, a mouse, and an Internet connection was necessary. Special care was taken to provide a properly working mouse and a normal mouse pad (not sticky or rough) so that subjects did not lose time for an irrelevant reason. A lab chair was provided. Subjects were allowed to adjust the height and position of the chair as they wished. Netscape Navigator 4.x was used for access to the world wide web. A stopwatch was used to measure time.

Procedure
The experimenter asked individuals whether they would like to participate in a psychological study about the Internet. People who complied with this initial request were taken to a separate PC and seated. The task was explained to all subjects in a standard manner. Subjects were also informed about their right to quit the experiment at any time they wanted. Each subject worked on only one of the four universities assigned to them by order.

After subject was seated and given instruction about the task he/she was asked whether he/she was ready to begin. If the answer was "yes", the timer was set to zero and the location of the mouse pointer on the screen (arranged to be the bottom right corner of the screen) was reminded to the subject. The address of the university was entered to the browser and the return key was hit. Time was started with the appearance of the first visible link on the screen. When subject clicked on a link on the homepage screen, time was stopped. The experimenter who was responsible with recording times reported the time spent on the homepage to the other experimenter who used a checklist to record data. Time was restarted with the appearance of the first visible link on the loading page and stopped when the subject requested another page. This procedure of starting and stopping times were applied in this manner throughout the experiment. This was done to eliminate Internet connection speed and page loading times as confounding variables. The link on the homepage chosen by the subject was also recorded with its label. The number of links that the subjects clicked on was counted. Hitting the back button was also recorded as one link. When a subject asked a question in the middle of the experiment, the experimenter told him/her that he could not respond. When a subject stated that he/she was giving up, we stopped the experiment and thanked to the subject. The experiment was unlimited in time but if the subjects used more than fifteen clicks the experiment was stopped. At the end of the experiment, subjects were given information about their performance. Two experimenters were always present and they collected data together.

Of the 29 participants, 9 were excluded because they did not meet our criteria for inclusion. 5 UCI subjects were dropped because 3 of them left the experiment and 2 passed the link limit. 4 USM subjects were dropped because 3 passed the link limit and 1 left the experiment. In this experiment, we recorded the time spent on the homepage, total time spent to reach the target location, and the number of links subjects used to finish the task. These make up our dependent variables. Table 1 lists each of these variables obtained from each of the remaining 20 subjects. University averages are shown in table 2. Figure 1 is the graphical presentation of table 2. Figures 2,3,4, and 5 include the presentation of DVs separately for each university. No statistical analyses were conducted using these data. Instead, we will continue with a descriptive analysis.

Since we ended up with a very small number of subjects (5 for each university), we do not believe we have a reliable set of data in hand. This is one of the reasons that this study remains a descriptive one. Yet another is that our study lacks scientific rigor in that it failed to take into account a number of variables that possibly confounded the results. However, we are very much satisfied with this initial attempt and hope that it will be the basis of further studies. Viewing the results, what we most clearly see is the relationship between our dependent variables. Our figures show that number of links and total time are directly proportional to each other. That is, the more links one uses, the more time one takes. Some subjects make an exception to this but figure 1 is suggestive of such a relationship. At first glance, one may have the impression that homepage time and total time follow the same trend. It may be thought that homepage time is just a portion of total time and is meaningless as a separate DV. However, total time is not divided equally among all pages. The homepage has a special significance. It is the starting point for a visitor and is the basis of the "first impression" of how the site is organized. A link on the homepage has more significance than links on subsequent pages since a mistake in distinguishing finer, more specific categories (links on subsequent pages) is more easily corrected than a mistake in distinguishing more general, global categories (links on the homepage). Therefore, the user may be expected to make a more thorough decision when choosing to follow a link on the homepage.

Figure 2. Individual dependent variable measures for UCI

UCI subjects spent more time on the homepage, followed by DU, UNC, and USM subjects. The UCI web site has a large and attractive search box on the top of its homepage. In fact, the search option was chosen by 3 UCI subjects, compared to 1 from UNC and none from the other two universities. The time taken to decide between following a link or using search, and/or the time to type the search word into the search box may have contributed to UCI subjects' increased homepage time. Another possibility is that subjects may have been confused by UCI's unconventional label "Instruction and Research" in place of "Academics". We had a valuable experience related to this. One subjects said "Okay but there's no 'academics' in here" when viewing the UCI homepage. One of our hypotheses was confirmed informally with this observation. Further support for the beneficial use of common labels comes from examining the homepage links which our subjects followed. 3 UCI subjects used search and the remaining two chose different links (one admissions, the other information and calendars). Had we more subjects in this experiment, we would observe a variety of links unrelated to each other. The other university subjects were definitely more homogeneous in this aspect. 3 DU subjects (%60) and 4 USM subjects (%80) used academics as their starting link. The label "For Students" found at the web site of UNC seems to create some confusion. The name 'student' may attract students because it promises to serve them as a group. 2 UNC subjects chose this link while 2 chose academics. Although UNC has the least visible search option (written in small characters) 1 UNC subject used it. Along with prior Internet experience, experience with a particular group of Internet sites (e.g., university sites) and/or knowledge of English may have confounded our data. We observed that most of the 9 excluded subjects chose links with irrelevant labels. The USM subject who started with choosing 'alumni and friends', which is completely irrelevant to our task, managed to meet our criteria.

Figure 3. Individual dependent variable measures for DU

Pop-up menus (DU) may also consume time since they are written in small characters and may be difficult to read. Some subjects seemed to be slightly surprised when a menu suddenly popped up. It may be that they were unfamiliar with pop-up menus and couldn't figure out whether the links contained in them were clickable or not. Two DU subjects did not even bother to test that but clicked on the main link instead, rendering the pop-up menu useless and ignoring the chance to chose a more specific link. It was observed that some subjects do take the time to read the rollover information (UNC) before deciding to click on a link. This takes some time, too. USM subjects spent the least time on the homepage. USM's homepage is the simplest page and it also takes the least space on the screen. The order of homepage times is very much plausible and in line with prior expectations.

Figure 4. Individual dependent variable measures for UNC

We had expected USM subjects to take the most total time. Instead, USM matched DU in shortest time. However, this is not an invalidation of our redundancy hypothesis since prior Internet experience occurred as a very critical confounding variable. While two USM subjects were slow and one was average, two were extremely fast in finishing the task. They just seemed to know exactly what to do which is most probably due to familiarity with web sites. As with UCI, we had a valuable experience concerning the redundancy issue. Speaking to himself, one subject uttered the words "But I had clicked on 'academics' before" when he encountered the redundant academics link. Also, we had to exclude 4 USM participants from our data. 1 of these quit the task and 3 passed the 15 link limit. In all cases, what made these subjects fail was the redundant academics link. All these subjects succeeded to the point of reaching the page containing the redundant link but did not know what to do thereafter. They thought the redundant link had the same function with the first academics link and they either used the back button or continued with an irrelevant link from that page.

Figure 5. Individual dependent variable measures for USM

This pilot study reached its goal of giving us an idea about what variables need to be taken into account in the study of the effects of site design on Internet users. Further research needs to be careful about prior Internet experience, experience with categories of sites, knowledge of English as the most important confounding factors. The results from this quasi-experiment were suggestive of the validity of both the redundancy and the common labels hypotheses. We also believed that in sites with rollovers and pop-ups time spent on the homepage would make up the highest percentage of total time. That is, people would spend more time on the homepage, but proceed quickly afterwards. Our data do not provide a clear picture concerning this issue. Along with controlling confounding variables, we need to formulate more specific hypotheses and think of supplementary measures. This study will hopefully guide us in doing that.

Anderson, J. R. (1995). Cognitive Psychology and its Implications. New York: Wiley.

Collins, A. M., & Quillian, M. R. (1969). Retrieval Time from semantic memory. Journal of Verbal Learning and Verbal Behavior, 8, 240-247.

Conrad, C. (1972). Congnitive economy in semantic memory. Journal of Experimental Psychology, 92, 149-154.

Quillian, M. R. (1966). Semantic Memory. Cambridge, MA: MIT Press.

To Page 1

Rating: Not rated yet