A museum informatics user test was performed with the volunteer assistance of Margaret Herndon, a 50 year old professional manager who frequents local prestigious museums with her husband, is a general user of the Internet (ex: shopping regularly online and reading news) and only uses museum websites in a cursory fashion, such as to obtain physical site visitation information. The museum website chosen for this usability test was from the Norton Simon Museum located in Pasadena, California (http://www.nortonsimon.org/). This is a museum that she frequents on occasion but doesn’t use the website.
Eight user tasks were presented and the tester recorded each of these tasks by paper, by watching, and by using the recording software Jing.
1. “Can you provide me with hours to the museum, and tell me when it is closed or if there are any free days to get in?”
2. “Can you find any information about any works of Vincent van Gogh?”
3. “Can you tell me what the current traveling shows going on are today, the temporary collections?”
4. “If you wanted to purchase an item from the current featured exhibition of Ingres, how would you do this online?”
5. “From the website, can you tell me who is Norton Simon, why is the collection so important and what the mission and vision of the museum is? What makes the Norton Simon Museum so special? With only using the website, can you elaborate on that?”
6. “Can you locate a floor map for the Norton Simon Museum and tell me where I can find the Asian collection?”
7. “Are there any materials, any guides online to the current exhibits, particularly ‘Portraits After Ingres’?”
8. “Can you locate a piece of art by its material, specifically sculpture? Can you find the European sculpture in the Norton and then locate any sculptures by the modernist Barbara Hepworth?”
User Task One:
“Can you provide me with hours to the museum, and tell me when it is closed or if there are any free days to get in?”
Margaret took a cursory look over the home page and couldn’t locate anything about hours. Looking at the navigation bar, she thinks it has to be under “Visit” or “Information.”
Margaret: “I don’t see hours. There’s something called visitor information.”
She selects “Visitor Information” under “Visit” since it looked like a likely option through her process of elimination. The subject finds what’s required of the task regarding hours but struggles with the next part.
Margaret: “Here are the hours. It is closed on Tuesday. That’s peculiar – normally museums are closed on Monday. This is very hard to read. It is very small and hard for my eyes. I can’t find free time information.”
She continues to scroll up and down and finding it hard to read as she squints. It begins to fatigue her. Finally, she finds text under “Admission Prices,” detailing that first Fridays of the month are free.
Fig 1. Visitor Information
Observation: The subject wanted to first find that information on the home page. When she could not, she started first by looking at the main navigation bar. The main navigation bar was relied on heavily throughout the test. I agreed with Margaret that is was a bit hidden to not have the words “Hours and Admission” listed since in that same “Visit’ dropdown, “Directions and Parking ” is clearly listed.
She comments: “Anything can go in Visitor Information. Then they should just have one page for all that information.”
Even though Margaret found the hours and admission information from the “Visit” navigation bar, she did not see a section called “Visit” on the home page – even though this was where she wanted to first look. This information is tucked in the lower-right hand corner of the page and is very hidden from view. Interestingly, it lists “Hours” here but not in the dropdown as Margaret had suggested that should have been better worded. Also, this selection takes you to the same “Visitor Information” page from the navigation bar. So, even the “Hours” designation on the home page should be “Hours and Admission.” This is an example of a user showing how even the most basic museum request need of “hours and admission” can be unnecessarily veiled.
Fig 2. Hidden Visit Option on Home Page
User Task Two:
“Can you find any information about any works of Vincent van Gogh?”
Margaret: “Assume under collections, different choices, European, I guess I could pick European Art 19th century. Well, now it takes me to ‘Highlights’ and no images look like van Gogh. Here’s ‘More Highlights’ and I see a screen of small image icons. It’s 21 pages. This will take a long time. I’m tired.” [Shoulders droop.]
Fig 3. Looking for van Gogh in European Art 19th Century
Margaret: “So, ways to find van Gogh. There’s ‘Advanced Search.’
Oh, here can type by artist. No results found? That is strange. Well, you can look under artist on this page – browse by and nothing listed.
[User tried to search under Monet and see the function worked. Finds Monet, so looks troubled. Margaret still can’t believe there is way to find van Gogh on here.] It’s not showing van Gogh and went under artist and 19th century and don’t want to go back there.”
“I know his work ‘Mulberry Tree’ is there. I can go by title ‘Mulberry Tree.’ Here it is. That’s strange. They have van Gogh and one of the special things about Norton Simon is they have a number of van Goghs, which is very unusual. I don’t know where else to look. Under gallery, there’s a picture of his paintings. ‘Browse by Gallery’ just gives ‘Exhibitions.’ I don’t know [dejected.] I just have to go to the museum to see the van Goghs.” User gives up.
Fig 4. Looking for van Gogh under a title work or in the gallery
Observation: This user task was complex and fraught with difficulty, frustration, and eventually ended without complete success. However, this was not a user error but issues with the interface and cataloging options. Since Margaret was a casual visitor to the Norton Simon, she knew that he had collected a significant amount of impressionist art, and in particular, a stunning selection of van Goghs. To not find van Gogh in the collections was not only extremely frustrating for her but an example of a serious flaw in the website. The first flaw was in the “Browse by Artist” option. This should have been an easy to find exercise since it was an alphabetical listing/choices for artists. However, when Margaret searched under “V” for van Gogh, it was not there. Those who put the alphabetical listing should note that often van Gogh is sometimes listed under “V” rather than “G.” Patrons don’t normally think of “Gogh” as last name of Vincent even though “van” is “correctly” alphabetized as a subordinate. When Margaret had to dig deeper and use the advanced search screen, she did not see the little pop-up line selection when she typed in “van Gogh” that offered a choice “Gogh, Vincent van.” It was incredibly small, in bright yellow, and was easy to miss.
Fig 5. The problem overseen – smart selection “Vincent van Gogh” missed and search by artist is under “Gogh” not “van”
User Task Three:
“Can you tell me what the current traveling shows going on are today, the temporary collections?”
Margaret: “Usually don’t they have them on the home page? [Exasperation, deep sigh.] There’s no home button. There’s no home button on the bottom either.”
Margaret spends a great deal of time looking in navigation bar, using the search option. Tester states to locate another option besides the browser back button. After clicking on practically everything on the page, she finds that the Norton Simon Museum logo is clickable and can go back to the home page with. New lesson is learned.
Margaret: “I didn’t realize that clicking on the museum logo would take you back to the home page. I still want my “home” selection. When I’ve used the Getty and Huntington websites, they both have a home button – because I always like to go back home so that I know where I am.”
User then completes primary task.
“I like the showing of images of the exhibitions. They have two exhibits: Rembrandt and Portraiture after Ingres. How annoying, it’s hard for me to read about them when they keep flashing images on the screen above. I liked it at first that they had the images flashing of the current shows but now they are just flashing images from the Norton Simon collection. How annoying. It’s making it difficult to read.” [Blinks, forehead creases, eyes, squint.]
Fig 6. “Current Exhibitions” under home page
The tester suggests there is another place to get information about the exhibits. Margaret looks in disbelief, as to say “What more? I’ve already found it.” But she obliges that “nasty” tester and continues to look.
Margaret looks under the “Exhibitions” choice on the main navigation bar. She chooses “Current Exhibitions” and it brings her to the immediate information. This subsequent task is completed with ease and swiftness.
Observation: As a tester, I tried to use words that didn’t simply say “Exhibitions,” since that would have possibly ended the task in one minute or so. Instead, I suggested other words such as “traveling show” or “temporary collections.” This caused Margaret to think a bit and think back that often museum websites post their “hot” temporary exhibits on the home page. This should have been straightforward except for three things – one, there was no home button she was used to in order to navigate to; two, it was difficult to read those exhibits when there were flashing images on the home page; and three, the tester required her to locate the information elsewhere besides the home page.
Many user studies have brought up the fact that no matter what level of Internet expertise, it is most common to go back to the home page to reorient oneself. Margaret was not able to this because she could not locate any home button or home hyperlink. This is not her problem. More experienced users realize that often the logo is what is used as a home page. However, this is not evident to several users, and really should be a choice selection no matter what. The ability to go to the home page is also the ability to reorient oneself in the searching and browsing capacity. Note the fascinating words she said: I want a home button “because I always like to go back home so that I know where I am.” This is psychologically and powerfully telling. I would suggest to the Norton Simon Museum to put a home button/home hyperlink as part of their navigation scheme based on this scenario. Also, when she did finally find the home page and saw that the current exhibits were shown as she thought on the home page, she was disturbed by the flashing images above and couldn’t read or focus enough. At first she liked it because the flashing images or “slide show” went over the current exhibits and included some text. But then the slide show started to show other images that had no context except for being in the museum. This showed how that element made her not want to stay on the home page very long and could signify that information is then missed on the home page. Lastly, she was able to determine that this information could be found under the “Exhibitions” option on the main navigation bar and thus reflected an easy option choice.
Figure 7. “Current Exhibitions” choice
User Task Four:
“If you wanted to purchase on the current featured exhibition of Ingres, how would you do this online?”
Margaret: “Well, normally you would go to the store to do any shopping. And I love to go to museum stores to shop. So, here’s “Store” [located on main navigation bar] and ooohh… I want to go directly to the store. They just give me choices [dropdown selection of books, DVDs, gifts, etc. then accidently clicks directly again on “Store” and it brings her in. What is not so obvious is that the toolbar looks the same as before but now they are store choices and unbeknownst to Margaret, she in a different website altogether: http://store.nortonsimon.org/] It looks like…. They don’t have any. [Moans.] Just want to search the store.”
She looks under “Books” and “Museum Publications” and gets overwhelmed by icon choices. Margaret doesn’t want to go through all the pages. [Sighs of exasperation.] She finds by accident that the site has a “Search Store” box and types in “Ingres.” She easily finds cards and a special exhibition book.
Margaret: “They should really bring out their exhibit. I don’t really like this website – shopping it. Look, it says it’s an online store. Look, I can’t go back to the museum. I want to get out of there. [Uuuuuh.] I don’t know how to go back. [Face gets read, brows arched, fuming.] I’ll just hit the back arrows. That’s annoying. Navigation bar looks the same. [Clicks by page by page and finally back to museum page.] That was nearly eight pages back. I don’t like that.”
Figure 8. Museum Store
Observation: Some significant items were noted in this task. First of all, Margaret is an avid shopper and spends most of her money at museum shops when at the museum. She likes to collect books and items from all the special exhibitions she attends – even if she never uses them later. It makes her feel that she has a “part” of the collection or of the “experience.” With its museum website, the Norton Simon should be targeting these eager buyers. However, Margaret’s exercise brings out some inherent defects in the store purchasing experience that could stop someone from shopping online. The three shortcomings included: 1) No home page to navigate back to the museum; 2) No visual differentiation of the navigation bar, the logo, positioning of times to tell the user that he is in a completely different website and that it is only the store site now; 3) Does not feature special exhibition and collection items on the store home page; and 4) Hides the “search store” option in same place as “search site” option and instead should be search within the body of the store site.
User Task Five:
“From the website, can you tell me who is Norton Simon, why is the collection so important and what the mission and vision of the museum is? What makes the Norton Simon Museum so special? With only using the website, can you elaborate on that?”
Margaret: “Let’s see here. I’m going back to the home page. Maybe there is something on the home page. [Finds nothing.] Where is “About?” Should be under “About.” [Looks under “Information” on main navigation bar.] Here it is. [Locates “Norton Simon Bio” on dropdown.] Why is it so special? They should really bring it out. [Looks at additional dropdown menu choice of “Museum History.”] Museum History. Doesn’t tell you anything about mission statement, Poor, very poor information. Just lots of text. They could have mentioned a quote of his here. Oh here. I’ve seen this documentary on in the museum theater. [Bottom of Norton Simon bio discusses documentary on Norton Simon with hyperlink to information on it for viewing in the physical theater space. Clicks on it but cannot be viewed online.]
It’s done by the Guggenheim. It’s very professional. They should really have this here. I should let them know.”
Figure 9. Hidden Jewel: Norton Simon Documentary
Observation: This was an exciting exercise with much enlightenment and new information given by the user. Here, Margaret has become a valuable resource to the Norton Simon and is actually providing solid re-design information without knowing it – simply being a good Norton Simon aficionado. First, this exercise brought up that the Norton Simon museum does not currently present or offer a mission or vision statement on the website. In addition, it is hard to gleam from all the text what the museum is all about. It would behoove them to change “Information” to “About” and in that “About” section really discuss and show the “aboutness” of this museum and its collection. Margaret noticed interesting quotes within the body of the text that she said she would have liked to see brought out. Most importantly, she recognized a crown jewel that the museum website is not using – a high-toned Guggenheim-produced 30-minute documentary that should be on the website prominently in the “About” section, giving potential visitors a great overview of Norton Simon and the collection.
User Task Six:
“Can you locate a floor map for the Norton Simon Museum and tell me where I can find the Asian collection?”
Margaret: “So, that means… [Raises voice.] I WANT THE HOME BUTTON. [Uses back button again to go to home page.] It’s not here. I think I remember… [Looks under “Visit” in main navigation bar.] There’s no floor map. [Pauses and keeps looking.] Here is something called ‘Gallery Map.’ Oh, that’s beautiful. That really should be on the front of the website. It shows what the museum looks like and what the collections are. Here it is. It is ‘South and Southeast Asian’ and it is in the lower level of the building. [User completes task in efficient manner.]
Figure 10. Floor Map is called “Gallery Map”
Observation: Again, we notice that terminology is an important navigation and information-seeking item. I used the term “floor map” which is common in museums. However, it took her just a bit of time as she was seeking specifically “floor map.” The name given by the Norton Simon Museum was “gallery map.” This was a bit of a mind-bender, but she selected it anyways. She was astounded not only the beauty of the map, but that it really showed visually the museum architecture, gardens, collections, layout, and mapping captured in one image. She noted that it should be used more prominently by the museum – seemed too hidden. Another fascinating piece that came out of this was again two-fold: 1) Terminology and 2) Home Page Hiding. There was actually another choice to find this map and it was located in the “Visit” section on the lower right-hand corner of the home page as “Interactive Map.” Here we see that even though Margaret suggested that the museum highlight this and put it on the home page, it had, except it didn’t highlight the map and hid it under different terminology altogether – “Interactive Map.” Through this activity, Margaret was able to identify significant flaws that took away from the excellent, interactive floor map that the website actually contained.
User Task Seven:
“Are there any materials, any guides online to the current exhibits, particularly ‘Portraits After Ingres’?”
Margaret: [Breathes heavily.] “Well, let’s see. That should be under ‘Exhibitions,’ then here ‘Current Exhibitions.’ And they have, I will go into that. Well, I don’t see anything. They give me choices – press release, podcasts, and lots of text. I don’t see anything. [Scrolls up and down, heavy breathing.] Now, what was the question again?
I don’t see guides. Maybe under ‘Information,’ under ‘Visit,’ under ‘Education?’ No. There should be something under ‘Guides.’ [Goes through each choice in navigation bar and finds a section called “Multimedia” and finds “Brochures.”]
Brochures? Is that it? Why would they stick it here – under multimedia? Here it is – a brochure on ‘Portraiture after Ingres.’ [Comes up in a viewer in small size with control buttons.] Oh, how nice – you can actually view it like in a book format. [Struggles with control buttons.] Oh, you can page through it like a real book. That’s neat – it makes ‘turning pages’ sounds. Oh, I want to make it bigger – I can’t read it. [Struggles to enlarge and finally does.] Oh, this is really nice. I love it. I can read it – very lovely. Very nice – I love their literature and I can still see it here. Oh, I’d like to print it. There is no print button. At least I can see this here – very nice. Oh, darn, now how do I get out of this and go back HOME?”
Figure 11. This is a brochure?
Observations: This exercise brought up some frustration by the user. It was interesting to note that the strain of looking for this item caused her to forget the question and task at hand. Locating literature or materials on the current exhibits was not clear. Margaret spent a great deal of time looking at all the options she could figure out. Finally, she discovered “Brochures” under “Multimedia.” She found it confusing to find this under multimedia. This brings up an inherent discrepancy again about how to define – brochures are often thought of in a print or print online fashion and not as multimedia. However, when you use this, it is somewhat multimedia-like because it paginates, makes sounds, is graphic, etc. Perhaps, it more logically should be termed “Interactive.” Also, there are some users who will not understand what the category “Multimedia” is. However, it could be beneficial as a “catch all” for interactive, audio, and video components, but wonder if this term is outdated. This would be interesting to further test. Again, this “brochure” function should be something to bring out more in the website and is too hidden from view. It as well can work as both a promotional device to get people into see the exhibits or for reviewing the material afterwards for better absorption.
User Task Eight:
“Can you locate a piece of art by its material, specifically sculpture? Can you find the European sculpture in the Norton and then locate any sculptures by the modernist Barbara Hepworth?”
Margaret: “Ok, there is a search button but it doesn’t have advanced search. [Goes back to home.] I don’t see anything here. How about ‘Collections’? [Dropdown with choices of time periods and browse options.] AAAH. I don’t like this dropdown. Browse by Artist, Title, Object Type, Provenance… I don’t see it. Here’s Advanced Search. Well, here it gives me artist, title, type, origin? What’s that – oh, origin – that’s different places? [Selects Object Type]. Now that’s strange – I wouldn’t have thought to look this under “object type.” [This dropdown lists the material types and user locates sculpture choice.] What a strange name. [This brings up 62 pages of small icons of all varieties of sculpture, but finding difficulty narrowing down.] This is too much to look at. What if I wanted to see certain sculptures, modern sculptures? Aaargh. I’ll go back to advanced search again under collections. Oh, I see, I can type in European under Origin and it gives me lots of choices. Oh, I see now I can search for Artist in this search. I’ll type ‘Hepworth’ and see. Oh, here she is. Here are her pieces. Oh, how nice. I really like her work. Yes, that’s very nice.” [User smiles finally, exhibits first signs of user satisfaction – perhaps because she has reached the end of this usability test.]
Figure 12. Collection “Advanced Search” Option
Observations: Oddly, this task produced the greatest user satisfaction and perhaps the only time I truly saw Margaret smile with contentment. I personally was surprised and thought this would be the most difficult task and one where she would throw up her hands in dismay or at least throw something at the tester. In fact, though it was difficult, she worked her way through and then located the Collection “Advanced Search” option. Here she had some good choices to narrow her selections from once she learned what those were. Once again, we look at issues of terminology. The use of “Object Type” to describe the material or form of art seems very intellectual or too “insider-like.” This threw her off immediately from the choice of “Browse by Object Type” under “Collections.” She only discovered what that meant after looking at this dropdown choice in “Advanced Search.” This exercise pleased her immensely because it allowed her to locate specific items in the collection and give her a sense of “power” and “ease” in accessing the collection.
Conclusion and Readings
In “Come on Down! A Game Show Approach to Illustrating Usability Evaluation Methods” by Michael Twidale and Paul Marty, the authors demonstrate the quality of material and insight that can be produced from quick or rapid usability testing. Though the game show testing methodology was not applied here, it still demonstrates the efficacy and profoundness of what emerges out of even simple, face-to-face user testing. Twidale and Marty comment on “the remarkable robustness of user testing and that any test is better than no test. This is demonstrated effectively in the user test with Margaret Herndon. This brief and informal user testing provided significant findings that would be highly beneficial to the Norton Simon Museum enterprise. In actuality, it would be cost-effective.
Margaret herself brought out changes that the museum should do in its website to promote itself – the hidden Guggenheim documentary on Norton Simon, the redesign of the Museum Store highlighting special exhibit items, and the observation that the “Floor Map” with too many names should be a big piece of the website. In an additional article by Twidale and Marty, “Lost in Gallery Space: A Conceptual Framework for Analyzing the Usability Flaws of Museum Web Sites,” the authors stressed the use of scenario presentations or questions, in high-speed format, to facilitate locating top-level usability flaws and patterns. A fascinating result of this user test supported their findings. In each task, particular flaws were brought to the surface quickly. Many of these were semantic-based, others were orientation-based and some were quite insightful in their blatant “errors.” One of the several usability issues mentioned by Twidale and Marty is the characteristic of “museum professional” mindset in designing websites. This is observed in controlled vocabulary lingo, such as Norton’s use of “Object Type” when Margaret was trying to find an art piece by material. Again, the authors emphasize what seems to be common sense but so terribly overlooked in seeing the value of usability and its cost-effectiveness over the expenditure on website design without it. They state succinctly that “not analyzing for usability can mean users will fail to discover and appreciate this rich content…” Through the user test, Margaret found these rich resources, such as the Interactive Floor Map, the unused Guggenheim documentary on Norton Simon, the Paginating Online Brochure, and the powerful Collection Advanced Search Screen. By performing such a fairly simple and inexpensive usability test, the corresponding results were provoking, essential, and, of course, usable.