We are working with the University of Michigan Exhibit Museum of Natural History to design an augmented reality interface for an iPhone app. In a museum setting, users would view an exhibit through the iPhone camera, and the app would recognize it and overlay information about the exhibit and visualizations about its relation to other exhibits. This is a great opportunity to brainstorm creative ways to explain works of art or science without a tour guide and without the limitations of audio-only pre-recorded tours, and really define how ubiquitous computing can be designed to better inform us of our world.

How does the app work?

Although each user has different interests, needs and habits, when information is aggregated from many users, patterns arise. In the museum, this information can be visualized to represent how people interact with exhibits. The ARMuseum application works by grabbing and storing from users, and using the information to generate maps and augmented reality annotations. One assumption we have made, based on interviews with the museum director, is that the exhibitions in the University of Michigan Exhibit Museum of Natural History don’t change often, so that all the data are useful in the long run. However, in another museum setting decisions may have to be made about data that was collected regarding exhibits that have moved or retired. The following are the data that will be collected during the tour in the museum:

1) First, the app collects data from each individual user by recognizing what exhibits the user is standing in front of via the iPhone camera, as well as the date and time, and for how long. It also downloads data relevant to the exhibit itself, like comments related to the exhibit and the position information for them, and this information drives the interactions possible to the user.

2) The information collected in step 1 is used to build a social map that can help us visually analyze the foot traffic and the popularity. When looking at the map, users can estimate how crowded or popular each route might be by seeing the difference of color and thickness of lines. The users can also filter the information by selecting different dates of weeks, months or so to see the changes dynamically on the map.

3) Users can view, reply and create comments for each exhibit, and the notes are displayed in an augmented reality format, marked by rectangles in the viewfinder. Information about the note’s 3D position around the exhibit, and date and time, will be saved into the database each time someone writes a note. This way, other users who are interested in the exhibits can view the notes at the same position as the authors, providing valuable contextual information for their exchange of knowledge and comments. To ensure that the highest-quality notes are the most prominent, reward good notes, and prevent vandalism, viewers can rate a note with a thumbs up or thumbs down. The notes are sorted automatically by ranking, but users can choose to sort by time posted.

Interactive social map

Time filtering

In addition to being able to see who goes where in the museum, the social map has several dynamic filters that let users of ARMuseum explore the data. One is the time filter, which enables a slider at the bottom of the screen that the user can touch and slide to see the paths build over time.

Click on the image below to see a larger version with an image map that makes all screens clickable.


The user can also switch to the floor-level view and see the social map overlaid in the viewfinder.

Attribute filtering

Bubbles indicate where users have favorited an exhibit. The scale of the bubble grows as more users tag items as a favorite.

Bubbles indicating zones where many users favorited and exhibit, or left dicussion “notes” can be viewed at the same time.

This screen shows the traces of other users’ paths, as well as indicating where notes and favorites were left.

Users can select an exhibit on the map to view additional information

Once an exhibit has been selected, an informational window displays social information.

Exhibits


“Stickies view” is selected by default. When someone approaches an exhibit, ARMuseum’s camera recognizes it and automatically pulls up notes overlaid in the view. The notes are skewed rectangles, designed so that people can figure out where to position themselves to see the same thing the note writer did when they wrote the note. This provides valuable context.


Selecting a particular rectangle brings up a note. The user can read the note, thumbs-up, thumbs-down or tap the note-writer’s name (”DanTheMan”) to see the note-writer’s profile.


Selecting “DanTheMan” offers options to see his profile, view his museum path, view his favorites on the map, and view his notes.


This is DanTheMan’s path, highlighted against all the other paths in dark red.


Selecting the “list view” simply lists all the notes left on the exhibit, with a two-line snippet. Selecting the note box brings up the full comment.


To write a note, select the “write” icon on the left. It brings up this dialog, which takes a picture and records the time as well as where in space the picture was taken. The user presses the camera icon on the bottom, which brings up the next dialog, writing a note.


The user can write a note and finish the process of annotating the exhibit.


Selecting the heart in the upper-right corner favorites it.