Multi-modal Info Kiosk

Multi-modal INFO Kiosk

"Great communication begins with connection"

Oprah Winfrey
Talk Show Host

Designing MUSE

Multimodal interactions provide more engaging experiences allowing users to perform complex tasks while searching for information. In this project, I am focusing on the interaction design of a multimodal info kiosk – called MUSE, Multimodal Info Kiosk for Shopping Experience – that enables users to communicate with the kiosk using visual information, touch and natural language. When using language, users can connect to the kiosk using their own mobile phone as input device. The connection is established by scanning a QR code displayed on the kiosk screen.

My work comprises a field study, interaction and graphic design, as well as work with our development team towards achieving a robust system architecture and implementation.

Challenges

Information kiosks are prevalent in office buildings, malls, airports, museums etc. offering a broad range of information, assistance and services. A common characteristic of these kiosks is their relatively rigid interaction style combining a graphical interface with touch input. As a result, the number of tasks users can perform is relatively limited. Our task was to re-design the kiosk UI in such was that the interaction becomes less rigid and less restricted.

field study

We conducted a one day field study across several shopping malls in Singapore to find out how people use info kiosks. We watched more than 50 people interacting with mall info kiosks. We took notes, pictures, videos, talked to customers and mall concierges. This data collection enabled us to make the following observations:

People tend to be confused by kiosks using simple glass instead of touch screen.
Directories are usually sorted by categories; however, most people seem to encounter difficulties searching through categories and prefer to search rather by shop name (see fig.1a)
Specific information is missing, i.e. information that is not directly related with a shop, such as a particular product brand, Wi-Fi configuration, car park fees etc. is not easy to find. People often use their mobile phone to search for such missing information online
Long waiting times during peak-hours – people usually have to queue for several minutes to use the kiosk or to consult a concierge.
Kiosks are often overloaded with info that people might not need, such as transportation (buses & MRT stops), nearest toilets and elevators, links for social media, i.e. Facebook, Twitter, Instagram, etc.
Understanding maps: searching a shop on the map seems to be the most common screen functionality. However, map navigation seems to be difficult to follow and people were often spotted taking pictures or spending time starring at maps in an attempt to remember location and navigation to the shop.

On average, it takes 3.5 min to find information using the kiosk; however, it takes about 2.5 sec to get the same information from a human receptionist at info counter.

Design SOLUTIONS

The information gathered during our field study allowed us to get a better understanding of shoppers’ needs. It also helped us to come up with the following design ideas:

Touch screen labelling: the kiosk should be labelled as ‘touch screen’ to invite people to use it interactively
Alphabetical directory search should be provided along with search for products by category
Missing information: since certain type of information seems difficult to find, we will implement the kiosk as a combination of graphical and natural language information exchange; in this way, the user interface (UI) is kept minimalistic while users can ask questions about topics the kiosk doesn’t explicitly display – see a dialogue example in tab. 1.

Long waiting times: the queues during peak hours can be avoided by enabling users to connect simultaneously with the screen; such multi-sessions can be established by using a third device – in this case, a mobile phone that connects users with the screen via a QR code.
Info declutter: the kiosk should present only information related to shops and restaurants available in the building
Maps: the QR connection will allow users to download and store navigation maps.

Our intention is to enable a faster and more efficient exchange of information, such as between humans. The key ingredient is getting automatic access to information relating to the building. Within the framework of a ‘smart city’, such information will soon be easily available.

mock-up Prototype

Based on our observation, we conclude that people using an info kiosk are mainly interested in three types of information:

What business/shops are in the building? – Business directory
Where can I buy a certain product? – Product / Category directory
How can I go to a certain business? Maps / Navigation

At client request, a 4th info category was added concerning Promotions offered by the shops.

We propose a simple, decluttered kiosk UI that mirrors shops information, product directory and map navigation. enabling multi-session through a QR code connection, natural language input, as well as downloading available information such as maps or promotion coupons. The information on the kiosk should be available in at least 2 languages: English and Chinese.

A first mock-up was created using Balsamiq as shown in Fig. 2.

GraphicAL uI

The graphic UI was initially provided in three different color skins: blue, light blue and brown (see fig. 3a, 3b, 3c). This enabled our client to decide on the most suitable color through comparison and by taking into account their own preferences, as well as other physical building design elements. The figures below show the main screen page when the menu button is engaged, i.e. background is blurred.

The following design elements were implemented:

Carrousel view to search for information alphabetically (see fig. 3 or by category (see fig. 4)
Shop cards with information and links to maps and shop promotions (see fig. 5)
A text area below the card carousel to capture the natural language interaction (see fig. 3); the text area appears only after users connect to the kiosk using the QR code (see fig. 4)
A microphone icon that reacts to speech input coming from the mobile phone

Once the users scan the QR code and get connected to screen, they can download maps, promotion coupons or any other information the kiosk can provide. To ask specific questions in natural language, users would use the microphone of their own mobile devices All graphical elements were created using Adobe Photoshop and Illustrator.

System Architecture

The kiosk is running Windows 7 Embedded Standard on a 1.99GHz quad-core Intel Celeron CPU and 4GB of RAM. The vertical touch screen measures 42″ across and has WLAN connectivity. The audio output is provided via an external speaker.

Users with a mobile device may connect to the kiosk, as long as the phone has a wireless internet connection, a QR scanner and a web browser. Once connected, the mobile runs a client for natural language input (see fig. 6).

The client consists of a text box to input speech (or text) and a button (“Ask me”) to submit the processed text to APOLLO – a dialogue platform developed in our department – for more information, read here. To start talking, users have to activate the speech recognition routines on their own devices.

Both kiosk and client GUI are HTML pages. As such, any potential user can connect without having to download a special application.

The design steps of the multimodal MUSE kiosk was published in:

“Designing MUSE: A Multimodal User Experience for a Shopping Mall Kiosk”