Ron George Experience Design

All over the web I see the word gesture used to describe every type of interaction on a natural user interface. Just because you use your finger or a stylus or an accelerometer, does not make it a “gesture.” Is this crucial? Not really to users, consumers, marketing, et al. But it is in being a good scholar and interaction designer to get your terminology straight. It also helps when speaking with other developers to have your vocabulary correct so they do not misinterpret your meaning or solutions. Let’s start with the classical, dictionary definitions:

Main Entry: ¹ges·ture
Pronunciation: ˈjes-chər, ˈjesh-
Function: noun
Etymology: Middle English, from Anglo-French, from Medieval Latin gestura mode of action, from Latin gestus, past participle of gerere
Date: 15th century

1 archaic : carriage, bearing
2 : a movement usually of the body or limbs that expresses or emphasizes an idea, sentiment, or attitude
3 : the use of motions of the limbs or body as a means of expression
4 : something said or done by way of formality or courtesy, as a symbol or token, or for its effect on the attitudes of others <a political gesture to draw popular support — V. L. Parrington>

Main Entry: ma·nip·u·late
Pronunciation: mə-ˈni-pyə-ˌlāt
Function: transitive verb
Inflected Form(s): ma·nip·u·lat·ed; ma·nip·u·lat·ing
Etymology: back-formation from manipulation, from French, from manipuler to handle an apparatus in chemistry, ultimately from Latin manipulus
Date: 1834

1 : to treat or operate with or as if with the hands or by mechanical means especially in a skillful manner
2 a : to manage or utilize skillfully b : to control or play upon by artful, unfair, or insidious means especially to one’s own advantage
3 : to change by artful or unfair means so as to serve one’s purpose : doctor

You can already start to see the differences for our purposes. One is emotional, symbolic, indirect. The other is direct or mechanical. There are 4 primary differences between the two and they are easily classified after you know them.

Manipulations

1. contextual – they only happen at specific location(s) or on specific object(s)
2. react immediately – there is a direct correlation in cause and effect between your interaction and the system (this does not include visual affordance)
3. can be single state, but are usually 3 or more states ( see Bill Buxton’s paper on Chunking and Phrasing )
4. direct (could possibly be considered indirect by way of augmenting your actual interactions with the reaction of the system) – your actions directly affect the system, object, or experience in some way

Gestures

1. not contextual – they can be anywhere in the system in location and time
2. the system waits for the series of events to complete to decide on how to react (again, this does not include visual affordance)
3. they contain at least 2 states
4. indirect – they do not affect the system directly according to your action. Your action is symbolic in some way that issues a command, statement, or state.

In Dan Saffer’s book, Designing Gestural Interfaces, (O’Reilly, 2009) on page 2 he states “for the purposes of this book, is any physical movement that a digital system can sense and respond to without the aid of traditional pointing devices such as a mouse or stylus.” That may be a simple way to define the types of interaction for his book, but generalizing them in that manner is incorrect. I think Professor Shneiderman’s seminal paper in 1983 was absolutely correct. Direct manipulation is just that, direct manipulation. When we start to discuss more complex chained movements that are commands, we need a new set of terminology. (http://en.wikipedia.org/wiki/Direct_manipulation_interface)

Manipulations are the lowest common denominator and the “catch-all.” They are the most prevalent and the most widely patterned because they are easy to design for, easy to understand, and very intuitive with expected results. Gestures are more complex and is what all designers strive to achieve. When trying to decipher if something is a manipulation or a gesture, unless it passes all 4 tests for gesture, it is a manipulation. There are very few true gestures in systems currently.

These have also been called direct gestures (manipulations) and indirect gestures (gestures). Calling them this is confusing the terms and can lead to errors in design or implementation. I leave you with a graphical representation of gestures vs manipulations.

I’m eager to hear any dissenting opinions. Please comment or drop me an email. I’ll also send a copy of this to Dan as well.