I’m subscribed to just about every RSS that contains the words NUI or Multi-Touch in the title. One of the better one’s is http://theclevermonkey.blogspot.com. He made a post the other day about how impressed he was with a set of videos on youtube of a mockup from an anonymous person based on what they thought Apple’s new tablet will consist of interaction-wise. It was titled “Move Over Jonathan Ive, Hear comes Jon Doe”. That grabbed my attention.

Jonathan Ives is one of the best designers working today. I think he has great talent and he got some wonderful opportunities that really allowed him to flourish. He is great, but the only reason he is great is because he has made several errors. Big errors, as we all do in our careers. Bill Buxton was talking to me about this very thing a few days ago. Great artists make errors, learn from them, and move on. Jonathan has made his share of errors, as well as his share of wins. So, no question about Jonathan’s skills, but who is this John Doe?

I made an offhand comment on theclevermonkey about how unimpressed with the gesture language that was demonstrated in the videos. He replied that I should write a constructive and helpful post to point out a few things. So here we are.

The first video in the series:

So, I will breeze over a few of the main concepts that I feel the person missed in their mockup. I only watched 2 of the videos, and will only comment on one, but here is what I gathered from it.

• In watching the gestures I see confusion between single tap and double tap. At around 4 seconds he does a single tap to change its state after he used single tap to get it into the state? Then does another double tap to zoom out? I’m not quite sure I understand the metaphor around each of the taps. If single tap is select, then how can it also be activate and unzoom/state change.  It seems here that the user is going through a pre-determined path of usability rather than real motions.
• At around 10 seconds after the OnScreenKeyboard use, he does a single tap all the way across the screen on a very small hit target. This seems problematic and does not obey Fitt’s Law.

Fitts’s law (often cited as Fitts’ law) is a model of human movement in human-computer interaction and ergonomics which predicts that the time required to rapidly move to a target area is a function of the distance to and the size of the target. Fitts’s law is used to model the act of pointing, either by physically touching an object with a hand or finger, or virtually, by pointing to an object on a computer display using a pointing device). It was proposed by Paul Fitts in 1954.

• The closing of the keyboard should be thought about in a broader term. Why have the button so small? Why have the button on the other side of the screen? Why not put buttons on both sides to accommodate ambidextrous bi-manual input. Why not change its color slightly to ease speed of use? A very thin row of black buttons is not conducive to a natural interface.
• at 17 seconds he uses double tap twice. Once to do an “execute” on the video clip, and then again on the File Explorer window to perform a maximize/full screen. Unless in this operation if you “execute” the File Explorer that also goes into a Slideshow mode? I’m hazy on this.
• at 20 seconds he does a double tap on an image in the slideshow and that propels him into a scatterview experience? Maybe that is a Mac thing that I am unaware of, but if he just performed an “execute” on the icon, why does it go into scatterview?
• at 22 seconds he uses two-fingers to draw a marquee around items which also produces a context menu. If you had a context menu pop up everytime you selected a group it would not be a good experience. Menus need to be deliberately activated, not activated as a secondary action. There would be emails sent. 🙂
• at 27 seconds he uses two-fingers to then move the carousel around? Why has movement now shifted chords? Two-fingers were just used to draw a marquee and do a group select, and now they are used for movement? This is a confused metaphor and not consistent.

Thats pretty much everything in the first video besides the simple movements with single touch. To really be critical I would also say that using single touch for movement on a background is a little easy don’t you think? Certain things need to be forgivable, the background being one of them.

My main critique over the set is that multi-touch means just that, Multi-Touch. That means more than just two fingers side by side, or even three fingers next to each other. It means thinking about systems in a whole new way. What about offhand manipulation? What about modifiers? Think about occlusion and ease of use. If you are playing a game do you really want to have 3 fingers on the screen to run? then keep them on the screen the whole time? As the author noted in the last movie he had doubts.

To be more helpful and instructive I’ll throw out a few ideas. Why not use the offhand as movement and the primary hand as action? So, with your left hand you place down a finger and hold it outside of the quick menu area, this establishes the center of movement, now with any other touch in a direction near your initial touch you are running. This leaves your other hand free to perform actions that might be needed. Now with another tap with your left hand your running is now ‘frozen’ forward so you can remove your hand. Your character is now running in a direction that you chose without much primary attention. Want to change directions? Use two-fingers and slide. Want to stop running? Two-fingers and slide down. Side-step? You get the idea. Using the initial two taps to set motion then two fingers at the same time to give more detailed instructions eases the attention you would need to give it. Think how fast you could do that?

That ends the lesson for tonight. Overall, great video. I was very impressed with the videos themselves actually. High quality for a mock-up. Now hopefully you understand my thoughts when I see a lot of passion in this effort, but not quite enough expertise. The thing is …. expertise can be learned. Passion cannot. I hope he continues down this path, we could always use more passionate designers out here.

One of the main thought threads going through work and through design as a whole is to know your history. Things have been designed, created, used, and not used over the years. It is of being a good scholar and being a good designer, to know your history. Give credit where it is due.

This is of a video of Alan Kay presenting Ivan Sutherland’s Sketchpad from 1963. Yes, let that sink in. He was using a pen on a screen in 1963, which is a full 20 years before Macintosh and about 12 before Microsoft came to be. It was a seminal paper when he wrote it, and it was way ahead of its time. Even today’s programs do not encompass all the features that he had.

Here he is on Wikipedia.
Here is a link to his paper, “Sketchpad, A Man-Machine Graphical Communication System.”

You know, there are not that many things greater than seeing something you worked on being put to practical use. I think the only bad thing would be if they were complaining about it. Here, they are absolutely loving it. Granted, I only did a little bit of the work and there is a large team of great designers responsible. I was in on it and loved being able to contribute.

I think the visuals are stunning as well. I love seeing how fluid the animations are. Lovely!

I have been off of the Surface Team for months but I talk to the current members almost daily. I’m still doing Natural User Interface work, just not directly for them. Its so great to see what is still coming out of that great team over there.

This video demonstrates a number of new multi-touch software applications, designed using N-trig’s DuoSense® technology, in collaboration with some of N-trig’s ISV partners, including FingerTapps, Natural User Interface Europe AB, and SpaceClaim. The applications show full multi-touch utilizing up to four fingers, and range across areas of interest from consumer gaming, multi-media management, retail shopping and high-end engineering applications. More info on www.n-trig.com.

The most important thing to pull out of this hardware is that it is dual-mode, which means it can handle touch with fingers as well as with a pen. This is exactly what I am hoping for in the future of all hardware.

A preview demonstration of modeling in 3D using a SpaceClaim’s prototype multi-touch user interface that will be added to SpaceClaim this fall. Also includes clips of ANSYS Workbench, Blue Ridge Numerics CFdesign, and Bunkspeed Hypershot, showing some finger dancing that you can do without multi-touch.

There is a lot of random talk about multi-touch out right now due to Apple’s recent “leak” about their possible tablet. Talking to a few Apple fans I came to a realization: they had no idea that multi-touch was out right now. Absolutely clueless that Windows 7 supports MT (multi-touch) and that there is a very large array of hardware that you can purchase to take advantage of it. I was actually a little amazed, but not because of the simple fact they didn’t know about it. I was amazed because they seemed so versed in the subject, but had not even done a simple Google Search to look around.

MT has been around quite a long time. Bill Buxton’s very thorough page written on the subject gives a great history and is far better than I would want to replicate here. I will quote a great line from it though, as it definitely deserves it.

Multi-touch technologies have a long history. To put it in perspective, the original work undertaken by my team was done in 1984, the same year that the first Macintosh computer was released, and we were not the first. Furthermore, during the development of the iPhone, Apple was very much aware of the history of multi-touch, dating at least back to 1982, and the use of the pinch gesture, dating back to 1983.  – http://www.billbuxton.com/multitouchOverview.html

I highly recommend reading through his page.

I have tested, played with, and tortured just about every multi-touch piece of hardware out there and I can tell you that there are some stellar pieces of equipment floating around. The Touchsmart’s are very nice. I currently use one of those as well as a few other devices.

Here is the Official Touchsmart Page as well as some nice movies showcasing the technology and features.

Remember to install the Windows 7 Touch pack if you get one to really see some of the great things that you can do using ….. both hands, or one hand, or whatever!

The Windows 7 Touch Pack includes:

• Microsoft Surface Globe
• Microsoft Surface Collage
• Microsoft Surface Lagoon
• Microsoft Blackboard
• Microsoft Rebound
• Microsoft Garden Pond

If anyone has any user experience feedback about these products, feel free to drop me a line.

InkSeine, Rethink your Ink!

If you don’t know what Microsoft Research does, go take a quick peek at their website. Microsoft Research is think tank at MSFT that explores, develops, designs, and tries to implement new technologies and thoughts. They are not harbored by having to make money. Its really just like a wonderful adult playground where your imagination is the only think stopping you.

They do some amazing work over there and some very interesting projects. They do everything from researching ways to display information, to ways to gather information from the bottom of the ocean. One thing I am particularly interested in is the project Ken Hinckley is working on, InkSiene.

Now the interesting aspects of this project are numerous, but he touches on some great things. Here is his list of features:

These are some of the cool features offered by InkSeine:

Blaze a super-fast trail by using radial menus for pen commands.
Just hover to preview the commands. Touch down in the center and
stroke towards the command you want to activate it.

Select existing ink right on the page to initiate a search.

Create queries in the context of your notes to keep track of searches,
reuse them, or return to them later.

Easily find emails, documents, ink notes, or web pages without
leaving your notes.

Drag links to the documents and web pages you find right into your
notebook.

and lots more…

I encourage everyone to go check it out and let me know what you think. To be honest, its not very often you get state of the art inking software for free, so go download and install it.

The interesting part about this is that Ken did this totally in a bubble based on user feedback. Rather than stick with the same methods as Windows, he went a different direction and innovated some remarkable things. I really like seeing something and thinking, “now that is smart.”

I currently have this running on a few machines to put it through its paces.

Check out his wonderful tutorial!

Here are the programs I am using or testing right now that all perform some sort of sync operation.

• Second Copy – $29, lots of profiles, lots of options. Needs to run all the time.
• Live Mesh – free, syncs with a cloud of 5 gig storage space. Very powerful and unique service.
• Windows Live Sync – free – powerful, does sync with other folders, but all the computers need to be online. Even if you are doing it on a network.
• Delicious Toolbar – I went to this for bookmark sync’ing because it does all browsers and it has some other decent functions like compare, tagging, etc.

Currently trying it out

• Good Sync – I am currently testing this one out. Not sure about it yet. Looks good, but only offers 3 simultaneous download profiles. You can do the paid version for $29 and do unlimited, but I don’t want to buy another piece of sync software. I like the look of the interface though.
• Sync Toy from Microsoft – free – small but powerful. Also has the ability to be scheduled in Task Scheduler so it doesn’t need to be running all the time.

Phased Out

• Google Calendar Sync – free – this only syncs one single google account and one calendar. Almost not worth mentioning.
• Foxmarks – they tried to turn this into a marketing thing. I had issues because it would only sync your bookmarks between Firefox installs and not IE.
• Sync2it – does bookmarks, but just not robust enough for any serious use.
• Cobian Backup – free, Open Source – theres also a paid version. I used it for a bit but he does not offer syncing. He only offers backups.

Here is the main story, http://www.techcrunch.com/..

Executive Summary: Hacker finds gmail address of employee. Goes to Gmail’s lost password function. Sees secondary email account is a hotmail account that is deactivated. Creates new hotmail account with that address. Recovers password. Changes password back for stealth. Then has access to Google Apps on the twitter.com domain.

Basically, here is what happened:

A young Frenchman named “Hacker Croll” got interested in web security, social engineering a few years ago. He is unemployed. He wanted to hack into Twitter.

• He starts doing web searches on Twitter, accumulating vast amounts of names and email addresses of Twitter employees
• From there he uses the “Forgot Password” on a Twitter employees gmail address.
• Unable to determine what it is by guessing, he asks for a hint. Gmail balances usability with security by offering users to have a second email account attached to the main email account in case of password resets. Gmail informs Hacker Croll that they sent a password reset to “******@h******.com”
• Hacker Croll guesses that it is probably a hotmail account, so uses the same username at hotmail.com to check the email address
• Hotmail recycles old usernames, so the username was deleted.
• Hacker Croll creates a new hotmail user account with the twitter employees username. Asks for the password reset from the Twitter Employee’s gmail and gets the reset.
• Hacker Croll then searches through the account and finds what the password was before he changed it, so he could reset it and not alert the Twitter Employee.

Now, he has completely shadowed a twitter employees account and has their ‘main’ reused password. He uses that password to gain access to Google Apps on the Twitter domain. There he hit the goldmine with emails, and email attachments. Then he took control of their personal email, work email, iTunes (iTunes has a security hole that you can see the complete credit card numbers), banking account information, ATT, MobileMe, Amazon, everywhere the person was a customer through the vast amount of emails he had control of.

Then the CEO of Twitter downplayed the attack, so Hacker Croll got offended and sent all of the documents to TechCrunch to prove the severity of the attack. Then, they published a wealth of internal Twitter memos, strategies, and other documents. Here is HC’s apology.

I would like to offer my personal apology to Twitter. I think this company has a great future ahead of it.

I did not do this to profit from the information. Security is an area that fascinated me for many years and I want to do my job. In my everyday life, I help people to guard against the dangers of the Internet. I learned the basic rules .. For example: Be careful where you click the files that you download and what you type on the keyboard. Ensure that the computer is equipped with effective protection against viruses, external attacks, spam, phishing … Upgrading the operating system, software commonly used … Remember to use passwords without any similarity between them. Remember to change them regularly … Never store confidential information on the computer …

I hope that my intervention will be repeated to show how easy it can be for a malicious person to gain access to sensitive information without too much knowledge.

Croll Hacker.

Here is Twitter’s Official Response:

Twitter, Even More Open Than We Wanted

About a month ago, an administrative employee here at Twitter was targeted and her personal email account was hacked. From the personal account, we believe the hacker was able to gain information which allowed access to this employee’s Google Apps account which contained Docs, Calendars, and other Google Apps Twitter relies on for sharing notes, spreadsheets, ideas, financial details and more within the company. Since then, we have performed a security audit and reminded everyone of the importance of personal security guidelines.

That begs to make you think about the balance between usability and security. The security as a whole is only as strong as its weakest link. Better check those secondary email addresses and ensure that they are just as safe and secure as your primary. Which reminds me, I need to go change some stuff… brb. 🙂

Twitter’s Internal strategy, http://www.techcrunch.com/…

The “Peanut Butter Manifesto” internal Yahoo Memo from back in the day, http://online.wsj.com/…