Mobile interaction research at Google

Posted by Xiaojun Bi, Ciprian Chelba, Tom Ouyang, Kurt Partridge and Shumin Zhai

Google takes a hybrid approach to research - research happens across the entire company, and affects everything we do. As one example, we have a group that focuses on mobile interaction research. With research backgrounds in human-computer interaction, machine learning, statistical language modeling, and ubicomp, the group has focused on both foundational work and feature innovations for smart touchscreen keyboards. These innovations help us make things like typing messages on your Android device easier for hundreds of millions of people each day.

We work closely with world-class engineers, designers, product managers, and UX researchers across the company, which enables us to rapidly integrate the fruits of our research into the Android platform. The first major integration was the launch of Gesture Typing in Android 4.2.

Rapidly developed from basic concepts up to product code, and built on years of Android platform groundwork on input method editors (IME) and input method framework (IMF), Gesture Typing uses novel algorithms to dynamically infer and display the user’s intended word right at the fingertip. Often the intended word is displayed even before the user has finished gesturing--creating a magical experience for the user. Seamlessly integrated with touch tapping, Gesture Typing also supports two-thumb use.

It is exciting and rewarding to do research inside a product team that enforces engineering and user experience discipline. At the same time, we as researchers also contribute to the broader research community; publication, whether in the form of papers, code, or data, bind a research community together. The following papers are based on our work over the last year, some with bright and hardworking student interns:

Octopus: Evaluating Touchscreen Keyboard Correction and Recognition Algorithms via “Remulation”
by Xiaojun Bi, Shiri Azenkot (U. of Washington), Kurt Partridge, Shumin Zhai
CHI 2013, in press (link to come)

FFitts Law: Modeling Finger Touch with Fitts’ Law
by Xiaojun Bi, Yang Li, Shumin Zhai
CHI 2013, in press (link to come)

Making Touchscreen Keyboards Adaptive to Keys, Hand Postures, and Individuals - A Hierarchical Spatial Backoff Model Approach
by Ying Yin (MIT), Tom Ouyang, Kurt Partridge, Shumin Zhai
CHI 2013, in press (link to come)

Bimanual gesture keyboard.
by Xiaojun Bi, Ciprian Chelba, Tom Ouyang, Kurt Partridge, and Shumin Zhai
UIST 2012

Touch Behavior with Different Postures on Soft Smart Phone Keyboards
by Shiri Azenkot (U. Washington) and Shumin Zhai
MobileHCI 2012

2011 EMEA Android Educational Outreach Program Awards Mobile Phones to Universities

Posted by David Harper, Head of University Relations, EMEA

As part of EMEA’s 2011 Android Educational Outreach program, we recently granted over 300 Android-powered mobile phones to 40 universities across Europe, the Middle East, and Africa. These phones will be used to support mobile related project work in university teaching and research. Our steering committee reviewed applications from 77 universities in 24 countries across the region and selected finalists based on each proposal’s scope to generate interest in mobile engineering, reach many students, and be applicable both within and outside the university.

This is the second year we have awarded mobile phones to universities. This is largely attributable to the enthusiastic feedback from last year’s recipients who were interested in continued support for Android project work. The phones donated last year were used in a range of interesting projects, including:

• George Candea, EPFL (Switzerland): The Pocket Campus, an application that helps students, graduates, staff and visitors to find their way around the EPFL campus was created as a course project. After the course, some of the students decided to continue development of the application. It has become so successful that it’s now EPFL’s campus-wide smartphone app.
• Andrew Rice, University of Cambridge (United Kingdom): Students in the summer programme developed Learn!, a flashcard-based learning application that is available in Android Market. This project investigated how one might incorporate features of modern phones such as multimedia capture and playback, data communications and significant computation power into a learning application.
• Alan Smeaton and colleagues, Dublin City University (Ireland): Undergraduate, master’s, and PhD students embarked on a wide variety of projects, which included lifelogging (recording everyday activities using the phone); measuring the strengths of wireless networks as an aid to mapping wireless propagation; and interface design for an augmented reality application.
• Nicolae Tapus, University Politehnica of Bucharest (Romania): Numerous applications were developed by students, including: TaxiFinder, an application that finds the closest taxi number with the lowest price, and Viewlity, an augmented reality engine for showing nearby points of interest (e.g., gas stations, restaurants, ATMs, places of worship) on an Android phone.
• Gerhard Tröster, ETH Zurich (Switzerland): Martin Wirz and his team are using mobile phones to conduct research in the field of wearable computing and machine learning. The devices are used to collect all kinds of sensor information (e.g., accelerometer, magnetometer, microphone, GPS) to infer personal activities, psychological behaviors and social phenomena.

We are looking forward to sharing the great projects resulting from this year’s Android Educational Outreach program early next summer.

QR code generator

---

QR Code is a two-dimensional bar code, invented in Japan in 1994, in a subsidiary company of Toyota, Denso Wave.
Initially, this QR code (Quick Response) was used to mark car parts used in the production process. After a while, due to its ability to integrate a relatively large amount of data and because it has no fixed size, it start to be used successfully in several other domains of activity. The highest penetration was in the mobile devices. This area is booming and QR code seemed to be the ideal solution for data encryption of information that can be transmitted and stored easily in smartphones, tablets and other mobile devices. The highest spreading has in mobile devices which use Google Android operating system.
The code consists of black modules arranged in a square pattern on a white background. The information encoded can be made up of any kind of data (e.g., binary, alphanumeric, or Kanji symbols).
Together with a data structure format, QR code can contain: a business card, a phone calendar event, a useful web address, geographical position, coordinates GPS, SMS, email, etc.
Present application implements the most common formats and generate QR code associated.

You can embed this application in your website by copying this snippet:

What You Capture Is What You Get: A New Way for Task Migration Across Devices

Posted by Yang Li, Research Scientist

We constantly move from one device to another while carrying out everyday tasks. For example, we might find an interesting article on a desktop computer at work, then bring the article with us on a mobile phone during the commute and keep reading it on a laptop or a TV when we get home. Cloud computing and web applications have made it possible to access the same data and applications on different devices and platforms. However, there are not many ways to easily move tasks across devices that are as intuitive as drag-and-drop in a graphical user interface.

Since last year, our research team started developing new technologies for users to easily migrate their tasks across devices. In a project named Deep Shot, we demonstrated how a user can easily move web pages and applications, such as Google Maps directions, between a laptop and an Android phone by using the phone camera. With Deep Shot, a user can simply take a picture of their monitor with a phone camera, and the captured content automatically shows up and becomes instantly interactive on the mobile phone.

This project was inspired by our observations that many people tend to take a picture of map directions on the monitor using their mobile phone camera, rather than using other approaches such as email. Taking pictures feels more direct and convenient, and fits well our everyday activity that is often more opportunistic. Instead of just capturing raw pixels, Deep Shot recovers the actual contents and applications on the mobile phone based on these pixels. You can find out how Deep Shot keeps user interaction simple and what happens behind the scenes here. Similar to WYSIWYG—What You See Is What You Get—for graphical user interfaces, Deep Shot demonstrates WYCIWYG—What You Capture Is What You Get—for cross-device interaction. We are exploring this interaction style for various task migration situations in our everyday life.



Deep Shot remains a research project at Google. With increasing capabilities of mobile phones and fast growing web applications, we hope to explore more exciting ways to help users carry out their everyday activities.

Instant Mix for Music Beta by Google

Posted by Douglas Eck, Research Scientist

Music Beta by Google was announced at the Day One Keynote of Google I/O 2011. This service allows users to stream their music collections from the cloud to any supported device, including a web browser. It’s a first step in creating a platform that gives users a range of compelling music experiences. One key component of the product, Instant Mix, is a playlist generator developed by Google Research. Instant Mix uses machine hearing to extract attributes from audio which can be used to answer questions such as “Is there a Hammond B-3 organ?” (instrumentation / timbre), “Is it angry?” (mood), “Can I jog to it?” (tempo / meter) and so on. Machine learning algorithms relate these audio features to what we know about music on the web, such as the fact that Jimmy Smith is a jazz organist or that Arcade Fire and Wolf Parade are similar artists. From this we can predict similar tracks for a seed track and, with some additional sequencing logic, generate Instant Mix playlists from songs in a user’s locker.

Because we combine audio analysis with information about which artists and albums go well together, we can use both dimensions of similarity to compare songs. If you pick a mellow track from an album, we will make a mellower playlist than if you pick a high energy track from the same album. For example, here we compare short Instant Mixes made from two very different tracks by U2. The first Instant Mix comes from "Mysterious Ways," an upbeat, danceable track from Achtung Baby with electric guitar and heavy percussion.

1. U2 "Mysterious Ways"
2. David Bowie "Fame"
3. Oingo Boingo "Gratitude"
4. Infectious Grooves “Spreck”
5. Red Hot Chili Peppers “Special Secret Song Inside”

Compare this to a short Instant Mix made from a much more laid back U2 cut, "MLK" from the album Unforgettable Fire. This track has delicate vocals on top of a sparse synthesizer background and no percussion.

1. U2 "MLK"
2. Jewel “Don’t”
3. Antony and the Johnsons “What Can I Do?”
4. The Beatles “And I Love Her”
5. Van Morrison “Crazy Love”

As you can hear, the “Mysterious Ways” Instant Mix is funky, with strong percussion and high-energy vocals while the “MLK” mix carries on with that track's laid-back lullaby feeling.

Our approach also allows us to create mixes from music in the long tail. Are you the lead singer in an unknown Dylan cover band? Even if your group is new or otherwise unknown, Instant Mix can still use audio similarity to match your tracks to real Dylan tracks (provided, of course, that you sing like Bob and your band sounds like The Band).

Our goal with Instant Mix is to build awesome playlists from your music collection. We achieve this by using machine learning to blend a wide range of information sources, including features derived from the music audio itself. Though we’re still in beta, and still have a lot of work to do, we believe Instant Mix is a great tool for music discovery that stands out from the crowd. Give it a try!

Further reading by Google Researchers:
Machine Hearing: An Emerging Field
Richard F. Lyon.

Sound Ranking Using Auditory Sparse-Code Representations
Martin Rehn, Richard F. Lyon, Samy Bengio, Thomas C. Walters, Gal Chechik.

Large-Scale Music Annotation and Retrieval: Learning to Rank in Joint Semantic Spaces
Jason Weston, Samy Bengio, Philippe Hamel.

Robot hackathon connects with Android, browsers and the cloud

Posted by Ryan Hickman and Mamie Rheingold, 20% Robotics Task Force

With a beer fridge stocked and music blasting, engineers from across Google—and the world—spent the month of October soldering and hacking in their 20% time to connect hobbyist and educational robots with Android phones. Just two months later we’re psyched to announce three ways you can play with your iRobot Create(R), LEGO(R) MINDSTORMS(R) or VEX Pro(R) through the cloud:

• App Inventor for Android
• Cellbots for Android
• Python library for the Scripting Layer 4 Android

For the month of October, we invited any Googler who wanted to contribute to connect robots to Google’s services in the cloud to pool their 20% time and participate in as much of the process as they could, from design to hard-core coding.

Thanks to our hardware partners (iRobot, LEGO Group, and VEX Robotics), we never suffered a shortage of supplies. Designers flew in from London, and prototypes were passed between engineers in Tel-Aviv, Hyderabad, Zurich, Munich and California. In Mountain View, we gathered around every Thursday night, rigging up a projector against the wall to share our week’s worth of demos while chowing on pizza. And here is what we produced (so far!):

• App Inventor: Low level Bluetooth support for connecting with many serial-enabled robots, and of course tight integration with LEGO MINDSTORMS.
• Cellbots for Android: Brand new Java app from Cellbots.com, which is open source and available for free in the Android Market.
• Python library: Modularized version of the popular Cellbots project, which is all open source code.

We hope these applications provide some fun and inspire you to build upon this lightweight connectivity between robots, Android, the cloud and your browser.

Letting everyone do great things with App Inventor

Posted by Karen Parker, App Inventor Program Manager

In July, we announced App Inventor for Android, a Google Labs experiment that makes it easier for people to access the capabilities of their Android phone and create apps for their personal use. We were delighted (and honestly a bit overwhelmed!) by the interest that our announcement generated. We were even more delighted to hear the stories of what you were doing with App Inventor. All sorts of people (teachers and students, parents and kids, programming hobbyists and programming newbies) were building Android apps that perfectly fit their needs.

For example, we’ve heard of people building vocabulary apps for their children, SMS broadcasting apps for their community events, apps that track their favorite public transportation routes and—our favorite—a marriage proposal app.

We are so impressed with the great things people have done with App Inventor, we want to allow more people the opportunity to do great things. So we’re excited to announce that App Inventor (beta) is now available in Labs to anyone with a Google account.

Visit the App Inventor home page to get set up and start building your first app. And be sure to share your App Inventor story on the App Inventor user forum. Maybe this holiday season you can make a new kind of homemade gift—an app perfectly designed for the recipient’s needs!

Korean Voice Input -- Have you Dictated your E-Mails in Korean lately?

Posted by Mike Schuster & Kaisuke Nakajima, Google Research

Google Voice Search has been available in various flavors of English since 2008, in Mandarin and Japanese since 2009, in French, Italian, German and Spanish since June 2010 (see also in this blog post), and shortly after that in Taiwanese. On June 16th 2010, we took the next step by launching our Korean Voice Search system.

Korean Voice Search, by focusing on finding the correct web page for a spoken query, has been quite successful since launch. We have improved the acoustic models several times which resulted in significantly higher accuracy and reduced latency, and we are committed to improving it even more over time.

While voice search significantly simplifies input for search, especially for longer queries, there are numerous applications on any smartphone that could also benefit from general voice input, such as dictating an email or an SMS. Our experience with US English has taught us that voice input is as important as voice search, as the time savings from speaking rather than typing a message are substantial. Korean is the first non-English language where we are launching general voice input. This launch extends voice input to emails, SMS messages, and more on Korean Android phones. Now every text field on the phone will accept Korean speech input.

Creating a general voice input service had different requirements and technical challenges compared to voice search. While voice search was optimized to give the user the correct web page, voice input was optimized to minimize (Hangul) character error rate. Voice inputs are usually longer than searches (short full sentences or parts of sentences), and the system had to be trained differently for this type of data. The current system’s language model was trained on millions of Korean sentences that are similar to those we expect to be spoken. In addition to the queries we used for training voice search, we also used parts of web pages, selected blogs, news articles and more. Because the system expects spoken data similar to what it was trained on, it will generally work well on normal spoken sentences, but may yet have difficulty on random or rare word sequences -- we will work to keep improving on those.

Korean voice input is part of Google’s long-term goal to make speech input an acceptable and useful form of input on any mobile device. As with voice search, our cloud computing infrastructure will help us to improve quality quickly, as we work to better support all noise conditions, all Korean dialects, and all Korean users.