AR to Realize World of Science Fiction -- Nikkei Electronics Asia -- November 2009 - Tech-On!

http://techon.nikkeibp.co.jp/article/HONSHI/2...

Augmented reality (AR) technology is coming into our homes, our cities and elsewhere in the near future. Dramatic advances in computer technology are making possible a world that once would only have been found in the realm of science fiction... The revolution is impacting everything from mobile phones to digital home electronics, forcing rapid evolution.

In the near future, reality will change into something like science fiction, thanks to augmented reality (AR) technology (Fig 1). AR emerged in the first half of 1990s, and is already extensively featured in film, television programs and the like. For example, a newscaster in the studio can be synthesized into a background image of the great outdoors. In addition to TV, there has also been considerable progress in military technology applications such as night-vision scopes and fighter pilot targeting displays, while practical applications appeared in medicine about two years ago.

In the near future AR will become a key part of our daily lives. As image processing algorithms become more refined and the amount of processing power required drops, the performance of microprocessors and graphics processing unit (GPU) integrated circuits (IC) is soaring. Professor Hirokazu Kato of the Nara Institute of Science & Technology (NAIST) of Japan, known for the public release of the "ARToolkit" C-language library for AR free of charge, points out "Even the iPhone boasts the processing power of a workstation of the 1990s."

The first consumer AR products are expected to appear before the end of 2009, including mobile phones and printed items, for example. Within a few years the trend will spread to home electronics, gathering strength to become an avalanche.

This change, simply, is a total revolution in the shape of home electronics, overturning our existing conceptions.

Virtual Reality with Uses
So what is AR, anyway? Basically, it's a technology that enhances human senses by overlaying computer-processed information, in realtime, on top of real-world information we can see, hear or touch.

The fundamental difference between AR and the virtual reality (VR) that attracted so much hype in the 1990s is the ratio between real-world information and virtual information (Fig 2). While VR attempts to replace all real-world information with virtual information, such as using computer graphics (GC) for all visual information, AR instead enhances (or augments) real-world information with virtual information. The ratio of virtual information is low.

Some researchers use a different yardstick to define AR. NAIST's Kato says "AR is a technology that is convenient, fun to use and safe for users." In other words, how the technology supports human activity is more important than how virtual information is weighted.

Sensors for the Five Senses
What is needed to actually utilize AR is the AR interface equipment, which consists of the computer, various sensors to serve the computer as vision, touch and other senses, and a display or similar device to output the processed information. Depending on what sensors, output devices, etc, are used, and how processed information is to be overlaid on reality, an enormous range of applications is possible.

For example, explains Takahito Iguchi, chief executive officer (CEO) of Tonchidot Corp of Japan, "Brand new forms of communication become possible through real space" by overlaying personal computer (PC), Internet or other information on real sight. If imagery invisible to the naked eye, such as human body imagery taken in infrared, or with a magnetic resonance imaging (MRI) system, is overlaid on an actual view of the subject, it becomes possible to actually see blood vessels, organs, etc, as if looking inside the person's body. The real object can also be hidden behind the image. In daily life it would be possible, for example, to have explanations of functions appear when you look at a button on a piece of equipment. The first AR prototype for consumer use, in fact, showed the position of a copier paper tray.

Visual AR Development
AR can mean enhancements to touch, hearing, balance, etc, but development is farthest along in AR interfaces dealing with vision (Fig 3). These AR interfaces can be generally categorized into four groups by image output device, namely (1) PC and similar displays, (2) mobile phones, (3) eyeglass-type head-mounted displays (HMD), and (4) projectors. All use cameras as sensors and have a certain degree of image processing capability.

PC and similar displays (1) may eventually combine the role of existing mirror stands in the home. "Displays can be used as a type of mirror with AR functions. Processed information can be added to your true appearance to, for example, simulate cosmetics or clothing, or compare your golf swing to that of Tiger Woods," explains Professor Jun Rekimoto, Interfaculty Initiative in Information Studies, University of Tokyo, Japan.

Mobile phones (2), on the other hand, are the AR interface closest to widespread adoption. Many multi-function products come with cameras, global positioning satellite (GPS) functions, geomagnetic sensors and acceleration sensors. Software is being developed now for these handsets that will show network-based information referenced to the handset position on the display (Fig 4).

Mimicking Clairvoyance
Applications for mobile phones weight information related to the user's current position with the scenery picked up by the handset's onboard camera. The weighted information is on the Internet, but the situation is quite a bit different from the way Internet searches retrieve information currently. The user will no longer have to use keyboard, mouse, etc, to implement the search, and the only results that will be displayed are those related to the current position.

For example, it would be possible to use electronic mail (E-mail) linked to a specific place instead of a bulletin board, searching through the pals living near to you via Twitter, a communication service available on the Web. Internet-based information such as good restaurants could be displayed along with direction and distance from the user's current position. This resembles clairvoyance, where objects can be seen through to reveal the needed information.

More Intuitive Vision
If HMDs (3), projectors (4) and similar AR interfaces achieve widespread use, they could have at least as great an impact on our lives as displays (1), mobile phones (2), etc (Fig 5). The reason is that HMDs and similar devices can do just about everything a display, mobile phone or similar device caned, through more intuitive visual augmentation.

First of all, there will no longer be any need to compare the actual image with the image shown on the screen. For example, assume AR technology is used to create a "pop-up" illustrated book. With a PC display or handset, for example, AR would only display the weighted image of the book inside the screen, but with HMDs the characters from the book could walk or dance directly onto the pages.

HMD development requires advanced technology, however, and the problems in developing HMDs usable in daily life are only just beginning to be solved. It will take time to reach products that are easy to use.

It is, however, relatively simple to use a projector to make an AR interface. Many of these systems overlay the virtual image on the actual object. This offers advantages in that there is no need to look at a separate display, and the image can be seen with the naked eye.

In medicine some products have already entered commercial use. The VeinViewer from Luminetx Corp of the US superimposes image-processed vein photographs on the patient's body. According to Professor Masahiko Inami, Graduate School of Media Design, Keio University, Japan, "Even nurses with minimal training can easily locate veins, reducing the number of times they make a mistake trying to draw a blood sample, for example."

Moving Images by Touch
Advanced AR related to senses other than vision is also being developed. For example, Professor Kato of NAIST is researching the method to lift and move the virtual imagery of AR with the hands (Fig 6a). "We're trying to develop a tangible user interface, where information can be touched," he explains. "Existing touch panels are more intuitive than keyboards or mice, for example, but you're still not touching the information directly." With AR, he added, it is possible to communicate, including with virtual characters, for an even more intuitive user interface.

Associate Professors Takuya Nojima and Hiroyuki Kajimoto of the University of Electro-Communications, Japan have joined Keio University's Professor Masahiko Inami in researching functions to assist in "force sensing," how force is applied in the hand or elsewhere, and implementing these functions into scalpels, for example (Fig 6b).

Already systems have been developed using an optical sensor on the tip of a scalpel to detect the boundary between the yolk and the white of an egg, for example, or the electrostatic charge sensor from a touch panel to detect the interface between oil and water. The results are similar to what might happen if a new sense organ were added to your fingertip. Connected to robot arms, these scalpels can be controlled in accordance with sensed information.

Changing Home Electronics
AR technologies like these will cause a dramatic change in home electronics. Tonchidot's Iguchi, for example, predicts that some types of home appliances will become extinct, replaced by AR. Home electronics interfaces like buttons will be replaced by the AR interface, and storage functions such as hard disk drive (HDD) recorders will be replaced by cloud computing. "Extinct" is a bit of an exaggeration, but it is quite possible that the cases, at least, will vanish.

The key point in these forecasts is the ratio of real to virtual information. As discussed above, AR has a much higher ratio of real-world information than VR, but in some implementations the virtual content might be very high, too.

When AR first began to be applied in home electronics, the idea was that it would display additional information and take the place of the user's manual. As AR became more useful, though, the importance of real-world information based on weighted information dropped: the role of the hardware as the product interface shrank. In other words, the AR interface can take the place of not only the TV's buttons, but the TV display functions as well. "Interfaces that were once hardware are being virtualized in AR," says Keio University's Inami.

The First Step: Games
Electronic Entertainment Expo (E3), the computer gaming show held in 2009, demonstrates that it is more than just a pipedream: just take a look at the AR-related technologies demonstrated by companies like Sony Computer Entertainment Inc (SCE) of Japan, Microsoft Corp of the US, or under development at the Massachusetts Institute of Technology (MIT) Media Laboratory in the US and elsewhere ( Fig 7).

SCE revealed a demonstration of a game controller at E3, using AR to overlay it with the image of a tennis racket. Compared to using a controller actually shaped like a tennis racket, it was obvious that AR significantly reduces the size of the hardware required. SCE plans to release the product in spring 2010.

Microsoft, meanwhile, demonstrated a game called Project NATAL, operated by player gestures with no controller at all. Both use displays as the AR interface. Microsoft also plans to ship in 2010.

Complete Virtualization
Some even more advanced developments have emerged when it comes to virtualizing home electronics interfaces: instead of changing just the interfaces for equipment, virtualize the whole thing, including operation. At SIGGRAPH 2009, the computer graphics conference held in Aug 2009, this was announced as "control of remotely interfaced systems using touch-based actions in living spaces" (CRISTAL), jointly by the MIT Media Laboratory, University of Tokyo and Keio University.

Using ceiling-mounted cameras and projectors, CRISTAL projects an image of the entire room on a white table. When the user touches the image of a piece of equipment, the camera detects it and activates the corresponding system. The images of home electronics displayed on the table work in the same way as icons on the PC screen, operating equipment. Keio University's Inami explains "Until now AR has overlaid information on images, but CRISTAL uses the weighted imagery instead to actually affect the corresponding objects."

Fingers as Viewfinders
There has also been an example of using projector-based AR as an interface: "Sixth Sense," announced by the MIT Media Laboratory in Feb 2009. AR researchers in the field expressed astonishment at the development.

The system consists of a small projector and camera hanging around the user's neck, a mobile phone in the pocket for communication, an earphone, and colored bags to fit over the fingers. The finger sacks are aligned to act as AR markers, determining the type of action, weighted information, etc. Total cost is only US$350.

For example, if a beautiful scene is framed by the fingers, the finger sacks switch the system to photograph mode, and the frame indicated by the fingers defines the photo... the shutter clicks, and the shot is taken. If the user holds his or her palm up in front of the projector, a handset keypad is displayed, with the camera detecting which keys are pressed and dialing the phone accordingly. Camera, phone and other functions are operated via the AR interface, allowing the hands to function as a universal interface.