Augmented reality makes shopping more personal

Shopping for groceries in supermarkets can sometimes be very exhausting, especially if they pile up similarly looking goods near each other. On the other hand, if you are exploring new products, you might have ended looking at the whole isle of products you haven’t tried before, and you don’t know does their composition suit your needs. Researchers at IBM’s lab in Haifa, Israel, are developing a new augmented reality (AR) app for smartphones which could change the way we shop in stores.

After users point their smartphone’s or tablet’s video camera to pan over products on the shelf, the application instantly displays recommendations and offers based on predefined specific needs of the user.

“We’re going way beyond simple facial recognition for products to provide superimposed information that points out the products shopper prefer – whether based on previous purchases, price, consumer rating, sodium content, environmentally friendly packaging, or other considerations”, said Amnon Ribak, project leader for the augmented shopping adviser.

For example, a shopper looking for a high-quality facial moisturizer can specify important characteristics of the product, such as having a sun protection factor of 15, is hypoallergenic, was not tested on animals, and that is on sale. As the shopper points a smart device camera at the shelf of moisturizers in the pharmacy, the app recognizes the merchandise and superimposed meta data of the product over the merchandize. It could also be used to highlight information about coupons or special discounts that may apply.

“The idea of standing in an aisle in the supermarket and having your mobile device point out the gluten-free cookies you need can be a real time saver”, said Ribak. “This has the potential to completely change the shopping experience from one of hunting, reading, and searching to simply picking up those products you prefer.”

Ribak leads a team of researchers who already contributed to advances in image processing such as license plate recognition at night or inclement weather for the Swedish toll road system, identification of malignant tumors in medical images, and optical recognition for ancient texts in libraries across Europe.

The augmented shopping adviser app relies on combination of techniques used in facial recognition, color and shape matching, and associations with surrounding products. The app can take into account the mobile device’s camera angle, and distance from a shelf to help distinguish between products.

“Our first mission was to create our own mini-supermarket in the lab, so we could test the various approaches and challenges involved”, said Ribak. “We’ve already submitted a number of patent applications based on the new techniques we discovered to overcome the challenges in recognizing products with less than ideal lighting, shadows, and reflections.”

Aside from assisting shoppers, the new application from IBM research could also help retail managers organize their stores, instantly point out what’s missing on the shelves, or summarize which products were on sale during the week. It could also provide important information about optimized shelf and store arrangement, since it would show the shopping habits of their customers.