Cyber seeing will soon be AI believing

The days of wondering where a celebrity bought a certain item of clothing he or she is sporting on the latest style pages, and trying in vain to describe it to Google to find it, may finally be numbered.

Imagine a technology that could recognise a dress in a social post, a YouTube video or a film, identify the brand that made it and then automatically connect the user to an ecommerce site to buy it? This is just one example of intelligent scene understanding (ISU). Put simply, the technology enables computer systems to recognise, analyse and understand objects more accurately and efficiently.

It’s a field of increasing importance within artificial intelligence, with researchers currently trying to establish the ISU system through pattern recognition and deep-learning models in order to achieve accurate and effective target identification, analysis and understanding.

One such scientist is Dr Kaizhu Huang, associate professor in the department of electrical and electronic engineering at Xi’an Jiaotong-Liverpool University in Suzhou, China. Huang is developing a research project based around ISU technology, which  “integrates object detection, feature extraction and object recognition into a whole framework”, he says.

“Preliminary results show that the invented new model has led to so far the best performance in benchmark object recognition data sets,” he says.

Automatic and intelligent understanding of scenes can substantially reduce the amount of work normally undertaken by humans. A potential example of its application could be at supermarkets like Amazon Go in Seattle, a prototype store that has dispensed with employees, queues and checkouts.

Huang also says the technology, which boasts “significantly better recognition accuracy”, would be useful in systems designed to hunt criminals, as well as for anomaly detection software: in cases where computer systems are processing large volumes of data, ISU can be used to spot and resolve issues before they impact on users.

It’s a tempting prospect for marketers, too. Patrick Rona, chief digital officer at McCann Worldgroup APAC, says that while ISU is not a new area, with a number of startups in both Asia and Europe working on ISU-based applications and services over the last four to five years, it could transform product placements by making them more subtle, more searchable and more buyable when pictured in social and ecommerce channels.

The visual dimension

As network connectivity has increased, the vast majority of information being posted by people on social has increasingly become visual — think Instagram for pictures or YouTube for videos.

“The opportunity for ISU services, then, is tremendous, as people will expect to index and make this content searchable and linkable to other services, like ecommerce,” says Rona, adding that ISU differs significantly from other ‘object-recognition’ systems such as the Samsung S8 phone’s Bixby AI assistant, unveiled earlier this year. “It’s very different in that ISU services would utilise the ‘cloud’, and the immense processing power and machine-learning capacity of the cloud, to work on a vastly larger scale and at a far faster pace.”

Amazon Go… the ecommerce giant’s experimental supermarket aims to revolutionise the shopping experience.

Dr Neil Wang, global partner and greater China president at research and business consulting firm Frost & Sullivan, elaborates further on the differences between tools such as Bixby and ISU. Bixby’s object-recognition system is based on the understanding and analysis of static photos, he says, with the tool recognising buildings and providing suggestions for users — such as noted restaurants near a particular building or landmark. Moreover, through the product images provided by users, Bixby can provide related product information and purchasing links. However, Wang says ISU is based on the understanding and analysis of a much more dynamic environment, such as video content or social posts.

An example of how ISU works in practice has been tried out by Asian social community hub Clozette, which has been working with Singapore startup ViSenze to test the technology. Using the ViSenze service, a Clozette user can find out more about a piece of clothing, handbag or pair of shoes that might be posted in pictures or videos by other Clozette users, and even be linked to an ecommerce portal to buy said items.

Digital departure

Another obvious consumer category that could benefit from ISU, believes Rona, is travel.  “The ability to identify a landmark or a place — say, a beautiful beach, mountain or urban street scene — and help people get there could be very powerful.”

ISU’s potential has certainly not gone unnoticed by this sector. Cindy Tan, vice-president of display at TripAdvisor Asia-Pacific, says that while the company hasn’t yet experimented directly with ISU, the field of machine learning has been central to the brand’s efforts to extract the most contextually relevant trip-planning insights from its massive amount of data, in order to help every one of its users discover and compare options and confidently make decisions to better enhance their trips.

“Machine-learning can distil the essence of what’s exciting or noteworthy about a place or point of interest at scale — this saves users time, as they can be more effective in their travel planning,” she says. “They need not trawl thousands of reviews to make the best decisions for their trip.”

Gita Sriram, business operations consultant at Arc Philippines, believes that the rising trend for ISU could potentially be the “perfect confluence” of digital and consumer marketing, leading to a whole new area of brand engagement. “Regardless of the retail shopping environment, an ISU modelling program offers shoppers options and upgrades based on their preferences and past behaviours, and this learning curve makes their retail journey far smoother and quicker if they so choose, because the process of de-selection is automated,” says Sriram.

This means that the shopper is spared the hassle of having to trawl a supermarket aisle with 300 shampoo options; instead, using an ISU algorithm, the shopper is presented with a manageable shortlist via a mobile app or shopping cart screen, based on their past behaviours but also environmental circumstances such as the daily weather — allowing them to make a final purchase decision without much thought. 

“ISU would also help marketers identify early adopters and disruptors when it comes to new products and services, so that the shopper behaviour of these individuals can be leveraged for wider brand campaigns,” says Sriram.

Time for reflection

With ISU technology tried but by no means rigorously tested, however, it may be a case of ‘wait and see’ before marketers and advertisers take the plunge. As Frost & Sullivan’s Wang outlines, the development of ISU is still at the initial stage, with the technology at present ‘imperfect’ for achieving accurate real-time recognition and analysis.

“Marketers looking to exploit this technology may not be capable of fully understanding users’ demand in the market to cater to their needs,” Sriram says.

Time is also a consideration. Building tools that can provide contextual relevance to consumers takes a fair amount of time, with any investment for the long-term requiring lengthy testing and learning.

“We began this investment in personalisation in 2014 and this allowed us to start building a deep set of experience in natural language processing, machine learning and algorithmic recommendations,” says TripAdvisor’s Tan. “We have learned a lot and continue to evolve these efforts based on the results of our testing. In addition, the technology we have built so far is being leveraged in key areas of the product like search, merchandising, discovery, trip planning, and in-destination recommendations.”

Neil Hudspeth, chief experience officer at Publicis One Japan, believes brands will need ‘toolkits’ to help them leverage platforms using ISU. “Brands will also have to decide where in the customer journey they want to be — top of the experience or at the base of the data stack,” he says. “However, let’s hope that image, shape and word bidding does not dictate results as we move into a more AI-facilitated world. Can you imagine when we use voice and object recognition to help us find a ‘bottle of Barolo on my way to Nicolas’ house’ and the result we’re given is a bottle of beer? Not very intelligent.”

The key for those looking to exploit ISU will be the maturity and accuracy of the technology and the associated applications and services.  Those companies that have a culture of innovation and are willing to get it wrong at times are likely to be the first to adopt the technology.  Greg Paull, principal at consultancy group R3 Worldwide believes that at present, not enough marketers are at the bleeding, or even leading, edge for some of these [technology] breakthroughs.

Finally, another challenge for marketers, according to McCann Worldgroup’s Rona, will be the talent and expertise available both internally and with agency and technology partners. It’s not just about the technical skill, he says; it’s also having the strategic marketing expertise to apply the technology in a meaningful and valuable way for consumers.