When Amazon began
While the final design relies on machine learning — a different type of artificial intelligence than the generative AI popularized by ChatGPT — gen AI played a crucial role in the store’s early development.
“What does it look like when a person takes an item off the shelf?” asked Jon Jenkins, vice president of Just Walk Out technology for Amazon Web Services, in an interview at the National Retail Federation’s Big Show this week in New York. “We used generative AI to create videos to feed into [a separate] AI system to train it to detect when people are taking things.”
Amazon’s checkout-free technology has since expanded greatly, and now has years worth of data to better train its systems. Changes based on real-world findings include the switch in some stores from asking shoppers to scan a payment card or Amazon app as they enter to asking them to scan when they leave with an item. “We don’t want to require you to provide us payment credentials when you walk in, if you’re just there to browse,” Jenkins said.
The addition of stadiums presented another challenge that AI was uniquely suited to solve, Jenkins said.
In the original convenience store model, everything was neat and tidy; soda and candy had their own shelves, and sensors such as cameras and scales could detect when a customer would pick something up and put it back. Soft items, like hats and shirts emblazoned with a sports team’s logo, are a bit more difficult to track this way because customers almost never put a shirt back, neatly folded, the way they found it.
The solution to this problem was an older technology repurposed in the AI-driven era: RFID.
“When my team first came to me with this, I’m like, it’s not 1990,” Jenkins said. “The argument they made was, RFID up to this point has always been used for the benefit of the merchant,” such as for loss prevention and inventory tracking, he said. “We looked at, how can we use RFID to benefit the shopper?”
Amazon used RFID tags — and a healthy dose of gen AI and machine learning — to train a new version of Just Walk Out that can detect individual items by its RFID tag. Each tag is unique to each item and can be linked to a shopper’s receipt and payment method.
AI comes into play at the store’s exit. The sensor has to be smart enough to detect the difference between someone holding a shirt near the checkout and someone carrying the shirt through it. Signal strength isn’t enough to do this, since a person could be inadvertently obscuring the RFID tag by bunching the item up in their arms, or keeping it at a distance by wearing their new RFID-tagged hat out of the store.
Machine learning plays a key role in determining whether an item has passed through the checkout gate or not, “given the variety of antennas that are placed very creatively in an intentional way in there,” Jenkins said.
This system also streamlines product returns and exchanges by eliminating the need for a receipt. The store can scan an item’s unique RFID tag to determine how a customer paid and credit the return to the proper payment method. This system can also help determine if a scammer is trying to return an item that was never sold, because the tag will not have that payment history associated with it.
The tags also have QR codes that the customer can scan to view the full receipt from that store’s visit, rather than just the record of the sale of the one item.
This approach isn’t universal. An RFID tag can’t be read as easily through metal and liquid, so it wouldn’t work for soda cans, Jenkins said. For a store that sells both apparel and snacks, a hybrid approach of cameras and RFID would be necessary, he said.
The type of store or product can also determine the best type of payment to use at checkout. For example, a third-party store could provide loyalty benefits to a customer who uses that store’s cobranded card, Jenkins said. Or it could bypass bank-issued cards altogether. In a health care facility, hospital workers could use payroll deductions to purchase food during their shifts, Jenkins said. In a university, the charge could go to a meal plan.
While Amazon’s use of AI is creating a tangible change for the shopper, many companies have been relying on AI for “unsexy” uses like fraud detection, according to Chris Adams, senior vice president of payments at Oracle.
But the tone of that conversation is shifting, Adams said in an interview at the NRF conference.
“A lot of the focus now is how that AI can help with the customer interactions and the journey, and so we’re seeing a lot of that focus now translating into the buyer’s experience,” Adams said.
Amazon’s Just Walk Out connects to Oracle’s point of sale, Adams said. And the ease of payment is just as critical as any other part of the system, he said.
“Consumers love the convenience as long as it doesn’t mean that you’re having to do a huge heavy lift before you can use it,” Adams said. “And I mean by that, like, having to join a loyalty program or set up another card.”
Other companies are similarly testing ways that AI can improve the checkout.
Wegmans Food Markets, for example, is experimenting with AI for product recommendations online and for computer vision and product recognition at self-checkout.
“Related to other stores of our size, we take a lot of pride that [our] checkout lanes move pretty fast,” Smita Katakwar, senior vice president of technology for Wegmans, said in a panel discussion. “But as we are attracting more and more people in our store, we have to continue to think about and anticipate what we can do better with the checkout experience.”
Its early tests of AI are “promising,” she said. “We believe AI is going to have a lasting impact on both customer and employee experience … It is too early to say that we have any concrete result here, but this is a journey that we are on for the long haul.”
