Theoretically, generative AI can help AR developers create virtual worlds, digital objects, and realistic characters that can intuitively respond to the user’s environment.

Generative AI refers to self-learning algorithms that use existing data, such as text, audio, or images, to produce realistic new content, while AR is a technology that allows the user to see the real world overlaid with a layer of digital content.

It could also support the creation of AR objects at scale, cheaper, and more quickly than currently possible. This will allow companies of all sizes to use generative AI-based digital objects in AR for customer engagement, marketing campaigns, and advertising. The use of generative AI in AR will only grow as the two technologies advance in the coming years.

AI is increasingly pervasive in AR development

AI technologies such as ML, conversational platforms, and AI chips power most of today’s AR devices and apps. AR developers use ML to improve the user experience (UX) by continually analysing user activities. Apple’s CoreML and Google’s TensorFlow Lite ML frameworks support ARKit and ARCore (also known as Google Play Services for AR), respectively. They allow developers to run ML models to improve object recognition in photos, convert speech to text, and enable gesture recognition. Eye tracking and facial recognition, fast becoming standard functions across all AR devices, use ML to improve UX.

Virtual assistants, such as Apple’s Siri, Amazon’s Alexa, Google’s Assistant, and Microsoft’s Cortana, enable the hands-free operation of AR devices. This is critical for some use cases, especially in enterprises. At the same time, efforts to develop low-power AI chips for AR devices are also underway by companies like Mythic. In March 2022, Qualcomm announced a $100m fund to support developers and companies integrating AR and AI in their metaverse platforms and experiences. Some companies are also working on specialized AI tools. Heru, for example, creates AI-powered vision diagnostics and augmentation software that can be used with commercially available AR devices.

Generative AI and content creation

Generative AI can support content creation and offer context-aware virtual elements, elevating user engagement. For instance, generative AI-supported AR applications for interior design can craft realistic, virtual furniture in a living room, allowing the user to visualize its appearance. The user inputs a text prompt and the platform converts it into a 3D virtual object. Start-up Devar is working on one such generative AI offering for WebAR. The user can then customize the virtual furniture (in size and design) to fit the space and requirement before purchasing.

Moreover, generative AI can make AR content adaptable in real time. For instance, in gaming, generative AI-supported virtual characters and objects can respond intuitively to the user’s environment, making the gaming world feel integrated with reality. Furthermore, AI-created characters can appear in AR to promote a message, brand, or film within a game. In April 2023, Snap introduced AR Lenses powered by generative AI and a ‘Cosmic Lens’ that turns the user’s surroundings into an immersive, animated sci-fi scene.

WebAR platform Geenee AR has partnered with Inworld AI, a generative AI-based platform for developing non-playing characters (NPCs), to allow brands to bring their NPCs to life through AR. For example, an AI-generated brand ambassador can provide in-depth product info and tutorials. This signals the development of new use cases across sectors, promising novel user experiences.

However, most initiatives combining generative AI and AR are experimental. Significant commercial opportunities will only emerge when both generative AI and AR have matured.