“Hey, Meta. Take a look at this and tell me which of these teas doesn't contain caffeine.”

I uttered these words while wearing a pair of Meta Ray-Bans at the tech giant's New York headquarters, while staring at a table with four packets of tea with caffeine labels erased with a magic marker. A small click in my ears was followed by Meta's AI voice telling me that the chamomile tea probably didn't contain caffeine. It was reading the labels and making judgments using generative AI.

I was demonstrating a feature being rolled out in Meta's second generation Ray-Ban glasses starting today, a feature that Meta CEO Mark Zuckerberg had already promised in September, when the new glasses were announced. The AI ​​features, which can access the cameras in Meta's glasses to view images and interpret them with generative AI, were expected to launch in 2024. Meta moved to introduce these features much faster than I expected, although the Early access is still a beta. In addition to adding Bing search to Ray-Bans as part of a new update that increases the power of the voice features already available in the glasses, Meta's glasses are quickly starting to gain a number of new abilities.

I was very impressed with the demonstration because I had never seen anything like it. I have it in parts: Google Lens and other phone tools already use cameras and AI together, and Google Glass – a decade ago – had some translation tools. That said, the easy-to-access way Meta's glasses have to invoke AI to identify things in the world around me seems pretty advanced. I'm excited to try a lot more.

A restaurant sign in Italian, with captions above and below asking an AI assistant to translate

The glasses do not have a display and only respond to responses. But the Meta View phone app saves the photos and AI responses for later.


Multimodal AI: How it works now

The resource has limits at the moment. It can only recognize what you see by taking a photo, which the AI ​​analyzes. You may hear the shutter release after making a voice request, and there is a pause of a few seconds before a response arrives. Voice instructions are also wordy: every voice prompt on the Meta glasses needs to start with “Hey, Meta.” and then you need to follow with “Take a look at this” to trigger the photo, followed immediately by whatever you want to ask the AI ​​for. “Hey, Meta, take a look at this and tell me a recipe with these ingredients.” “Hey Meta, take a look at this and make a funny caption.” “Hey, Meta, take a look at this. What plant is that?”

Every response from the AI ​​and the photo it saw is stored in the Meta View phone app that pairs with the glasses. I like this because it's a visual/written record for later, like notes to jog your memory. I could see wandering around somewhere and asking questions, using it as a form of Google search for my eyes, while shopping or who knows what.

A photo of grilling, with captions asking an AI assistant to help you cook A photo of grilling, with captions asking an AI assistant to help you cook

I haven't tried Meta's glasses while cooking – yet.


It could also have possible uses for relief purposes. I wore a pair of non-prescription Meta test glasses and asked what I was seeing. Answers may vary in detail and accuracy, but may serve as a warning. He knew I was showing my glasses, which were said to have bluish lenses (blue-black frames, very close).

Sometimes you may have hallucinations. I asked the glasses about the fruit in a bowl in front of me and they said there were oranges, bananas, dragon fruit, apples and pomegranates. It was correct, except for the pomegranates. (There were none of those.) I was asked to make a caption for a large stuffed panda in front of a window. I did some cute ones, but one was about someone who was alone and looking at their phone, which didn't fit.

I looked at a menu in Spanish and asked the glasses to show me spicy dishes. He read some dishes and translated some important ingredients for me, but I asked again about meat dishes and he read everything in Spanish.

The possibilities here are wild and fascinating, and possibly incredibly useful. Meta admits that this early release will be to discover bugs and help evolve the way the glasses' AI works. I found there were a lot of “Hey, Meta, check this out” moments. But this process could change, who knows. When engaged in immediate image analysis, asking direct follow-up questions may work without saying “Look at this” again, but I'm sure my success will vary.

A hand pointing at a mountain, with bubbles asking the AI ​​to help caption a photo A hand pointing at a mountain, with bubbles asking the AI ​​to help caption a photo

When will subtitles be useful and when will hallucinate?


The future of wearable AI is getting interesting

This AI, which Meta calls “multimodal AI” because it uses cameras and voice chat together, is a precursor to future AI in which the company plans to mix many forms of input, including more sensory data. from Qualcomm AI-focused chipset in Meta's new Ray-Bans he already seems ready to take on more. It is also a process that Meta plans to make more transparent over time.

Meta CTO, Andrew Bosworth told me in September that although the glasses now need a voice command to activate and “see” so they don't burn through battery life, eventually they “will have sensors low enough in power to be able to detect an event that triggers an awareness that drives AI. That's really the dream we're working toward.” Meta is also already researching AI tools that combine multiple forms of sensory data, ahead of future more advanced wearables.

At the moment, know that it is an early access beta. Meta is using anonymized query data to help improve its AI services during the initial access phase, which may worry people who want more privacy. I don't yet know the specific details of acceptance, but more discreet controls on data sharing appear to be in place once the final AI features are released, likely next year.

All of this reminds me of exactly what Humane is aiming for with its Wearable AI Pin, a device I haven't even seen in person yet. While Human's product is expensive and needs to be worn on clothing, Meta's glasses cost $300 and are already on store shelves. As watches, VR headsets, and smart glasses evolve their AI capabilities, things could look very different for the future of wearable technology and its level of assistive consciousness.

It's becoming clear that a new frontier of AI wearables is already underway, and Meta's glasses are coming first.

Editor's note: CNET is using an AI engine to help create some stories. For more information, see this post.



Please enter your comment!
Please enter your name here