Zuckerberg’s one-hour “live conversation” in the Metaverse amazed the world

"For a time, I forgot that the person in front of me was not a real person."

"This is a conversation I had with Mark Zuckerberg in the metaverse. We are hundreds of miles apart physically, but I feel like we are face to face with him. Because our images are all built with photo-accurate 3D models. Model, this technology is amazing! I think this is how people will communicate on the Internet in the future," said Lex Fridman.

The day before yesterday, a remote conversation between Massachusetts Institute of Technology (MIT) scientist Lex Fridman and Meta CEO Mark Zuckerberg became the biggest hit on social networks. The video of their hour-long interview in the metaverse quickly reached tens of millions of views.

Now, in the metaverse, our images can be almost the same as real people, from the details of the whole body to the expressions and movements, it is so natural that it is amazing.

So in the real world, we need to pay attention to managing expressions:

The effect is so real that it makes people feel uncomfortable.

In the "First Metaverse Conversation" on the Lex Fridman Podcast, Zuckerberg showed off Meta's new Codec virtual reality avatar, which uses scanning technology to build a 3D model of the user's face, which is then driven by machine learning implementation. During the conversation, the new headset will detect the user's facial expressions and map them onto the 3D avatar in real time to form actions.

Zuckerberg said of the Metaverse conversation, "The platform is basically sending an encoded version of your avatar over the network in a way that's even more bandwidth efficient than transmitting video."

Of course, in the metaverse, you can also customize your image, such as modifying the lighting:

Unlike Meta's original avatars, which were ridiculed as legless, dull-eyed, and uncomfortable, the new generation's avatars appear extremely real, with negligible latency, allowing you to feel like you're connected to others in the virtual world. together.

"This goes to the core of the vision of augmented reality, which is to provide a sense of presence no matter where you are in the world," Zuckerberg said.

All this obviously puts more load on the rendering hardware. During the conversation, both Zuckerberg and Fridman were wearing the latest version of the Quest Pro, which has built-in facial and eye-tracking capabilities. It should be noted that this is not the Quest 3 released a few days ago. Quest Pro may not be officially released until 2025.

Codec Image requires a scan of the user's face to build a 3D virtual model.

Zuckerberg’s interview was in the metaverse, and the content of the conversation was also in the metaverse. During the hour-long conversation, Zuckerberg talked about the metaverse, augmented reality, large models and other topics.

Zuckerberg said that in fact, most of the communication between people is not done through language, but through details such as expressions. Meta is trying to use more cartoon avatars to capture expressions, delivering a realistic experience while also having a certain sense of realism.

He suggested that users could edit their virtual doppelgangers to display more exaggerated emotions, and joked that he might use avatar technology to tweak his famous deadpan delivery.

As impressive as avatars are, it may be a while before they are widely used.

It’s important to note that the figures shown in this interview were created using state-of-the-art, hours-long scanning involving hundreds of cameras, which is currently out of the reach of the average consumer.

But Zuckerberg said in the interview that Meta's plan is to allow people to scan very quickly using their smartphones.

"You just pick up your phone and shake it in front of yourself for a few minutes, say a few words, make a bunch of expressions... and then the AI can produce something of the quality that we have now," Zuckerberg Ge told Lex Fridman.

Zuckerberg said Meta plans to integrate the technology into its products over the next few years.

The current realistic projections are a significant improvement over the cartoon avatars used in Horizon Worlds, Meta's free VR game and social platform.

Zuckerberg’s metaverse selfie.

Last year, Zuckerberg was ridiculed on social media for posting a selfie of his Metaverse avatar outside the virtual Eiffel Tower, with commenters comparing the image to a 1990s video game. Zuckerberg admitted at the time that the avatar was "basic."

Unexpectedly, technology has improved so much after just one year.

As AI scientist Lex Fridman lamented during the interview: "I really don't know how to describe this feeling. It's like we are in the same room, like the future."

Zuckerberg's ongoing efforts are changing people's perception of Horizon Worlds. He told Lex Fridman that realistic avatars might help attract people who are put off by imagery in the Metaverse. "People were blown away by the experience, but we did get some feedback from people saying the avatars were in this environment," he told Lex Fridman. It doesn’t feel that realistic.”

Clearly, today's metaverse capabilities will bring about huge changes in the way we interact. Admittedly, this is the most impressive demonstration of Zuckerberg’s Metaverse project ever.

"It has changed, and our perception of the metaverse has begun to change." This is an epoch-making technology.

If Meta does this in consumer products soon, it could become a game-changing application.

Reference content: