Runway 'Act-One' Can Create Character Animations Using Simple Video Inputs

Runway Act One

Technology has come a long way, and AI is one of the advancements that create possibilities like never before.

One example, is animations in films and videos games, where actors and actresses have to use motion capture headset with specialized cameras and sensors to capture their facial movements. This gear included a head-mounted camera that recorded facial expressions in high detail. The equipment is covered in markers, often small dots placed on the face.

These markers allow the computer to map the actors and the actress expressions onto the digital models of their characters, frame by frame.

The process is called facial motion capture or facial performance capture.

AI is changing this, and it's able to achieve a similar feat, but without the contraption.

Runway AI, the American company headquartered in New York City that specializes in generative artificial intelligence research and technologies, focuses on creating products and models for generating videos, images, and various multimedia content.

And here, it has introduced what it calls the 'Act-One' feature, which is a way to "generate expressive character performances using simple video inputs."

In a web page, the company wrote:

"At Runway, our mission is to build expressive and controllable tools for artists that can open new avenues for creative expression. Today, we're excited to release Act-One, a new state-of-the-art tool for generating expressive character performances inside Gen-3 Alpha."

"Act-One can create compelling animations using video and voice performances as inputs. It represents a significant step forward in using generative models for expressive live action and animated content."

Read: Runway Introduces 'Gen-3 Alpha' Image-To-Video AI: Focusing On Speed And Quality

The goal is to be able to capture "the essence of a performance."

"Traditional pipelines for facial animation often involve complex, multi-step workflows. These can include motion capture equipment, multiple footage references, manual face rigging, among other techniques."

"The goal is to transpose an actor's performance into a 3D model suitable for an animation pipeline. The key challenge with traditional approaches lies in preserving emotion and nuance from the reference footage into the digital character."

With Act-One, actor and actress can have their expression recorded and portrayed by their characters, without requiring extra equipment.

"Act-One is able to translate the performance from a single input video across countless different character designs and in many different styles," wrote Runway on its X account.

Initially, Act-One is focused "mostly" on the face "for now," according to Cristóbal Valenzuela, co-founder and CEO of Runway.

Runway’s approach offers significant advantages for animators, game developers, and filmmakers alike.

The model is able to accurately captures the depth of an actor or actress performance, while remaining versatile across different character designs and proportions.

This opens up exciting possibilities for creating unique characters that express genuine emotion and personality.

Act-One’s standout feature is its ability to produce cinematic-quality, realistic outputs from multiple camera angles and focal lengths, giving creators unprecedented flexibility.

This versatility allows for emotionally resonant character performances, capturing details that previously required costly equipment and complex workflows.

With Act-One, the emotional depth and unique style of an actor/actress performance can now be faithfully translated, even in challenging scenes.

This shift brings characters to life in fresh, impactful ways, opening up opportunities for richer storytelling across both live-action and animated formats.

Previously, Runway's tools, such as Gen-2 and Gen-3 Alpha, enabled video-to-video AI conversion, allowing users to upload footage and apply AI-driven “reskinning” effects.

Act-One, however, is purpose-built for advanced facial mapping and effects, setting a new standard in character-driven visual storytelling.

Since the generative AI boom that started soon after OpenAI introduced ChatGPT, the race that is becoming more intense is seeing a lot of tools that do pretty much the same thing.

But Runway's focus differs than most.

Runway, which quickly becomes one of the leading AI video platforms, shows how AI can help enhance filmmaking in general through character consistency, which is something most AIs lack.

In other words, Act-One is like a new approach to AI video generators, which allows a more modern puppeteering: it allows users to film themselves or have an actor/actress perform and then use AI to completely change the way they look.

It's able to accomplish this by allowing users to upload a video along with a control image, and let the AI work afterward.

As AI video tools get better in reading human motion, lip-synching and character development, Runway with Act-One manage to tackle one of the biggest AI weaknesses.

"The consistency and performance is unmatched with Act-One," explained Valenzuela.

In a post on X, Valenzuela noted a shift in how the industry approaches generative models.

"We are now beyond the threshold of asking ourselves if generative models can generate consistent videos. A good model is now the new baseline. Generating pixels with prompts has become a commodity. The difference lies (as it has always been) in what you do with a model - how you think about its applications and use cases, and what you ultimately build."

"There has been an obsession with technology, with people trying to figure out what to do with models. Now is the time to start from the other side and work backwards to the technology. The why matters. Finally."

According to Runway, Act-One charges 10 credits per second with a minimum of 5 seconds.

What this means, a video that is 5 seconds long will require 50 credits.

After the 5 second minimum, each additional second is charged 10 credits, with partial seconds accounted for and rounded up to the nearest decimal. In an example, a 5.6 seconds video costs 56 credits.

Users can hover over the duration modal to see the calculated credit cost before generating.

They can click on the Generate button after confirming that they're content with the selected inputs and credit costs.

"Your video will begin processing in your current session, where each video will be available for review once complete," explained Runway in a dedicated help page.

Published: 
29/10/2024