Hollywood is all smoke and mirrors. Through clever editing and visual effects, Films can create illusions that trick viewers into believing what isn't real.
One of the most common ways is by using what's called the green screen, or also known as chroma keying. The popular "smoke and mirrors" technique in movies and TV production allows filmmakers to replace a solid green background with virtually any image, video, or environment, creating the illusion that actors are in a different setting.
Because green is a contrasting color to the human skin tones, it's easy for the visual effect team have the green color "keyed out" during post-production.
The team can then replace the void with the desired imagery, like in exotic locations, outer space, or some fantasy worlds.
Adobe wants to try this using generative AI.
A team from Adobe Research has teamed up with Hong Kong University of Science and Technology (HKUST) to introduce 'TransPixar,' an AI that could revolutionize how visual effects are produced for films, games, and interactive media.
TransPixar brings a crucial enhancement to AI-generated videos that were not possible in previous AI-powered video generators.
They include the ability to create transparent elements like smoke, reflections, and other ethereal effects that seamlessly blend into scenes.
In order words, whereas all AI tools can only handle solid visuals that are opaque, TransPixar can imitate a green screen, but without the green screen.
The TransPixar team developed the AI using a method that builds on existing AI models rather than starting from scratch.
"We introduce new tokens for alpha channel generation, reinitializing their positional embeddings, and adding a zero-initialized domain embedding to distinguish them from RGB tokens," explained Luozhou Wang, lead author and researcher at HKUST. "Using a LoRA-based fine-tuning scheme, we project alpha tokens into the qkv space while preserving RGB quality."
"Alpha channels are crucial for visual effects, allowing transparent elements like smoke and reflections to blend seamlessly into scenes," added Yijun Li, project leader at Adobe Research and one of the
"However, generating RGBA video, which includes alpha channels for transparency, remains a challenge due to limited datasets and the difficulty of adapting existing models."
In demonstrations, the system showed impressive results generating diverse effects from simple text prompts.
From generating an asteroid belt swirling chaotically through space, a cloud of dust erupting and dispersing like an explosion and more, the technology can also animate still images with transparency effects, opening up new creative possibilities for artists and designers.
The research team has made their code publicly available on GitHub and deployed a demo on Hugging Face, allowing developers and researchers to experiment with the technology.
This breakthrough arrives at a pivotal moment, with demand for visual effects skyrocketing across entertainment, advertising, and gaming industries.
As consumers want more content, production houses are always in needs to consider their budgets because visual effects can be very expensive. Not to mention that effects artists maybe hard to come by.
Technology like TransPixar can enhance how conventional VFX production works, because it offers a solution that make effects faster to create, less expensive, and more consistent in quality.
And because the technology can lift some of the labor-intensive manual work by artists to achieve realistic transparent effects, using just prompts, the tool should come in handy for smaller studios that couldn't afford expensive effects work.
While the system still needs significant computing power to process longer videos, its potential impact on the creative industry is clear.