For many people creating doodles was part of growing up. Children loves scribbling their minds. This AI however, can reimagine them.
Tech giant Meta has officially open-sourced an AI project, where anyone can turn their doodles into something spectacular, in terms of graphics. This is because the AI can turn doodles to life, by converting drawings into real animations.
With the project called 'Animated Drawing', the Facebook’s parent company hopes that developers can create bigger and brighter art experiences.
In a blog post, Meta said that:
" [...] Our researchers envisioned using object detection models, pose estimation models, and image processing-based segmentation methods to quickly create a digital version of the drawing. This could then be deformed and animated using traditional computer graphics techniques. At Meta, we see this vision as an opportunity to connect people and communities through the power of creativity."
It all began in 2021, when Meta's team at FAIR decided to apply recent advances in computer vision towards a fast and intuitive pipeline to animate the human-like figures found in character drawings.
The technology involves envisioning using object detection models, pose estimation models, and image processing-based segmentation methods to quickly create a digital version of the drawing. With the tool, users can upload their hand-drawn doodle, to then tweak or make adjustments to where the joints should be located.
It's the digital version of the drawing that can then be deformed and animated using traditional computer graphics techniques.
At that time, there was a problem: character drawings, at the scale needed to train a computer vision model, were scarce.
The researchers needed annotations, such as bounding boxes, segmentation masks, and joint locations, to train a more powerful AI. Because no such dataset existed at the time, the researchers at Meta resourced to creating the Animated Drawings Demo, and made it a publicly-accessible web-based tool where creators could upload their drawings, view/correct a few model predictions, and receive an animation starring a character in the drawings.
The researchers hoped to eventually collect 10,000 drawings with which to improve their model’s performance.
And yes they succeeded, because within the first few months, users had uploaded and provided consent to use over 1.6 million images.
But this was when Meta realized that people used it for purposes other than the original intention of the AI.
While the demo required users to upload human figures, users apparently uploaded anything form quadrupeds, birds, fish, and many other forms. They even uploaded company logos, stuffed animals, anime characters, pets, action figures, and a variety of other things that people wanted to animate.
Meta also realized that users wanted a much more comprehensive tool, like for example, they wanted transparent backgrounds, support for different skeleton types, multiple interacting characters, sound effects, background scenery, and text overlays.
"Clearly, there was interest in more drawing-to-animation-based experiences," said Meta.
So instead of extending the demo to support all the new feature requests, the researchers decided to just release an open-sourced version of the models and code used by the Animated Drawings Demo, along with a dataset of around 180,000 drawings.
Meta hopes that it can help inspire other developers to create more extensive drawing-to-animation experiences.