Nonetheless from “A teddy bear washing dishes,” as generated by Google Imagen Video.
At present, Google introduced the event of Imagen Video, a text-to-video AI mode able to producing 1280×768 movies at 24 frames per second from a written immediate. At the moment, it is in a analysis section, however its look 5 months after Google Imagen factors to the speedy improvement of video synthesis fashions.
Solely six months after the launch of OpenAI’s DALLE-2 text-to-image generator, progress within the discipline of AI diffusion fashions has been heating up quickly. Google’s Imagen Video announcement comes lower than per week after Meta unveiled its text-to-video AI software, Make-A-Video.
In line with Google’s analysis paper, Imagen Video consists of a number of notable stylistic talents, reminiscent of producing movies primarily based on the work of well-known painters (the work of Vincent van Gogh, for instance), producing 3D rotating objects whereas preserving object construction, and rendering textual content in quite a lot of animation types. Google is hopeful that general-purpose video synthesis fashions can “considerably lower the issue of high-quality content material era.”
The important thing to Imagen Video’s talents is a “cascade” of seven diffusion fashions that rework the preliminary textual content immediate (reminiscent of “a bear washing the dishes”) right into a low-resolution video (16 frames, 24×48 pixels, at 3 fps), then upscales it into progressively larger resolutions with larger body charges with every step. The ultimate output video is 5.3 seconds lengthy.
Video examples introduced on the Imagen Video web site vary from the mundane (“Melting ice cream dripping down the cone”) to the extra implausible (“Flying via an intense battle between pirate ships on a stormy ocean.”) They comprise apparent artifacts, however present extra fluidity and element than earlier text-to-image fashions reminiscent of CogVideo that debuted 5 months in the past.
One other Google-adjacent text-to-video mannequin additionally formally debuted in the present day. Referred to as Phenaki, it may create longer movies from detailed prompts. That, together with DreamFusion, which might create 3D fashions from textual content prompts, exhibits that aggressive improvement on diffusion fashions continues quickly, with the variety of AI papers on arXiv growing exponentially at a charge that makes it tough for some researchers to keep up with the most recent developments.
Coaching information for Google Imagen Video comes from the publicly obtainable LAION-400M image-text dataset and “14 million video-text pairs and 60 million image-text pairs,” in accordance with Google. In consequence, it has been educated on “problematic information” filtered by Google however nonetheless can comprise sexually specific and violent content material —in addition to social stereotypes and cultural biases. The agency can be involved its software could also be used “to generate pretend, hateful, specific or dangerous content material.”
In consequence, it is unlikely we’ll see a public launch any time quickly: “Now we have determined to not launch the Imagen Video mannequin or its supply code till these considerations are mitigated,” says Google.