Meta introduces Movie Gen text-to-video-and-sound generator

A screenshot of a short video generated with Meta’s Movie Gen, with a watermark on the bottom left
| Photo Credit: Meta

Meta introduced its Movie Gen text-to-video-and-sound generator, that uses generative AI to create video clips of up to 16 seconds at a rate of 16 frames per second and high-fidelity audio up to 45 seconds.

“Given a text prompt, we can leverage a joint model that has been optimized for both text-to-image and text-to-video to create high-quality and high-definition images and videos. This 30B parameter transformer model has the ability to generate videos of up to 16 seconds at a rate of 16 frames per second,” said Meta in a blog post.

Some tasks that the models can carry out include text-to-video synthesis, video personalisation, video editing, video-to-audio generation, and text-to-audio generation, said Meta in its research paper.

Movie Gen can be used to create video clips as well as edit them based on text prompts. It can also turn photos of a person into a short video clip, apart from generating and extending soundtracks. The company shared hyper-realistic clips of a baby hippo swimming, a child running across a beach, as well as a video clip of a man from a photo carrying out a scientific experiment.

While the tech is not yet widely available, Meta suggested that in the future, its users across social media platforms could generate their own videos or edit their multimedia before sharing it.

“By taking a collaborative approach, we want to ensure we’re creating tools that help people enhance their inherent creativity in new ways they may have never dreamed would be possible. Imagine animating a “day in the life” video to share on Reels and editing it using text prompts, or creating a customized animated birthday greeting for a friend and sending it to them on WhatsApp,” said Meta.

There are still widespread concerns about the potential of AI-powered video generation and its possible impact on the individuals’ privacy, the copyright of artists, and children’s safety.