OpenAI Sora, the AI innovator’s latest video generation model is now available to the public (provided you have the right subscription plan). However, this revolutionary model has been generating hype (and a little controversy) for a while now.
OpenAI first introduced Sora to the world on February 15th, 2024, by sharing various AI-generated videos and a research paper on X (Twitter).
Although Sora isn’t the first generative AI model to focus on video generation, it does appear to be one of the models with the most capacity for realism that we’ve seen so far. Over the last few months, various testers have already shown just how powerful this model can be, sharing their own AI-generated videos on social media.
Now that Sora is more “openly available”, there could be a massive change in the type of content we see online. So, what exactly is Sora, how does it work, and what are the benefits (and limitations) to consider going forward?
What is OpenAI Sora?
OpenAI Sora is a text-to-video AI generation model developed by OpenAI – the company best known for the world’s most popular generative AI chatbot, ChatGPT. This model marks a new era in OpenAI’s roadmap, allowing it to move beyond text-based content generation (with ChatGPT), and image generation (with tools like Dall-E).
Experimenting with the model is similar to using OpenAI’s existing tools. You give the bot a prompt, like “A horse running in an open field”, and it will create a video. Of course, the more specific you are with what you want, the more likely you are to end up with great results.
Notably, Sora is still in its very early stages (OpenAI has already noted that this new model still has a lot of limitations), but its already making waves in the content development space – and it has a bunch of features to explore. Before we dive into those though, let’s take a look at how Sora works.
How Does OpenAI Sora Work?
On a basic level, the technology that powers OpenAI Sora is similar to the tech that allows users to search for pictures on the web. If you show an AI model enough photos of a specific thing, like a cat or a mouse, it can generally identify the same “thing” in other images.
On a deeper level, the inner workings of Sora are a little more complex. OpenAI actually provided a deep-dive overview of the tech, if you want to check that out. However, what you really need to know is that OpenAI is trained on huge volumes of data, which allow it to identify specific images.
It’s based on a diffusion model, which essentially means that the tool breaks an image down into several different parts and then works towards “cleaning” an output through a series of feedback loops. Similar to most of OpenAI’s models (and other generative AI tools), Sora also uses transformer tech.
Transformers use various sophisticated data evaluation techniques to process massive amounts of data and identify the most “important” parts of that data. The transformer model creates a high-level “layout” for your video, while the diffusion model fills in the details.
To ensure it captures the “essence” of a prompt, Sora also uses a recaptioning technique (something also available in DALL-E 3). That means before it creates a video, it uses a GPT to re-write a prompt with a lot more detail.
Who Can Use OpenAI Sora? The Path to Availability
OpenAI usually takes a “slow and steady” approach to releasing new models. In February 2024, when the Sora model was initially announced, it was available only to “red teamers”. Those are the people who basically test the stability and security of products.
In November, a group of beta testers claimed to have leaked early access to the model because they felt that OpenAI were exploiting artists for “free labor”. They criticized the company for having hundreds of artists provide unpaid testing support and feedback. Obviously, OpenAI quickly shut down access after this leak, but decided to launch Sora officially in December, all the same.
The latest version of the current model (Sora Turbo) is currently available in some regions worldwide as a standalone product (on the Sora.com website). Alternatively, ChatGPT Plus ($20 per month) and ChatGPT Pro ($200 month) users can access the model too.
Through ChatGPT, the tool is available in most regions, except for certain locations in the European Economic Area (like the UK and Switzerland). However, what you can do with the model depends on your plan, for instance:
- ChatGPT Plus: Users get up to 50 priority videos (720p resolution with a 5 second duration)
- ChatGPT Pro: Users get up to 500 priority videos, and unlimited relaxed videos, with up to 1080p resolution, a 20 second duration, and 5 concurrent generations. These users can also download their videos without a watermark.
Right now, ChatGPT Team, Enterprise, and Edu users don’t have access to OpenAI Sora at all, but this could change when the company has refined the model further.
What Can OpenAI Sora Do? The Features
Right now, OpenAI Sora is still in its early stages. OpenAI may even have released the product earlier than intended, after the leak. Still, it can already create HD-quality videos up to a minute long (without sound), based on text prompts. It can create all kinds of characters and scenes, and even produce simulations of video games (like Minecraft).
Obviously, the videos vary in quality, as you may know if you’ve seen the examples of the videos that OpenAI has shared. People might move in odd ways, and images aren’t always 100% accurate. But many creators are still viewing the tool as revolutionary.
Here are some of the features you can access on the platform right now.
Prompt-Based Video Generation
Although Sora can create videos up to a minute in length, users can only create videos up to 20 seconds long for the time being. Once you’ve created an account (with ChatGPT Plus or Pro), you can visit the model via ChatGPT or on the Sora website.
To generate a video, just enter a text prompt in the field at the bottom of the screen. You’ll also be able to upload an image or video file in your initial prompt to give the bot more context, by clicking the “+” button. Notably, though, there are limitations on what kind of content you can upload. For instance, you have to have complete ownership rights for anything you use.
Once you submit your prompt, OpenAI Sora will generate a video, which you can hover over and preview in your library. Then you can start editing and adjusting the output with the additional features in the video editor.
The Remix Feature
Remix, in the Sora editor, allows users to essentially “reimagine” the videos they’ve already created by altering specific components but maintaining the essence of the original content. For instance, you can change colors, add a new background, or tweak brightness levels and focus areas.
This could be a great way for content creators to use generative AI to refresh old content, personalize it to different users, or explore new variations of video clips.
Re-cut in Sora
Re-cut is a feature in OpenAI Sora that allows a user to choose and isolate specific frames in a video, extending them to build out a more comprehensive scene. This tool is fantastic for enhancing specific moments in a video, or drawing more attention to valuable visuals. You can also use it to improve the flow between scenes in a video, boosting the storytelling potential.
The Loop Feature
The “Loop” feature in OpenAI Sora makes it easier to create a video that basically “repeats itself”. Imagine you want to create a hypnotic image of a spinning top turning, or a background video for a song on your playlist. The tool tries to stitch the video together naturally, so the loops feel smooth and naturally (rather than like they’re starting again from scratch).
Sora’s Storyboard
With the Sora Storyboard, creators can produce specific shots at certain “frame points” in a video’s timeline. This gives them a lot more control over the visual narrative – hence the name. For instance, you could choose to have the first set of frames focused on a specific landscape, then ask the bot to adjust the next lot of frames to show the landscape from a different perspective.
The Blend Feature
Sora’s blending feature allows users to combine different styles and video elements to create unique compositions. It’s a bit like asking DALL-E to combine different artistic concepts to create a new image. You’ll be able to mix footage, colors, and different artistic approaches to experiment with a wide range of outputs.
Style Presets
The Style presets in Sora provide users with predefined aesthetic templates they can apply directly to videos. These presets make it a lot easier to achieve a specific look, whether you’re searching for something vibrant, cinematic, or more professional.
OpenAI Sora: The Opportunities, and Challenges
Even in its early stages, there are a lot of use cases for something like OpenAI’s Sora – that’s probably why so many competitors are creating similar products. As tools like this become more advanced, the potential could be endless. For instance, companies could use AI tools to create more immersive extended reality content, or personalized content for entertainment, education and more.
Currently, some of the main use cases for Sora include:
Accelerating Content Creation
Sora is an excellent tool for creating videos from scratch – and extending or customizing existing videos to make them more unique or impactful. Just like generative AI tools have made it easier for users to create images and text-based content at speed, Sora will revolutionize how we produce video.
This could mean we see many more companies and artists creating video-based content at scale without the need to invest in complex and expensive equipment. Plus, it should mean that creators can produce more “personalized” content, whether they’re creating educational videos for learners, onboarding content for customers, or anything else.
Transforming Marketing and Advertising
Already, there’s amazing potential for Sora to upgrade social media marketing practices. It’s a great tool for creating short-form videos for platforms like YouTube Shorts, TikTok, and Instagram Reels. You don’t even need a high-quality smartphone to start producing engaging content.
Elsewhere in the marketing world, companies will be able to create promotional videos, product demos, and adverts, without the typical costs. Imagine being able to create a video showcasing someone wearing your ski boots while exploring the Swiss Alps – without having to pay for travel or models. That’s the kind of thing OpenAI Sora could accomplish.
Prototyping and Synthetic Data Generation
Even if you’re not using AI video in a final “product”, it’s a great tool for demonstrating ideas to potential stakeholders quickly. For instance, filmmakers could use AI to generate scenes that they might use before they’re actually shot, and designers could create videos showcasing products with specific features.
AI video generation is also a good option for creating “synthetic” data, which could be used to train computer vision systems and other tools. For instance, with an AI-generated video, you could potentially train a computer-vision system to better recognize animals with specific features, or understand how cars move on the road.
The Biggest Challenges with OpenAI Sora
Just like most AI tools, OpenAI’s Sora might be generating excitement among some users – but it’s surfacing concerns for others. First, as evidenced by the leak mentioned above, some artists and creators are concerned about the information being used to train Sora.
They believe that OpenAI ignored the concept of ethical AI by “transcribing” content from the web to train their model, rather than licensing content from artists. Beyond that, the major concerns include:
Potential to Generate Harmful Content
Without the right guardrails in place, any AI content generator has the potential to create inappropriate content. This is one of the main reasons why many governments and regulatory bodies are implementing stricter rules about how AI should be used for content generation.
Fortunately, OpenAI says its already taking steps to address this issue. For instance, the company says it’s blocking the model’s ability to create specific types of content, such as “child abuse” videos, and sexual deepfakes. They’ll also be continuing to monitor the model for any potential signs of misuse from customers, and upgrading their guardrails in the future.
Misinformation and Deepfakes
Obviously, one of the most compelling things about OpenAI Sora is that it can create fantastical scenes that don’t really exist in real life. This also makes it easier to create “deepfake” videos where situations and people are changed to portray something untrue.
If this content is generated and presented as truth, (either accidentally or on purpose), it creates serious ethical issues. Already, many individuals feel they can no longer trust what they see online, or differentiate between real and synthesized content. This will be a difficult issue to overcome in the years ahead, but its something OpenAI has also acknowledged.
Biases and Discriminatory Content
Beyond the issues above, all generative AI models can potentially create biased or discriminatory content based on their given data sets. An incomplete or problematic data set can perpetuate biases, potentially increasing the likelihood of problematic content.
Again, this is a difficult issue to overcome, but it’s something that OpenAI is attempting to address by providing tools like Sora with comprehensive data sets. Although, as mentioned above, we don’t know exactly what kind of training data was used for the model.
The Future of AI Video Generation: What’s Next?
Generative AI tools for video creation could very well mark the next era of AI-driven content development. As mentioned above, while various companies have experimented with similar tools, few have delivered the same results as Sora.
While Sora still has its limitations – particularly in regard to generating unrealistic physics and making mistakes with images of people and animals – it’s a huge step up in the market. The introduction of this model has already been enough to prompt a lot of competition in the marketplace.
Already Runway Generation 3 is delivering similar results for content creators, and Google has its own prompt-to-video generator (Google Veo), now available through Vertex AI. You can even use elements of Veo when creating YouTube shorts.
Sora is likely to be a tool that continuous to drive ongoing innovation and competition in the field of generative AI. Who knows what we’ll be able to create with the help of AI models in the years to come, if companies like OpenAI continue to push the boundaries.
