Google says that it is testing a new feature on its YouTube Stories app that allows users to change video backgrounds in real time, without the need to use a green screen, or complicated software.
Known as “video segmentation,” the process of separating foreground from background in a video is notoriously difficult to do. If you’re not a pro, and/or not using a green screen. Doing it would need hours of work, if not a professional video setup. But Google says it is developing AI that will be able to do mobile real-time video segmentation on the fly – right within its beta Stories app. And all you will need is your smartphone.
The AI technique uses very much the same process that a human would undertake; it takes each frame and identifies and cuts out the background. The difference is that it gets done really fast. As this is done, the background can be replaced as well. How did Google make it work? Well, after being taught how to segment one image, the AI can make comparisons with the next and the next etc.
If there are small differences between several frames (i.e. someone is talking into the camera) the AI adjusts the mask only slightly. When there are major differences between frames, the AI creates a new mask.
Of course, Google had to teach the AI to do this, so a lot of human input was necessary to begin with. Yet, the result is pretty impressive, especially if you think that this process can be run by a smartphone. After adjusting for speed and quality, the final result is very smooth, allowing users to replace a background in real time, at 100 fps on an Apple iPhone 7, and at over 40fps on a Google Pixel 2.
The feature is being beta tested currently, and rolling out slowly, but that will allow Google to figure out how to improve it, and hopefully add new options; AR for example.
You might also like
More from Youtube
YouTube Announces #YouTubeBlack Voices Creator Class Of 2022
YouTube's second edition of its $100 million #YouTubeBlack Voices Fund has announced 135 worldwide winners.
YouTube Is Looking To Innovate Its Product Offering With NFTs And DAOs
YouTube CEO Susan Wojcicki hinted at the possibility that YouTube will expand its range of creator offerings to include NFTs.
YouTube Significantly Improves User Experience On TVs
Google has announced new features that improve the user experience on its TV app, to save users time when using …
YouTube Is Enhancing Its Studio Mobile App
YouTube Studio is bringing its mobile app in line with its desktop version for creators to better manage their channels …
YouTube Introduces New Tools And Features In Support Of Shorts
YouTube's product development team has announced the addition of several updates that will enhance the creation and experience of Shorts.
YouTube Launches Three New Creator Features
YouTube has expanded channel permissions, added customizable text in chapters and links to places mentioned in a video.
YouTube To Fix Systemic Errors In How It Handles Copyright Claims
YouTube issued a transparency report highlighting that 2.2 million copyright claims have incorrectly caused creators a loss of revenue.
YouTube Lists Top 10 Best US Creators And Videos of 2021
Since end-of-year Rewind was canceled, YouTube is offering another way to celebrate 2021's most successful creators and their works.
YouTube Now Lets Creators Reuse Details From Previous Uploads
In the latest episode of Creator Insider, YouTube has announced that it is now letting creators reuse details from previous …
