Smart Video Retargeting

The Invention

With the recent advent of mobile video displays and their expected proliferation, there is an acute need to display videos on smaller displays than originally intended. There are two main issues recognized today; one is the need to change the aspect ratio of the video (the ratio of the frame's width and height) and the second is the need to down-sample the video while maintaining enough resolution of objects-of-interest. An example for the first challenge is the display of widescreen movie on a TV screen. An example of the need for smart down-sampling is the adaptation of a ball game to a cellular screen, where the ball needs to remain large enough to be easily seen.

The Need

The current industry solutions are non-existent or primitive and include ignoring the problem, cropping each frame at the sides of the frame and changing the aspect ratio of the smaller screen by adding black stripes above and below the frame or simply by stretching the content. The gap between the huge need and poor solutions can be explained by several misconceptions that make the problem look much harder than it is. At first glance the problem of deciding what information in the video can be discarded without degrading the user's experience seems as hard as the problem of AI itself. In reality, we achieve excellent results by very efficient and robust means.

We have developed a method that can infer which regions in each frame are of importance – these regions are then used to reconstruct a down-sampled frame. We achieve this using novel image processing computer vision algorithms that determine each region’s importance, map it into an optimal optimization framework and transforms the original frame to a downsized frame.

Figure 1: An example of retargeting. (Bottom) original frame from standard benchmark news sequence ”Akiyo”; (Top Left) the downsized frame achieved by our method; (Right) a conventionally down-sampled frame

Potential Applications

Our technology allows real-time and offline broadcasting of video content to mobile devices and other displays (a good example is live sport event). This is a huge market in the US, Europe and ASIA. In addition, our technology can become a part of the video playback pipe-line in home or mobile devices.

Using our technology, images and videos can be altered to any size and better fit web-page design and other applications. Our technology can therefore find its place in commercial video and image editing software, and also be employed by the many companies in the competitive market of online video and images.

Patent

Patents pending in Europe and US.

Links

http://www.cs.tau.ac.il/%7Ewolf/presentations/RetargetClip.006b.wmv

Project manager

Oren Calfon
VP Business Development, ICT

Project researchers

Daniel Cohen-Or
T.A.U Tel Aviv University, Exact Sciences
School of Computer Science

Lior Wolf
T.A.U Tel Aviv University, Exact Sciences
School of Computer Science