Visual Information Systems. A Course Project On Trailer Classification. Shreeharsh Kelkar

Transcription

1 Visual Information Systems A Course Project On Trailer Classification Shreeharsh Kelkar sak2010@columbia.edu 1

2 1.1 Goal of This Project The goal of this project is not just trailer classification, but film classification. However considering that the average length of a normal film is approximately 2 hrs, it becomes difficult to effectively apply techniques such as shot segmentation, motion characterization to such a long sequence of images. A trailer, on the other hand, can be considered to be a compact representation of a film. Indeed a trailer is basically a short clip, which effectively summarizes a film. Thus the problem of film classification can be reduced to one of trailer classification. A trailer can be anything between 1 minute to 5 minutes long. Interestingly since a trailer is supposed to package a film, it generally contains many of the film s high points designed to capture the potential viewer. Thus the trailer of a film emphasizes the type of film it is marketing. Thus classifying the trailer also turns out to be better than classifying the whole movie. (For e.g. an action movie trailer will generally contain lots of action even though the film may have a large number of non-action shots interspersed with action sequences.) At this stage we might also speculate on the uses that this automatic trailer classification may be put to. It is not hard to conclude that in the future video may be delivered to the user using high-speed networks such as the Internet. Thus the user will now have access to a multitude of channels, all of which have diverse video content. In such circumstances, the user may require this automatic classification in order to find a program, which suits his interests. Interestingly trailer classification may also be used to characterize the amount of violence in the film (since violence in films can be easily distinguished by the motion content of the film). This could be useful for other tasks such as film censorship. The rest of this document is organized as follows. Section 1.2 is a brief introduction to the concept of film genre. Section 1.3 gives the overview of how a trailer may be classified. Section 1.4 deals with the implemented method along with the results obtained. Lastly in section 1.5, I have attempted to try and obtain certain characteristics in which trailers of different genre films differ from each other. 2

3 1.2 Film Genres Film Genres are various forms or identifiable types, categories, classifications or groups of films that have similar, familiar or instantlyrecognizable patterns, filmic techniques or conventions that include one or more of the following: settings (and props), content, themes, plot, narrative events, motifs, styles, structures, situations, icons (e.g., six-guns and tengallon hats in Westerns), characters (or characterizations), and stars. The main genres are action/adventure, comedy, drama, and the musical. The term action/adventure is an all-encompassing term that comprises genres such as war movies, gangster (crime) movies, science-fiction (like the Star Wars Trilogy), horror movies and the Western. On the other hand the comedy, drama and the musical genres are essentially characteroriented (or relationship oriented films). Recently film types such as buddy films, caper films, aviation films etc have surfaced but for most part they fall into any one of the above-mentioned categories. Consider films said to be action/adventure films. Action films usually include high energy, stunts and chases, possibly with rescues, battles, fights, escapes, non-stop motion, spectacular rhythm and pacing, and sometimes exotic locations. Character oriented films, on the other hand are serious (or comic) presentations, portraying realistic characters, settings, life situations, and stories. These films are also characterized by most part, by an absence of high-energy stunts and chases. It is primarily this absence of motion that can be used to differentiate character films from action films. Action films and character-oriented films (and consequently their trailers) also differ in the way they approach background music, narrative and other film techniques. These differences will also be explored later in Section

4 1.3 Overview of Trailer Classification Trailers can be classified as action/character by two basic characteristics: motion activity and average shot length. It is difficult to quantify the abstract features of each film genre such as the character complexity since these are basically qualitative in nature. However we can safely say that action movies have busy imagery highspeed chases, car-crashes, and explosions while character-oriented movies on the other hand will have long duration shots. This is because these movies consist mainly of conversations and a meaningful dialogue cannot be shown in a shot which doesn t even last a few seconds. On the other hand, showing flying debris (bomb explosion) results in a highmotion scene with fewer frames in a shot. Thus in a limited way, the shot length and the motion-content of a trailer can be used for classification. To establish the amount of motion in a scene, (and this must be independent of camera motion and changes in lighting) in [1] the authors have used tangent distance between successive images. The key idea behind tangent distance is that, when subject to spatial transformations, images describe manifolds in a very high dimensional space. A metric invariant to those transformations should measure the distance between these manifolds instead of measuring the distance between other properties of the images themselves. However since these manifolds are very complex the distance between the tangents to the manifolds is measured instead. Instead of this the optical flow in each shot could be determined. The average of the optical flows of all the shots in the trailer could be used to quantify the amount of motion in the trailer. 4

5 The average shot length can be determined by using the twin threshold shot detection algorithm. Exact partitioning into shots is not necessary since trailers mostly rely on fast cuts rather than on slow fades or wipes. The trailer is now characterized by the feature pair (A, L), where A is the average motion activity and L is the average shot length. 1.4 Implementation & Results The following method was used to classify the trailers. A clip from a trailer, which seemed to be most representative of its genre, was chosen. The clip had duration not exceeding 10 seconds (or when decomposed into its frames, not more than 100 frames). 1. The twin threshold shot detection algorithm was then applied to the clip. The average shot length (L) of the clip was determined. 2. The average difference between the color histograms of subsequent frames was also determined. Seven clips were analyzed. The movies along with their IMDB (Internet Movie Database) genre are tabulated below: Movie The Lord of the Rings: The Fellowship of the Ring (L) Shrek (S) Forrest Gump (F) Memento (M) The Fast and the Furious (FF) X-Men (X) Se7en (7) IMDB Genre Fantasy/Adventure Adventure/Romance/ Animation/Fantasy/ Family/Comedy Drama/Comedy Drama/Crime/ Mystery/Thriller Thriller/Action Action Sci-Fi Crime/Thriller/Drama/Mystery The symbol in the bracket has been used to represent the movie in the (A,L) feature space. 5

6 After calculating the (A,L) features for each of these clips was plotted as shown in the figure below. We see in the graph above that character-oriented movies tend to cluster in the lower right corner while action movies cluster in the leftmost part of the graph. 1.5 Some Trailer Characteristics All trailers generally have some similarities, which may be used in aiding this classification: 1. A film trailer starts out slowly (average shot length is higher) and then becomes faster and faster (the shot length decreases) as it approaches its end. This is primarily because the trailer is gradually leading the viewer to the dramatic center point of the actual film. 2. Dramatic films have trailers that are generally narrated, action movies generally use captions to talk about the movie plot. 3. The background music for dramas and character movies in general is slow and primarily instrumental. (Violins and other instruments 6

7 are mostly used.) Action film trailers on the other hand have background music that abounds in drums and harsh sounds. A particular film (or trailer) genre can be identified by its look. It is hard to determine what exactly this look consists of but it is usually the result of a large no of things: the color scheme, the placement of the characters in relation to the background, the shapes of the objects etc. In this project an attempt to get the color histogram of the entire trailer clip was made. To do this the following method was used. The histogram intersection of all the frames within a shot was determined. The average of the intersections over the total no of shots in the clip was then calculated. Two of the resultant histograms are shown in the figure below. From left to right, these are respectively of the movies, The Fast and The Furious and Forrest Gump. Though generalizations cannot be made, the histogram of Forrest Gump seems to be richer in colors than that of The Fast and the Furious. The trailer classification done above suffers from the following drawbacks 1. Only 7 trailers were analyzed. 2. Only a small clip from trailer itself was considered (generally not exceeding 100 frames). 3. Method was not real time. 4. RGB color histograms were exclusively used. Perhaps a more perceptually similar color space like the HSV space would have been more useful. 7

8 1.6 References Papers Nuno Vasconcelos & Andrew Lippman, Towards semantically meaningful feature spaces for the characterization of video content Giridharan Iyengar & Andrew Lippman, Semantically controlled contentbased retrieval of video sequences J. M. Corridoni, A. Del Bimbo, Film Semantic Analysis Books Alberto Del Bimbo, Visual Information Retrieval Trailers obtained from 8