Semantics of Narrative in Collective, Distributed Problem-Solving Environments based on Correspondence Analysis and Hierarchical Clustering

Transcription

1 Correspondence Analysis & Related Methods, CARME 2011 Semantics of Narrative in Collective, Distributed Problem-Solving Environments based on Correspondence Analysis and Hierarchical Clustering Fionn Murtagh (1, 2), Adam Ganz (2), Joe Reddington (2) (1) Science Foundation Ireland (2) Royal Holloway, University of London

2 Narrativization: our aims include Build, from an arbitrarily large number of texts, a story-line Achieve a fixed target story-line size or length Ensure preservation of data rights (implying: knowing and protecting data sources at all stages) Map story transitions so as to allow interactive reading and other adaptive usage frameworks Determine smooth story transitions so as to allow story subplot embedding, or other information placement

3 Correspondence Analysis inside Correspondence Analysis is A Tale of Three Metrics - Chi squared metric - appropriate for profiles of frequencies of occurrence - Euclidean metric, for visualization, and for static context - Ultrametric, for hierarchic relations and for dynamic context

4 Triangular inequality holds for metrics Vertical x y z Example: Euclidean or as the crow flies distance d(x, z) d(x, y)+d(y, z) Horizontal

5 Ultrametric Euclidean distances makes a lot of sense when the population is homogeneous Ultrametric distance makes a lot of sense when the observables are heterogeneous, discontinuous Latter is especially useful for determining: anomalous, atypical, innovative cases

6 Property Property 1

7 Property Property 1

8 Property Property 1

9 Property Property 1

10 Property Property 1

11 Property Property 1

12 Property Property 1

13 Property Property 1

14 Property Property 1

15 Property Isosceles triangle: approx equal long sides Property 1

16 Strong triangular inequality, or ultrametric inequality, holds for tree distances d(x, z) Height max{d(x, y),d(y, z)} d(x, z) =35 x y z d(x, y) =35 d(y, z) =10 Closest common ancestor distance is an ultrametric

17 Other Publications Deal With the Importance of Ultrametricity In the (high dimensional) limit, everything is hierarchical Ultrametricity is ubiquitous and pervasive

18 Hierarchical clustering on the time-line or other total order on the set of observables Sequence of n-1 pairwise agglomerations using the usual algorithm, given n observables Complete link criterion used But: agglomerands must be adjacent Adjacency between clusters q and q is defined as at least i and i adjacent, with i in q, i in q Theorem: the complete link method under this adjacency constraint is guaranteed not to give rise to inversions (or reversals) in the agglomerative sequence Proof: F Murtagh, Multidimensional Clustering Algorithms, Physica-Verlag, 1985

19 Convergence between Film, TV, Games, Print Media, Social Networking, Telecoms - and Overlapping Deployment in Education and Training, Entertainment, Healthcare and Assisted Living, Culture Multiplay degree environment From game to TV or film: Final Fantasy, Tomb Raider From film to TV and games: Indiana Jones TV to film, game, online: Simpsons, Lost Online to TV: Sofia s Diary (Bebo to Fiver)

20 Interaction and decision making - Casablanca (1942) Script half completed when production began Dialog for some scenes written while shooting in progress

21 Our way of analyzing semantics Based always on data ie filmscript text We discern story semantics arising out of the orientation of narrative This is based on the web of interrelationships First now I will examine caesuras and breakpoints in the flow of narrative

22 Caesuras and Break-Points Hierarchy represents change - caesuras, breakpoints - very well We use a contiguity (ie scene sequence) constrained complete link agglomerative hierarchical clustering algorithm This is based on a Euclidean embedding, resulting from all words and scenes being projected into a Euclidean space - Correspondence Analysis does that In the following we use subscenes of a scene and also hierarchically cluster orientations of vectors

23 Analysis of Casablanca s Mid-Act Climax, Scene 43 subdivided into 11 beats (subscenes) McKee divides this scene, relating to Ilsa and Rick seeking black market exit visas, into 11 beats Beat 1 is Rick finding Ilsa in the market Beats 2, 3, 4 are rejections of him by Ilsa Beats 5, 6 express rapprochement by both Beat 7 is guilt-tripping by each in turn Beat 8 is a jump in content: Ilsa says she will leave Casablanca soon In beat 9, Rick calls her a coward, and Ilsa calls him a fool In beat 10, Rick propositions her In beat 11, the climax, all goes to rack and ruin: Ilsa says she was married to Laszlo all along Rick is stunned

24 MPrincipal plane of 11 beats in scene 43 m-10 M20 m-20 M-20 M-15 M-10 M-05 M00 M05 M10 M15 MFactor 1, 2, 126% 122% of inertia Principal plane of 11 beats in scene Factor 2, 122% of inertia Factor 1, 126% of inertia 210 words used in these 11 beats or subscenes

25 MPrincipal plane of 11 beats in scene 43 m-10 M20 m-20 M-20 M-15 M-10 M-05 M00 M05 M10 M15 MFactor 1, 2, 126% 122% of inertia Principal plane of 11 beats in scene Factor 2, 122% of inertia Factor 1, 126% of inertia

26 MPrincipal plane of 11 beats in scene 43 m-10 M20 m-20 M-20 M-15 M-10 M-05 M00 M05 M10 M15 MFactor 1, 2, 126% 122% of inertia Principal plane of 11 beats in scene 43 Factor 2, 122% of inertia Repulsion 00 Attraction Beat 8: Ilsa to leave Casablanca! Beat 11: Ilsa married to Laszlo all along! Factor 1, 126% of inertia

27 MHierarchical clustering of 11 beats, using their orientations m00 M00 M02 M04 M06 M08 M10 1M 2M 3M 4M 5M 6M 7M 8M 9M Hierarchical clustering of 11 beats, using their orientations Full dimensionality analysis Note caesura in moving from beat 7 to 8, and back to 9 Less so in moving from 4 to 5 but still quite pronounced

28 McKee, Methuen, 1999 Casablance is based on a range of miniplots McKee: its composition is virtually perfect Text is the sensory surface of the underlying semantics

29 Style analysis of scene 43 based on McKee Monte Carlo tested against 999 uniformly randomized sets of the beats In the great majority of cases (against 83% and more of the randomized alternatives) we find the style in scene 43 to be characterized by: small variability of movement from one beat to the next greater tempo of beats high mean rhythm

30 Analysis of style and structure, using 999 randomizations, of sequence of 77 scenes In all, 9 attributes were used Entire Casablanca plot is well-characterized by the variability of movement from one scene to the next (attribute 2) Variability of movement from one beat to the next is smaller than randomized alternatives in 82% of cases Similarity of orientation from one scene to the next (attribute 3) is very tight, ie smaller than randomized alternatives We found this to hold in 95% of cases The variability of orientations (attribute 4) was also tighter, in 82% of cases Attribute 6, the mean of ups and downs of tempos is also revealing In 96% of cases, it was smaller in the real Casablanca, as opposed to the randomized alternatives This points to the balance of up and down movement in pace

31 Work of J Eliashberg, Wharton, U Penn Use features characterizing scripts to predict boxoffice success

32 Other Features We have various practical ways of determining the texture of text So we can match genre and style We could have highly multi-outcome games, for example, created in a crowd sourcing way - and ensure homogeneity and cohesion

33 Looked at: characterization of narrative, using script --- analysis Next: narrativization, or assembling narrative --- synthesis Example of multiple versions of the same storyline (cf creating a game from the story) Collaborative, distributed creation of narrative

34 Text Synthesis Aristotle s Poetics (c 350 BC) Outlines and episodization - Stories should first be set out in universal terms on that basis, one should then turn the story into episodes and elaborate it reasoning is the speech which the agents use to argue a case or put forwards an opinion

35 Project TooManyCooks: Applying Software Design Principles to Fiction Writing Joe Reddington (Comp Sci), Doug Cowie (English) and myself!!!!

36 Support environment for collaborative, distributed creating of narrative Pinpointing anomalous sections Assessing homogeneity of style over successive iterations of the work Scenario experimentation and planning This includes condensing parts, or elaborating Similarity of structure relative to best practice in chosen genre

37 From testimonial, The Writers Desk (London) Overall we at The Writers Desk have found this analysis to be of a great assistance to our editorial work in all aspects of the editorial process From helping to identify target markets, to highlighting problem areas within the novel, to illustrating writing style, these analysis reports have become a rather important part of our working operations

38 Segmentation and Nodal Points in Narrative We used 9 versions of a Scottish lowlands ballad, Lady Maisry Ballad relates to woman who became pregnant by an English lord, and as a result is burnt to death by her own family Interest in nodal points in narrative include, beyond literature, the importance of such locations in the narrative for introducing change, eg to create derivative or parallel versions, or to support user interaction in the context of the storyline Initial 4 stanzas (out of 27) from ballad 65B 65B1! IN came her sister,! Stepping on the floor;! Says, It s telling me, my sister Janet,! That you re become a whore 65B2! A whore, sister, a whore, sister?! That s what I ll never be;! I m no so great a whore, sister,! As liars does on me lee 65B3! In came her brother,! Stepping on the floor;! Says, It s telling me, my sister Janet,! That you re become a whore 65B4! A whore, brother, a whore, brother?! A whore I ll never be;! I m no so bad a woman, brother,! As liears does on me lee

39 Opening stanzas of 9 variants All 219 stanzas as dots B C H K A E G D 4 3 F Factor Factor Factor 2 Last stanzas of 9 variants All 219 stanzas as dots F H A K E G D C Factor 1 B 2

40 65A: Lady Maisry 65B: Lady Maisry 65C: Lady Maisry G: Lady Maisry 65H: Lady Maisry 65K: Lady Maisry

41 To Read Further F Murtagh, Correspondence Analysis and Data Coding with R and Java, Chapman and Hall/CRC Press, 2005 See chapter 5 on text analysis Software at wwwcorrespondancesinfo F Murtagh, A Ganz and J Reddington, New methods of analysis of narrative and semantics in support of interactivity, Entertainment Computing, advance access online, 2010 F Murtagh, A Ganz, S McKie, J Mothe and K Englmeier, "Tag clouds for displaying semantics: The case of filmscripts", Information Visualization Journal, 9, , 2010 F Murtagh, "The Correspondence Analysis platform for uncovering deep structure in data and information", Sixth Boole Lecture, Computer Journal, 53, , 2010 F Murtagh, A Ganz and S McKie, "The structure of narrative: the case of film scripts", Pattern Recognition, 42, , 2009 (See discussion in Z Merali, "Here's looking at you, kid Software promises to identify blockbuster scripts", Nature, 453, p 708, 4 June 2008) T Cooks, The Shadow Hours, 2009, 231 pp T Cooks, The Delivery, MPG Books Group, 2009, 213 pp And more See

42