1 Interactive Computational Intelligence: A Framework and Applications
Agenda Overview Methodology IGA Knowledge-based encoding Partial evaluation based on clustering Direct manipulation of evolution Applications Media retrieval Media design Concluding remarks 2
Overview Interactive Computational Intelligence Interaction/Evolution Human Computer Interface (Emotion) User Modeling AI (Soft Computing) (Efficiency) FL GA 3 NN
Interactive Evolutionary Computation (IEC) Overview Difficulty of optimization problem in interactive system Outputs must be subjectively evaluated (graphics or music) Definition Evolutionary computation that optimizes systems based on subjective human evaluation Fitness function is replaced by a human user Target system f(p1,p2,,pn) System output My goal is Interactive EC Subjective evaluation From Takagi (2001) 4
Overview Technical Problems in IEC Human fatigue problem Common to all human-machine interaction systems IEC Acceleration of EC convergence with a small population size and a few generation numbers Usually 10 or 20 search generations Limitation of the individuals simultaneously displayed on a monitor Limitation of the human capacity to memorize the timesequentially displayed individuals Requirement to minimize human fatigue I m tired 5
Overview Framework Direct Manipulation Media Retrieval & Design Media Retrieval & Design IEC IEC Partial Evaluation Domain Knowledge Conventional Proposed 6
Methodology Interactive Genetic Algorithm Initial Population GA Fitness Function IGA User Selection Crossover / Mutation Fitness Evaluation Population Reproduction 7
Methodology Knowledge-based Encoding New Search Space Effective encoding? Domain Knowledge Removing impractical solutions Using partially known information about the final solution Focusing on a specific search space Encoding in a modular manner Search Space 8
Partial Evaluation Based on Clustering Methodology Issues Clustering algorithm selection, indirect fitness allocation strategy 9
Methodology Direct Manipulation of Evolution It is very similar to the final solution but a bit different. IGA Direct Manipulation Proof of usefulness of the direct manipulation N-K Landscape 10
Applications Overview Humanized Media Retrieval & Design Media Retrieval Media Design Image Retrieval Video Retrieval Music Retrieval Fashion Design Flower Design Intrusion Pattern Design 11
Content-based Image Retrieval Applications: Image Retrieval Keywords-based method Much time and labor to construct indexes in a large databases Performance decrease when index constructor and user have different point of view Inherent difficulty to describe image as a keyword Content-based method Image contents as a set of features extracted from image Specifying queries using the features 12
Applications: Image Retrieval Motivation System for content-based image retrieval QBIC (IBM, 1993) QVE (Hirata and Kato, 1993) Chabot (Berkeley, 1994) Needs for image retrieval based on human intuition Difficult to get perfect expression by queries Lack of expression capability 13
System Structure Applications: Image Retrieval 14
Chromosome Construction Applications: Image Retrieval R G B 0 1 2 3 49 50 Original image Transformed image Chromosome 15
Convergence Test Applications: Image Retrieval The case of gloomy image retrieval Initial population The 8th population 16
Applications: Image Retrieval Subjective Test by Sheffe s Method Confidence interval 95% 99% 17
Applications: Video Retrieval Motivation Evolution of computing technology (Huge computing storage, networking) Necessary for management of video data transfer, processing and retrieval Change of view point Query by keyword Query by content Difficult to represent human s semantic information Query by semantics Emotion-based retrieval 18
Hierarchical Structure of Video Applications: Video Retrieval 19
System Architecture Applications: Video Retrieval 20
Chromosome Construction Applications: Video Retrieval 21
System Interface Applications: Video Retrieval 22
User s Satisfaction Applications: Video Retrieval 23
Emotional Music Retrieval Applications: Music Retrieval Music information retrieval Immature field Query by singing Query by content Uploading pre-recording humming via web browser Typing the string to represent the melody contour Preprocessing Query by humming transforms user s humming into symbolized representation Conventional music retrieval system Fast and effective extraction of musical patterns and transformation of user s humming into systematic notes Problem : User cannot remember some parts of melody 24
Power Spectrum Analysis Applications: Music Retrieval 1/f )] log 10 [ ( f S Z a b a : classic b : jazz c : rock d : news channel c d ( log 10 f ) 25
System Architecture Applications: Music Retrieval 3Retrieved music IGA : Music Database 4 User evaluates the music retrieved 5 Generation of new offspring 1Query 2Retrieval of music by the query 26
Chromosome Construction Applications: Music Retrieval 27
Convergence Test Applications: Music Retrieval Distance 60 50 40 30 20 10 1 m=0.5 c=0.5 2 m=0.2 c=0.5 3 m=0.5 c=0.2 4 m=0.2 c=0.2 0 0 1 2 3 4 5 6 7 8 9 10 generation 28
Convergence Test (2) Applications: Music Retrieval 29
Change of Consumer Economy Applications: Fashion Design Manufacturer Oriented Before the Industrial Revolution : Customers have few choices on buying their clothes After the Industrial Revolution : Customers can make their choices with very large variety Consumer Oriented Near Future : Customers can order and get clothes of their favorite design 30
Applications: Fashion Design Need for Interaction-based System Almost all consumers are non-professional at design To make designers contact all consumers is not effective Need for the design system that can be used by non-professionals 31
Applications: Fashion Design Fashion Design Definition To make a choice within various styles that clothes can take Three shape part of fashion design Silhouette Detail Trimming 32
System Architecture Applications: Fashion Design OpenGL Program Models of each part Combine Display Decode Interactive Genetic Algorithm GA operation Reproduce User Fitness 33
Knowledge-based Encoding Applications: Fashion Design A B C D E F Total 23 bits A : Neck and body style(34) B : Color(8) E : Skirt and waistline style(9) F : Color(8) C : Arm and sleeve style(11) D : Color(8) Search space size =34*8*11*8*9*8 =1,880,064 34
Direct Manipulation Interface Applications: Fashion Design 35
Applications: Fashion Design Fitness Changes for Each Encoding Scheme 70 60 Average Fitness 50 40 30 20 10 0 Knowledge-based Encoding Sequential Encoding 1 2 3 4 5 6 7 8 9 10 Generation 36
Hypercube Analysis Applications: Fashion Design 1110 65 20 0110 mutton 0111 pagoda 60 75 1111 No mapped design 1110 No mapped design 65 20 0110 mutton 0111 pagoda 60 75 1111 No mapped design No mapped design 0010 flare 80 100 1010 tucked 0010 flare 80 100 1010 tucked 0011 french 1011 no sleeves 0011 french DM 1011 no sleeves 0001 china 85 35 0101 melon 80 50 1101 No mapped design 1100 No mapped design 0001 china 85 35 0101 melon 80 50 1101 No mapped design 1100 No mapped design 45 0100 mandarin 1001 tight 15 45 0100 mandarin 1001 tight 15 0000 bishop 1000 poncho 0000 bishop 1000 poncho One-Mutant Neighbors using Genetic Operator Shortest Path Using DM Method 37
Applications: Flower Design Motivation Proposed Method Representation of knowledge-based genotype using the structure of real flower Automatic generation of creatures Process Directed graph: Define the genotype of flower structure IGA: Evaluation of created individuals and automatic creation Math Engine: Representation of phenotype Objective Creation of character morphology similar to reality Find optimal solution in small solution space using knowledgebased genotype representation such as directed graph Automatic creation of character using IGA 38
Applications: Flower Design Related Works Tree, flower and feather modeling L-System Box-based artificial characters (Karl Sims) evolution Graph model representation Golem Robotic form and controller evolution Graph-based data structure 39
Overall Procedure Applications: Flower Design Create initial edges between nodes Set dimension and color of each part Set parameters of parts and edges Create 3D world in simulation Create rigid parts in genotype Perform 3D rendering Evaluate fitness of each individual No Satisfied individual? Yes Selection Stop Crossover/Mutation Create individuals of next generation 40
Chromosome Construction Applications: Flower Design 41
Convergence Test Applications: Flower Design 10 Fitness value 8 6 4 2 0 1 2 3 4 5 6 7 8 9 10 Generation Average fitness value Best fitness value 42
Analysis Using Sheffe s Method Applications: Flower Design 99% 95% -3-2 -1 0 1 2 3 43
Motivation Applications: Intrusion Pattern Design Various intrusion detection system For evaluating, IDS uses known intrusions Possibility of having different patterns within identical intrusion type Difficult to detect transformed intrusions Problems For better evaluation, intrusion patterns are needed more Difficult to generate all possible proper intrusion patterns manually Automatic generation is needed 44
Motivation (2) Applications: Intrusion Pattern Design Evolutionary pattern generation Needs a variety of intrusion patterns It is possible to generate new patterns by combining primitives Generation of transformed intrusion patterns using simple genetic algorithm IGA-based intrusion pattern generation Difficult to estimate the fitness of patterns automatically Using interactive genetic algorithm for the estimation 45
Applications: Intrusion Pattern Design Method Intrusive behavior can be divided into 5 steps Proposed by DARPA Possible path of U2R attack TRANSPORT ENCODING EXECUTION ACTIONS CLEAN UP web download archive shell variables change file permissions editor octal character shell script display files remove files ftp simple encryption encrypted shell interaction delete files restore permissions mail character stuffing generate chaft output in bg transfer files edit audit logs floopy encoding delayted execution alter files 46
System Architecture Applications: Intrusion Pattern Design Known Intrusion Patterns Initial Population IGA GA Fitness Evaluation Intrusion Patterns 000100... 001100... Generate Audit Data Exploit Code 47
Concluding Remarks Humanized computing framework using IEC Knowledge-based encoding Partial evaluation based on clustering Direct manipulation of evolution Applications in media retrieval & design Image, video and music retrieval Fashion, flower and intrusion pattern design 48