Semantizing Complex 3D Scenes using Constrained Attribute Grammars

Transcription

1 Semantizing Complex 3D Scenes using Constrained Attribute Grammars A. Boulch, S. Houllier, R. Marlet and O. Tournaire slight variant of the material presented at SGP 2013

2 Motivation Semantizing complex objects in 3D scenes Bare geometry Semantized geometry 2

3 Motivation Semantizing complex objects in 3D scenes Bare geometry 2 Semantized geometry (rendered)

4 Motivation Semantizing complex objects in 3D scenes Building industry for the renovation market point cloud! building model Semantized geometry 2

5 Motivation Semantizing complex objects in 3D scenes Building industry for the renovation market point cloud! building model for architects building sketch! rendering Semantized geometry (rendered) 2

6 Motivation Semantizing complex objects in 3D scenes Building industry for the renovation market for architects point cloud! building model building sketch! rendering Game industry for graphic designers basic level design! rendering Semantized geometry (rendered) 2

7 Motivation Semantizing complex objects in 3D scenes Building industry for the renovation market point cloud! building model for architects building sketch! rendering Game industry for graphic designers basic level design! rendering Object mining in shape databases for semantic queries 3D object! semantic labeling Semantized geometry 2

8 Outline Constrained attribute grammars Scene interpretation Bottom-up parsing Experiments 3

9 Outline Constrained attribute grammars Scene interpretation Bottom-up parsing Experiments 4

10 Grammars to express hierarchical decomposition Sentence Building Subject Verb Complement Facade Roof Floor Article Adjective Noun Window Wall The super computer rules Discovery One 4

11 Complex, non-hierarchic relations between components Sentence Building Subject Verb Complement Facade Roof Floor Article Adjective Noun Window Wall adj The super computer rules Discovery One agreement (singular) 5

12 Constrained attribute grammars G =(N, T, P, S) N: nonterminals($ complex forms) e.g., window, wall T :terminals($ geometric primitives) e.g., polygon, cylinder P: production rules($ hierarchical decomposition and constraints) S: startsymbols($ root shapes) e.g., building 6

13 Basic rules Decomposition of a complex object y of type Y into its constituents x i of type X i : Y y! X 1 x 1,...,X n x n Example: step s! riser r, tread t tread Rule application: riser top-down view: y decomposes into x 1,...,x n bottom-up view: given some x 1,...,x n,createanewobjecty 7

14 Basic rules Decomposition of a complex object y of type Y into its constituents x i of type X i : Y y! X 1 x 1,...,X n x n Example: step s! riser r, tread t tread Rule application: riser top-down view: y decomposes into x 1,...,x n bottom-up view: given some x 1,...,x n,createanewobjecty riser? tread? 7

15 Constraints Conditional rule application (conjonction of predicates): Y y! X 1 x 1, X 2 x 2,... h cstr 1 (x 1 ), cstr 2 (x 1, x 2 ),...i Example: tread riser r! polygon p h vertical(p) i tread t! polygon p h horizontal(p) i step s! riser r, tread t h edgeadj(r, t), above(t, r) i riser 8

16 Attributes Features attached to each grammar element: at creation time (primitives) at rule application (synthetized attributes) Examples: length, width bounding box... r.width r.length 9

17 Predicates Predicates on grammar elements: adj, edgeadj orthogonal, parallel vertical, horizontal... t.length Predicates on attributes:, >, apple,... r.length =, 6=... Example: r.length == t.length 10

18 Collections of similar elements Grouping elements via recursion (Y as set of X s): Y y! Xx Y y! Xx, Y y 2 Grouping elements via specific collection operators: Y y! coll(x ) xs Useful operators: maximal collections... maxconn: maximal set of connected components maxseq: maximal sequence Example: stairway sw! maxseq(step, adjedge) ss 11

19 Collections of similar elements Grouping elements via recursion (Y as set of X s): Y y! Xx Y y! Xx, Y y 2 Grouping elements via specific collection operators: Y y! coll(x ) xs Useful operators: maximal collections maxconn: maximal set of connected components maxseq: maximal sequence... [see complete stairway grammar] Example: stairway sw! maxseq(step, adjedge) ss 11

20 Outline Constrained attribute grammars Scene interpretation Bottom-up parsing Experiments 12

21 Scene interpretation: parse tree Tree representation of a grammatical analysis of the scene: leaves: terminals representing primitives non-leaf nodes: instanciations of grammar rules 12

22 Scene interpretations: parse forest Set of parse trees with systematic sharing (DAG) Compact representation of all possible interpretations 13

23 Ambiguity and the exclusivity constraint Example (assuming no height ordering constraint): step 1 step s! riser r, tread t h edgeadj(r, t) i step 2 Stairway Exclusivity constraint: at most 1 occurrence of a grammar element per interpretation Step 1 Step 2 Riser 2 Tread 1 Riser 1 14

24 Outline Constrained attribute grammars Scene interpretation Bottom-up parsing Experiments 15

25 Parse forest computation Bottom-up parsing: construction of the parse forest from leaves (terminals) to roots (start symbols) create one terminal for each primitives iteratively create new grammar elements from existing ones e.g., given grammar rule step s! riser r, tread t given existing instances riser r 23, tread t 18 create new instance step s 5 merge identical trees on the fly stop iterating when no rule applies 15

26 Rule application order Simple rules: use any order Maximal operators: wait for all subelements to ensure maximality maxseq 16

27 Rule application order Simple rules: use any order Maximal operators: wait for all subelements to ensure maximality maxseq 16

28 Rule application order Simple rules: use any order Maximal operators: wait for all subelements to ensure maximality reverse topological sort of nonterminal dependency graph maxseq 16

29 Mastering the combinatorial explosion Usual drawback of bottom-up analysis: combinatorial explosion all trees all sets, all sequences,... all combinations Our solution: tree sharing: construction of a parse forest (exp.! lin.) maximal operators, with efficient implementation (> exp.! polyn.) constraint propagation: predicate ordering ) early pruning 17

30 Maximal operators maxseq allseq vs 18

31 Maximal operators maxseq max(allseq) vs 18

32 Constraint propagation as predicate ordering Asimple2Dexample pair p! seg s 1, seg s 2, h orthogonal(s 1, s 2 ), adj(s 1, s 2 ), vertical(s 1 ) i 19

33 Constraint propagation as predicate ordering Example of poor ordering pair p! seg s 1, seg s 2, h orthogonal(s 1, s 2 ), adj(s 1, s 2 ), vertical(s 1 ) i 1. orthogonality Complexity: O(#seg 2 ) constraints to test 20

34 Constraint propagation as predicate ordering Example of poor ordering pair p! seg s 1, seg s 2, h orthogonal(s 1, s 2 ), adj(s 1, s 2 ), vertical(s 1 ) i 1. orthogonality Complexity: O(#seg 2 ) satisfied constraints 20

35 Constraint propagation as predicate ordering Example of poor ordering pair p! seg s 1, seg s 2, h orthogonal(s 1, s 2 ), adj(s 1, s 2 ), vertical(s 1 ) i 1. orthogonality 2. adjacency Complexity: O(#seg 2 + #seg maxdeg) = O(#seg 2 ) constraints to test 20

36 Constraint propagation as predicate ordering Example of poor ordering pair p! seg s 1, seg s 2, h orthogonal(s 1, s 2 ), adj(s 1, s 2 ), vertical(s 1 ) i 1. orthogonality 2. adjacency Complexity: O(#seg 2 + #seg maxdeg) = O(#seg 2 ) satisfied constraints 20

37 Constraint propagation as predicate ordering Example of poor ordering pair p! seg s 1, seg s 2, h orthogonal(s 1, s 2 ), adj(s 1, s 2 ), vertical(s 1 ) i 1. orthogonality 2. adjacency 3. verticality Complexity: O(#seg 2 + #seg maxdeg + #seg) = O(#seg 2 ) constraints to test 20

38 Constraint propagation as predicate ordering Example of poor ordering pair p! seg s 1, seg s 2, h orthogonal(s 1, s 2 ), adj(s 1, s 2 ), vertical(s 1 ) i 1. orthogonality 2. adjacency 3. verticality Complexity: O(#seg 2 + #seg maxdeg + #seg) = O(#seg 2 ) satisfied constraints 20

39 Constraint propagation as predicate ordering Example of good ordering pair p! seg s 1, seg s 2, h orthogonal(s 1, s 2 ), adj(s 1, s 2 ), vertical(s 1 ) i 1. verticality Complexity : O(#seg) constraints to test 21

40 Constraint propagation as predicate ordering Example of good ordering pair p! seg s 1, seg s 2, h orthogonal(s 1, s 2 ), adj(s 1, s 2 ), vertical(s 1 ) i 1. verticality Complexity : O(#seg) satisfied constraints 21

41 Constraint propagation as predicate ordering Example of good ordering pair p! seg s 1, seg s 2, h orthogonal(s 1, s 2 ), adj(s 1, s 2 ), vertical(s 1 ) i 1. verticality 2. adjacency Complexity : O(#seg + #seg maxdeg) = O(#seg maxdeg) constraints to test 21

42 Constraint propagation as predicate ordering Example of good ordering pair p! seg s 1, seg s 2, h orthogonal(s 1, s 2 ), adj(s 1, s 2 ), vertical(s 1 ) i 1. verticality 2. adjacency 3. orthogonality Complexity : O(#seg + #seg maxdeg + #seg maxdeg) = O(#seg maxdeg) O(#seg 2 ) satisfied constraints 21

43 Constraint propagation as predicate ordering 1. Unary predicates =constraintsimplyingonly1element:complexityo(#element) Example: riser r! polygon p h vertical(p) i 22

44 Constraint propagation as predicate ordering 1. Unary predicates 2. Invertible predicates that are partially instantiated =constraintswithsmallcardinalitywhensomeargumentsarefixed Example: step s! riser r, tread t h edgeadj(r, t) i 22

45 Constraint propagation as predicate ordering 1. Unary predicates 2. Invertible predicates that are partially instantiated 3. General predicates =remainingrelations Example: step s! riser r, tread t h orthogonal(r, t) i 22

46 Outline Constrained attribute grammars Scene interpretation Bottom-up parsing Experiments 23

47 Semantization pipeline Grammar CAD Model (triangle soup) Polygon extraction Parsing Semantized model Point cloud Primitive detecion 23

48 CAD models Preprocessing Region growing over triangles for polygon creation Computation of exact and approximate adjacency graphs 24

49 CAD models Detection of stairs (more examples in supplem. material) 25

50 CAD models Detection of walls, roofs and openings 26 (more examples in supplem. material)

51 Real data: photogrammetry Preprocessing (point cloud) Problems: clustering using RANSAC (or region growing) polygons bounded by alpha shapes missing primitives false primitives wrong adjacencies Solution: use of a relaxed grammar 27 looser bounds 1-2 missing items OK 22 openings out of 31

52 Quasi-real data: simulated LIDAR Planes by region growing in depth image Polygons as oriented bounding rectangles Adjacency based on pixels in depth image 28

53 Size and parsing time (CAD models) #of #of Parsing time (s) Name triangles polygons stairs openings LcG LcA LcC LcD LcF

54 Precision and recall (%, CAD models) # of # of Stairs Openings Name stairs steps Prec. Rec. # Prec. Rec. LcG LcA LcC LcD LcF

55 Future work Principled way to deal with partial or missing primitives Exploitation of occlusion/visibility information Scoring of interpretations: pick best tree(s) 31

56 Conclusion Constrained attribute grammars: appropriate to semantize complex objects high-level specification language being expert is enough, computer scientist not required efficient even on large models This work: well-delimited first step: perfect data extensions required for incomplete/noisy data On the web sites.google.com/site/boulchalexandre/ 32