Grammar Rule Extraction and Transfer in Buildings

Transcription

1 Grammar Rule Extraction and Transfer in Buildings Asad Khalid Ismail Lahore University of Management Sciences Sector U, DHA, Lahore Zuha Agha Lahore University of Management Sciences Sector U, DHA, Lahore Abstract Inspired by Muller s Procedural modeling approach, we wanted to extract certain definitive architectural features from buildings and transfer them to another building structure. After various attempts at extraction, our final approach focused on automatic layout extraction for the target structure and given grammar parameters in a rule file, the application and generation of a building. We also focused on transference of features like windows from one architectural structure to the other and used Esri City Engine for the grammar based building generation. 1. Introduction Inspired from the Muller et al, Procedural modeling of buildings [1], the aim of this project was to generate a variety of architecture models by defining the geometric and visual detail of buildings as a set of rules invoked on some primitives according to stochastic shape grammar and shape interactions. These rules encapsulate the layout, shape, orientation and visual details of the buildings. Once defined, these rules can be reused and modified to create variations in models procedurally. The construction of complex models can no longer be achieved by split grammar alone since several factors have to be taken into account such as variations in mass configurations, occlusion queries to avoid placing windows and doors at the intersection of building models and testing snap lines, just to name a few. This paper attempted to reach an optimal solution to such issues to achieve a greater level of detail and precision in building these complex models. Our ideal vision for the project included extracting the feature information from any given 3D structure and after having learnt all this feature set, to generate another, different, building using grammar primitives and features from the first one. Thus the resulting output would be a combination of both buildings, built on the layout and the structure of the original building, but using features and architectural elements in the form of grammar elements from the other one. Thus, for example, our ideal envisioning would be an SSE building structure modeled with the architectural elements and features of Masjid Wazir Khan, for example. Thus, in a way, we could say that if the SSE was designed by the architect of Masjid Wazir Khan, this would be the final output encapsulating all the stylistic and architectural elements of the Masjid applied to the SSE building. 2. Literature Review We first approached the problem having only a high level view of both procedural modeling and grammar through the paper Procedural Modeling of Buildings, Muller et al. We went on to deeply study this paper and also another paper by Muller called Image Based procedural modeling of facades This paper dealt with extracting 3D models from single frontal façade images and use a procedural modeling approach to derive a set of grammar rules with which to define the building. The main hierarchal relationship between first dealing with floors, then subdividing into tiles, then window extraction from 1

2 tiles was an interesting thinking paradigm which we later employed in our work. The hierarchal relationship is given below [2]: capable of both extracting object geometry, as well as annotating its type as shown in the pipeline in the figure below: The main problem with not exactly implementing this approach was the fact that they had carried out machine learning, probabilistic approach to calculate and sub-divide the building structure, not a geometry based argument for subdivision that we wanted. We then went through the paper Automatic extraction of building features from image data: how far are we? [3] a survey paper that gives an overview of the past and current work done on automatic extraction of building features, especially windows, from image data. It first defines an ideal pipeline to do so from photo and range images and then tries to analyze how far we are from realistically attaining this goal. A practical difficulty in the problem of automatic extraction is obtaining building features input. The desired building features have to be given as typed objects. That is, the type and geometry of a particular object are known. This requires that the object recognition algorithm is But the scenario is not that ideal as we do not have prior knowledge of the type of the objects in a scene, and a range image treats an entire scene as an entity; so, it is difficult to automatically determine which subset of points belongs to which object. Following this, we directed our attention to another recent research paper called High-Level Bottom-Up Cues for Top-Down Parsing of Facade Images [4] which discusses the idea of façade parsing to detect repetitive patterns by construction of a derivation tree corresponding to a given configuration of terminal shapes. The root of the tree is the axiom S and all the nodes correspond to basic shapes, with terminal basic shapes at the leaves. Tree parsing can be performed in either a top-down or a bottom-up fashion. But the parsing and repetitive pattern detection methods introduced in this paper involved probabilistic methods, Random Forest classifier and Q-learning algorithms rather than a geometric approach so we decided to employ some other strategy. 2

3 From there on, we also went through a series of papers describing multiple approaches for extraction of features from buildings. A notable paper which used edge detection histogram values to isolate the window was Extraction and Integration of Window in a 3D Building Model from Ground View images [5]. This paper gave us the idea to try out k-means clustering on a frontal facade image, which will be discussed later. Figure 1 : Own Result. City Engine Grammar Rule File 3. Experimentation We carried out various approaches to try to extract both the layout and the primitive features from both 2D frontal facades as well as 3D models of buildings City Engine and Grammar rules We first discovered that Muller et al s procedural modeling paper had been adapted into commercial procedural modeling and grammar rule generation software called ESRI City Engine. We learnt various grammar rule file generation and how to generate a generic building given any layout, given below [6]: Figure 2 : Own output. Grammar Rule Output 3

4 The next three images show how grammar rules are applied to various lots and how the same grammar rule file can generate various buildings: textures/colors quite different from their surrounding elements. Thus, if we could somehow partition the image into two main clusters, all the windows would emerge in one of them and then after applying basic image processing operations like erosion and dilation, we could use corner and edge detectors to find a bounding box that identified the window primitive. Some of the results of our k-means approach are given below: Figure 3a. b. c. Own output. Grammar rule outputs 3.2. K-Means Clustering Approach We initially tried to extract primitives like windows and doors from 2 dimensional frontal façade images. The initial implementation approach we tried was based on k-means clustering, and operated under the assumption that windows had Figure 4 : Own Output. k-means clustering 4

5 Figure 6 : Own Output. k-means clustering Figure 5 : Own Output. k-means clustering 3.3. Parsing 3D models and layout extraction After implementing the k-means clustering approach, we decided to move on to 3D models of buildings and to extract relevant features from them directly. The main motivation for this was extracting the building layout from a 3D model in order to apply a particular grammar rule file to it, so that we would be able to generate a building on the current layout using a completely different architechtural style. With this decision, the initial challenge we faced was parsing a 3D model structure in a data processing format. We wanted to extract vertices and faces information from the 3D model and effectively capture this data in MATLAB for further procession. For this, we needed to understand the base structure of popular 3D model formats like.3ds and.obj formats and thus parse these structures to gather the required information. After interpreting both 3ds and obj formats, we decided to parse the obj file format and read in the vertices and faces in MATLAB From there on, our initial approach was based on extracting base vertices for the 3D model for the minimum z-value, ie, the layout vertices on this plane. Our initial approach in joining these vertices was as follows: We first calculated the centroid position for all these 5

6 points. From the centroid, we drew a vector to all these points and the angle traversed from this vector in a clock-wise angle was used to join the points. This worked well for simple polygonal structures but didn t work well for cases where there were sudden extruding edges from the layout. A result from this approach is given below: Figure 7 : Own output. Layout generation 3.4. Automatic extraction of layout We discovered that the above approach of joining vertices in order of ascending angles from their centre gives the correct results for some simple polygonal layouts but fails for most concave polygons. Figure 8 : Own output. Layout generation So, to solve this problem we attempted to find the layout faces from the file. We used the vertices extracted for ground floor, at minimum value of y, and store them in a matrix minvy. Then using minvy, we manually calculated the face index values to generate a.obj file of the layout. Once that was done, the file generated was imported in meshlab for display. We tried this to learn how faces are defined in.obj files, so that information about faces could be extracted from our parsed input 3d file. In a.obj file format, faces are defined using lists of vertex, texture and normal indices. A valid vertex index matches the corresponding vertex elements of a previously defined vertex list. f v1 v3 v5 6

7 There are several other ways to define faces including face/ texture/ normal vertices and face//normal vertices as given by the two examples given below. f v1/vt1/vn1 v2/vt2/vn2 v3/vt3/vn3.. f v1//vn1 v2//vn2 v3//vn3... Once the vertices needed to form the layout are known, every vertex index is stored and the list of faces in the input.obj file is parsed to find faces formed by indices of layout coordinates only. If no other vertex index, other than the layout vertices indices is used to define the face in the file, it implies that face belongs to the layout polygon. That face gives us information about how to join those vertices, so we need not apply the angle approach anymore to find the order of vertices connected. All such faces are stored in another matrix and their new indices are re-computed. The layout faces and vertices information obtained is then written into a file in obj format. The output is then displayed in meshlab to ensure if it correct. In this way, we came up with an automated way to find the layout of a 3d model. Figure 9 : Own Output. Layout generation 4. Layout Transfer We started off with learning how grammar works in City engine and rules for writing grammar files. We played around with grammar rules to change the height, number of floors, splitting of façade into tiles and layout of a building to any random polygon. To transfer the extracted layout from a 3d model onto a simple building in city engine, we imported the extracted layout of 3d building in city engine and transformed the static model to a shape. The shape was then assigned as a Lot and a rule grammar file of the simple input building was applied to that lot. The layout of the simple building is now transformed to that of the extracted model. Even though we did not get to extract architectural elements, we still attempted to transfer windows and doors of another style along with their texture given their obj files taken up from an online source. 7

8 5. Conclusion and Future Work Layout transfer is the first milestone we have achieved towards our ultimate objective to design an application that can transfer complete architecture style and texture of one building onto another. This is not an easy challenge as it not only involves the difficult task of detection and extraction of all kinds of relevant and representative architectural elements but also incorporating these architectural elements into the grammar of the building to be transformed. It is not necessary that the architectural primitives found in one building model are of the same type as those found in another, which means transfer of elements has to be carried out in such a way that the building generated is plausible. All of these are important factors to be taken into account to pursue future work in this area. 6. Software Used Esri City Engine, MeshLab and MATLAB. Figure 10a. b. c. d. Own output. Grammar generation using City Engine 7. References [1] Müller, Pascal, et al. Procedural modeling of buildings. Vol. 25. No. 3. ACM, [2] Müller, Pascal, et al. "Image-based procedural modeling of facades." ACM Transactions on Graphics 26.3 (2007): 85. [3] Yue, Kui and Krishnamurti, Ramesh, "Automatic extraction of building features from image data: how far are we?" (2009). School of Architecture. Paper 48. [4] Ok, David, et al. "High-Level Bottom-Up Cues for Top-Down Parsing of Facade Images." 3D Imaging, Modeling, Processing, Visualization and Transmission (3DIMPVT), 2012 Second International Conference on. IEEE, [5] Lee, Sung Chun, and Ram Nevatia. "Extraction and integration of window in a 3D building model from ground view images." Computer Vision and Pattern Recognition, CVPR Proceedings of the 2004 IEEE Computer Society Conference on. Vol. 2. IEEE,