Identifying Clones in the Linux Kernel DRAFT

Transcription

1 Identifying Clones in the Linux Kernel. Casazza,. Antoniol, U. Villano, E. Merlo, M. Di Penta, gec, dipenta, University of Sannio, Faculty of Engineering - Benevento, Italy University of Naples Federico II, DIS - Naples, Italy École Polytechnique de Montréal - Montréal, Canada Abstract Large multi-platform software systems are likely to encompass hardware-dependent code or sub-systems. However, analyzing multi-platform source code poses several challenges, intrinsically due to the variety of supported configurations. Often, the system was originally developed for a single platform, and then new target platforms were added.this practice promotes the presence of duplicated code, also said cloned code. This paper presents results on the clone percentage of a multi-platform/multi-million lines of codes, Linux kernel version After a brief description of the procedure followed for code analysis and clone identification, the obtained results are commented. Keywords: clone detection, source code analysis, metric extraction 1. Introduction Large multi-platform software systems are likely to encompass hardware-dependent code or sub-systems. Analyzing multi-platform source code, however, poses several challenges, intrinsically due to the variety of supported configurations. Often, the system was originally developed for a single platform, and then it was ported to support a larger product family: in other words, new target platforms were added. When porting a software system, developers may decide to copy an entire working sub-system and then modify the code to cope with the new hardware architecture and dependencies. This technique ensures they will not have any unplanned effect on the original piece of code they have just copied. However, this evolving practice promotes the appearing of clones in the code. Several papers investigated both the identification and the extent of cloned components in industrial software systems [1, 5, 7, 8, 9, 1]. In this paper the focus is on evaluating the extent of the cloning practice in a large multiplatform software system using a metric-based clone detection technique. A multi-million lines of codes system, Linux kernel release 2.4., was used as case study. Linux kernel is almost entirely written in C, with few assembler boot files and TCL/TK, Perl configuration scripts. Linux is an open source UNIX-like operating system, created by Linus Torvalds with developers throughout the world. Originally it was developed for 32-bit x86-based PCs (386 or higher). Nowadays the kernel 2.4. also runs on Compaq Alpha AXP, Sun SPARC and UltraSPARC, Motorola 68, PowerPC, ARM, Hitachi SuperH, IBM S/39, MIPS, HP PA-RISC, Intel IA-64 and DEC VAX. Ports are currently in progress to the AMD x86-64 architecture. In this paper, two or more code fragments (i.e., functions) are considered to be clones if the proposed metric returns exactly the same values. The Linux 2.4. kernel source code has been parsed and analyzed; a set of software metrics characterizing functions extracted. Once software metrics are available, clones could be identified [9]. In the absence of any documentation on the structure of the Linux Kernel, we desumed it for simplicity is sake from the directory structure of source code distribution. As mentioned later [2, 4, 3], this is only partially true. Under this assumption, the Linux 2.4. kernel consists of 9 major subsystems: arch, drivers, fs, init, ipc, kernel, lib, mm, net and scripts. Each major subsystem (e.g., drivers), according to the hierarchy of the source code directory organization, is further composed of a number of other sub-subsystems. For example, arch contains a tree for each supported architecture (i.e., i386, m68k, alpha, etc.). Each tree (e.g., alpha) in its turn is organized into several other directories (i.e., kernel, lib, mm, etc.). The evaluation of the Linux 2.4. kernel clones extent has been performed at different levels. Clones have been identified between both major subsystems which, accord-

2 Release Series Initial Release Numb. of Releases Time to Start of Next Release Series Duration of Series.1 9/17/ months 2 months.1 12/3/ months 27 months 1. 3/13/ month 12 months 1.1 4/6/ months 11 months 1.2 3/7/ months 14 months 1.3 6/12/ months 12 months 2. 6/9/ months 32 months 2.1 9/3/ months 29 months 2.2 1/26/ months still current 2.3 5/11/ months 12 months 2.4 1/4/1 4 still current ing to our choice, are assumed to be 1st level directories of source tree. For example clones have been identified between fs and arch. Furthermore, the same analysis has been performed between minor subsystems, such as arch/m68k and arch/i386 (are considered to be minor subsystems all the directories nested into 1st level directory). In particular, an experimental activity was performed to address two main research questions: which is the cloning extent within the Linux 2.4. kernel major subsystems; which is the cloning extent within the subsystems related to the different supported platforms; The rest of this paper is organized as follows. First, basic notions on clone detection are summarized in Section 2. Then, Section 3 presents the case study. The method used to to analyze the Linux 2.4. kernel and for identifying clones is presented in Section 4 and the experimental activity is summarized in 5. Finally, a discussion about the obtained experimental results is presented in Section Background Notions The goal of this paper is to study the potential impact of clones on system evolution in term of percentage of actual system size. Thus the presented results are not tied to the specific clone detection techniques adopted. In particular, the clone detection technique described below has been derived from a previously published paper [9]. In [9], each function in a software system is characterized by its name together with 21 metrics extracted with Datrix software metrics tool (available at Then, any pair of functions can be compared on the basis of these extracted characteristics to yield a rating scored on an ordinal scale. In this paper the focus is on the presence of duplicated or nearly duplicated code, so that the exact metrics identity is required to classify two functions as clones. The assumption corresponds to the ExactCopy and DistinctName classes presented in [9]. Table 1. Linux Kernels Most Important Events 2 When studying software system evolution, function identity is usually disregarded in favor of a different concept: clone clusters. A clone cluster can be seen as a set of similar code fragments which contains identical fragments or fragments exhibiting negligible difference (if any), from a given fragment prototype. Clone clusters are detected following a metric-based approach. 3. Case Study Linux is a Unix-like operating system that was initially written as a hobby by a Finnish student, Linus Torvalds [12]. The first Linux version,.1, was released in Since then, the system has been developed by the cooperative effort of many people, collaborating over the Internet under the control of Torvalds. In 1994, version 1. of the Linux Kernel was released. The current version is 2.4, released in January 21, the latest stable release is (March 31, 21). Unlike other Unixes (e.g., FreeBSD), Linux it is not directly related to the Unix family tree, in that its kernel was written from scratch, not by porting existing Unix source code. The very first version of Linux was targeted at the Intel 386 architecture. At the time the Linux project was started, the common belief of the research community was that high operating system portability could be achieved only by adopting a microkernel approach. The fact that now Linux, which relies on a traditional monolithic kernel, runs on a wide range of hardware platforms, from PalmPilots to Sparc, MIPS and Alpha workstations, clearly points out that portability can also be obtained by the use of a clever code structure. Linux is based on the Open Source concept: it is developed under the NU eneral Public License and its source code is freely available to everyone. Open Source refers to the users freedom to run, copy, distribute, study, change and improve the software. An open source software system includes the source code, and it explicitly promotes the source code as well as compiled forms distribution/redistribution. Very often open source code is

3 arch drivers fs init ipc kernel lib mm net scripts arch 1% 15.5% 7.5% % % 3.5% % 1.5% 8.8%.5% drivers 6.8% 1% 3.92% % % 1.6% %.97% 4.93%.2% fs 9.6% 9.9% 1% % % 4.1% % 4.1% 7.2%.5% init % % % 1% % % % % % % ipc 1.4% 2.8% % % 1% % % % 1.4% % kernel 13.6% 1.4% 6.3% % % 1% % 1.9% 6.1% % lib % % % % % % 1% % 2.2% % mm 4.8% 5.1% 5.4% % % 1.1% % 1% 2.5% % net 6.7% 8.1% 5.7% % % 2.5% % 1.7% 1%.2% scripts 1.1% 3.4% 2.2% % % % % % 1.13% 1% Table 2. Cloning Percentage among Linux Kernel Subsystem distributed under artistic license and/or the Free Software Foundation PL (NU eneral Public License). Open source distribution license shall not restrict any party from selling or giving away the software as a component of an aggregate software distribution containing programs from several different sources. Moreover, the license does not prevent modifications and derived works, and must allow them to be distributed under the same terms as the license of the original software. However, its most peculiar characteristic is that is not an organizational project, in that it has been developed through the years thanks to the efforts of volunteers from all over the world, who contributed code, documentation and technical support. Linux has been produced through a software development effort consisting of more than 3 developers distributed over 9 countries on five continents [11]. As such, it is a bright example of successful distributed engineering and development project. In the meantime, due to the nature of the decentralized, voluntary basis development effort and due to the lack of formalized processes it is very likely that a certain percentage of clones will show up. A key point in Linux structure is modularity. Without modularity, it would be impossible to use the open-source development model, and to let lot of developers work in parallel. High modularity means that people can work cooperatively on the code without clashes. Possible code changes have an impact confined to the module into which they are contained, without affecting other modules. After the first successful portings of the initial 8386 implementation, the Linux kernel architecture was redesigned in order to have one common code base that could simultaneously support a separate specific tree for any number of different machine architectures. The use of loadable kernel modules, introduced with the 2. kernel version [6], further enhanced modularity, by providing an explicit structure for writing modules containing hardware-specific code (e.g., device drivers). Besides making the core kernel highly portable, the introduction of modules allowed a large group of people to work simultaneously on the kernel without central control. The kernel modules are a good way to let programmers work independently on parts of the system that should be independent. An important management decision was establishing, in 1994, a parallel release structure for the Linux kernel. Evennumbered releases were the development versions on which people could experiment with new features. Once an oddnumbered release series incorporated sufficient new features and became sufficiently stable through bug fixes and patches, it would be renamed and released as the next higher even-numbered release series and the process would begin again. Linux kernel version 1., released in March 1994, had about 175, lines of code. Linux version 2., released in June 1996, had about 78, lines of code. Version 2.4, released in January 21, has more than 2.5 millions lines of code. Table 1, which is an updated version of the one published in [11], shows the most important events in the Linux kernel development time table, along with the number of releases produced for each development series. 4. The Method Parsing programming language such as C or C++ poses several challenges; besides intrinsic programming language peculiarities (e.g., union, struct, classes, function pointers, etc.), pre-processor directive must be handled. Preprocessing directives are usually managed by a dedicated compiler component, the pre-processor (e.g., the NU cpp). However, parsing multi-platform code where preprocessing directives are platform-dependent is equivalent to the projection of the source code on a given hardware/software configuration. To obtain information on several platforms, two approaches are feasible: 3 to pre-process and to parse on several different machines and configurations; to project the source code on a generic anonymous configuration.

4 Z!#"$!! %&(' )!*" ),+ $ )- % $*%../!#" $,1 $ -23'!#"/4!! $!.! $,'5 6 7!8 9 : ; 9</1.=' )! % >'?"@ ) / /.A' $ /! $,' $/B1 )!C,'?"@ ) /D2 ' $ )- % $/@ $,1 -$-EF%.. 1 )! (' )+IH JK!7J 59 L J H<9/B1.! $ >' ' $ )- % $/? $ /B1.=' )!N )'" )/ /!!N)! >' )/1 ='!#"/D )! $ >'?" )/DE*%..(1. ) O %P C % )- 4) O %#") O %/.='P) O % Q ) O! C/.A') O % Q ) O! / ) O % Q ) O! Q ) O! C42 ) O % Q ) O! C1 ") O % Q ) O! C-2*) O % Q ) O! 1 ) O % Q ) O! CF2-; M< <,1 The first approach requires several hardware/software configurations which may or may not be feasible, depending on the available resources and the analyzed software (the Linux kernel can be compiled on more than 1 different hardware architectures). On the contrary, by defining an anonymous configuration, no specific architecture will be identified. The latter approach is well suited to the task of identifying function clones across several platformdependent sub-system, without the need of recompiling the kernel for every hardware platform. 4.1 Function Identification ( ST 7U R!#"$!! %&(' )!*" ),+ $ )F % $*%../!#" $,1 $ -23'!#"/4!! $!.! $B'P5 6 7!8 9 : ; 9<,+V/B1.3' )! % >'?"@ ) / /.3' $ /! $,' $ /B1 )!C,'?"@ ) /D2 $ )F % $ /@$ >1 -$4E*%.. 1 )! (' )+IH JK!7J 59 L J H<9/B1.! $ >' ' $ )4 % $ /@$ >+W /B1.3' )!N )'" )/ /!!N)! >' )/1 ='!#"/D )! $ >'?" ) XC % ) %C! $!. %! O,'? D $*" $ /.3' $ Q C! /.3' $ Q! B/ $ Q C! 42 $ Q C! 1 "@ $ Q C! F2Y$ Q! 1 $ Q C! F24; M < <1 Figure 1. Two examples of clones found. Large C system are likely to encompass a variety of intermixed programming stiles, programming patterns, idioms, coding standard and naming conventions. Most noticeably, both the ANSI-C and the Kernighan & Ritchie style will possibly be present. To localize and extract function definitions, we adopted an approach inspired by islanddriven parsing: once islands (e.g., the function bodies or the signatures) were identified the in-between code was scanned, and function definition extracted by means of a hand-coded parser. Due to the presence of pre-processor directives, function bodies are not guaranteed to balance branches nor to be at lexical level zero. An heuristic was adopted to avoid applying pre-processing, while removing pre-processor statements from source code. Since preprocessor directives, i.e., ifdef, must be balanced, a preliminary parsing of the preprocessor statements was used to project the source on if branch, in other words the ifdef conditions were forced to true, extracting the then branches. The pre-processor elimination step generates sources with removed pre-processor directives regardless of the hardware/software architecture. Unfortunately, there are few cases where the adopted heuristic gives raise to syntactically wrong C code. Namely, a C scope start (i.e., ) may be opened in the then part of an ifdef with condition EXP, whereas the scope end ([ ) is located in a different preprocessor statement in the else part, that also means that the scope is closed by a combination of expressions where EXP must be negated (e.g., the ultrastor.c scsi driver). Due to the low number of such cases (2 cases over more that 48 functions), these were considered pathological situation, detected and signaled for manual intervention Clone Identification When a system evolves, functionalities are added, modified and/or removed: thus, it is likely that code distribution varies. Following the approach proposed in [9], code fragments (functions in the following) were compared on

5 e the basis of software metrics accounting for layout, size, control flow, function communication and coupling. More precisely, functions were compared on the basis of the following software metrics: the number of passed parameters; the number of LOC; the cyclomatic complexity; the number of used/defined local variables; the number used/defined non local variables; the numbers of arithmetic and logic operators (++, >=, etc.); the numbers of function calls and return/exit points; the numbers of structure/pointer access fields. Unlike the procedure followed in the past, (e.g., [9]), function name and file/unit name were not considered. Indeed, function identity may lead to consider different two functions that are in different files of a given release, or a function that migrates from a file or in a subsequent release. Let be the tuple of metrics characterizing a function, where each is the -th software metric chosen to describe (e.g., number of passed parameters, number of LOC, cyclomatic complexity, number of used/defined local variables, and number used/defined non local variables). In theory, different sub-systems or functions could be represented by different collection of metrics, thus discriminating among different features. For any given function, let be the clone cluster. is the subset of function, belonging to the considered software system/sub-system, that exhibits software metric values!"$# identical of similar to %&'# : (*),+.-"/ ;:<>=@?BAC$DFEH,=@?BAI3JD <K=@?'69L )NM These represent necessary conditions: the OQP was used to state that metric values, "$#, may be chosen to meet the specific goal. In order to collect exact, or nearly exact, function duplicates, OQP is implemented by the equality, while to identify similar functions a threshold may be adopted: =?BAC$D'EHR=?SAI3JD'TU=@?BAC$DWVX=?BAI3JDYV[Z\$AH]&D*4^4 Z_QAH]&D*V`=?SAI3JDYV`=?SAC$D5< =?a6 L ) where bncs "S# and bndfb# are the lower/upper bounds. Inside this range of values, is considered a clone of. 5 e f e f e g f e e f iven two different software systems, say and, information about the cloning extent between such software systems can be measured in terms of common ratio. The common ratio between and is defined as the ratio of the number of functions belonging to, having gih Uj when compared to function in, with respect to the numer of functions contained in. In other words, it is the percentage of functions having clones in with respect to size. 5. Experimental Activity As far as code analysis, program understanding and reverse software engineering practices are concerned, the peculiar characteristics of the Linux kernel make it an ideal candidate as tested for automated code examination and comprehension tools. It is based on the Open Source concept, and so there are no obstacles to discussing its implementation. It is not toy software, but one that is representative of real-world software systems. In addition, it is decidedly too large to be examined manually. Our experimental activity was driven by the two research questions, specified in the Introduction. Namely, dealing with the cloning percentage into major Linux architectural components and the percentage of duplicate code between different supported platforms. However, no published architectural documentation describes the system at a high level of abstraction, and maybe such a document does not even exist. Bowman et al. derived both the conceptual architecture (the developers system view) and the concrete architecture (the implemented system structure) of the Linux kernel [2, 4, 3]. They started from a manual hierarchical decomposition of the system structure, consisting of the assignment of source files to subsystems, and of subsystems hierarchically to subsystems. As shown in [3], most of the times, the extracted subsystems correspond to directories in the source code tree. For simplicity s sake, we have assumed that each directory of the source tree contains a subsystem (at a proper level of the system hierarchy). Our search for cloned code was performed by comparing the code contained in any two directories. Once metrics were evaluated, all the functions of each subsystem were matched against all the functions of all the other major subsystem. Figure 1 shows two different examples of function clones identified. The first clone pair (top of the figure), is an example of a function copied from mips to mips64 memory management subsystem. The second clone pair is a cross-system example: although the accessed data structure has different field names, the action actually performed is the same, i.e. to remove an item from a concatenated list. The cloning extent between two subsystem was measured in terms of common ratio. Re-

6 alpha arm i386 ia64 mips mips64 ppc s39 sh sparc sparc64 alpha 1% % 5.% % 5.% 1% 1% 5% 5% 5% % arm % 1% 2.4% % 8.6% 4.9% 4.9% % 1.2% 9.8% % i % 7.1% 1% % 7.1% 7.1% 1.7% 14.2% 32.1% % % ia64 % % % 1% % % % % 4.7% 9.5% % mips.6% 8.7% 1.3% % 1% 19.4% 4.2% %.6% 4.1 % % mips64 2.5% 3.8% 2.5% % 38.4% 1% 3.8% % 1.2% 2.5 % % ppc 3.2% 4.9% 4.9% % 8.1% 4.9% 1% 1.6% 3.2 % % % s39 5.2% % 21.1% % % % 5.2% 1% 5.2% % % sh 2.3% 2.3% 2.9% 2.3% 2.3% 2.3% 4.6% 2.3% 1% % % sparc.3% 2.2% %.6% 2.8%.6% % % % 1 % 1.9% sparc64 % % % % % % % % % 16.6% 1 % Table 3. Cloning Percentage among mm Subsystem Architecture-Dependent Code alpha arm i386 ia64 mips mips64 ppc s39 sh sparc sparc64 alpha 1% 2.7% 6.2% 7.6% 4.3% 7.1% 4.8 % 2.1% 1.6% 2.6% 1.9% arm 5.5% 1% 12.9% 8.9% 3.3% 3.3% 1.8% 8.2% 9.7% 4.4% 5.5% i % 7.7% 1% 7.9% 3.3% 2.4% 6.6% 4.4% 6.9% 5.1% 2.9% ia64 7.1% 4.2% 8.4% 1% 2.3% 2.3% 7.1% 5.2% 4.2% 7.3% 5.5 % mips 5.1% 5.5% 7.2% 3.4% 1% 16.3% 9.4% 9.1% 8.1% 7.7% 9.8% mips64 3.6% 3.6% 4.8% 4.2% 2.4% 1% 8.4% 4.2% 6.6% 4.8% 15.6% ppc 7.3% 8.1% 9.7% 9.5% 6.1% 6.7% 1% 6.8% 9.8% 1.1% 7.9% s % 17.1% 7.3% 17.1% 4.5% 3.1% 1.9% 1% 4.8% 17.4% 3.9% sh % 18.2% 12.9% 14.1% 8.9% 18.2% 6.9% 1% 15.9% 12.3 % sparc 3.7% 5.3% 4.9% 8.3% 3.2% 2.9% 7.7% 5.6% 5.1% 1% 14.7% sparc64 1.9% 4.6% 4.9% 3.4% 6.2% 5.1% 6.7% 2.2% 2.5% 12.4% 1% Table 4. Cloning Percentage among Core Kernel Architecture-Dependent Code sults are shown in Table 2. In the tables, the rows represent the e system/sub-system, while the columns are the f system/sub-system. A similar approach was followed to evaluate the cloning extent within the subsystem related to the different supported platforms. In particular, Tables 3 and 4 show the common ratio between the subsystem mm and kernel related to the different platform supported by Linux 2.4. kernel. 6. Discussion Table 2 shows the results obtained comparing the ten first-level subdirectories of the kernel source tree. These are arch (architecture-dependent code), drivers (drivers), fs (file systems), init (initialization code), ipc (interprocess communication subsystem), kernel (architectureindependent core kernel), lib (libraries), mm (memory manager), net (network interface), and script (system scripts). Table 3 and 4, instead, show the cloning percentage among the architecture-dependent code of the mm subsystem and of the core kernel (arch/*/mm and arch/*/kernel subdirectories, respectively). In all cases, the number of clones is at a physiologic level. Taking into account the results of [3], it seems that the implementation of similar subsystems (e.g., subsystems for similar architectures, file systems sharing a common ancestor,...) has been carried out by resorting to code reuse (function dependencies across different subsystems) rather than to cloning. The proposed tables show notable exceptions, nevertheless. For example, the cloning percentage between the mips64 and mips memory manager subsystems is Conclusion We have presented the analysis of a large multi-platform, multi-million lines of code software system: the Linux kernel. Software metrics at function level were extracted and duplicate code between architectural component of the kernel evaluated. Even if the analyzed software system has not been developed through a well-defined development process, but by the cooperative work of relatively uncoordinated developers, the clone percentage has to be considered low. The answers to the research questions can be summarized as follows: cloning percentage between sub-systems can be considered at a physiological level; 6 recently introduced architectures tend to exhibit a higher cloning percentage. Future activity will be devoted to evaluate the influence of clone size on the clone percentage, and to study the evolution of clones across releases. This will allow us to investigate if a refactory activity is carried out through succes-

7 sive software versions, leading to the remotion of a certain amount of clones. References [1] B. S. Baker. On finding duplication and near-duplication in large software systems. In Proceedings of the Working Conference on Reverse Engineering, July [2] I. Bowman. Conceptual architecture of the linux kernel. Technical report, Technical Report, University of Waterloo, itbowman/cs746/a1/. [3] I. Bowman, R. Holt, and V. Brewster. Linux as a case study: its extracted software architecture. In Proceedings of the International Conference on Software Engineering, pages IEEE Computer Society Press, [4] I. Bowman, S. Siddiqi, and M. Tanuan. Concrete architecture of the linux kernel. Technical report, Technical Report, University of Waterloo, itbowman/cs746/a2/. [5] E. Buss, R. D. Mori, W. entleman, J. Henshaw, H. Johnson, K. Kontogiannis, E. Merlo, H. Muller, J. M. S. Paul, A. Prakash, M. Stanley, S. Tilley, J. Troster, and K. Wong. Investigating reverse engineeering technologies for the cas program understanding project. IBM Systems Journal, 33(3):477 5, [6] J. de oyeneche and E. de Sousa. Loadable kernel modules. IEEE Software, 16(1):65 71, January [7] J. H. Johnson. Identifying redundancy in source code using fingerprints. In CASCON, pages , October [8] K. Kontogiannis, R. D. Mori, R. Bernstein, M. aller, and E. Merlo. Pattern matching for clone and concept detection. Journal of Automated Software Engineering, 3:77 18, March [9] J. Mayrand, C. Leblanc, and E. Merlo. Experiment on the automatic detection of function clones in a software system using metrics. In Proceedings of the International Conference on Software Maintenance - IEEE Computer Society Press, pages , Monterey, CA, Nov [1] T. J. McCabe. Reverse engineering, reusability, redundancy: the connection. American Programmer, 3:8 13, October 199. [11] J. Moon and L. Sproull. Essence of distributed work: The case of the linux kernel. Technical report, First Monday, vol. 5, n. 11 (November 2), [12] L. Torvalds. The linux edge. Communications of the ACM, 42(4):38 39, Dec