Free/Open Source Software: scientific and technological challenges for complex systems engineering

Transcription

1 Free/Open Source Software: scientific and technological challenges for complex systems engineering Roberto Di Cosmo Laboratoire Preuves Programmes et Systèmes Université Paris Diderot Paris 7 Paris, France and INRIA May 17th 2010 Journées Inria-Industrie, Toulouse Roberto Di Cosmo (Paris Diderot / INRIA) Free Software Complexity May 17th / Toulouse 1 / 38

2 Outline 1 The Open Source Infrastructure and its Complexity 2 Coccinelle 3 Mancoosi 4 IRILL: research and innovation on Free Software Roberto Di Cosmo (Paris Diderot / INRIA) Free Software Complexity May 17th / Toulouse 2 / 38

3 Outline 1 The Open Source Infrastructure and its Complexity 2 Coccinelle 3 Mancoosi 4 IRILL: research and innovation on Free Software Roberto Di Cosmo (Paris Diderot / INRIA) Free Software Complexity May 17th / Toulouse 3 / 38

4 Reminder: FOSS free (as in free beer, or gratuit) software which has not to be payed (today) free (as in free speech, or libre) software granting 4 freedoms to its users: 1 0 freedom to use the software 1 freedom to study the source code of the software and to adapt it to user needs 2 freedom to distribute software copies 3 freedom to distribute modified software copies two points of view: the freedom one as outlined above ( free software ), the technical one pivoting around source code availability ( open source ) 1 there are of course also obligations, which vary according to the license: GPL, BSD, Mozilla, MIT/X, AGPL,... Roberto Di Cosmo (Paris Diderot / INRIA) Free Software Complexity May 17th / Toulouse 4 / 38

5 Challenges and opportunities FOSS is radically changing the way software is conceived, developed, maintained, deployed, tested, proven, marketed and sold. Reminder: economics : the study of how society chooses to allocate scarce resources to produce, exchange, and consume goods and services. Ruffin, Gregory, Principles of Economics, 1990 proprietary software Without scarcity, there is no economy. based on an artificial scarcity of copies of a piece of software... free software removes the scarcity of copies, exposing the other scarce resources: know-how, infrastructure, process, industrialization, customization skills etc. Roberto Di Cosmo (Paris Diderot / INRIA) Free Software Complexity May 17th / Toulouse 5 / 38

6 Challenges and opportunities FOSS is radically changing the way software is conceived, developed, maintained, deployed, tested, proven, marketed and sold. Reminder: economics : the study of how society chooses to allocate scarce resources to produce, exchange, and consume goods and services. Ruffin, Gregory, Principles of Economics, 1990 proprietary software Without scarcity, there is no economy. based on an artificial scarcity of copies of a piece of software... free software removes the scarcity of copies, exposing the other scarce resources: know-how, infrastructure, process, industrialization, customization skills etc. Roberto Di Cosmo (Paris Diderot / INRIA) Free Software Complexity May 17th / Toulouse 5 / 38

7 Challenges and opportunities FOSS is radically changing the way software is conceived, developed, maintained, deployed, tested, proven, marketed and sold. Reminder: economics : the study of how society chooses to allocate scarce resources to produce, exchange, and consume goods and services. Ruffin, Gregory, Principles of Economics, 1990 proprietary software Without scarcity, there is no economy. based on an artificial scarcity of copies of a piece of software... free software removes the scarcity of copies, exposing the other scarce resources: know-how, infrastructure, process, industrialization, customization skills etc. Roberto Di Cosmo (Paris Diderot / INRIA) Free Software Complexity May 17th / Toulouse 5 / 38

8 Challenges and opportunities FOSS is radically changing the way software is conceived, developed, maintained, deployed, tested, proven, marketed and sold. Reminder: economics : the study of how society chooses to allocate scarce resources to produce, exchange, and consume goods and services. Ruffin, Gregory, Principles of Economics, 1990 proprietary software Without scarcity, there is no economy. based on an artificial scarcity of copies of a piece of software... free software removes the scarcity of copies, exposing the other scarce resources: know-how, infrastructure, process, industrialization, customization skills etc. Roberto Di Cosmo (Paris Diderot / INRIA) Free Software Complexity May 17th / Toulouse 5 / 38

9 Challenges and opportunities FOSS is radically changing the way software is conceived, developed, maintained, deployed, tested, proven, marketed and sold. Reminder: economics : the study of how society chooses to allocate scarce resources to produce, exchange, and consume goods and services. Ruffin, Gregory, Principles of Economics, 1990 proprietary software Without scarcity, there is no economy. based on an artificial scarcity of copies of a piece of software... free software removes the scarcity of copies, exposing the other scarce resources: know-how, infrastructure, process, industrialization, customization skills etc. Roberto Di Cosmo (Paris Diderot / INRIA) Free Software Complexity May 17th / Toulouse 5 / 38

10 Some numbers that say it all IT Market World: 550 billions euros in 2007 France: 30 billions euros in 2007 FLOSSImpact: LL = 30% of the world IT market in 2012 Gartner: Mark Driver, sept in 2011, 80% of commercial software contain Open Source... 49,7% of critical software come from Open Source France PAC 2008 study: 1st country of OCDE for FOSS LL = 3,6% of French IT market (1105Me, +32%!) OPIIEC 2008 study: LL strategic, but we miss education in Computer Science and Free Software Roberto Di Cosmo (Paris Diderot / INRIA) Free Software Complexity May 17th / Toulouse 6 / 38

11 Complex Software Systems Complex software systems are built from large number of components, which have to be deployed together; the most challenging are those that change frequently. The free/open source software (FOSS) infrastructure is a complex system archetype: no central authority / software architect quick (release early, release often) and distributed development strong component interdependency large code bases freely accessible (for developers, students, researchers,... ) Complexity arises at different levels... Roberto Di Cosmo (Paris Diderot / INRIA) Free Software Complexity May 17th / Toulouse 7 / 38

12 FOSS: complex software... [ kernel mapper project ] Roberto Di Cosmo (Paris Diderot / INRIA) Free Software Complexity May 17th / Toulouse 8 / 38

13 ... with superlinear growth [ Robles et al. ] Bad news: any change in core system API propagates in thousands of different call sites!good news: the Coccinelle project is building novel abstractions to master the problem. Roberto Di Cosmo (Paris Diderot / INRIA) Free Software Complexity May 17th / Toulouse 9 / 38

14 ... with superlinear growth [ Robles et al. ] Bad news: any change in core system API propagates in thousands of different call sites!good news: the Coccinelle project is building novel abstractions to master the problem. Roberto Di Cosmo (Paris Diderot / INRIA) Free Software Complexity May 17th / Toulouse 9 / 38

15 FOSS: complex interdependencies... Components Proposed 1968 by Douglas McIlroy as a remedy to the software crisis. Some Characteristics of Components: 1 Multiple-use 2 Encapsulated 3 A unit of independent deployment and versioning 4 Composable with other components Problem: Interaction between (3) and (4). GNU/Linux distributions are among the largest component based systems today! Roberto Di Cosmo (Paris Diderot / INRIA) Free Software Complexity May 17th / Toulouse 10 / 38

16 FOSS: complex interdependencies... Components Proposed 1968 by Douglas McIlroy as a remedy to the software crisis. Some Characteristics of Components: 1 Multiple-use 2 Encapsulated 3 A unit of independent deployment and versioning 4 Composable with other components Problem: Interaction between (3) and (4). GNU/Linux distributions are among the largest component based systems today! Roberto Di Cosmo (Paris Diderot / INRIA) Free Software Complexity May 17th / Toulouse 10 / 38

17 FOSS: complex interdependencies... Components Proposed 1968 by Douglas McIlroy as a remedy to the software crisis. Some Characteristics of Components: 1 Multiple-use 2 Encapsulated 3 A unit of independent deployment and versioning 4 Composable with other components Problem: Interaction between (3) and (4). GNU/Linux distributions are among the largest component based systems today! Roberto Di Cosmo (Paris Diderot / INRIA) Free Software Complexity May 17th / Toulouse 10 / 38

18 Inter-package relationships are complex... [ EDOS project ] Package: gnubg Version: Depends: gnubg-data, ttf-bitstream-vera, libartsc0 (>= ),..., libgl1-mesa-glx libgl1,... Conflicts:... And this changes everyday! Bad news: pulling our component out of the spaghetti dish is an NP-complete problem! Good news: the EDOS and Mancoosi projects have attacked the core of the dependency resolution phase, using formal methods to formalize and address them. Roberto Di Cosmo (Paris Diderot / INRIA) Free Software Complexity May 17th / Toulouse 11 / 38

19 Inter-package relationships are complex... [ EDOS project ] Package: gnubg Version: Depends: gnubg-data, ttf-bitstream-vera, libartsc0 (>= ),..., libgl1-mesa-glx libgl1,... Conflicts:... And this changes everyday! Bad news: pulling our component out of the spaghetti dish is an NP-complete problem! Good news: the EDOS and Mancoosi projects have attacked the core of the dependency resolution phase, using formal methods to formalize and address them. Roberto Di Cosmo (Paris Diderot / INRIA) Free Software Complexity May 17th / Toulouse 11 / 38

20 Complex software in industry, ten years from now Free Software is making software artefacts transparent: no artificial barriers among architectural layers... theoretical possibility to follow the flow of computation from the more abstract layers down to the moving bits in the hardware new levels of quality assurance and certification can be expected this is not easy: we need new theory and new tools it will take time, effort and good cooperation between industrial partners, higher education and research actors Industry need to invest in Free Software fundamental research: the sooner, the better. In the air&space Industry, you (should) know all this quite well. Roberto Di Cosmo (Paris Diderot / INRIA) Free Software Complexity May 17th / Toulouse 12 / 38

21 Complex software in industry, ten years from now Free Software is making software artefacts transparent: no artificial barriers among architectural layers... theoretical possibility to follow the flow of computation from the more abstract layers down to the moving bits in the hardware new levels of quality assurance and certification can be expected this is not easy: we need new theory and new tools it will take time, effort and good cooperation between industrial partners, higher education and research actors Industry need to invest in Free Software fundamental research: the sooner, the better. In the air&space Industry, you (should) know all this quite well. Roberto Di Cosmo (Paris Diderot / INRIA) Free Software Complexity May 17th / Toulouse 12 / 38

22 Complex software in industry, ten years from now Free Software is making software artefacts transparent: no artificial barriers among architectural layers... theoretical possibility to follow the flow of computation from the more abstract layers down to the moving bits in the hardware new levels of quality assurance and certification can be expected this is not easy: we need new theory and new tools it will take time, effort and good cooperation between industrial partners, higher education and research actors Industry need to invest in Free Software fundamental research: the sooner, the better. In the air&space Industry, you (should) know all this quite well. Roberto Di Cosmo (Paris Diderot / INRIA) Free Software Complexity May 17th / Toulouse 12 / 38

23 Complex software in industry, ten years from now Free Software is making software artefacts transparent: no artificial barriers among architectural layers... theoretical possibility to follow the flow of computation from the more abstract layers down to the moving bits in the hardware new levels of quality assurance and certification can be expected this is not easy: we need new theory and new tools it will take time, effort and good cooperation between industrial partners, higher education and research actors Industry need to invest in Free Software fundamental research: the sooner, the better. In the air&space Industry, you (should) know all this quite well. Roberto Di Cosmo (Paris Diderot / INRIA) Free Software Complexity May 17th / Toulouse 12 / 38

24 Complex software in industry, ten years from now Free Software is making software artefacts transparent: no artificial barriers among architectural layers... theoretical possibility to follow the flow of computation from the more abstract layers down to the moving bits in the hardware new levels of quality assurance and certification can be expected this is not easy: we need new theory and new tools it will take time, effort and good cooperation between industrial partners, higher education and research actors Industry need to invest in Free Software fundamental research: the sooner, the better. In the air&space Industry, you (should) know all this quite well. Roberto Di Cosmo (Paris Diderot / INRIA) Free Software Complexity May 17th / Toulouse 12 / 38

25 Complex software in industry, ten years from now Free Software is making software artefacts transparent: no artificial barriers among architectural layers... theoretical possibility to follow the flow of computation from the more abstract layers down to the moving bits in the hardware new levels of quality assurance and certification can be expected this is not easy: we need new theory and new tools it will take time, effort and good cooperation between industrial partners, higher education and research actors Industry need to invest in Free Software fundamental research: the sooner, the better. In the air&space Industry, you (should) know all this quite well. Roberto Di Cosmo (Paris Diderot / INRIA) Free Software Complexity May 17th / Toulouse 12 / 38

26 Complex software in industry, ten years from now Free Software is making software artefacts transparent: no artificial barriers among architectural layers... theoretical possibility to follow the flow of computation from the more abstract layers down to the moving bits in the hardware new levels of quality assurance and certification can be expected this is not easy: we need new theory and new tools it will take time, effort and good cooperation between industrial partners, higher education and research actors Industry need to invest in Free Software fundamental research: the sooner, the better. In the air&space Industry, you (should) know all this quite well. Roberto Di Cosmo (Paris Diderot / INRIA) Free Software Complexity May 17th / Toulouse 12 / 38

27 Two concrete examples of what can be done We will now focus on a short highlight of two ongoing research actions at each of the two levels of complexity: managing complexity for a big software component : the Coccinelle project and the Linux kernel managing complexity for a big software component collection : the Mancoosi project and the GNU/Linux distributions Roberto Di Cosmo (Paris Diderot / INRIA) Free Software Complexity May 17th / Toulouse 13 / 38

28 Outline 1 The Open Source Infrastructure and its Complexity 2 Coccinelle 3 Mancoosi 4 IRILL: research and innovation on Free Software Roberto Di Cosmo (Paris Diderot / INRIA) Free Software Complexity May 17th / Toulouse 14 / 38

29 Coccinelle: flexible abstractions to master collateral evolutions Thanks to Gilles Muller and Julia Lawall for the following slides. See for more information. The collateral evolution problem: Library functions change. Client code must be adapted. Change a function name, add an argument, etc. Linux context: Many libraries: usb, net, etc. Very many clients, including outside the Linux source tree. Roberto Di Cosmo (Paris Diderot / INRIA) Free Software Complexity May 17th / Toulouse 15 / 38

30 Coccinelle: flexible abstractions to master collateral evolutions Thanks to Gilles Muller and Julia Lawall for the following slides. See for more information. The collateral evolution problem: Library functions change. Client code must be adapted. Change a function name, add an argument, etc. Linux context: Many libraries: usb, net, etc. Very many clients, including outside the Linux source tree. Roberto Di Cosmo (Paris Diderot / INRIA) Free Software Complexity May 17th / Toulouse 15 / 38

31 An example Evolution: A new function: kzalloc = Collateral evolution: Merge kmalloc and memset into kzalloc fh = kmalloc(sizeof(struct zoran fh), GFP_KERNEL); if (!fh) { dprintk(1, KERN_ERR "%s: zoran open(): allocation of zoran fh failed\n", ZR_DEVNAME(zr)); return -ENOMEM; } memset(fh, 0, sizeof(struct zoran fh)); Roberto Di Cosmo (Paris Diderot / INRIA) Free Software Complexity May 17th / Toulouse 16 / 38

32 An example Evolution: A new function: kzalloc = Collateral evolution: Merge kmalloc and memset into kzalloc fh = kzalloc(sizeof(struct zoran fh), GFP_KERNEL); if (!fh) { dprintk(1, KERN_ERR "%s: zoran open(): allocation of zoran fh failed\n", ZR_DEVNAME(zr)); return -ENOMEM; } memset(fh, 0, sizeof(struct zoran fh)); Roberto Di Cosmo (Paris Diderot / INRIA) Free Software Complexity May 17th / Toulouse 17 / 38

33 A simple semantic patch Abstract away the details of a expression x; expression - x = kmalloc(e1,e2); + x = kzalloc(e1,e2);... - memset(x, 0, E1); A single snippet... updates 355/564 files! Roberto Di Cosmo (Paris Diderot / INRIA) Free Software Complexity May 17th / Toulouse 18 / 38

34 Coccinelle architecture Semantic patch C code AX(A[(ϕ 1 ϕ 2 )Uϕ 3 ]...) CTL formula Control Flow Graph Model checking algorithm Identification of the nodes to be modified Modification of the identified code Roberto Di Cosmo (Paris Diderot / INRIA) Free Software Complexity May 17th / Toulouse 19 / 38

35 Practical results Collateral evolutions Semantic patches for over 60 collateral evolutions. Applied to over 5800 Linux files from various versions, with a success rate of 100% on 93% of the files. Bug finding Generic bug types: Null dereference, initialization of unused variables, etc. Bugs in the use of Linux APIs: Incoherent error checking, memory leaks, etc. Over 450 patches created using Coccinelle accepted into Linux Starting to be used by other Linux developers Probable bugs found in gcc, postgresql, vim, amsn, pidgin, mplayer Roberto Di Cosmo (Paris Diderot / INRIA) Free Software Complexity May 17th / Toulouse 20 / 38

36 Outline 1 The Open Source Infrastructure and its Complexity 2 Coccinelle 3 Mancoosi 4 IRILL: research and innovation on Free Software Roberto Di Cosmo (Paris Diderot / INRIA) Free Software Complexity May 17th / Toulouse 21 / 38

37 Mancoosi: upgrading GNU/Linux distributions How do you compose a system by selecting components from dozens of thousands developend independently? A new idea from FLOSS: GNU/Linux distributions as intermediaries between FOSS projects and their users Roberto Di Cosmo (Paris Diderot / INRIA) Free Software Complexity May 17th / Toulouse 22 / 38

38 Mancoosi: upgrading GNU/Linux distributions How do you compose a system by selecting components from dozens of thousands developend independently? A new idea from FLOSS: GNU/Linux distributions as intermediaries between FOSS projects and their users Roberto Di Cosmo (Paris Diderot / INRIA) Free Software Complexity May 17th / Toulouse 22 / 38

39 Distributions: a somehow successful idea... Central notion in distributions (to abstract over the complex underlying infrastructure): package, together with package management software... Roberto Di Cosmo (Paris Diderot / INRIA) Free Software Complexity May 17th / Toulouse 23 / 38

40 Packages, metadata, installation Package = { some files some scripts metadata Identification Inter-package rel. Dependencies Conflicts Feature declarations Other Package maintainer Textual descriptions... Example Package: aterm Version: Section: x11 Installed-Size: 280 Maintainer: Göran Weinholt... Architecture: i386 Depends: libc6 (>= ds1-4), libice6 xlibs (>> 4.1.0),... Conflicts: suidmanager (<< 0.50) Provides: x-terminal-emulator... a package is the elemental component of modern distribution systems (not GNU/Linux specific) a working system is deployed by installing a package set ( 1000/2000 for GNU/Linux distro) Roberto Di Cosmo (Paris Diderot / INRIA) Free Software Complexity May 17th / Toulouse 24 / 38

41 Installation process Phase User request Constraint resolution Package retrieval Trace # apt-get install aterm Reading package lists... Done Building dependency tree... Done The following extra packages will be installed: libafterimage0 The following NEW packages will be installed aterm libafterimage0 0 upgraded, 2 newly installed, 0 to remove and 1786 not upgraded. Need to get 386kB of archives. After unpacking 807kB of additional disk space will be used. Do you want to continue [Y/n]? Y Get: 1 testing/main libafterimage [301kB] Get: 2 testing/main aterm [84.4kB] Fetched 386kB in 0s (410kB/s) Pre-configuration { Selecting previously deselected package libafterimage0. (Reading database files and directories currently installed.) Unpacking libafterimage0 (from.../libafterimage0_ _i386.deb)... Unpacking Selecting previously deselected package aterm. Unpacking aterm (from.../aterm_ _i386.deb)... { Setting up libafterimage0 ( )... Configuration Setting up aterm ( )... each phase can fail (it actually happens quite often... ) efforts should be made to identify errors as early as possible Roberto Di Cosmo (Paris Diderot / INRIA) Free Software Complexity May 17th / Toulouse 25 / 38

42 Installation process Phase User request Constraint resolution Package retrieval Trace # apt-get install aterm Reading package lists... Done Building dependency tree... Done The following extra packages will be installed: libafterimage0 The following NEW packages will be installed aterm libafterimage0 0 upgraded, 2 newly installed, 0 to remove and 1786 not upgraded. Need to get 386kB of archives. After unpacking 807kB of additional disk space will be used. Do you want to continue [Y/n]? Y Get: 1 testing/main libafterimage [301kB] Get: 2 testing/main aterm [84.4kB] Fetched 386kB in 0s (410kB/s) Pre-configuration { Selecting previously deselected package libafterimage0. (Reading database files and directories currently installed.) Unpacking libafterimage0 (from.../libafterimage0_ _i386.deb)... Unpacking Selecting previously deselected package aterm. Unpacking aterm (from.../aterm_ _i386.deb)... { Setting up libafterimage0 ( )... Configuration Setting up aterm ( )... each phase can fail (it actually happens quite often... ) efforts should be made to identify errors as early as possible Roberto Di Cosmo (Paris Diderot / INRIA) Free Software Complexity May 17th / Toulouse 25 / 38

43 Installation woes # sudo apt-get install debhelper Reading Package Lists... Done Building Dependency Tree... Done The following extra packages will be installed: armagetron armagetron-common autoconf bonobo-activation codebreaker debconf debconf-i18n debconf-utils dialog esound-common fb-music-high fontconfig frozen-bubble-data grepmail gv intltool-debian libaiksaurus-data libaiksaurus0c102 libatk1.0-0 libatk1.0-dev libbonobo-activation4 libbonobo2-0 libbonobo2-common libdb3 libdbd-mysql-perl libdbi-perl libeel2-data libesd0... The following packages will be REMOVED: autoconf2.13 frozen-bubble frozen-bubble-lib gconf2 gnomemeeting itk3.1-dev libbonoboui2-0 libbonoboui2-common libdigest-md5-perl libforms0.89 libgconf2-4 libgnome2-0 libgnome2-common libgnomeui-0 libgnomevfs2-0 libgnomevfs2-common libgtk1.2-dev libgtk2.0-0png3 libgtk2.0-dev libmime-base64-perl libpango1.0-dev libsdl-mixer1.2-dev libsdl-perl libsdl-ttf1.2-dev libsdl1.2-dev libsmpeg-dev libstorable-perl nautilus tk8.3-dev tktable-dev x-window-system x-window-system-core xaw3dg-dev xlib6g xlib6g-dev xlibmesa-dev xlibmesa3 xlibosmesa3 xlibs-dev xlibs-pic xpdf xpdf-reader The following NEW packages will be installed: armagetron-common debconf-i18n fb-music-high fontconfig intltool-debian libaiksaurus-data libaiksaurus0c102 libeel2-data libfilehandle-unget-perl libfontconfig1 libforms1 libgdbm3 libgnutls7 libgsf-1 libice-dev libice6 libidl0 liblzo1 libmagick5.5.7 libmail-mbox-messageparser-perl libmysqlclient12 libncursesw5 libnet-daemon-perl libnewt0.51 libpaper1 libplrpc-perl libsdl-console packages upgraded, 80 newly installed, 42 to remove and 858 not upgraded. Need to get 67.1MB of archives. After unpacking 26.9MB will be used. Do you want to continue? [Y/n] Abort. Roberto Di Cosmo (Paris Diderot / INRIA) Free Software Complexity May 17th / Toulouse 26 / 38

44 Installation woes # sudo apt-get install debhelper Reading Package Lists... Done Building Dependency Tree... Done The following extra packages will be installed: armagetron armagetron-common autoconf bonobo-activation codebreaker debconf debconf-i18n debconf-utils dialog esound-common fb-music-high fontconfig frozen-bubble-data grepmail gv intltool-debian libaiksaurus-data libaiksaurus0c102 libatk1.0-0 libatk1.0-dev libbonobo-activation4 libbonobo2-0 libbonobo2-common libdb3 libdbd-mysql-perl libdbi-perl libeel2-data libesd0... The following packages will be REMOVED: autoconf2.13 frozen-bubble frozen-bubble-lib gconf2 gnomemeeting itk3.1-dev libbonoboui2-0 libbonoboui2-common libdigest-md5-perl libforms0.89 libgconf2-4 libgnome2-0 libgnome2-common libgnomeui-0 libgnomevfs2-0 libgnomevfs2-common libgtk1.2-dev libgtk2.0-0png3 libgtk2.0-dev libmime-base64-perl libpango1.0-dev libsdl-mixer1.2-dev libsdl-perl libsdl-ttf1.2-dev libsdl1.2-dev libsmpeg-dev libstorable-perl nautilus tk8.3-dev tktable-dev x-window-system x-window-system-core xaw3dg-dev xlib6g xlib6g-dev xlibmesa-dev xlibmesa3 xlibosmesa3 xlibs-dev xlibs-pic xpdf xpdf-reader The following NEW packages will be installed: armagetron-common debconf-i18n fb-music-high fontconfig intltool-debian libaiksaurus-data libaiksaurus0c102 libeel2-data libfilehandle-unget-perl libfontconfig1 libforms1 libgdbm3 libgnutls7 libgsf-1 libice-dev libice6 libidl0 liblzo1 libmagick5.5.7 libmail-mbox-messageparser-perl libmysqlclient12 libncursesw5 libnet-daemon-perl libnewt0.51 libpaper1 libplrpc-perl libsdl-console packages upgraded, 80 newly installed, 42 to remove and 858 not upgraded. Need to get 67.1MB of archives. After unpacking 26.9MB will be used. Do you want to continue? [Y/n] Abort. Roberto Di Cosmo (Paris Diderot / INRIA) Free Software Complexity May 17th / Toulouse 26 / 38

45 Installation woes # sudo apt-get install debhelper Reading Package Lists... Done Building Dependency Tree... Done The following extra packages will be installed: armagetron armagetron-common autoconf bonobo-activation codebreaker debconf debconf-i18n debconf-utils dialog esound-common fb-music-high fontconfig frozen-bubble-data grepmail gv intltool-debian libaiksaurus-data libaiksaurus0c102 libatk1.0-0 libatk1.0-dev libbonobo-activation4 libbonobo2-0 libbonobo2-common libdb3 libdbd-mysql-perl libdbi-perl libeel2-data libesd0... The following packages will be REMOVED: autoconf2.13 frozen-bubble frozen-bubble-lib gconf2 gnomemeeting itk3.1-dev libbonoboui2-0 libbonoboui2-common libdigest-md5-perl libforms0.89 libgconf2-4 libgnome2-0 libgnome2-common libgnomeui-0 libgnomevfs2-0 libgnomevfs2-common libgtk1.2-dev libgtk2.0-0png3 libgtk2.0-dev libmime-base64-perl libpango1.0-dev libsdl-mixer1.2-dev libsdl-perl libsdl-ttf1.2-dev libsdl1.2-dev libsmpeg-dev libstorable-perl nautilus tk8.3-dev tktable-dev x-window-system x-window-system-core xaw3dg-dev xlib6g xlib6g-dev xlibmesa-dev xlibmesa3 xlibosmesa3 xlibs-dev xlibs-pic xpdf xpdf-reader The following NEW packages will be installed: armagetron-common debconf-i18n fb-music-high fontconfig intltool-debian libaiksaurus-data libaiksaurus0c102 libeel2-data libfilehandle-unget-perl libfontconfig1 libforms1 libgdbm3 libgnutls7 libgsf-1 libice-dev libice6 libidl0 liblzo1 libmagick5.5.7 libmail-mbox-messageparser-perl libmysqlclient12 libncursesw5 libnet-daemon-perl libnewt0.51 libpaper1 libplrpc-perl libsdl-console packages upgraded, 80 newly installed, 42 to remove and 858 not upgraded. Need to get 67.1MB of archives. After unpacking 26.9MB will be used. Do you want to continue? [Y/n] Abort. Roberto Di Cosmo (Paris Diderot / INRIA) Free Software Complexity May 17th / Toulouse 26 / 38

46 Installation woes # sudo apt-get install debhelper Reading Package Lists... Done Building Dependency Tree... Done The following extra packages will be installed: armagetron armagetron-common autoconf bonobo-activation codebreaker debconf debconf-i18n debconf-utils dialog esound-common fb-music-high fontconfig frozen-bubble-data grepmail gv intltool-debian libaiksaurus-data libaiksaurus0c102 libatk1.0-0 libatk1.0-dev libbonobo-activation4 libbonobo2-0 libbonobo2-common libdb3 libdbd-mysql-perl libdbi-perl libeel2-data libesd0... The following packages will be REMOVED: autoconf2.13 frozen-bubble frozen-bubble-lib gconf2 gnomemeeting itk3.1-dev libbonoboui2-0 libbonoboui2-common libdigest-md5-perl libforms0.89 libgconf2-4 libgnome2-0 libgnome2-common libgnomeui-0 libgnomevfs2-0 libgnomevfs2-common libgtk1.2-dev libgtk2.0-0png3 libgtk2.0-dev libmime-base64-perl libpango1.0-dev libsdl-mixer1.2-dev libsdl-perl libsdl-ttf1.2-dev libsdl1.2-dev libsmpeg-dev libstorable-perl nautilus tk8.3-dev tktable-dev x-window-system x-window-system-core xaw3dg-dev xlib6g xlib6g-dev xlibmesa-dev xlibmesa3 xlibosmesa3 xlibs-dev xlibs-pic xpdf xpdf-reader The following NEW packages will be installed: armagetron-common debconf-i18n fb-music-high fontconfig intltool-debian libaiksaurus-data libaiksaurus0c102 libeel2-data libfilehandle-unget-perl libfontconfig1 libforms1 libgdbm3 libgnutls7 libgsf-1 libice-dev libice6 libidl0 liblzo1 libmagick5.5.7 libmail-mbox-messageparser-perl libmysqlclient12 libncursesw5 libnet-daemon-perl libnewt0.51 libpaper1 libplrpc-perl libsdl-console packages upgraded, 80 newly installed, 42 to remove and 858 not upgraded. Need to get 67.1MB of archives. After unpacking 26.9MB will be used. Do you want to continue? [Y/n] Abort. Roberto Di Cosmo (Paris Diderot / INRIA) Free Software Complexity May 17th / Toulouse 26 / 38

47 Installation woes # sudo apt-get install debhelper Reading Package Lists... Done Building Dependency Tree... Done The following extra packages will be installed: armagetron armagetron-common autoconf bonobo-activation codebreaker debconf debconf-i18n debconf-utils dialog esound-common fb-music-high fontconfig frozen-bubble-data grepmail gv intltool-debian libaiksaurus-data libaiksaurus0c102 libatk1.0-0 libatk1.0-dev libbonobo-activation4 libbonobo2-0 libbonobo2-common libdb3 libdbd-mysql-perl libdbi-perl libeel2-data libesd0... The following packages will be REMOVED: autoconf2.13 frozen-bubble frozen-bubble-lib gconf2 gnomemeeting itk3.1-dev libbonoboui2-0 libbonoboui2-common libdigest-md5-perl libforms0.89 libgconf2-4 libgnome2-0 libgnome2-common libgnomeui-0 libgnomevfs2-0 libgnomevfs2-common libgtk1.2-dev libgtk2.0-0png3 libgtk2.0-dev libmime-base64-perl libpango1.0-dev libsdl-mixer1.2-dev libsdl-perl libsdl-ttf1.2-dev libsdl1.2-dev libsmpeg-dev libstorable-perl nautilus tk8.3-dev tktable-dev x-window-system x-window-system-core xaw3dg-dev xlib6g xlib6g-dev xlibmesa-dev xlibmesa3 xlibosmesa3 xlibs-dev xlibs-pic xpdf xpdf-reader The following NEW packages will be installed: armagetron-common debconf-i18n fb-music-high fontconfig intltool-debian libaiksaurus-data libaiksaurus0c102 libeel2-data libfilehandle-unget-perl libfontconfig1 libforms1 libgdbm3 libgnutls7 libgsf-1 libice-dev libice6 libidl0 liblzo1 libmagick5.5.7 libmail-mbox-messageparser-perl libmysqlclient12 libncursesw5 libnet-daemon-perl libnewt0.51 libpaper1 libplrpc-perl libsdl-console packages upgraded, 80 newly installed, 42 to remove and 858 not upgraded. Need to get 67.1MB of archives. After unpacking 26.9MB will be used. Do you want to continue? [Y/n] Abort. Roberto Di Cosmo (Paris Diderot / INRIA) Free Software Complexity May 17th / Toulouse 26 / 38

48 Installation woes # sudo apt-get install debhelper Reading Package Lists... Done Building Dependency Tree... Done The following extra packages will be installed: armagetron armagetron-common autoconf bonobo-activation codebreaker debconf debconf-i18n debconf-utils dialog esound-common fb-music-high fontconfig frozen-bubble-data grepmail gv intltool-debian libaiksaurus-data libaiksaurus0c102 libatk1.0-0 libatk1.0-dev libbonobo-activation4 libbonobo2-0 libbonobo2-common libdb3 libdbd-mysql-perl libdbi-perl libeel2-data libesd0... The following packages will be REMOVED: autoconf2.13 frozen-bubble frozen-bubble-lib gconf2 gnomemeeting itk3.1-dev libbonoboui2-0 libbonoboui2-common libdigest-md5-perl libforms0.89 libgconf2-4 libgnome2-0 libgnome2-common libgnomeui-0 libgnomevfs2-0 libgnomevfs2-common libgtk1.2-dev libgtk2.0-0png3 libgtk2.0-dev libmime-base64-perl libpango1.0-dev libsdl-mixer1.2-dev libsdl-perl libsdl-ttf1.2-dev libsdl1.2-dev libsmpeg-dev libstorable-perl nautilus tk8.3-dev tktable-dev x-window-system x-window-system-core xaw3dg-dev xlib6g xlib6g-dev xlibmesa-dev xlibmesa3 xlibosmesa3 xlibs-dev xlibs-pic xpdf xpdf-reader The following NEW packages will be installed: armagetron-common debconf-i18n fb-music-high fontconfig intltool-debian libaiksaurus-data libaiksaurus0c102 libeel2-data libfilehandle-unget-perl libfontconfig1 libforms1 libgdbm3 libgnutls7 libgsf-1 libice-dev libice6 libidl0 liblzo1 libmagick5.5.7 libmail-mbox-messageparser-perl libmysqlclient12 libncursesw5 libnet-daemon-perl libnewt0.51 libpaper1 libplrpc-perl libsdl-console packages upgraded, 80 newly installed, 42 to remove and 858 not upgraded. Need to get 67.1MB of archives. After unpacking 26.9MB will be used. Do you want to continue? [Y/n] Abort. Roberto Di Cosmo (Paris Diderot / INRIA) Free Software Complexity May 17th / Toulouse 26 / 38

49 Installation woes # sudo apt-get install debhelper Reading Package Lists... Done Building Dependency Tree... Done The following extra packages will be installed: armagetron armagetron-common autoconf bonobo-activation codebreaker debconf debconf-i18n debconf-utils dialog esound-common fb-music-high fontconfig frozen-bubble-data grepmail gv intltool-debian libaiksaurus-data libaiksaurus0c102 libatk1.0-0 libatk1.0-dev libbonobo-activation4 libbonobo2-0 libbonobo2-common libdb3 libdbd-mysql-perl libdbi-perl libeel2-data libesd0... The following packages will be REMOVED: autoconf2.13 frozen-bubble frozen-bubble-lib gconf2 gnomemeeting itk3.1-dev libbonoboui2-0 libbonoboui2-common libdigest-md5-perl libforms0.89 libgconf2-4 libgnome2-0 libgnome2-common libgnomeui-0 libgnomevfs2-0 libgnomevfs2-common libgtk1.2-dev libgtk2.0-0png3 libgtk2.0-dev libmime-base64-perl libpango1.0-dev libsdl-mixer1.2-dev libsdl-perl libsdl-ttf1.2-dev libsdl1.2-dev libsmpeg-dev libstorable-perl nautilus tk8.3-dev tktable-dev x-window-system x-window-system-core xaw3dg-dev xlib6g xlib6g-dev xlibmesa-dev xlibmesa3 xlibosmesa3 xlibs-dev xlibs-pic xpdf xpdf-reader The following NEW packages will be installed: armagetron-common debconf-i18n fb-music-high fontconfig intltool-debian libaiksaurus-data libaiksaurus0c102 libeel2-data libfilehandle-unget-perl libfontconfig1 libforms1 libgdbm3 libgnutls7 libgsf-1 libice-dev libice6 libidl0 liblzo1 libmagick5.5.7 libmail-mbox-messageparser-perl libmysqlclient12 libncursesw5 libnet-daemon-perl libnewt0.51 libpaper1 libplrpc-perl libsdl-console packages upgraded, 80 newly installed, 42 to remove and 858 not upgraded. Need to get 67.1MB of archives. After unpacking 26.9MB will be used. Do you want to continue? [Y/n] Abort. Roberto Di Cosmo (Paris Diderot / INRIA) Free Software Complexity May 17th / Toulouse 26 / 38

50 Installation woes # sudo apt-get install debhelper Reading Package Lists... Done Building Dependency Tree... Done The following extra packages will be installed: armagetron armagetron-common autoconf bonobo-activation codebreaker debconf debconf-i18n debconf-utils dialog esound-common fb-music-high fontconfig frozen-bubble-data grepmail gv intltool-debian libaiksaurus-data libaiksaurus0c102 libatk1.0-0 libatk1.0-dev libbonobo-activation4 libbonobo2-0 libbonobo2-common libdb3 libdbd-mysql-perl libdbi-perl libeel2-data libesd0... The following packages will be REMOVED: autoconf2.13 frozen-bubble frozen-bubble-lib gconf2 gnomemeeting itk3.1-dev libbonoboui2-0 libbonoboui2-common libdigest-md5-perl libforms0.89 libgconf2-4 libgnome2-0 libgnome2-common libgnomeui-0 libgnomevfs2-0 libgnomevfs2-common libgtk1.2-dev libgtk2.0-0png3 libgtk2.0-dev libmime-base64-perl libpango1.0-dev libsdl-mixer1.2-dev libsdl-perl libsdl-ttf1.2-dev libsdl1.2-dev libsmpeg-dev libstorable-perl nautilus tk8.3-dev tktable-dev x-window-system x-window-system-core xaw3dg-dev xlib6g xlib6g-dev xlibmesa-dev xlibmesa3 xlibosmesa3 xlibs-dev xlibs-pic xpdf xpdf-reader The following NEW packages will be installed: armagetron-common debconf-i18n fb-music-high fontconfig intltool-debian libaiksaurus-data libaiksaurus0c102 libeel2-data libfilehandle-unget-perl libfontconfig1 libforms1 libgdbm3 libgnutls7 libgsf-1 libice-dev libice6 libidl0 liblzo1 libmagick5.5.7 libmail-mbox-messageparser-perl libmysqlclient12 libncursesw5 libnet-daemon-perl libnewt0.51 libpaper1 libplrpc-perl libsdl-console packages upgraded, 80 newly installed, 42 to remove and 858 not upgraded. Need to get 67.1MB of archives. After unpacking 26.9MB will be used. Do you want to continue? [Y/n] Abort. Roberto Di Cosmo (Paris Diderot / INRIA) Free Software Complexity May 17th / Toulouse 26 / 38

51 The Mancoosi project [ ] project strep FP7 ( ) focus coherence and maintenance of a FOSS distribution installation (user point of view) Utils for package installation management (package managers / meta-installer) installation removal upgrade downgrade... Roberto Di Cosmo (Paris Diderot / INRIA) Free Software Complexity May 17th / Toulouse 27 / 38

52 Goals, approach and strategy Mancoosi s goal: enable safe, efficient maintainability of the Open Source software infrastructure built out of software packages A twofold approach to reach this objective: design algorithms and tools to tackle the upgradeability problem; this allows to find good upgrade paths w.r.t. the user s needs. conceive and implement a transactional update process; this allows to roll-back an unsatisfactory upgrade. A long term, ecosystem strategy to reach this objective: bring users, distributions, developers and researchers together! set up virtous cycles Roberto Di Cosmo (Paris Diderot / INRIA) Free Software Complexity May 17th / Toulouse 28 / 38

53 Goals, approach and strategy Mancoosi s goal: enable safe, efficient maintainability of the Open Source software infrastructure built out of software packages A twofold approach to reach this objective: design algorithms and tools to tackle the upgradeability problem; this allows to find good upgrade paths w.r.t. the user s needs. conceive and implement a transactional update process; this allows to roll-back an unsatisfactory upgrade. A long term, ecosystem strategy to reach this objective: bring users, distributions, developers and researchers together! set up virtous cycles Roberto Di Cosmo (Paris Diderot / INRIA) Free Software Complexity May 17th / Toulouse 28 / 38

54 Goals, approach and strategy Mancoosi s goal: enable safe, efficient maintainability of the Open Source software infrastructure built out of software packages A twofold approach to reach this objective: design algorithms and tools to tackle the upgradeability problem; this allows to find good upgrade paths w.r.t. the user s needs. conceive and implement a transactional update process; this allows to roll-back an unsatisfactory upgrade. A long term, ecosystem strategy to reach this objective: bring users, distributions, developers and researchers together! set up virtous cycles Roberto Di Cosmo (Paris Diderot / INRIA) Free Software Complexity May 17th / Toulouse 28 / 38

55 Goals, approach and strategy Mancoosi s goal: enable safe, efficient maintainability of the Open Source software infrastructure built out of software packages A twofold approach to reach this objective: design algorithms and tools to tackle the upgradeability problem; this allows to find good upgrade paths w.r.t. the user s needs. conceive and implement a transactional update process; this allows to roll-back an unsatisfactory upgrade. A long term, ecosystem strategy to reach this objective: bring users, distributions, developers and researchers together! set up virtous cycles Roberto Di Cosmo (Paris Diderot / INRIA) Free Software Complexity May 17th / Toulouse 28 / 38

56 Goals, approach and strategy Mancoosi s goal: enable safe, efficient maintainability of the Open Source software infrastructure built out of software packages A twofold approach to reach this objective: design algorithms and tools to tackle the upgradeability problem; this allows to find good upgrade paths w.r.t. the user s needs. conceive and implement a transactional update process; this allows to roll-back an unsatisfactory upgrade. A long term, ecosystem strategy to reach this objective: bring users, distributions, developers and researchers together! set up virtous cycles Roberto Di Cosmo (Paris Diderot / INRIA) Free Software Complexity May 17th / Toulouse 28 / 38

57 All in a picture Mancoosi architecture In Mancoosi, there are really two main activities, corresponding to the architecture of our design, and they are loosely coupled User request WP4,WP5 Platform management interface Optimal upgrade path computation WP4, WP5 Transactional component manager WP2, WP3 Software Platform Roberto Di Cosmo (Paris Diderot / INRIA) Free Software Complexity May 17th / Toulouse 29 / 38

58 Upgrade resolution Abstract view of packages through the metadata lens. Roberto Di Cosmo (Paris Diderot / INRIA) Free Software Complexity May 17th / Toulouse 30 / 38

59 Mancoosi technology preview: visualizing Systems of Systems GNU/Linux software distributions: probably the most complex system of systems on the planet HOW can you visualize such a system of systems? Roberto Di Cosmo (Paris Diderot / INRIA) Free Software Complexity May 17th / Toulouse 31 / 38

60 Mancoosi technology preview: visualizing Systems of Systems GNU/Linux software distributions: probably the most complex system of systems on the planet HOW can you visualize such a system of systems? Roberto Di Cosmo (Paris Diderot / INRIA) Free Software Complexity May 17th / Toulouse 31 / 38

61 Draw a graph, no deep knowledge kde-amusements and kde-toys in Debian 3.1 (aka sarge, 2005) were unnecessary components, overlapping kde-games and kde-edu.they are waste, and are gone today (2009). Should be easy to spot, only 244 components are concerned! Can you see this fact here? Roberto Di Cosmo (Paris Diderot / INRIA) Free Software Complexity May 17th / Toulouse 32 / 38

62 Draw a graph, no deep knowledge kde-amusements and kde-toys in Debian 3.1 (aka sarge, 2005) were unnecessary components, overlapping kde-games and kde-edu.they are waste, and are gone today (2009). Should be easy to spot, only 244 components are concerned! Can you see this fact here? Roberto Di Cosmo (Paris Diderot / INRIA) Free Software Complexity May 17th / Toulouse 32 / 38

63 Draw a graph, no deep knowledge kde-amusements and kde-toys in Debian 3.1 (aka sarge, 2005) were unnecessary components, overlapping kde-games and kde-edu.they are waste, and are gone today (2009). Should be easy to spot, only 244 components are concerned! Can you see this fact here? Roberto Di Cosmo (Paris Diderot / INRIA) Free Software Complexity May 17th / Toulouse 32 / 38

64 Draw a graph, no deep knowledge kde-amusements and kde-toys in Debian 3.1 (aka sarge, 2005) were unnecessary components, overlapping kde-games and kde-edu.they are waste, and are gone today (2009). Should be easy to spot, only 244 components are concerned! Can you see this fact here? Roberto Di Cosmo (Paris Diderot / INRIA) Free Software Complexity May 17th / Toulouse 32 / 38

65 Let s try a more systemic approach Here is a graph of strong dominators for the KDE subsystem: See the problem, now? More on this at Roberto Di Cosmo (Paris Diderot / INRIA) Free Software Complexity May 17th / Toulouse 33 / 38

66 Let s try a more systemic approach Here is a graph of strong dominators for the KDE subsystem: See the problem, now? More on this at Roberto Di Cosmo (Paris Diderot / INRIA) Free Software Complexity May 17th / Toulouse 33 / 38

67 Generality of the approach Everything developed in Mancoosi is already relevant for some, and will be increasingly relevant to all component based technologies Linux : packages Eclipse : plugins Firefox : extensions Java : beans Web services : predictable service assembly Roberto Di Cosmo (Paris Diderot / INRIA) Free Software Complexity May 17th / Toulouse 34 / 38

68 Conclusions FOSS is reopening markets, and provinding new opportunities, if we can take over the challenges! They are many: scientific : master the complexity of FOSS educational : adapt CS curricula to the FOSS revolution economical : build an efficient, viable ecosystem of actors Roberto Di Cosmo (Paris Diderot / INRIA) Free Software Complexity May 17th / Toulouse 35 / 38

69 Outline 1 The Open Source Infrastructure and its Complexity 2 Coccinelle 3 Mancoosi 4 IRILL: research and innovation on Free Software Roberto Di Cosmo (Paris Diderot / INRIA) Free Software Complexity May 17th / Toulouse 36 / 38

70 IRILL in the big picture To boldly attack these challenging problems, Inria is contributing to create IRILL, together with other academic and industy partners: IRILL (Innovation et Recherche en Informatique sur le Logiciel Libre): a center to host researchers, teachers, engineers, and free software developers to work together on the new scientific, technological and educational challenges of Free Software See more info on Roberto Di Cosmo (Paris Diderot / INRIA) Free Software Complexity May 17th / Toulouse 37 / 38

71 Questions? Roberto Di Cosmo (Paris Diderot / INRIA) Free Software Complexity May 17th / Toulouse 38 / 38