Inferred System Descriptions ELS <

Transcription

1 Inferred System Descriptions <

2 Perspective github : nine repositories; (SLOC) = 188,758 dydra : :depends-on x 22; (SLOC) = 317,746 github : < dydra : <

3 Perspective github : nine repositories; (SLOC) = 188,758 dydra : :depends-on x 22; (SLOC) = 317,746 build systems build (1984) defsystem (1981) define-system (1984) asdf (2002) asdf (2010) asdf (2014) mkdefsystem (1990) xcvb (2009) asdf : < github : < dydra : <

4 Perspective github : nine repositories; (SLOC) = 188,758 dydra : :depends-on x 22; (SLOC) = 317,746 build systems build (1984) defsystem (1981) define-system (1984) asdf (2002) asdf (2010) asdf (2014) mkdefsystem (1990) xcvb (2009) :depends-on : why? asdf : < github : < dydra : <

5 Perspective: must it be manual? :depends-on : why? build (robbins 1984) "patching" "more precise change analysis" description of large systems (pitman 1984) "transitivity of dependency information" "tight bonding to syntax" build : < dls : <

6 Results systems

7 Perspective: should be possible? "patching" "more precise change analysis" "transitivity of dependency information" "loose bonding to syntax" LISP a programming system called LISP (for LISt Processor) LISP : <

8 Perspective: should be possible? "patching" "more precise change analysis" "transitivity of dependency information" "loose bonding to syntax" LISP a programming system called LISP (for LISt Processor) "Recursive Functions of Symbolic Expressions..." LISP : <

9 Abstraction Alternatives ignore any system definition macro-expansion just to normalize syntax emulate just package symbol identity extract relations from source code do not compile do not build do not execute

10 Abstraction Vocabulary doap : : packaged product level scro : : just conceptual structure compre : : components and depends, but no hierarchy, versions, or generation swonto : : data processing in jena iao : : the entities are too generic swo : swoinowl/inferredswo/swo_inferred.owl : overdrawn : 4338 entities sioc : : process oriented oflossc : (404) : sioc specialized for open source process code : : provides starting point

11 Abstraction Vocabulary string pathname Subclass of FaslFile (external) Namespace (external) File (external) haspart generated Package (external) Subclass of wasderiv... Compilation (external) Subclass of Subclass of Subclass of compiled datetime Subclass of endedattime SourceFile (external) Collection (external) Constituent (external) Activity Subclass of Class (external) authoredon (external) Thing filecontai... Subclass of Subclass of Subclass of Subclass of Entity (external) Operator (external) Test (external) called Subclass of Subclass of Function (external) Subclass of Subclass of Subclass of Macro (external) GenericFunction (external) hasmethod Method (external)

12 Goals identify all packages to load to use a given operator construct system based on requirements at any of three levels derive changes to load between two versions per system per file per function (test)

13 Results: Metadata

14 Results: Metadata

15 Start with Text file2 test (defun iex:function (arg) (+ (iex:method1 arg) 2)) file1 (defclass iex:class () ((slot1 :initform 1 :accessor iex:class-slot1) (slot2 :initform 2 :accessor iex:class-slot2))) (defgeneric iex:method1 (object) (:method ((object iex:class)) (+ 1 (iex:method1 (iex:class-slot1 object)))) (:method ((object t)) object)) (defmethod iex:method2 ((object iex:class)) (+ (iex:method1 object) (iex:class-slot2 object))) file2 (defun test.function1 () (eql (iex:method1 (make-instance 'iex:class)) 2)) (defun test.function2 () (eql (iex:method2 (make-instance 'iex:class)) 4)) (system :example :components ((:file "file1") (:file "file2" :depends-on "file1") (:file "file3" :depends-on "file2") (:file "test" :depends-on "file2 "file1")

16 Just the Text file2 test (defun iex:function (arg) (+ (iex:method1 arg) 2)) file1 (defclass iex:class () ((slot1 :initform 1 :accessor iex:class-slot1) (slot2 :initform 2 :accessor iex:class-slot2))) (defgeneric iex:method1 (object) (:method ((object iex:class)) (+ 1 (iex:method1 (iex:class-slot1 object)))) (:method ((object t)) object)) (defmethod iex:method2 ((object iex:class)) (+ (iex:method1 object) (iex:class-slot2 object))) file2 (defun test.function1 () (eql (iex:method1 (make-instance 'iex:class)) 2)) (defun test.function2 () (eql (iex:method2 (make-instance 'iex:class)) 4)) (system :example :components ((:file "file1") (:file "file2" :depends-on "file1") (:file "file3" :depends-on "file2") (:file "test" :depends-on "file2 "file1")

17 Abstract Symbols and Relations example Package file1.lisp haspart file2.lisp haspart file3.lisp haspart package.lisp haspart test.lisp haspart ISDF-EXAMPLE-IMPLEMENTATION contains ISDF-EXAMPLE contains ISDF-EXAMPLE-TEST contains File " T10:23:29Z"^^< authoredon CLASS CLASS-SLOT1 CLASS-SLOT2 METHOD1 METHOD1 METHOD1 " T10:22:26Z"^^< authoredon METHOD2 " T10:22:26Z"^^< authoredon FUNCTION " T10:22:26Z"^^< authoredon COMMON-LISP-USER " T10:22:26Z"^^< authoredon TEST.FUNCTION1 TEST.FUNCTION2 Namespace Class Function Method GenericFunction ISDF-EXAMPLE:CLASS-SLOT1+CLASS ISDF-EXAMPLE:CLASS-SLOT2+CLASS ISDF-EXAMPLE:METHOD2+CLASS

18 Identify Types and Relations file1.lisp file2.lisp file3.lisp test.lisp FUNCTION TEST.FUNCTION2 TEST.FUNCTION1 METHOD2 METHOD1 CLASS-SLOT2 CLASS-SLOT1

19 Establish Source References file2.lisp file1.lisp test.lisp file3.lisp METHOD2 CLASS-SLOT2 METHOD1 TEST.FUNCTION2 TEST.FUNCTION1 FUNCTION CLASS-SLOT1

20 Capabilities: Infer Source Requirements file2.lisp file1.lisp file3.lisp test.lisp

21 Process : Quicklisp as Corpus $ ls /tmp/quicklisp/ wc -l 1669 $ ls quicklisp/versions/ / wc -l 1317 $ find quicklisp/versions/ / -name '*.asd' wc -l 3134 $ find quicklisp/versions/ / -name '*' - d wc -l 5839 $ find quicklisp/versions/ / -name '*.asd' \ while read dir; do dirname $dir; done \ sort uniq wc -l 1670 $ find quicklisp/versions/ / -name '*.lisp' wc -l $/development/source/library/org/quicklisp/versions/ # sloccount.... Total Physical Source Lines of Code (SLOC) = 4,307,928

22 Process : Crawl Source Code * (time (scm:abstract-system (ensure-system "quicklisp" :location #p"quicklisp/versions/ /"))) Evaluation took: seconds of real time seconds of total run time ( user, system) [ Run times consist of seconds GC time, and seconds non-gc time. ] 14.83% CPU 3,444,829 forms interpreted 17 lambdas converted 7,624,481,182,395 processor cycles 5,746,702,672 bytes consed * (/ ) *

23 Process : Result Repository

24 Process : Infer Dependency prefix ssd: < prefix code: < clear silent graph <urn:dydra:file-requirements> ; insert { graph <urn:dydra:file-requirements> {?definitionsource ssd:?referencesource } } where{ select?definitionsource?referencesource where { graph?package {?definitionsource code:?definition. # eliminate function/function definitions?definitionsource a < code:?reference.?referencesource a < code:?definition. filter(?definitionsource!=?referencesource ) } } }

25 Process : Infer Dependency prefix ssd: < prefix code: < clear silent graph <urn:dydra:system-requirements> ; insert { graph <urn:dydra:system-requirements> {?definitionpackage ssd:?referencepackage.?referencepackage ssd:requires?definitionpackage } } where{ select?definitionpackage?referencepackage where { { graph?definitionpackage {?definitionsource code:?definition.?definitionsource a < # eliminate function/function definitions } } { graph?referencepackage {?referencesource code:?reference.?referencesource a < code:?definition. } } filter (?definitionpackage!=?referencepackage) } }

26 Results: Quicklisp Graphs * (quicklisp-sparql " select?p (count(?o) as?count) FROM <urn:dydra:all> where {?s?p?o } group by?p order by desc(?count)" :message "predicate count") predicate count p count < label < < comment < < < < < endedattime 10 generated 10 used 10 wasderivedfrom 10

27 Results: Quicklisp Graphs * (quicklisp-sparql " select count(*) where { graph <urn:dydra:file-requirements> {?s?p?o} }" :message "intra-system file cross-references") intra-system file cross-references COUNT

28 Results: Quicklisp Graphs * (quicklisp-sparql " prefix ssd: < select count(*) where { graph <urn:dydra:system-requirements> {?s ssd:requires?o} }" :message "inter-system cross-references") inter-system cross-references COUNT * (quicklisp-sparql " prefix ssd: < select count(*) where { graph <urn:dydra:system-requirements> {?s ssd:?o} }" :message "inter-system cross-references") inter-system cross-references COUNT

29 BKNR: Infer Source Requirements blob.lisp tests.lisp xml-tutorial.lisp object-tests.lisp category-index.lisp xml-object.lisp xml-update.lisp xml-export.lisp tutorial.lisp object.lisp actor.lisp tutorial.lisp indices-tests.lisp xml-import.lisp convert.lisp txn.lisp encoding-test.lisp acl-mp-compat.lisp indices.lisp xml-class.lisp reader.lisp encoding.lisp runtime-statistics.lisp class.lisp capability.lisp skip-list.lisp indexed-class.lisp money.lisp utils.lisp protocol.lisp dump-core.lisp fswrap.lisp mop-bug.lisp shop.lisp slot-a LISP : <

30 BKNR: Load Order * (bknr-sparql " prefix ssd: < prefix prov: < prefix pav: < prefix code: < select distinct?loadfilename?dependentsource #?sequencenr from <urn:dydra:all> where { { select * where { bind ( ' T00:00:00Z'^^xsd:dateTime as?system_load_time ) { select?source where {?source a < optional {?othersource ssd:?source } filter (!bound(?othersource) ) } }?source ssd:+?dependentsource. bind (rdf:seq(xsd:integer) as?sequencenr ) } }?source pav:authoredon?timemodified. filter (?timemodified >?SYSTEM_LOAD_TIME )?dependentsource code:location?sourcepathname.?dependentsource pav:authoredon?sourcetime.?dependentsource rdfs:label?sourcefile. optional { [] prov:used?dependentsource ; prov:generated?dependentbinary ; prov:endedattime?binarytime.?dependentbinary code:location?binarypathname.?dependentbinary rdfs:label?binaryfile. } bind (if( bound(?binarytime), if ((?binarytime >?sourcetime),?binaryfile,?sourcefile ),?sourcefile ) as?loadfilename ) } order by desc(?sequencenr) " :message "source/binary load list (but absent binaries)")

31 BKNR: Load Order source/binary load list (but absent binaries) loadfilename dependentsource convert.lisp encoding-test.lisp object.lisp txn.lisp blob.lisp object-tests.lisp tests.lisp tutorial.lisp encoding.lisp xml-object.lisp xml-tutorial.lisp xml-export.lisp xml-import.lisp xml-update.lisp indices.lisp xml-class.lisp category-index.lisp indexed-class.lisp indices-tests.lisp tutorial.lisp capability.lisp class.lisp actor.lisp NIL *

32 Deficiencies macros ambiguous packages : non-universal symbols non-lisp syntax ".lisp" file content colleen git/sample.uc.lisp bknr-datastore git/src/xml-impex/tutorial.lisp read-eval read-time, compile-time, load-time, run-time no tracking for defparameter/-constant/-var circularity implicit package definitions ambiguous methods call sites

33 Deficiencies macros ambiguous packages : non-universal symbols non-lisp syntax ".lisp" file content colleen git/sample.uc.lisp bknr-datastore git/src/xml-impex/tutorial.lisp read-eval read-time, compile-time, load-time, run-time no tracking for defparameter/-constant/-var circularity implicit package definitions ambiguous methods call sites

34 Deficiencies : ambiguous methods bknr-datastore git/src/indices-tests.lisp (defmethod run-test :around ((test index-test-class) &optional (output *debug-io*))... ) unit-test git/unit-test.lisp (defun run-all-tests (&key (verbose nil) (output *standard-output*) (unit :all)) (let (*unit-test-results*) (run-test test output)... ) would require either propagation or run-time instrumentation to distinguish specific methods for call sites.

35 Thank you <