Information Processing Letters Vol. 30, No. 2, pp , January Acad. Andrei Ershov, ed. Partial Evaluation of Pattern Matching in Strings

Transcription

1 Information Processing Letters Vol. 30, No. 2, pp , January 1989 Acad. Andrei Ershov, ed. Partial Evaluation of Pattern Matching in Strings Charles Consel Olivier Danvy LITP DIKU { Computer Science Department University of Paris VI University of Copenhagen 4 place Jussieu, couloir 45-55, 2e etage Universitetsparken Paris Cedex Copenhagen France Denmark chac@litp.ibp.fr danvy@diku.dk Abstract This article describes how automatically specializing a fairly naive pattern matcher by partial evaluation leads to the Knuth, Morris & Pratt algorithm. Interestingly enough, no theorem proving is needed to achieve the partial evaluation, as was previously argued, and it is sucient to identify a static component in the computation to get the result { a deterministic nite automaton. This experiment illustrates how a small insight and partial evaluation can achieve a non-trivial result. Keywords Partial evaluation, program specialization, Knuth, Morris & Pratt algorithm. Introduction Finding whether a given string occurs within another is a typical problem to which partial evaluation can be applied. Given a general pattern matching program PM whose semantics is a function: Pattern-String Subject-String! Boolean one can specialize PM with respect to a given pattern string, and obtain by partial evaluation a program whose semantics has the functionality: where, for some alphabet, Subject-String! Boolean Pattern-String = Subject-String = We propose such an approach here, and more precisely to demonstrate that specializing a fairly simpleminded pattern matcher yields the ecient Knuth, Morris & Pratt algorithm [Knuth, Morris & Pratt 77]. Partial evaluation is a semantics-preserving program transformation based on propagating constants, unfolding and specializing procedures. Presently, partially evaluating the program PM with respect to a pattern string pat: 1

2 Mix PM pat gives a so-called residual program whose functionality is: Subject-String! Boolean and which performs all the operations of PM on the subject string { all those depending only on the pattern have been achieved by the partial evaluator. Further, self-applying the partial evaluator [Futamura 71], that is, specializing the partial evaluator with respect to the pattern matching program: Mix Mix PM makes it generate a program compiling a pattern into the program above 1. The semantics of the residual program is a function: Pattern-String! (Subject-String! Boolean) which illustrates that a self-applicable partial evaluator implements the Curry function. Self-applicable partial evaluators exist since [Jones, Sestoft & Sndergaard 85]. All the programs described here have been run with the partial evaluator of [Jones, Sestoft & Sndergaard 88] and with the Schism partial evaluator [Consel 88], both of which are self-applicable. We will use Mix generically for naming any partial evaluator. Below, we reproduce the parts processed with Schism because they are written in Scheme [Rees & Clinger 86] and are thus readable without further description. Section 1 presents the naive algorithm and section 2 proposes one single transformation and its partial evaluation. Section 4 optimizes further the construction. Section 5 compares the present approach with related work and puts it into perspective. 1 A naive approach A fairly simple-minded approach to the problem of nding whether a given pattern occurs within another string is to iteratively compare it with every sux of the second string. (define (kmp p d) (define (loop p d pp dd) (or (null? p) (or (null? p) (start p d))) (define (start p d) (loop (cdr p) (cdr d) pp dd) (restart pp (cdr dd)))))) ; naive! (restart p d))) (define (restart p d) (loop (cdr p) (cdr d) p d) (start p (cdr d)))) This strategy is not optimal. It is a strong starting point in [Knuth, Morris & Pratt 77] to have remarked that where to restart after a mismatch could be deduced from the pattern alone. The result is a much better algorithm: a preprocessing time of order O(m) and a matching time of order O(n) rather than O(m n), m and n being the length of the pattern string and the subject string, respectively. Let us illustrate this, by partially evaluating that program with respect to a pattern string (a b a). By Mix's binding time analysis [Jones, Sestoft & Sndergaard 85] [Jones, Sestoft & Sndergaard 88], 1 In Mix Mix PM, Mix designates a program text and Mix designates its meaning, that is an input-output function. The result is a (residual) program text. 2

3 variables p and pp are identied as static { that is, computable from the pattern alone. The residual program's functions are versions of the original functions specialized to values of p and pp, renamed to be distinguished from the original versions. They are of form kmp p (d), loop p;pp (d; dd), start p (d) and restart p (d), where pp = (a b a) and p ranges over suxes of (a b a). (define (kmp-0 d) (start-1 d)) (define (start-1 d) (define (restart-2 d) (loop-3 (cdr d) d) (define (loop-3 d dd) (loop-4 (cdr d) dd) (restart-2 (cdr dd))))) (define (loop-4 d dd) (loop-5 (cdr d) dd) (restart-2 (cdr dd))))) (define (loop-5 d dd) '#t) The resulting code is poor. Up to the occurrence of a mismatch, a substring of the subject string is known (since it has been proven iteratively to be equal to the pattern so far, and the pattern is known) but this knowledge is thrown away and the subject string re-scanned at the state start-1. 2 A still naive approach Noticing that up to the point of mismatch, the pattern string is identical to the subject string, we propose to iterate rst on two instances of the pattern string up to the point of mismatch and then on pattern and the subject strings, rather than only iterating along the pattern and the subject strings. This is exactly equivalent since the pattern and the subject string are identical up to that mismatch point. To express it graphically: Pattern String mismatch Subject String On the left of the mismatch, the pattern string and the subject string are equal. What we propose is, rather than shifting the pattern one place to the right and re-scanning the pattern and the subject strings up to the mismatch point and further: ----> ----> Pattern String Subject String > ----> we instead match two instances of the pattern string up to the point of mismatch and then scan the pattern and the subject strings further. ----> ----> Pattern String Subject String Pattern String > ----> To match the pattern against a part of itself, any algorithm will do. We use the same strategy of going iteratively along the pattern string. The number of iterations, however, is already given (the length of the substring that matched before the mismatch occurred). 3

4 (define (kmp p d) (define (restart p d) (or (null? p) (start p d))) (or (null? (cdr p)) (define (start p d) (loop (cdr p) (cdr d) p))) (start p (cdr d)))) (restart p d))) (define (loop p d pp) (or (null? (cdr p)) (loop (cdr p) (cdr d) pp))) (let ((np (static-kmp pp (cdr pp) (- (length (cdr pp)) (length p))))) (if (equal? np pp) ; any match at all? (if (equal? (car pp) (car p)) ; where to continue? (start pp (cdr d)) (restart pp d)) (loop np d pp))))) (define (static-kmp p d n) (static-loop p d n p d n)) ; n records the number of elements to match (define (static-loop p d n pp dd nn) (if (zero? n) p (static-loop (cdr p) (cdr d) (sub1 n) pp dd nn) (static-kmp pp (cdr dd) (sub1 nn))))) This meaning-preserving transformation has the eect that in the rst part of the algorithm (loop), control is determined by the subject string and in the second (static-loop), it is determined by the pattern string, that is, intensionally, by the \static" variables p, pp, n and nn. That situation is ideal for a partial evaluator. 3 Partial evaluation Partially evaluating the code given in the end of section 2 with respect to a pattern has the eect to specialize it and produces a surprisingly optimized code. In particular, it does not \back up" the input string. Here is the new residual program for the string (a b a): (define (kmp-0 d) (define (loop-3 d) (start-1 d)) (define (start-1 d) (loop-4 (cdr d))) (define (loop-4 d) (define (restart-2 d) (or (equal? 'a (car d)) (loop-3 (cdr d))) The partial evaluator has processed everything that depends on the pattern string, leaving only the actions depending on the subject string. The resulting program expects a string and scans it linearly. 4

5 What we get is a deterministic nite automaton: it is deterministic because the original program is deterministic; and it is nite because the pattern is nite. At this point we can state a conclusion: the eect of the Knuth, Morris & Pratt algorithm has been automatically achieved by simply identifying a part of the program that depends solely on the pattern string, which is static { that is: available at partial evaluation time. In particular, we have not needed to compute any next or failure table to drive the pattern matching. More generally, this has been done quite naively and automatically rather than cleverly and by hand, as in the original algorithm. Further, any pattern will be compiled into a non-backtracking matcher, running in time O(length(subject-string)). Still one information has not been exploited: the character causing the mismatch. 4 Further optimization Because the character causing the mismatch is ignored, we can expect some redundancy to remain in the residual program, and actually this redundancy can be pointed out. Specializing the program with respect to (a b a b c) yields: (define (kmp-0 d) (define (loop-4 d) (start-1 d)) (define (start-1 d) (loop-5 (cdr d))) (define (loop-5 d) (define (restart-2 d) (loop-6 (cdr d))) (loop-3 (cdr d))) (loop-3 d))) (define (loop-6 d) (define (loop-3 d) (or (equal? 'c (car d)) (loop-4 d))) (loop-4 (cdr d))) where in state loop-5, we go to state loop-3 and repeat the same test, to see whether (car d) is b. We can use the information about the last mismatch, taking it into account while statically matching the pattern against itself: (define (static-loop p d n pp dd nn) (if (zero? n) (if (and (> nn 0) ; possible to continue? (equal? (car p) (car d))) ; mismatch again? (static-restart pp (cdr dd) (sub1 nn)) p) (static-loop (cdr p) (cdr d) (sub1 n) pp dd nn) (static-restart pp (cdr dd) (sub1 nn))))) Specializing this new algorithm with respect to (a b a b c), we obtain a residual program identical to the one above except at the state loop-5, where the redundancy has vanished: 5

6 (define (loop-5 d) (loop-6 (cdr d))) The full eect of the Knuth, Morris & Pratt algorithm is now achieved, and we include in the appendix the same optimal code as section 3 of [Knuth, Morris & Pratt 77]. 5 Comparison with related work [Knuth, Morris & Pratt 77] is the seminal paper on fast pattern matching in strings. Because partial evaluation is essentially program specialization, we get algorithmic (\compiled") versions of the original next table that determines which character in the pattern should be tested after a mismatch. Further, by partially evaluating the original Knuth, Morris & Pratt algorithm with respect to a next table, we have obtained a residual program structurally equivalent to the one above. During the rst workshop on Partial Evaluation and Mixed Computation [Bjrner, Ershov & Jones 88], the question was asked recurrently whether a partial evaluator could treat the Knuth, Morris & Pratt algorithm. It is argued in [Futamura & Nogi 88] that it needs using a theorem prover. This note shows that theorem proving it is not necessary for this, and that separating out the portion of the algorithm that will be repeated statically is sucient to obtain straightforwardly the Knuth, Morris & Pratt algorithm. Conclusion This paper illustrates how partial evaluation can be used for obtaining the Knuth, Morris & Pratt pattern matching algorithm from a fairly naive method. Interestingly enough, this has been done automatically whereas it was done by hand originally. To conclude, let us underline that the Knuth, Morris & Pratt algorithm is two-fold: it oers both ecient processing in time O(m) and ecient matching in time O(n), m and n being the length of the pattern string and the subject string, respectively. Presently, we get the latter by partial evaluation, so the generated matcher also runs in time O(n). Generating a matcher generator can be done at an earlier stage by self-applying Mix. Generating the matcher 2 takes more time than O(m) (apparently O(m 2 )), which is less ecient than the sophisticated Knuth, Morris & Pratt construction of the next table. This is not unreasonable, since a partial evaluator can specialize any program at all. It will not matter in the common situation where the same (short) pattern is to be matched against many (long) subject strings. Finally let us point out that a pattern matching program could be specialized as well with respect to the subject string. However the residual program can be huge. This would need some better insight into sux trees [Weiner 72]. It is our hope that this work contributes to present partial evaluation as an active help for creating and designing algorithms and programs. Acknowledgements To Neil D. Jones for his thoughtful interaction and support, and Peter Sestoft and Andrzej Filinski for their re-reading. This work has been achieved while the rst author was visiting DIKU. 2 Either by applying the matcher generator to a pattern string or by partially evaluating the general pattern matcher with respect to a pattern string. 6

7 References [Bjrner, Ershov & Jones 88] Dines Bjrner, Andrei P. Ershov, Neil D. Jones (eds.): Partial Evaluation and Mixed Computation, Gl. Averns, Denmark, North-Holland (1988) [Consel 88] Charles Consel: New Insights into Partial Evaluation: the Schism Experiment, proceedings of the second European Symposium on Programming ESOP '88, Lecture Notes in Computer Science No 300 pp , Harald Ganzinger (ed.), Nancy, France (March 1988) [Futamura 71] Yoshihiko Futamura: Partial Evaluation of Computation Process { an Approach to a Compiler-Compiler, Systems, Computers, Controls Vol. 2, No 5 pp (1971) [Futamura & Nogi 88] Yoshihiko Futamura, Kenroku Nogi: Generalized Partial Computation, from Partial Evaluation and Mixed Computation pp , Dines Bjrner, Andrei P. Ershov and Neil D. Jones (eds.), North-Holland (1988) [Knuth, Morris & Pratt 77] Donald E. Knuth, James H. Morris, Vaughan R. Pratt: Fast Pattern Matching in Strings, SIAM Journal on Computing, Vol. 6, No 2 pp (June 1977) [Jones, Sestoft & Sndergaard 85] Neil D. Jones, Peter Sestoft, Harald Sndergaard: An Experiment in Partial Evaluation: the Generation of a Compiler Generator, proceedings of the First International Conference on Rewriting Techniques and Applications, Jean-Pierre Jouannaud (ed.), Lecture Notes in Computer Science No 202 pp , Dijon, France (June 1985) [Jones, Sestoft & Sndergaard 88] Neil D. Jones, Peter Sestoft, Harald Sndergaard: MIX: a Self- Applicable Partial Evaluator for Experiments in Compiler Generation, to appear in the international journal LISP and Symbolic Computation (1988) [Rees & Clinger 86] Jonathan Rees, William Clinger (eds.): Revised 3 Report on the Algorithmic Language Scheme, Sigplan Notices, Vol. 21, No 12 pp (December 1986) [Weiner 72] P. Weiner: Linear Pattern Matching Algorithms, IEEE Symposium on Switching and Automata Theory, Vol. 14 pp 1-11, IEEE, New York (1972) 7

8 Appendix This appendix presents the residual program obtained by specializing our pattern matching program with respect to the string (a b c a b c a c a b). This example is interesting because it is isomorphic to the one in section 3 of [Knuth, Morris & Pratt 77]. (define (kmp-0 d) (start-1 d)) (define (loop-7 d) (define (start-1 d) (if (equal? 'c (car d)) (loop-8 (cdr d))) (define (restart-2 d) (define (loop-8 d) (loop-3 (cdr d))) (loop-9 (cdr d))) (define (loop-3 d) (define (loop-9 d) (if (equal? 'c (car d)) (loop-4 (cdr d))) (loop-10 (cdr d))) (define (loop-4 d) (loop-6 d))) (if (equal? 'c (car d)) (define (loop-10 d) (loop-5 (cdr d))) (loop-11 (cdr d))) (define (loop-5 d) (define (loop-11 d) (or (equal? 'b (car d)) (loop-6 (cdr d))) ) (define (loop-6 d) (loop-7 (cdr d))) 8