Outline. Two combinatorial optimization problems in machine learning. Talk objectives. Grammar induction. DFA induction.

Transcription

1 Outline Two comintoril optimiztion prolems in mchine lerning 1 Feture selection ICTEAM Institute Université ctholique de Louvin Belgium My 1, 011 P. Dupont (UCL Mchine Lerning Group) Comintoril optimiztion in ML My 1, P. Dupont (UCL Mchine Lerning Group) Comintoril optimiztion in ML My 1, Tlk ojectives Descrie simple ML prolems nd formulte them s constrined comintoril optimiztion prolems 1 Feture selection Trigger discussions to see whether CP pproch my help to etter ddress them nd possily ootstrp from ML to CP Grmmr induction Also known s grmmticl inference Grmmr induction is out lerning forml grmmr from set of positive strings from its lnguge, nd possily negtive strings s well Dt Induction Grmmr S >S S > λ The positive nd negtive strings form lerning smple nd the grmmr, or n lterntive representtion, generlizes it Often simplest generliztion is sought (Ockhm s rzor formlized in computtionl lerning theory) Proilistic extensions nd sttisticl estimtion lgorithms hve een proposed P. Dupont (UCL Mchine Lerning Group) Comintoril optimiztion in ML My 1, 011. P. Dupont (UCL Mchine Lerning Group) Comintoril optimiztion in ML My 1, 011.

2 The miniml DFA consistency prolem Lerning regulr lnguge is the most studied cse Interesting pplictions in computtionl iology, nturl lnguge processing, softwre engineering,... A regulr lnguge cn e equivlently represented y cnonicl Deterministic Finite-stte Automton (DFA) L = ( ) S λ A A B B B A 0 1 Lerning regulr lnguge solving the miniml DFA consistency prolem Note: theoreticl results show tht solving this NP-hrd prolem on growing lerning smple leds to the correct lnguge identifiction in finite time [, 1] P. Dupont (UCL Mchine Lerning Group) Comintoril optimiztion in ML My 1, 011. Stte-merging lgorithm Algorithm STATE-MERGING DFA INDUCTION ALGORITHM Input: A positive nd negtive smple (S +, S ) Output: A DFA A consistent with (S +, S ) // Compute PTA, let N denote the numer of its sttes PTA Initilize(S + ); π {{0}, {1},..., {N 1}} // Min stte-merging loop while (B i, B j ) ChoosePir(π) do π new Merge(π, B i, B j ) if Comptile(PTA/π new, S ) then π π new return PTA/π P. Dupont (UCL Mchine Lerning Group) Comintoril optimiztion in ML My 1, 011. Lnguge generliztion through stte merging The Merge function lso reduces non-determinism S + = {λ,,,, } Prefix-Tree Acceptor (PTA) Quotient utomton {,} {3,} S = {, } Under resonle ssumptions, the trget mchine is in the PTA prtition set [] Merging 3 nd Merging nd (for determiniztion) Merging 8 nd 0 9 Merging nd 9 0 Merging 9 nd 0 Merging 10 nd P. Dupont (UCL Mchine Lerning Group) Comintoril optimiztion in ML My 1, 011. P. Dupont (UCL Mchine Lerning Group) Comintoril optimiztion in ML My 1,

3 An lterntive representtion: grph coloring prolem The Stmin winner S + = {λ,,,, } S = {, } 8 The winning lgorithm DFASAT from Mrijn Heule nd Sicco Verwer (Delft, Leuven) preliminry version ws proposed in [9] mix of stte merging + grph coloring + reduction to SAT Augmented PTA with positively ccepting sttes (= grey) nd negtively ccepting sttes (= lck) The Merge function reduces non-determinism nd checks such coloring constrints Sttes hving different colors my not e merged A miniml grph coloring prolem Find deterministic grph with miniml numer of nodes stisfying the coloring constrints [, 3] P. Dupont (UCL Mchine Lerning Group) Comintoril optimiztion in ML My 1, P. Dupont (UCL Mchine Lerning Group) Comintoril optimiztion in ML My 1, Chllenge Solve grid of prolems for incresing lphet size nd decresing lerning smple sizes [1] Why CP looks interesting to tckle this prolem? This comintoril optimiztion prolem is CSP DFASAT uses some ingredients lso found in CP A dedicted prolem representtion Redundnt cluses Symmetry Constrint propgtion is nturl in this prolem {, 8} implies {0, } incomptiility! Note: the competition is over ut you cn still try to outperform the winning lgorithm! stmin.chefe.net P. Dupont (UCL Mchine Lerning Group) Comintoril optimiztion in ML My 1, Some form of cnnot-link constrints on grph structure + determinism Mndtory merge (must link) constrints hve lso een proposed [10] P. Dupont (UCL Mchine Lerning Group) Comintoril optimiztion in ML My 1,

4 Why is it chllenging? The serch spce is the prtition set of the PTA stte set A concrete exmple from Stmin: 1, 000 positive nd negtive lerning strings The ugmented PTA hs 0, 000 sttes The numer of prtitions of set of m elements into k non empty susets is O( k m k! ) (exct computtion through Stirling numer) m = 0, 000 ; k = mchines with 0 sttes (nd yet, one should serch other trget sizes s well) The more lerning dt you get the lrger the serch spce while it should e simpler from ML viewpoint P. Dupont (UCL Mchine Lerning Group) Comintoril optimiztion in ML My 1, A motivting exmple Feture selection Moleculr iology in one slide! Gene expressions vry for mny different, ut possily dependent, resons: chemicl or physicl environment of the cells, growth of the orgnism, regultion of complex metolic processes, susceptiility to develop some illness or to respond to tretment,... P. Dupont (UCL Mchine Lerning Group) Comintoril optimiztion in ML My 1, Outline Feture selection DNA microrrys Feture selection 1 DNA chips mesure the level of expression of ll genes in single experiment Feture selection P. Dupont (UCL Mchine Lerning Group) Comintoril optimiztion in ML My 1, P. Dupont (UCL Mchine Lerning Group) Comintoril optimiztion in ML My 1,

5 Feture selection Gene selection: mchine lerning viewpoint Feture selection Issues with t-test selection gene 1 gene... gene d clss lel smple 1 x 1,1 x 1,... x 1,d y 1 smple x,1 x,... x,d y smple n x n,1 x n,... x n,d y n The numer d of genes or proe sets,000 The numer n of smples (tissues, ptients) 100 The clss lels y come from clinicl sttus: responsive or not to tretment, good or d dignosis/prognosis, type of pthology,... Biomrker Selection Find smll suset of ( 0) genes to predict the outcome y of new smples C 0, possiilities... you need relile estimtes of the mens nd vrinces of ech feture (hrd with few smples) you need to correct for multiple testing the selection is univrite the dependencies etween fetures re not considered P. Dupont (UCL Mchine Lerning Group) Comintoril optimiztion in ML My 1, P. Dupont (UCL Mchine Lerning Group) Comintoril optimiztion in ML My 1, Feture selection A sic feture selection: t-test relevnce index Mny feture selection methods hve een proposed [, 13] A sic pproch Compute the men feture vlues in ech clss Assess whether the mens significntly differ etween clsses select the top rnked fetures ccording to the p-vlues of t-test P. Dupont (UCL Mchine Lerning Group) Comintoril optimiztion in ML My 1, Feture selection Mximum relevnce minimum redundncy [1] Multivrite ojective Notes: Find suset S X of k mximlly relevnt nd minimlly redundnt fetures Relevnce cn e mesured y the mutul informtion with the response I(S; Y ) Redundncy cn e mesured y the mutul informtion etween vriles I(S 1,..., S k ) Mutul informtion is difficult to estimte in high dimensions ut pproximtions or lterntive mesures (e.g. rnk correltion) exist The mrmr pproch uses greedy serch to optimize this ojective Question: Could CP help to etter solve this CSP? P. Dupont (UCL Mchine Lerning Group) Comintoril optimiztion in ML My 1,

6 Feture selection If stndrd feture selection looks too simple to you Some references I Feture selection Questions How to use some uncertin nd prtil prior knowledge out relevnt fetures? or out feture dependencies? How to select common fetures on distinct ut relted tsks (trnsfer or multi-tsk lerning)? Note: mthemticl progrmming pproches hve een proposed to ddress those prolems [8,, 11]. Cn CP complement or outperform those methods? [1] D. Angluin, On the complexity of minimum inference of regulr sets, Informtion nd Control 39 (198), [] F. Coste nd J. Nicols, How considering incomptile stte mergings my reduce the serch tree, Grmmticl Inference, ICGI 98 (Ames, Iow), Lecture Notes in Artificil Intelligence, no. 133, Springer Verlg, 1998, pp [3] P. Dupont, B. Lmeu, C. Dms, nd A. vn Lmsweerde, The QSM lgorithm nd its ppliction to softwre ehvior model induction, Applied Artificil Intelligence (008), 11. [] P. Dupont, L. Miclet, nd E. Vidl, Wht is the serch spce of the regulr inference?, Grmmticl Inference nd Applictions, ICGI 9 (Alicnte, Spin), Lecture Notes in Artificil Intelligence, no. 8, Springer Verlg, 199, pp. 3. P. Dupont (UCL Mchine Lerning Group) Comintoril optimiztion in ML My 1, P. Dupont (UCL Mchine Lerning Group) Comintoril optimiztion in ML My 1, Conclusions Feture selection Some references II Feture selection nd feture selection cn e formulted s CSPs These re comintoril optimiztion prolems with very lrge serch spces, even with reltively smll lerning smples Those prolems could trigger dditionl collortions etween CP nd ML [] E.M. Gold, Complexity of utomton identifiction from given dt, Informtion nd Control 3 (198), [] I. Guyon, S. Gunn, M. Nikrvesh, nd L. Zdeh (eds.), Feture extrction, foundtions nd pplictions, Series Studies in Fuzziness nd Soft Computing, vol. 0, Springer, 00. [] T. Helleputte nd P. Dupont, Feture selection y trnsfer lerning with liner regulrized models, Europen Conference on Mchine Lerning, Lecture Notes in Artificil Intelligence, no. 81, 009, pp. 33. [8] T. Helleputte nd P. Dupont, Prtilly supervised feture selection with regulrized liner models, Interntionl Conference on Mchine Lerning, 009. P. Dupont (UCL Mchine Lerning Group) Comintoril optimiztion in ML My 1, 011. P. Dupont (UCL Mchine Lerning Group) Comintoril optimiztion in ML My 1, 011.

7 Some references III Feture selection [9] M. Heule nd S. Verwer, Exct DFA identifiction using SAT solvers, Interntionl Colloquium on Grmmticl Inference, Lecture Notes in Artificil Intelligence, vol. 339, Springer Verlg, 010, pp. 9. [10] B. Lmeu, C. Dms, nd P. Dupont, Stte-merging DFA induction lgorithms with mndtory merge constrints, Lecture Notes in Artificil Intelligence, vol. 8, 008, pp [11] G. Oozinski, B. Tskr, nd M.I. Jordn, Joint covrite selection nd joint suspce selection for multiple clssifiction prolems, Sttistics nd Computing (009), 1. P. Dupont (UCL Mchine Lerning Group) Comintoril optimiztion in ML My 1, 011. Some references IV Feture selection [1] H. Peng, F. Long, nd C. Ding, Feture selection sed on mutul informtion criteri of mx-dependency, mx-relevnce, nd min-redundncy, IEEE Trnsctions on Pttern Anlysis nd Mchine Intelligence (00), no. 8, [13] Y. Seys, I. Inz, nd P. Lrrñg, A review of feture selection techniques in ioinformtics, Bioinformtics 3 (00), no. 19, 0 1. [1] N. Wlkinshw, K. Bogdnov, C. Dms, B. Lmeu, nd P. Dupont, A frmework for the competitive evlution of model inference techniques, 1st Interntionl workshop on Model Inference In Testing, 010. P. Dupont (UCL Mchine Lerning Group) Comintoril optimiztion in ML My 1, 011.