Integer Linear Programming

Transcription

1 Integer Linear Programming Micha Elsner April 5, 2017

2 2 Integer linear programming A framework for inference: Reading: Clarke and Lapata 2008 Global Inference for Sentence Compression An Integer Linear Programming Approach ILP details in section 3

3 3 Discrete optimization problems Inference in the HMM/CRF turned out to be tricky: Had to search over exponentially large space of T 1:n Developed dynamic program to do so in polynomial time (Viterbi) Relied on the Markov property to break problem into parts Algorithmic efficiency tied to model structure Modifying model requires (potentially inefficient) changes to inference These inference issues are general in structured models (output consists of many separate but related predictions)

4 4 The dynamic programming approach Model must be designed with Markov structure so DP is efficient Modifications require complicated reprogramming Deal with inefficiency with approximate methods Beam search: zero out low numbers in trellis Faster, but no guarantee of getting best tag sequence anymore Or with exact methods (A-star, coarse-to-fine) Generally take significant effort to design A lot of research effort directed toward inference! Resulting algorithms can be fast and good even for large problems

5 The declarative approach ILP (and related tools) follow a different mindset: Model design can be (mostly) arbitrary Modifications generally easy to include Always find the exact best solution But: Performance exponential in worst case Generally: small problems run really fast Phase transition at some point Hard to predict where Anything slightly larger is impossible Depending on problem, ILP may be fine for your final version Or only ok for prototype Either way, spend less time on inference during development 5

6 6 Integer linear programming Integer linear programming Black-box toolkit for solving constrained (discrete) optimization problems: Problem is to assign values to a set of variables x 1:n Which take integer values (often 0/1) (Or real values actually easier but less useful) To maximize/minimize a linear function f (x 1:n ) So that solution does not violate linear constraints

7 Linearity Linear function Equation of a line is y = m 1 x 1 + m 2 x b Sum of terms Variables x multiplied by constant coefficients m Constants b Not allowed: variable times variable: x 1 x 2 Variable times itself (powers): x 2 Log, exp, other special fns Dot-product form (as in max-ent) is linear 7

8 8 A trivial example Variables: x: integer in [0, 10] Objective: max x

9 9 Also trivial Variables: x: integer in [0, 10] Objective: max x Such that: x 7

10 10 Example PuLP code (also on Carmen) #LpMaximize means maximize the objective problem = LpProblem("counting", LpMaximize) #variable named "counter" in [0,10] var1 = LpVariable("counter", 0, 10, LpInteger) #add the objective problem += var1 #next we add a constraint problem += (var1 <= 7) #solve the problem... problem.solve() #check that this worked print "Problem status:",\ LpStatus[problem.status] #access information about the solution print "Value of", var1.name,\ "is", var1.varvalue

11 11 A more complicated problem Making furniture (From Clarke and Lapata, from Winston and Venkataraman) Table requires 1hr labor, 9sq ft wood: sell for $8 Chair requires 1hr labor, 5sq ft wood: sell for $5 We have 6hr labor, 45 sq ft wood Want to maximize profits How many tables shall we make? How many chairs?

12 12 Setting up Variables: Integer number of tables t (0 ) Integer number of chairs c (0 ) Note that fractional furniture is not possible Objective: (maximize our profit) We make $8 for a table Profit from tables is 8t (This is linear, right?) max 8t + 5c What would happen if we solved it right now?

13 13 Constraints We can t use more than 6hr: A table requires 1hr Hours from tables: 1t max 8t + 5c 1t + 1c 6 (Do we also need 1t + 1c 0?) We can t use more than 45sq ft wood: 9t + 5c 45

14 14 The problem max 8t + 5c st: 1t + 1ct 6 9t + 5c 45 t [0, ] c [0, ] Solution: t = 5, c = 0 for profit of $40 Can t make anything else; 5 9 = 45 uses up the wood

15 15 Practical: the Viterbi decoder We ll step through the problem of building a (really slow) Viterbi decoder: T 1:n = argmax P(T 1:n W 1:n ) Let s start with the emissions (no transitions): Variables: x i,t [0, 1] is 1 if Ti = t Objective (sum over all words, over all tags for that word): What will we get? max n log(p(w i T i = t))x i,t i=1 t

16 16 Constraint: you have to actually tag something! n max log(p(w i T i = t))x i,t i=1 t st: x i,t 1 t i We add i different constraints (one per word) Each constraint involves T variables (one per tag type)

17 17 Transitions What we need: If Ti = t and Ti+1 = t, objective has a term for P(t t) Transition probabilities Straightforward way to write this: What s wrong with this? log(p(t t))x i,t x i+1,t

18 18 Dealing with non-linearity Solution: more variables Let v i,t,t [0, 1] be 1 if T i = t and T i+1 = t If we could make the vs do this, we could have: Objective: max n log(p(w i T i = t))x i,t + i=1 t n i=1 t t log(p(t t))v i,t,t

19 19 Making the v do what we want x i,t x i+1,t sum v i,t,t We see that v i,t,t = 1 if the sum of the x is 2: v i,t,t x i,t + x i+1,t 1 Encodes that x i,t x i+1,t v i,t,t Do we also need that v i,t,t x i,t x i+1,t? We encode logic as arithmetic!

20 20 The ILP n max log(p(w i T i = t))x i,t + i=1 t n i=1 t t log(p(t t))v i,t,t st: x i,t 1 i t v i,t,t x i,t + x i+1,t 1 i, t, t x i,t [0, 1] v i,t,t [0, 1] Plus some messing around with start and end dummy states

21 21 Modifying inference Easy to use these tools to add: Long-distance dependencies Force tag to have a value No more than one NNP in a string If a word appears repeatedly in a sentence, has to get the same tag each time Useful for names, unknowns All these can be difficult in the dynamic program On the other hand, it s really slow My Viterbi running in about.1-.2 sec on sample sentences My ILP usually sec, sometimes 90 sec (However, this may not be the best ILP encoding)

22 22 For instance Finkel and Manning Enforcing transitivity in coreference resolution Problem: Link all mentions in document that refer to the same entity Hillary Clinton said she was looking forward to leaving the Cabinet to spend more time with Bill Clinton. The former Secretary of State... Important to get consistent clusters Clinton ok with Hillary and Bill she ok with Clinton, Hillary, not Bill Don t link she Clinton Bill Used classifier to decide whether to link each pair ILP to extract final clustering maximizing pairwise link probs

23 23 Another example Elsner and Santhanam Learning to fuse disparate sentences Not necessarily the best work, but easy to steal slides! Input The bodies showed signs of torture. They were left on the side of a highway in Chilpancingo, in the southern state of Guerrero, state police said. Output The bodies of the men, which showed signs of torture, were left on the side of a highway in Chilpancingo, state police told Reuters.

24 Generic framework for sentence fusion 24

25 25 Simple paraphrasing Add relative clause arcs between subjects and verbs (Alternates police said / police, who said )

26 26 Merging/selection A fused tree: a set of arcs to keep/exclude The bodies, which showed signs of torture, were left by the side of a highway

27 27 Constraints Not every set of selected arcs is valid...

28 28 Decipherment Ravi and Knight Attacking Decipherment problems optimally with low-order N-gram models Task: decrypt substitution cipher Source text is in English But letters are replaced according to a key Insight: can use LM over characters to decide how English a proposed decipherment is... Solution is most English decipherment

29 29 ILP setup Variable for each transition at each time (as in our HMM) Key vars (encode which characters are enciphered as which)