Pre-Conference Workshops

Transcription

1 Pre-Conference Workshops Michael Bussieck Steve Dirkse Fred Fiand Lutz Westermann GAMS Development Corp. GAMS Software GmbH

2 Outline Part I: An Introduction to GAMS Part II: Stochastic programming in GAMS Part III: The GAMS (Object-Oriented) API's Part IV: Code embedding in GAMS 2

3 3

4 Motivation Avoid Unreadable/Slow Code GAMS code for parallel assignment and equation definition is compact, elegant, and efficient GAMS uses relational data tables as a base data structure Traditional data structure are not available in GAMS No arrays, lists, dictionaries, trees, graphs, GAMS can represent such traditional data structures but GAMS code becomes quickly unreadable t(tt) = t(tt-1); // advances set element t to t+1 Performance with naïve representation is very inefficient t(tt) = ord(tt)=tcnt; // advances set element t to t+1 Writing code that executes efficiently requires deep understanding of underlying GAMS internal data structures and often results in even more unreadable code 4

5 Motivation Data Input/Transformation at Compile Time GAMS data input (ASCII) follows strict syntax Practical GAMS models get data (via ASCII input files) that is often not in a proper shape Hence GAMS code is often augmented with execution of scripts and programs to get data files into a GAMS readable shape GAMS even ships a selection of POSIX text utilities (sed, grep, awk, ) on Windows to support a somewhat standardized way of transforming text files into GAMS readable format Scripts spawned by GAMS cannot (easily) access data that is already available in GAMS GAMS has no string editing facilities to e.g. modify labels change content of compile time variables Solution : $xxx new and weird compile time constructs, e.g. $setnames, $splitoption, 5

6 Motivation Other Connecting libraries for special algorithms (e.g. graph algorithms like connected components, matrix operations like Cholesky factorization) to GAMS is not easy Current solution has issues $unload/$call/$load or execute_unload/execute/execute_load Performance: disk IO + process creation Knowledge of data API (GDX or OO-API) Remapping of relational data (plus concept of UELs) into other data structures Add compile time directives to perform a single special task (e.g. $splitoption) Introduce unreadable option or put_utility syntax to perform a single special task (e.g. option a<b;) Object Oriented API/Framework versus Embedded Code OO-API: Framework in control Embedded Code: GAMS in control 6

7 Embedded Code Support the use of external code during GAMS compile and execution time Provide support for off-line debugging of embedded code Share GAMS symbols (sets, parameters, variables, and equations) structure and content with the external code in memory Communication of the data between GAMS and the embedded code inspired by the existing interface to GDX in many ways: Records access by both labels and label indices Data in GAMS can be merged with or replaced by data from embedded code Data from embedded code can be send to GAMS database filtered or domain checked Provide automatically generated, additional source code for common tasks Allows the user to concentrate on the task at hand and not the mechanics 7

8 Example Calculate i j 4-quantiles of k numbers: Set i /... /, j /... /, k /... / Table p(i,j,k) k1 k2... i1.j i1.j ; Set q; Parameter quantiles(i,j,q); $onembeddedcode Python: import pandas as pd df = pd.dataframe(list(gams.get('p'))) q = df.groupby([0,1]).quantile([0,.25,.5,.75,1]) gams.set('quantiles', [ (*k[:-1],str(k[-1]),v[3]) for k,v in q.iterrows() ]) $offembeddedcode q<quantiles quantiles display q, quantiles; 8

9 Split Example Data Set cc / "France - Paris", "France - Lille", "France - Toulouse", "Spain - Madrid", "Spain - Cordoba", "Spain - Seville", "Spain - Bilbao", "USA - Washington DC", "USA - Houston", "USA - New York", "Germany - Berlin", "Germany - Munich", "Germany - Bonn" / country / system.empty / city / system.empty / mcccountry(cc,country) mcccity (cc,city); Hands-On embeddedsplit 9

10 Split Example Embedded Code $onembeddedcode Python: country = set() city = set() mcccountry = [] mcccity = [] for cc in gams.get("cc"): r = str.split(cc, " - ", 1) country.add(r[0]) city.add(r[1]) mcccountry.append((cc,r[0])) mcccity.append((cc,r[1])) gams.set("country",list(country)) gams.set("city",list(city)) gams.set("mcccountry",mcccountry) gams.set("mcccity",mcccity) $offembeddedcode country city mcccountry mcccity 10

11 Split Example Output Display country, city; SET country France, USA, Spain, Germany SET city Seville, Washington DC, New York, Paris Munich, Madrid, Toulouse, Berlin Bonn, Lille, Houston, Bilbao Cordoba 11

12 Sorting Example Set i / i1*i10 /; Parameter a(i) Random Data aindex(i) Sorted index of a; a(i) = uniformint(1,10*card(i)); embeddedcode Python: a = list(gams.get("a")) tmp = [r[0] for r in sorted(enumerate(a), key=lambda x:x[1][-1])] aindex = len(a)*[-1] for idx in range(len(tmp)): aindex[tmp[idx]] = (a[tmp[idx]][0], idx+1) gams.set("aindex",aindex) endembeddedcode aindex Hands-On embeddedsort 12

13 Sorting Example Output Display a, aindex; PARAMETER a Random Data , , , , , , , , , PARAMETER aindex Sorted Index of a , , , , , , , , ,

14 Exchange via Files $onembeddedcode Python: 10 f = open('i.txt', 'w') for i in range(int(gams.arguments)): f.write('i'+str(i+1)+'\n') f.close() $offembeddedcode Set i / $include i.txt /; Display i; SET i i1, i2, i3, i4, i5, i6, i7, i8, i9, i10 Hands-On exchangeviafiles 14

15 Exchange via Environment Variables Set i / i1*i5 /; Parameter b(i) / i1 2, i2 7, i3 59, i4 2, i5 47 /; Set k "from 0 to max(b)" / k0*k? /; $onembeddedcode Python: import os kmax = int(max([b[1] for b in list(gams.get("b"))])) gams.printlog('max value in b is ' + str(kmax)) os.environ["maxb"] = str(kmax) $offembeddedcode $if x%sysenv.maxb%==x $abort MAXB is not set Set k "from 0 to max(b)" / k0*k%sysenv.maxb% /; Scalar card_k; card_k = card(k); Display card_k; PARAMETER card_k = Hands-On exchangeviaenvvars 15

16 Multiple Independent Python Sessions $if not %sysenv.gmspythonmultinst%==1 $abort.noerror Set command line option pymultinst=1 Set i / i1*i3 /; Parameter h(i) ord_i / 0 /; loop(i, ord_i = ord(i); embeddedcode Python: i = int(list(gams.get("ord_i"))[0]) gams.printlog(str(i)) pauseembeddedcode h(i) = embeddedhandle; ); loop(i, continueembeddedcode h(i): gams.printlog(str(i)) endembeddedcode ); Hands-On embeddedmultiinstance 16

17 Multiple Independent Python Sessions Log --- Initialize embedded library embpycclib.dll --- Execute embedded library embpycclib.dll Initialize embedded library embpycclib.dll --- Execute embedded library embpycclib.dll Initialize embedded library embpycclib.dll --- Execute embedded library embpycclib.dll Execute embedded library embpycclib.dll Execute embedded library embpycclib.dll Execute embedded library embpycclib.dll

18 Performance Considerations Set i / i1*i50 /, p(i,i); Alias (i,ii); Parameter c(i,i); c(i,ii) = uniform(-50,50); Set iter / 1*100 /; Scalar tcost, mintcost / +inf /; loop(iter, embeddedcode Python: import random i = list(gams.get("i")) p = list(i) random.shuffle(p) for idx in range(len(i)): p[idx] = (i[idx], p[idx]) gams.set("p", p) endembeddedcode p tcost = sum(p, c(p)); if (tcost < mintcost, mintcost = tcost); ); Display mintcost; EXECUTION TIME = SECONDS Hands-On performance 18

19 Performance Considerations Set i / i1*i50 /, p(i,i); Alias (i,ii); Parameter c(i,i); c(i,ii) = uniform(-50,50); embeddedcode Python: import random pauseembeddedcode Set iter / 1*1000 /; Scalar tcost, mintcost / +inf /; loop(iter, continueembeddedcode: i = list(gams.get("i")) p = list(i) random.shuffle(p) for idx in range(len(i)): p[idx] = (i[idx], p[idx]) gams.set("p", p) pauseembeddedcode p tcost = sum(p, c(p)); if (tcost < mintcost, mintcost = tcost); ); continueembeddedcode: pass endembeddedcode Display mintcost; EXECUTION TIME = SECONDS 19

20 Performance Considerations Set i / i1*i50 /, p(i,i); Alias (i,ii); Parameter c(i,i); c(i,ii) = uniform(-50,50); embeddedcode Python: import random i = list(gams.get("i")) pauseembeddedcode Set iter / 1*1000 /; Scalar tcost, mintcost / +inf /; loop(iter, continueembeddedcode Python: p = list(i) random.shuffle(p) for idx in range(len(i)): p[idx] = (i[idx], p[idx]) gams.set("p", p) pauseembeddedcode p tcost = sum(p, c(p)); if (tcost < mintcost, mintcost = tcost); ); continueembeddedcode: pass endembeddedcode Display mintcost; EXECUTION TIME = SECONDS 20

21 Performance Considerations Set i / i1*i50 /, p(i,i); Alias (i,ii); Parameter c(i,i); c(i,ii) = uniform(-50,50); embeddedcode Python: import random i = list(gams.get("i",keytype=keytype.int)) pauseembeddedcode Set iter / 1*1000 /; Scalar tcost, mintcost / +inf /; loop(iter, continueembeddedcode Python: p = list(i) random.shuffle(p) for idx in range(len(i)): p[idx] = (i[idx], p[idx]) gams.set("p", p) pauseembeddedcode p tcost = sum(p, c(p)); if (tcost < mintcost, mintcost = tcost); ); continueembeddedcode: pass endembeddedcode Display mintcost; EXECUTION TIME = SECONDS 21

22 Syntax: GAMS Compile Time: $onembeddedcode[s V] Python: [arguments] Python code {Python code} $offembeddedcode {output symbol} $onembeddedcode[s] Python: [arguments] Starts a section with Python code Parameter substitution is activated The optional arguments can be accessed in the Python code $onembeddedcodev Python: [arguments] Same as $onembeddedcode but parameter substitution is disabled (the Python code is passed on verbatim) $offembeddedcode {output symbol} Ends a section with Python code The optional output symbol(s) get updated in the GAMS database 22

23 Syntax: GAMS Execution Time: EmbeddedCode[S V] Python: [arguments] Python code {Python code} endembeddedcode {output symbol} EmbeddedCode[S] Python: [arguments] Starts a section with Python code Parameter substitution is activated The optional arguments can be accessed in the Python code EmbeddedCodeV Python: [arguments] Same as EmbeddedCode but parameter substitution is disabled (the Python code is passed on verbatim) endembeddedcode {output symbol} Ends a section with Python code The optional output symbol(s) get updated in the GAMS database 23

24 Syntax: GAMS Execution Time: pauseembeddedcode {output symbols} continueembeddedcode[s V] [handle]: [arguments] pauseembeddedcode {output symbol} Pauses a section with Python code The optional output symbol(s) get updated in the GAMS database continueembeddedcode[s] [handle]: [arguments] Continues a previously paused section with Python code Parameter substitution is activated The optional arguments can be accessed in the Python code The optional handle (pointing to a specific paused embedded code section) could be retrieved by the function embeddedhandle. If omitted, the last section paused will be continued. continueembeddedcodev [handle]: [arguments] Same as continueembeddedcode but parameter substitution is disabled (the Python code is passed on verbatim) 24

25 Syntax: Python The Python Class ECGamsDatabase serves as interface between GAMS and Python. An instance of this class is automatically created when an embedded code section is entered and can be accessed using the identifier gams. Several methods can be used to interact with GAMS: gams.get(symbolname, [ ]) Retrieves iterable object representing the symbol symbolname Several optional parameters allow to modify format of the data gams.set(symbolname, data[, merge][, domcheck]) Sets data for the GAMS symbol symbolname Data takes a Python list of items representing records of the symbol Optional parameter merge specifies if data in a GAMS symbol is merged or replaced Optional parameter domcheck specifies if Domain Checking is applied 25

26 Syntax: Python gams.getuel(idx) Returns the label corresponding to the label index idx gams.mergeuel(label) Adds label to the GAMS universe if it was unknown and returns the corresponding label index Note: New labels cannot be added at execution time gams.getuelcount() Returns the number of labels gams.printlog(msg) Print msg to log gams.arguments Contains the arguments that were passed to the Python interpreter at start-up of the embedded code section gams.epsaszero Flag to read GAMS EPS as 0 [True] or as a small number ( E-300) [False] gams._debug Debug flag for additional output 26

27 Some Examples of Python Embedded Code Splitting of labels (compile time) Permutation Sorting Calculation of quantiles Power set Matching Parsing of specially structured ASCII input TSP subtour elimination Benders Decomposition using Message Passing Interface (MPI) on High-Performance Computing (HPC) infrastructure 27

28 Plot example Hands-On meanvarplot 28

29 Next steps More examples High Performance Libraries for specific tasks FORTRAN (Factorization of matrix) C/C++ (Expansion Planning Power Systems) Support of other popular frameworks (compiled and interpreted) C/C++ C#/.NET, Java, R, Connect of powerful libraries e.g. boost::graph, Provide a configurable build system that supports building the required libraries (for compiled languages) at GAMS compile time Provide a documented API to allow integration of individual user embedded code libraries Asynchronous/parallel use of embedded code This feature is currently in beta status. Any feedback to support@gams.com is highly appreciated. 29

30 Thank You Meet us at the GAMS booth! GAMS Development Corp. GAMS Software GmbH