Inner Secrets of GRAPNEL Code Generation. Daniel Drotos. Peter Kacsuk. University of Miskolc, Department of Automation

Transcription

1 Inner Secrets of GRAPNEL Code Generation Daniel Drotos Peter Kacsuk University of Miskolc, Department of Automation H-3515 Egyetemvaros, Miskolc, Hungary Abstract This paper describes structure and internal behavior of GRAPNEL programming environment Development of the GRAPNEL environment was supported by EC within COPERNICUS Research Project No 5383 This project has implemented a graphical programming environment for parallel programming The developed environment helps programmers to develop, write, execute, and verify parallel programs for distributed environments more quickly 1 Introduction GRAPNEL is a graphical programming language which has been developed to help programmers to develop parallel programs for distributed parallel environments Using grapnel the programmer can use graphical symbols to dene important structure of his/her program and processes and connections between the processes Only the communication actions must be dened graphically, the other parts of the code of the processes can be placed in a \sequential" blocks which can contain any regular source codes The developed environment consists of several tools GRED [5] is the user interface with a graphical editor and it is the integration platform of the other tools The programmer uses GRED to edit his/her program and invoke compiler programs and to start debugger [10, 11, 12], simulator [17, 18], mapping [8, 9, 14] and other tools [16] Graphical code helps the programmer to produce code more eciently Some tools had to be developed to help the computer to understand and handle the graphical code One of them is a translator tool which converts graphical representation of the program into ordinary C source code which is understandable by ordinary C compilers The code generated by the translator tool uses API function calls to implement actions dened in the graphical elements The API has been developed to hide details of target system and make it more easy to port the GRAPNEL system into a new target system Following sections describe source code generation and API system in more detail: Section 2 describes internal structure of GRAPNEL programs Section 3 describes code generator tool Section 4 describes structure of the generated programs and how their are using API functions to implement their functionalities 2 Layers of GRAPNEL programs Representation of GRAPNEL programs can be split to several layers (Figure 1) 21 Hardware layer The lowest layer is the hardware of the computer and its operating system GRAPNEL system uses message passing paradigm so the operating system should support messages between processes Most systems do it but in some cases it is dicult to use dierent IPCs of dierent systems Generally we use a special software layer on top of the operating system which hides the dierences of the message passing 22 Message passing layer This layer should be a widely used system The PVM or MPI is a good choice because they are ported

2 to a lot of operating systems Because this layer hides operating system dependencies the GRAPNEL system can be hardware and operating system independent 23 GRAPNEL API layer Because the previous layer can be implemented by dierent kind of message passing system, an other software layer is required which hides dependencies of the previous layer This layer is an Application Programming Interface and because its physical representation is a C library, it is called as Grapnel Library This API layer can support any kind of message passing system, eg PVM, MPI, or an operating system directly This API consist of GRAPNEL (or GRP shortly) functions and higher layers use these GRP functions to start processes and sending messages GRED GRP program is running Every time when higher layer calls a GRAPNEL function to start a process or send a message to somebody, the API function makes a record in the trace-le The API functions do not implement their own trace generation methods but they are using standard tracing tools instead to produce standard format of trace-le After the execution the generated trace-le can be visualized [3] 24 GRED layer This is the highest layer of the GRAPNEL system At this layer the program can be represented by graphically [2] This representation is easy to understand for the developer but it is not for programs Because of this the GRED software saves the graphical program in a plain text le called GRP le This le is used by programs and utilities of the GRAPNEL system One of these utilities is the Grapnel Compiler which can translate GRP le into C-source The generated sources use GRP function calls from GRAPNEL API to do their job The highest layer can be represented in three dierent form: API C-source Grapnel Library (API) PVM MPI Graphically It is used by the programmer to develop the program GRP le It is used by the GRAPNEL programs to process developed program C-source It is used by the system to do the work that must be done by the developed program 3 Translating graphical elements into C source les HARDWARE Hardware + OS Figure 1 Layers of GRAPNEL programs GRAPNEL API is the lowest layer which is really included into the GRAPNEL system and is developed by the development team The API for PVM is available already and support for other systems such as MPI and QNX operating system is under development The API has a very important functionality It is that it creates trace information during the developed The GRAPNEL system is a graphical programming language This means that a programmer can build up his/her program using graphical elements placing them on the screen and connecting them together In this context every graphical element represents a small or even a bigger piece of the program and the connections describe in which order these parts of the program must be executed The programmer uses a Graphical Editor (GRED [5]) instead a text editor when (s)he makes programs It is very easy to make programs in this way but unfortunately there is not an execution system that would be able to execute graphical symbols directly Developed translator program called Grapnel Compiler can translate the graphical representation of the program into the standard C source code [4] The programs generated by the translator can

3 be compiled with any standard C compiler and can be executed in general way First step of the translation is made by the graphical editor The editor makes a text le called GRP le that describes the whole system built by the programmer The GRP le contains information not only about the program but the screen of the editor, location of the windows and the graphical symbols, etc and information requested by the debugging, monitoring and animating tools The translator has to select information and pick up explanations about the program The syntax of the GRP le is very strictly and it is dened in BNF (Backus-Naur Form) It can be considered that the GRP le is the source of a program that uses the GRP syntax and the Grapnel Compiler has to translate it into the source of an other program that uses the syntax of the C programs The syntax denition of the GRP le allows Grapnel editor to split information about elements of the program into several parts and to place them into the GRP le separately This was the main problem during the implementation of the Grapnel Compiler 31 Lexical analysis Standard programming tools of the UNIX operating system [1] are used to implement the Grapnel Compiler The rst step of the implementation of any compiler is making a lexical analyzer The analyzer routine reads characters of the source code sequentially and tries to reconcile the read character string with the syntactical units dened in the BNF syntax These syntactical units are described in a special form and this description is used to make the lexical analyzer routine with the tool called lex The lex is a very effective programming facility that requires a description of the syntax rules and makes the lexical analyzer routine in C source Required description is very simply, for example let's see the following rule: [-+]?[0-9]+[eE][-+]?[0-9]+ { FloatParam= atof(yytext); return(floatnumber); } This rule is from the source of the lexical analyzer of the Grapnel Compiler and describes the syntax of the oating point numbers The left side of this rule is an expression The lex translates this expression into the C instructions and this code checks the input string and if it matches with the expression then executes actions described in the right hand side of the rule This action is a compound C instruction It converts the recognized input string that is in the yytext variable into the oating point number and places it in the FloatParam variable After the conversion the analyzer returns to the caller and informs it that the recognized token was a FLOATNUMBER The caller of the analyzer must make a decision that the token can appear in the source at the actual point of the processing or not If the analyzer founds a matched expression then it avoids to check of the next rules so the order of the denition of the rules is very important Last three rules are special in some respect [a-za-z_][0-9a-za-z_]* return(lookreserved()); \n LineNum++; ; The rst one of these three rules contain an expression that corresponds to the identiers and reserved words of the GRP le If the input may be an identi- er the analyzer calls a function made by the developer This function picks up the identier and looks for it in the database of the reserved words If the word is in the database then analyzer will return token attached to that reserved word otherwise it will return the token called ID that is a token of identiers The expression in the second rule will match with the end of the line character If analyzer detects the end of the line then increments the line counter The number of the actual line is very important information when the user or the Grapnel editor is informed about the wrong lines of the source GRP le The last rule has got a special expression that will match with every string read from the input le and force the analyzer to do nothing It is very important because without this action the analyzer copies unrecognized inputs for example the empty lines into the standard output This side eect is unexpected and can confuse the user of the compiler 32 First phase of translation As it has been explained earlier the GRP le may split information about elements of the program into several parts Because of this characteristic of the GRP le the Grapnel Compiler needs use two phases during the compilation In the rst phase the compiler scans the GRP le and collects information about the elements of the program such as processes, ports, connections, etc During this activity the compiler calls lexical analyzer to get the next token from the source and checks if it is valid token or not It is not so easy to write a C program that accomplishes this parsing operation There is an other UNIX programming tool called yacc that helps to realize this routine Yacc

4 uses a description of the syntax to implement the parser routine This description is very similar to the BNF Yacc parser denition le contains rules like BNF does Let's see an example: HeaderPart: HEADERPART {ErrorCode= ceneedbraceopen;} BRACEOPEN HeaderSList {ErrorCode= ceneedbraceclose;} BRACECLOSE {ErrorCode= ceneedapppart;} ; HeaderSList: HeaderSList HeaderSection ; Left side of the rules denes a non-terminal symbol that represents a rule of the syntax Right side describes the rule The description is a list of terminal and/or non-terminal symbols Terminal symbols are tokens returned by the lexical analyzer Every nonterminal symbol used in the rules must appear in the left side of one or more other rule Rules are closed with semicolon In the example shown above the HeaderPart is symbol of the part of the syntax that describes how the information about headers must be written in the GRP le This part of the le must begin with HEADERPART token This token belongs to reserved word \headerpart" The reserved word must be followed by a \f" character and a list of header sections This part of the le must be closed with a \g" character (token BRACECLOSE) In this rule the HeaderSList is a non-terminal symbol and it is described in an other rule Any part of the syntax can be represented by one or more rules We can write these alternative forms in one rule separated with \j" Description of the list of the header sections means that the list can be empty or can be a list followed by a header section In the rules we may write C instructions which will be executed after the recognition of the previous token Using this capability we implement actions that must be taken during the rst phase of the compilation The most important work of the translator is to build internal data structures that represent the program For example if the parser nds information about a process then it makes a new data structure and places information into it If the parser nds information about the same process during the scanning Graphical Editor Simulator GRP file Application: t4 { Programpart { Process: t41 HeaderSection { Global {@int i Mapping Tool Visualization Figure 2 GRP le used by GRAPNEL system then it picks up the information and extends the existing data structure and places new information into the extended structure Parser part of the Grapnel Compiler is written in C++ language using object oriented programming paradigm and it is available as a separate library which contains a class hierarchy This class library is used by other part of the GRAPNEL environment such as GRED and other tools to read in and handle GRP les or manipulate them in other ways (Figure 2) 33 Second phase of translation When all the requested information about processes, ports, and other program elements are collected from the GRP le, the second phase of the translation can be done Second phase is source code generation The Grapnel Compiler makes one source le for the application server program and one le for every process The generated source les are unique executable C programs that can communicate with each other using Grapnel system calls These function calls are inserted into the source les by the translator 331 C les generated by the translator The translator rst makes the code of the application server program The compiler uses a template le to make the source le This template le is inserted into

5 the executable code of the compiler The translator includes process creation functions into the template using information about processes found in process collection After the application server program the compiler generates source codes of the processes Every process is implemented in a separate source le Like the application template le, the process template le is inserted into to executable code of the compiler too Every source code of the processes starts with include section It depends on information located in the header section of the GRP le Next part is the denition part of the global variables This part and the beginning part of the main function where local variables and the channels are dened are included to the template by the translator After variable denitions the translator inserts C instructions into the template Several Grapnel system calls will be inserted before the code made by programmer These system calls start the process (inform application server that the process is being run) and initialize the channels used by the process To make the source code of these function calls the translator takes information from the collection of the ports that is built during the rst phase of the translation The functionality of the process is dened by the programmer and it is represented by the program items These items correspond to graphical elements of the program respectively Every item represents a small piece of the executable code and the connections between the items dene the order of the execution The items must be translated into the C code in the appropriate order First and last items in the chain have special type They used to sign the beginning and the end of the chain only The program items have connections These connections are references to other items The object of the process collects these references in a collection The translator looks for the rst item and starts the code generation using that item After the code generation the compiler looks for an other item connected to the actual one Every item must be processed only once There are several types of the program items and some of them must be processed recursively 332 Other les generated by the translator The code generator creates a \Makele" that can be used to produce the executables from the C code by invoking the \make" command The third type of les produced by the translator is a cross reference le (Figure 3) This le contains information about the correspondences between graphical program items in the GRAPNEL code and textual code fragments in the C les (ie which graphical program item a particular line in the generated C code belongs to) SEQ for (cyc= 1, { grp_pre_send( grp_pack( grp_post_send( grp_printf( Figure 3 Cross references between graphical and textual forms 4 Generated Programs and the GRAP- NEL API Grapnel applications are client-server programs (see Figure 4) It means that there is a server process in every application which administrates other \normal" processes Grapnel Compiler generates separated programs for every process of the application and an additional program for every applications The additional program will be the server of the application and programs of processes will work as clients of the server There are several channels that connect processes together Channels between processes are dened by the programmer There are other \pseudo" channels between server process and client processes 41 Tasks of the server program It checks the version number of the libraries which the program is linked with The server program refuses the execution if the program is linked with a wrong version of a library The server spawns the processes dened by the programmer in the developed application Static processes are started during start phase of the application Dynamic process creation is under development It is not required to start all the processes successfully, the server will continue but the application can work in wrong way When the server creates processes, it sends some messages to the newly spawned task and informs

6 hjskjk sd ksj ds skf jkllk jsd kjfd kjsk fd kjshf ksjhf ks skjf sjf sjks ks kfsks server with GRAPNEL library as well as some other libraries which implement functions to handle connections to other tools of the environment (Figure 5) client4 clent3 TARGET LIBRARY (eg pvm) Programs generated by grp2c GRAPNEL LIBRARY Target independent (grpstuff) client1 client2 Target dependent (eg grppvm) OTHER LIBRARIES (slib, gred) Figure 4 Client-server model of GRAP- NEL programs it about some important things such as the parameters which the application is invoked with After process creation phase, the server creates channels Channels are identied by a unique number, called tag Server attaches tags for every channels and sends them to the processes because they should know about tags of channels that they are connected to Channels dened by the programmer in the Graphical Editor are represented by black solid lines on Figure 4 Blue dashed lines represent predened channels between application server program and processes At last the server goes into a loop and listens for messages from the processes and serves their requests Processes can ask the server to do I/O operations, for example print out normal or error messages Processes normally must inform the server if they are going to die The server's main loop nishes when the last process exits 42 Using GRAPNEL API As mentioned before the programs generated by the Grapnel Compiler are using GRAPNEL API function calls to communicate each other and the server program The GRAPNEL API consists two dierent libraries, collections of target dependent and target independent functions The application must be linked Figure 5 Libraries in GRAPNEL system Functions dened in the GRAPNEL API are grouped into several classes 421 General functions Functions belonging to this class are for general use They are used to check version number of the libraries, parsing command line parameters and so on These functions are used by application server program and the processes as well 422 Functions for the server program Some functions of the GRAPNEL API including process and channel creation can be used by the application server program only Mapping tool of the GRAP- NEL environment [15] can analyze GRP le and produce mapping information for the system This information is automatically read by the application server program and used to spawn processes on specied CPU resource (execution resources are referenced as \host" in PVM context) Channel creation is a simple operation It sends some important data of the channel to the both processes that the channel connects together Data includes channel identier, and name and identier of the connected process Server's main loop is implemented in a GRAPNEL function It waits for messages from the processes and accepts requests to make I/O operations

7 423 Functions for the processes Some functions can be used in processes only They are receiving messages from application server program when the process starts and informs the server when the process quits The most important API functions are for communication Implemented message passing operations are SEND, RECEIVE, and ALTERNATIVE RECEIVE generated by the translator is the send function itself which sends the prepared message to addressed process This function call accepts a parameter which species if the process should be blocked on this send operation or not Let's suppose that following variables are dened in myproc process and we are going to send them to other process and receive data into them from the other process: int i1, i2; double d[5]; and the \protocol" attached to port 1 is: Figure 6 Connected processes Figure 6 shows connected processes Processes can have ports and ports of processes can be connected These ports and connections denes communication channels The Grapnel Compiler places dierent function in generated programs to prepare message and send it Preparing the message means picking up all the required data and packing them together into a message To produce proper API calls the Grapnel Compiler must know type and name of the variables which the programmer would like to send through the channel The programmer can attache additional information to graphical elements Data types must be attached to the ports and name of the variables must be attached to communication actions in processes Figure 7 shows communication actions Actions are connected to one of the available ports so the Grapnel Compiler can put required informations together Figure 7 Communication actions in process \myproc" on gure 6 SEND For send operation the Grapnel Compiler produces one or more pack API function calls to pack variables into the message Next function call double[5]; It denes data type as array of 5 double elements Port 2 denes: int[1]; int[1]; protocol for two independent integer variable Figure 7 shows communication actions in myproc process and describes which port they are connected to In O1 communication action i1 and i2 variables are dened to be sent Code produced by the code generator for this communication action is shown in Figure 8 There are some comments generated in C source code to help identify graphical blocks in the C source grp BeginBlock function call is related to trace le generation This function generates a special trace event to identify entering into a graphical block Next function calls prepare data to send, then pack data into the message and - nally send the message Number of packing function calls depends on number of data types attached to the port RECEIVE For receive operation the Grapnel Compiler produces one function call which receives the message and one or more unpack API function calls to pick up data fragments from the message and place them in variables There is a receive communication action in myproc process called I1 which is connected to other process through port 1 Figure 9 shows code generated by the Grapnel Compiler to implement I1 graphical block There is only one unpack function generated to pick up data from the message but it picks up 5 elements of the array together

8 /* Source of block 17 (CAO) Line= 112 */ #ifdef TapeInstrumented grp_beginblock(code_cao, 17); #endif grp_pre_send(&channs[to_myproc2_2], Magic MagicBlocked); grp_pack(&i1, (1), INT); grp_pack(&i2, (1), INT); grp_post_send(&channs[to_myproc2_2], Magic MagicBlocked); ; End of source of block 17 (CAO) Line= 123 */ Figure 8 Code generated by Grapnel Compiler for SEND operation Source of block 16 (CAI) Line= 100 */ #ifdef TapeInstrumented grp_beginblock(code_cai, 16); #endif grp_pre_recv(&channs[from_myproc2_1], &Grp_Recv/*magic*/); grp_unpack(&d, (5), DOUBLE); grp_post_recv(&channs[from_myproc2_1]); ; End of source of block 16 (CAI) Line= 110 */ Figure 9 Code generated by Grapnel Compiler for RECEIVE operation

9 ALTERNATIVE RECEIVE Alternative receive operation (Figure 10) is similar to simple receive but it accepts message from more than one processes system can be important for industrial real-time applications References [1] John R Levine, Tony Mason, Doug Brown: lex & yacc, O'Reilly & Associates, Inc, 1990, 1992 [2] G Dozsa, P Kacsuk, T Fadgyas, \GRADE: A Graphical Programming Environment for PVM Applications", In Proc of PDP'97: 5th EUROMI- CRO Workshop on Parallel and Distributed Processing, Jan 22-24, 1997, London, pp 358{365 Figure 10 ALT communication action in a process The translator generates dierent unpack API function calls for dierent ports but they are placed in a switch instruction and right ones are selected by port number where the actual message arrived in 424 User functions These functions can be used by the programmer in sequential blocks In the sequential blocks programmer can place any code fragments but communication User functions of GRAPNEL API can be used to ask server program to make I/O operations including printing out messages and reading data in 5 Conclusions Availability of powerful programming environments for heterogeneous networks is getting more and more important GRADE provides a complex programming environment where the programmer can concentrate on high level abstractions without worrying about the low level details of communication primitives Currently the GRADE environment supports PVM as target system and it runs on UNIX hosts Grapnel Compiler generates C source les from graphical representation of the program Graphical symbols are language independent so it is possible to modify the translator tool to generate source les for other programming languages The development team is going to support Fortran language which is still very important in high performance computing business New GRAPNEL API implementations are going to be developed as well to support more message passing systems for example MPI and QNX operating system Supporting QNX operating [3] Copernicus Program Software Engineering for Parallel Processing, Final Report, The PROVE Visualization Tool, March 1997, pp 86{92 [4] Copernicus Program Software Engineering for Parallel Processing, Final Report, The GRP2C Precompiler, March 1997, pp 38{89 [5] Copernicus Program Software Engineering for Parallel Processing, Final Report, The GRED Visual Editor, March 1997, pp 35{38 [6] P Kacsuk, G Dozsa and T Fadgyas: Designing parallel programs by the graphical language GRAPNEL, Microprocessing and Microprogramming, 41: , 1996 [7] P Kacsuk, JC Cunha, G Dozsa, J Lourenco, T Fadgyas, T Antao: A graphical development and debugging environment for parallel programs, Parallel Computing, 22: , 1997 [8] L Hluchy, M Dobrucky, J Astalos, Hybrid approach to task allocation in distributed systems, in Lecture Notes in Computer Science 1277, Springer 1997, pp210{216 [9] L Hluchy, M Dobrucky, J Astalos, Hybrid solving method for task allocation in distributed systems, in Proceedings of the Seventh International Conference on Articial Intelligence and Information - Control Systems of Robots, I Plander ed, World Scientic 1997, pp189{201 [10] J Cunha and J Lourenco: An Experiment in Tool Integration: the fddbgg Parallel and Distributed Debugger, EUROMICRO Journal of Systems Architecture, 2 nd Special Issue on Tools and Environments for Parallel Processing, 1997 [11] J C Cunha and J Lourenco: An Integrated Testing and Debugging Environment for Parallel and

10 Distributed Programs, EUROMICRO 97, Proceedings of the 23 rd EUROMICRO Conference, IEEE Computer Society Budapest, Hungary, 1997, pp 291{298 [12] J C Cunha and J Lourenco and T Ant~ao: A Debugging Engine for a Parallel and Distributed Environment, Proceedings of DAPSYS'96, 1st Austrian- Hungarian Workshop on Distributed and Parallel Systems, Hungarian Academy of Sciences KFKI, Miskolc, Hungary, 1996 [13] T Delaitre, S Poslad, S C Winter, Center for Parallel Computing, University of Westminster: Case Studies in Performance Modeling and Simulation of Distributed Information Systems, 14 th UK Teletrac symposium on Performance Engineering in Information systems, IEE Electronics and Communications Division, 22{26 March 1997, UMIST, Manchester [14] P Bouvry, J Chassing, D Trystram CWI (The Netherlands), IMAG-LMC (France): Ecient solutions for mapping parallel programs Proc f the EUROPAR'95 conference, Springer Verlag LNCS 966 [15] L Hluchy, M Dobrucky, D Dobrovodsky: Distributed Static Mapping and Dynamic Load Balancing Tools under PVM, DAPSYS'96 Proceedings, 1996 [16] Henryk Krawczyk, Bogdan Wiszniewski, Faculty of Electronics, Telecommunications and Informatics, Technical University of Gdansk: Interactive Testing Tool for Parallel Programs, Software Engineering for Parallel and Distributed Systems, I Jelly, I Gordon, P Croll eds Chapman Hall, London, 1996, pp 98{109 [17] Emilio Luque, Remo Suppi, Joan Sorribes: Models and Simulation, Internal research report Computer Architecture and Operating System Group Computer Science Dept University Autonoma of Barcelona (AUB) July 1994 [18] T Delaitre, E Luque, R Suppi, S Taylor: Simulation of Parallel Systems in SEPP, up'94-8 th Symposium on Computer & Microprocessor Applications October 1994 Budapest Hungary