APAN Conference 2000, Beijing. Ninf Project

Transcription

1 APAN Conference 2000, Beijing Ninf Project Kento Aida(4), Atsuko Takefusa(4), Hirotaka Ogawa(4), Osamu Tatebe(1), Hidemoto Nakada(1), Hiromitsu Takagi(1), Yoshio Tanaka (1), Satoshi Matsuoka(4), Mitsuhisa Sato(2), Satoshi Sekiguchi(1), Umpei Nagashima(3) Electrotechnical Laboratory(1) Real World Computing Partnership(2) National Institute of Materials and Chemical Research(3) Tokyo Institute of Technology(4) URL:

2 Towards Global Computing Rapid increase in speed and availability of WAN enables: sharing distributed data resources , file archives, WWW sharing distributed computational power Grid, Global Computing Starting Grid / global computing projects

3 Global Computing client request server client WAN result result server client Network based Information Library network infrastructure for the Grid RPC based system Ninf Ninf, NetSolve, RCS, Legion, Javelin, Globus,...

4 Outline overview of Ninf system Ninf RPC protocol Client API Providing Ninf library program applications scheduling Metaserver Bricks conclusions

5 Ninf Overview Ninf DB data Meta Meta Internet Meta results Other Global Computing Systems e.g., NetSolve via Adapters Ninf Computational Ninf Ninf Library Ninf Library (Ninf Ninf Library (Ninf Executable) Ninf Library (Ninf Ninf Library (Ninf Executable) Library (Ninf (Ninf Executable) Ninf Client Ninf query Client Library request : Ninf_call( linpack,..); : Program Ninf RPC protocol

6 Ninf Overview (cont( cont d.) Ninf server + Ninf RPC protocol transparent execution of Ninf library program on computational server Ninf database server direct query on accurate constant database Ninf Client Interface easy-to-use programming interface Ninf Metaserver scheduling of computation, asynchronous and automatic parallel computation,

7 Ninf RPC Protocol Exchange interface information at run-time No need to generate client stub routines (cf. SunRPC) No need to modify a client program when server s libraries are updated. Client Program Ninf library program Client Library Interface Info Stub Program Interface Info Ninf Interface Info Interface Request Interface Info. Argument Result

8 Ninf Client API A client user only needs to specify the name of libraries to invoke. Ninf_call( ninf://host:port/entry_name, ninf://host:port/ arg, ) Implemented API: C, C++, Fortran, Java, Lisp,Mathematica, Excel,... Client Ninf_call

9 Ninf Client API (cont( cont d.) Example double A[n][n],B[n][n],C[n][n]; /* /* Data Decl.*/ :: dmmul(n,a,b,c); /* /* Call local function*/ Ninfy double A[n][n],B[n][n],C[n][n]; /* /* Data Decl.*/ :: Ninf_call( dmmul,n,a,b,c); /* /* Call Ninf Executable*/

10 Direct Web Access URL can be specified as an argument. computation of Ninf library using data on a specified Web server (ex.) important constant for physics or chemistry storing interim results to a Web Ninf_call( dmmul, n, n, WEBSERVER/A,B, C); Client Program Request B C Ninf WEBSERVER A Ninf Ninf Executable (dmmul)

11 Ninf Client API(2) - asynchronous call - Asynchronous Call Ninf_call_async( FUNC,...); Waiting the reply Ninf_wait(ID); Ninf_wait_all(); Ninf_wait_and(IDList, len); Ninf_wait_or(IDList, len); Ninf_cancel(ID); Client A B Ninf_call_async Ninf_call_async Ninf_wait_all

12 Ninf Client API(3) - Transaction- Aggregate invocation of user specified code region Macrodataflow execution by Metaserver Ninf_transaction_start(); Ninf_call( dmmul,n,a,b,c); Ninf_call( dmmul,n,d,e,f); Ninf_call( dmmul,n,c,f,g); Ninf_transaction_end(); A B D F dmmul dmmul C F dmmul G

13 Ninf Client API(4) - Callback - side routine can callback client side routine. (ex.) Display of interim results of computation on servers to a client machine Client Ninf_call CallbcakFunc void CallbackFunc(...){. /* /* define callback routine */ */ } Ninf_call( Func, arg..,.., CallbackFunc); /* /* call with pointer to to the function */ */

14 Providing Ninf Library (1)Writing interface description for a library (computational) program in Ninf IDL Ninf IDL file (2)Running Ninf interface generator stub programs and Makefile (3)Compiling the library program and linking with the stub programs Ninf executables (4)Registering the Ninf executables with Ninf server

15 Generating Ninf executables Ninf Clients Ninf_call("goo",...) Ninf_call("bar",...) Ninf_call("foo",...) Ninf IDL file xxx.idl Ninf_gen stub main programs Ninf _stub_foo.c _stub_bar.c _stub_goo.c module.mak stubs.dir stubs.alias _stub_foo _stub_bar _stub_goo Ninfserver.conf Library program yyy.a yyy.a

16 Ninf Interface Description (Ninf IDL) Define dmmul(long mode_in int int n, n, mode_in double A[n][n], mode_in double B[n][n], mode_out double C[n][n]) description Required libxxx.o CalcOrder n^3 Calls C dmmul(n,a,b,c); IDL information: library function s name, and its alias (Define) arguments access mode, data type (mode_in, out, inout,...) required library for the routine (Required) computation order (CalcOrder) source language (Calls)

17 Application on Ninf: NinfCalc+ Matrix Calculator implemented as an Applet in browser linear system solver, eigen-value problem, click and quick operation direct Web access Huge matrices are stored to Web servers. interactive control of huge matrix calculation via thin network Web Browser NinfCalc+ Ninf perform C=AxB Web server matrices A matrices B Data Storage matrices C

18 Application on Ninf : Excel Ninf Arguments for Ninf_call are specified on the Excel worksheet A B C D E F Ninf_call( dmmul, 2, 2, A, A, B, B, C) C) Ninf C= A x B

19 Application on Ninf : NetCFD Ninf computational component for CFD Parallelized CFD program is ninfied on Ninf server. providing an interface to a parallel CFD program running on MPP, PC cluster,... Callback function

20 Ninf over TransPAC Demonstration of Ninf system over TransPAC SC98, SC99 Demo. at SC98

21 Scheduling An effective scheduling is required! /Network status dynamically change. Status information is distributed in WAN. s may be located inside Firewall. Our approaches Metaserver monitoring server/network status, performing scheduling development of effective scheduling systems Bricks simulating scheduling in Grid modeling computation in Grid, discussion of scheduling algorithm by simulation

22 Meta Architecture Client Side Directory Service Meta Side Proxy Proxy Client Scheduler Probe Module Proxy Proxy Client Client Proxy Schedule query Data Throughput Measurement Load query Proxy

23 Scheduling Algorithm Metaserver needs to select the most suitable server for client jobs using effective scheduling algorithm. Performance evaluation/estimation for scheduling algorithm measurement on real Grid system practical measurement difficult to perform large-scale experiments difficult to have reproducible results partial solution

24 Bricks Simulation system for scheduling in Grid Queuing Model A global computing system is modeled by queuing network Features evaluation/estimation of scheduling performance by simulation verification of existing scheduling modules for actual Grid environment

25 The Model in Bricks (Queuing System) Client A λns Qns1 μns λs A μs λnrnr Qnr1 μnrnr Client A Networks / s are represented as queues Client B Client C Qns2 Qns3 Qns4 Qs1 B Qs2 C Qnr2 Qnr3 Qnr4 Client B Client C External network traffic / server jobs represent congestion Qs3

26 Software Architecture of Bricks Scheduling Unit Predictor NetworkPredictor Predictor Scheduler ResourceDB NetworkMonitor Monitor Client Network Network Global Computing Env.. Unit

27 Verification of Scheduling Modules with Bricks Replaceable component Components in Scheduling Unit are easily replaceable with other software modules (modules for actual Grid environment). (ex.) Monitor/Predictor NWS Verification of modules with low cost Developers of scheduling module can verify/debug/evaluate their developed modules by Bricks with low cost. The verification in actual Grid env. needs expensive cost!

28 Related Work RPC based systems use existing programming languages NetSolve [Casanova and Dongarra, Univ. Tennessee] The same basic API as Ninf_call (now interchangeable) load-balancing with a daemon process called Agent. RCS [Arbenz, ETH Zurich] PVM-based systems using parallel distributed language etc. Legion [Grimshaw, Univ. Virginia] An user distributes his programs written with the parallel object-oriented language Mentat. Javelin [Schauser et al., UCSB] High portability due to using Java and WWW.

29 Ninf-NetSolve Collaboration Ninf Ninf Ninf Ninf Ninf Ninf Ninf-Netsolve Adapter NetSolve NetSolve NetSolve Ninf Ninf Client Client Netsolve-Ninf Adapter NetSolve Client Client Ninf client can use NetSolve server via adapter NetSolve client can use Ninf server via adapter

30 Related Work (cont( cont d.) systems for scheduling AppLes [Berman, UCSD] application level scheduler NWS [Wolski, UCSD/NPACI] monitoring and predicting congestion of server/network toolkits: Globus [Argonne/USC]

31 Conclusions Ninf is a RPC based global computing infrastructure. Ninf RPC protocol easy-to-use client API direct web access, transaction, Applications NinfCalc+, Excel Ninf, NetCFD,... Demo. over TransPAC [SC98, SC99] Scheduling Metaserver Bricks

32 Conclusions (cont( cont d.) Ninf platforms : Solaris1,2, DEC, UNICOS, Linux, FreeBSD Client: server platforms + Win32 Contact