s Protocol Buffer Knight76 at gmail.com

Size: px
Start display at page:

Download "s Protocol Buffer Knight76 at gmail.com"

Transcription

1 s Protocol Buffer Knight76 at gmail.com

2 1. RPC, Not RPC 2.. Proto (structed data) Proto Compiler C++ Java python

3 References Protocol Buffers: A new open source release e8ddrwvug Home Page

4 Protocol Buffer Not socket XML Parsing, serialization (debugging) Portable : IDL Heavy Optimization Language /

5 XML Simple 3~10 20~100

6 ? index server request/response protocol 48,162 different message types 12,183.proto files /

7 Stub Serializing, Parsing / IDL Portable struct java, c++, python 3 rd party lib (. Lite Good Document BSD license

8 ? Output binary PB Reflection json Map, set

9

10 Download proto compiler in google code (Option) eclipse plugin

11 Proto compiler ist

12

13 Eclipse #1 Protoclips Windows-Preference-ProtoBuf Plugin Perferences protoc.exe compiler path.

14 Eclipse #2 PB Project -> Add Google ProtoBuf Compiler Nature

15 Eclipse #3 src/test.proto test/test.java But, PB lib maven!

16 Eclipse #4 Pom.xml <dependencies> <dependency> <groupid>com.google.protobuf</groupid> <artifactid>protobuf-java</artifactid> <version>2.4.1</version> </dependency> </dependencies>

17 Eclipse #4

18

19 PB Format-> // text.txt package test; message Request { required string command = 1; optional string param = 2; } /cygdrive/c/protocolbuffer $./protoc.exe test.txt --java_out=. Test TestTxt.java (17KB, Protocol buffer ) // c++ out test.txt.pb.cc (12K) test.txt.pb.h (8K) // python out /test/txt_pb2.py (2K) naming

20

21 Descriptor internal_static_test_request_descriptor Reflection Message / Message.Builder interfaces. Json (ajax ),, option java_package = "com.example.foo.bar"; option java_outer_classname = ProtocolData"; PB Enum java enum

22 64 MB CodedInputStream/CodedOutputStream (ZeroCopyInputStream/ZeroCopyOutputStrea m ) SetTotalBytesLimit

23 package.proto message Protocol buffer language guide Protoc.exe( ) java, python, c++.(immutable) protoc, accessor. get/set/has

24 message package Type Bool, int32, uint32, float, double, string, bytes,. Enum Nested type Default value importing Modifier required :. RuntimeException, IOException optional : option hasxxx() Repeated : 0 List

25 message protoc Write/Read, serialization set ( bit )

26 Versioning But, java, c++.. (GOOGLE_PROTOBUF_VERIFY_VERSI ON ) Incompatible

27

28 DEMO #1-0 PB // text.txt package test; message Request { required string command = 1; optional string param = 2; }

29 DEMO #1-1 package com.example.test; import java.io.fileoutputstream; import test.test.request; public class Writer { public static void main(string[] args) throws Exception { Request request = Request.newBuilder().setCommand("commit").setParam("every files").build(); FileOutputStream output = new FileOutputStream("r.os"); request.writeto(output); output.close(); } }

30 DEMO #1-2 package com.example.test; import java.io.fileinputstream; import test.test.request; public class Reader { public static void main(string[] args) throws Exception { Request request = Request.parseFrom(new FileInputStream("r.os")); System.out.println("command : " + request.getcommand()); if (request.hasparam()) { System.out.println("params : " + request.getparam()); } } }

31 command : commit params : every files DEMO #1-3

32 package tutorial; option java_package = "com.example.tutorial"; option java_outer_classname = "AddressBookProtos"; message Person { required string name = 1; required int32 id = 2; optional string = 3; enum PhoneType { MOBILE = 0; HOME = 1; WORK = 2; } DEMO #2 message PhoneNumber { required string number = 1; optional PhoneType type = 2 [default = HOME]; } repeated PhoneNumber phone = 4; } message AddressBook { repeated Person person = 1; }

33 DEMO #3 1 c++ #include <iostream> #include <fstream> #include <string> #include "test.pb.h" using namespace std; int main(int argc, char* argv[]) { Request request; request.set_command("init"); request.set_param("0"); fstream out("streams", ios::out ios::binary ios::trunc); request.serializetoostream(&out); out.close(); } return 1;

34 DEMO #3 1 c++ #include <iostream> #include <fstream> #include <string> #include "test.pb.h" using namespace std; int main(int argc, char* argv[]) { Request request; fstream in( streams", ios::in ios::binary); if (!request.parsefromistream(&in)) { cerr << "Failed to parse streams." << endl; exit(1); } cout << command: " << request.command() << endl; if (request.has_param()) { cout << param: " << request.param() << endl; } }

35

36 Protocol Buffer Language Guide KR/apis/protocolbuffers/docs/proto.html

37 API KR/apis/protocolbuffers/docs/reference/java/index.html

38 PB Format rmat

39 ( json/xml )

40 End of Document

Googles Approach for Distributed Systems. Slides partially based upon Majd F. Sakr, Vinay Kolar, Mohammad Hammoud and Google PrototBuf Tutorial

Googles Approach for Distributed Systems. Slides partially based upon Majd F. Sakr, Vinay Kolar, Mohammad Hammoud and Google PrototBuf Tutorial Protocol Buffers Googles Approach for Distributed Systems Slides partially based upon Majd F. Sakr, Vinay Kolar, Mohammad Hammoud and Google PrototBuf Tutorial https://developers.google.com/protocol-buffers/docs/tutorials

More information

Protocol Buffers, grpc

Protocol Buffers, grpc Protocol Buffers, grpc Szolgáltatásorientált rendszerintegráció Service-Oriented System Integration Dr. Balázs Simon BME, IIT Outline Remote communication application level vs. transport level protocols

More information

CS 378 Big Data Programming

CS 378 Big Data Programming CS 378 Big Data Programming Lecture 9 Complex Writable Types AVRO, Protobuf CS 378 - Fall 2017 Big Data Programming 1 Review Assignment 4 WordStaQsQcs using Avro QuesQons/issues? MRUnit and AVRO AVRO keys,

More information

Random File Access. 1. Random File Access

Random File Access. 1. Random File Access Random File Access 1. Random File Access In sequential file access, the file is read or written sequentially from the beginning. In random file access, you can skip around to various points in the file

More information

CS 417 9/18/17. Paul Krzyzanowski 1. Socket-based communication. Distributed Systems 03. Remote Procedure Calls. Sample SMTP Interaction

CS 417 9/18/17. Paul Krzyzanowski 1. Socket-based communication. Distributed Systems 03. Remote Procedure Calls. Sample SMTP Interaction Socket-based communication Distributed Systems 03. Remote Procedure Calls Socket API: all we get from the to access the network Socket = distinct end-to-end communication channels Read/write model Line-oriented,

More information

Distributed Systems. 03. Remote Procedure Calls. Paul Krzyzanowski. Rutgers University. Fall 2017

Distributed Systems. 03. Remote Procedure Calls. Paul Krzyzanowski. Rutgers University. Fall 2017 Distributed Systems 03. Remote Procedure Calls Paul Krzyzanowski Rutgers University Fall 2017 1 Socket-based communication Socket API: all we get from the OS to access the network Socket = distinct end-to-end

More information

Semesterprojekt Implementierung eines Brettspiels (inklusive computergesteuerter Spieler) Remote Procedure Calls

Semesterprojekt Implementierung eines Brettspiels (inklusive computergesteuerter Spieler) Remote Procedure Calls Semesterprojekt Implementierung eines Brettspiels (inklusive computergesteuerter Spieler) Wintersemester 16/17 Remote Procedure Calls Patrick Schäfer patrick.schaefer@hu-berlin.de Marc Bux buxmarcn@informatik.hu-berlin.de

More information

CS2141 Software Development using C/C++ C++ Basics

CS2141 Software Development using C/C++ C++ Basics CS2141 Software Development using C/C++ C++ Basics Integers Basic Types Can be short, long, or just plain int C++ does not define the size of them other than short

More information

Google Protocol Buffers for Embedded IoT

Google Protocol Buffers for Embedded IoT Google Protocol Buffers for Embedded IoT Integration in a medical device project Quick look 1: Overview What is it and why is useful Peers and alternatives Wire format and language syntax Libraries for

More information

Lecture Notes CPSC 224 (Spring 2012) Today... Java basics. S. Bowers 1 of 8

Lecture Notes CPSC 224 (Spring 2012) Today... Java basics. S. Bowers 1 of 8 Today... Java basics S. Bowers 1 of 8 Java main method (cont.) In Java, main looks like this: public class HelloWorld { public static void main(string[] args) { System.out.println("Hello World!"); Q: How

More information

CSCI-1680 RPC and Data Representation John Jannotti

CSCI-1680 RPC and Data Representation John Jannotti CSCI-1680 RPC and Data Representation John Jannotti Original Slides from Rodrigo Fonseca Today Defining Protocols RPC IDL Problem Two programs want to communicate: must define the protocol We have seen

More information

Advanced I/O Concepts

Advanced I/O Concepts Advanced Object Oriented Programming Advanced I/O Concepts Seokhee Jeon Department of Computer Engineering Kyung Hee University jeon@khu.ac.kr 1 1 Streams Diversity of input sources or output destinations

More information

CSCI-1680 RPC and Data Representation. Rodrigo Fonseca

CSCI-1680 RPC and Data Representation. Rodrigo Fonseca CSCI-1680 RPC and Data Representation Rodrigo Fonseca Today Defining Protocols RPC IDL Problem Two programs want to communicate: must define the protocol We have seen many of these, across all layers E.g.,

More information

Fundamental File Processing Operations 2. Fundamental File Processing Operations

Fundamental File Processing Operations 2. Fundamental File Processing Operations 2 Fundamental File Processing Operations Copyright 2004, Binnur Kurt Content Sample programs for file manipulation Physical files and logical files Opening and closing files Reading from files and writing

More information

Convenient way to deal large quantities of data. Store data permanently (until file is deleted).

Convenient way to deal large quantities of data. Store data permanently (until file is deleted). FILE HANDLING Why to use Files: Convenient way to deal large quantities of data. Store data permanently (until file is deleted). Avoid typing data into program multiple times. Share data between programs.

More information

C++ Structures Programming Workshop 2 (CSCI 1061U)

C++ Structures Programming Workshop 2 (CSCI 1061U) C++ Structures Programming Workshop 2 (CSCI 1061U) Faisal Qureshi http://faculty.uoit.ca/qureshi University of Ontario Institute of Technology C++ struct struct keyword can be used to define new data types

More information

Linked List using a Sentinel

Linked List using a Sentinel Linked List using a Sentinel Linked List.h / Linked List.h Using a sentinel for search Created by Enoch Hwang on 2/1/10. Copyright 2010 La Sierra University. All rights reserved. / #include

More information

Dave DiFranco Apache Thrift

Dave DiFranco  Apache Thrift Dave DiFranco david.difranco@oracle.com ddif@alum.mit.edu Apache Thrift http://thrift.apache.org Goals of Thrift Scalable, cross-language services development Allow developers to use the right language

More information

Scientific Computing

Scientific Computing Scientific Computing Martin Lotz School of Mathematics The University of Manchester Lecture 1, September 22, 2014 Outline Course Overview Programming Basics The C++ Programming Language Outline Course

More information

This chapter introduces the notion of namespace. We also describe how to manage input and output with C++ commands via the terminal or files.

This chapter introduces the notion of namespace. We also describe how to manage input and output with C++ commands via the terminal or files. C++ PROGRAMMING LANGUAGE: NAMESPACE AND MANGEMENT OF INPUT/OUTPUT WITH C++. CAAM 519, CHAPTER 15 This chapter introduces the notion of namespace. We also describe how to manage input and output with C++

More information

Chapte t r r 9

Chapte t r r 9 Chapter 9 Session Objectives Stream Class Stream Class Hierarchy String I/O Character I/O Object I/O File Pointers and their manipulations Error handling in Files Command Line arguments OOPS WITH C++ Sahaj

More information

DNA Sequence Reads Compression

DNA Sequence Reads Compression DNA Sequence Reads Compression User Guide Release 2.0 March 31, 2014 Contents Contents ii 1 Introduction 1 1.1 What is DSRC?....................................... 1 1.2 Main features.......................................

More information

CS 376b Computer Vision

CS 376b Computer Vision CS 376b Computer Vision 09 / 25 / 2014 Instructor: Michael Eckmann Today s Topics Questions? / Comments? Enhancing images / masks Cross correlation Convolution C++ Cross-correlation Cross-correlation involves

More information

CS2141 Software Development using C/C++ Stream I/O

CS2141 Software Development using C/C++ Stream I/O CS2141 Software Development using C/C++ Stream I/O iostream Two libraries can be used for input and output: stdio and iostream The iostream library is newer and better: It is object oriented It can make

More information

Agenda. The main body and cout. Fundamental data types. Declarations and definitions. Control structures

Agenda. The main body and cout. Fundamental data types. Declarations and definitions. Control structures The main body and cout Agenda 1 Fundamental data types Declarations and definitions Control structures References, pass-by-value vs pass-by-references The main body and cout 2 C++ IS AN OO EXTENSION OF

More information

Streams in C++ Stream concept. Reference information. Stream type declarations

Streams in C++ Stream concept. Reference information. Stream type declarations Stream concept A stream represent a sequence of bytes arriving, being retrieved, being stored, or being sent, in order. A stream is continuos and offer sequential access to the data. Each byte can be read

More information

External data representation

External data representation External data representation https://developers.google.com/protocol-buffers/ https://github.com/protobuf-c/protobuf-c http://www.drdobbs.com/webdevelopment/after-xml-json-thenwhat/240151851 http://www.digip.org/jansson/

More information

B4M36ESW: Efficient software

B4M36ESW: Efficient software B4M36ESW: Efficient software Lecture 7: Data structure serialization, RPC Michal Sojka michal.sojka@cvut.cz April 16, 2018 1 / 40 Outline 1 Introduction 2 Less efficient data serialization XML JSON 3 Faster

More information

PROGRAMMING IN C++ CVIČENÍ

PROGRAMMING IN C++ CVIČENÍ PROGRAMMING IN C++ CVIČENÍ INFORMACE Michal Brabec http://www.ksi.mff.cuni.cz/ http://www.ksi.mff.cuni.cz/~brabec/ brabec@ksi.mff.cuni.cz gmichal.brabec@gmail.com REQUIREMENTS FOR COURSE CREDIT Basic requirements

More information

RProtoBuf: An R API for Protocol Buffers

RProtoBuf: An R API for Protocol Buffers RProtoBuf: An R API for Protocol Buffers Romain François Dirk Eddelbuettel Version 0.2.6 as of October 3, 2012 Abstract Protocol Buffers is a software project by Google that is used extensively internally

More information

Exceptions, Case Study-Exception handling in C++.

Exceptions, Case Study-Exception handling in C++. PART III: Structuring of Computations- Structuring the computation, Expressions and statements, Conditional execution and iteration, Routines, Style issues: side effects and aliasing, Exceptions, Case

More information

10/23/02 21:20:33 IO_Examples

10/23/02 21:20:33 IO_Examples 1 Oct 22 22:07 2000 extractor1.c Page 1 istream &operator>>( istream &in, Point &p ){ char junk; in >> junk >> p.x >> junk >> p.y >> junk; return in; 2 Oct 22 22:07 2000 extractor2.c Page 1 istream &operator>>(

More information

CS 103 Unit 14 - Streams

CS 103 Unit 14 - Streams CS 103 Unit 14 - Streams 1 2 I/O Streams '>>' operator used to read data from an input stream Always skips leading whitespace ('\n', ' ', '\t') and stops at first trailing whitespace '

More information

CS 326 Operating Systems C Programming. Greg Benson Department of Computer Science University of San Francisco

CS 326 Operating Systems C Programming. Greg Benson Department of Computer Science University of San Francisco CS 326 Operating Systems C Programming Greg Benson Department of Computer Science University of San Francisco Why C? Fast (good optimizing compilers) Not too high-level (Java, Python, Lisp) Not too low-level

More information

Chapter 2. Procedural Programming

Chapter 2. Procedural Programming Chapter 2 Procedural Programming 2: Preview Basic concepts that are similar in both Java and C++, including: standard data types control structures I/O functions Dynamic memory management, and some basic

More information

Programming II with C++ (CSNB244) Lab 10. Topics: Files and Stream

Programming II with C++ (CSNB244) Lab 10. Topics: Files and Stream Topics: Files and Stream In this lab session, you will learn very basic and most common I/O operations required for C++ programming. The second part of this tutorial will teach you how to read and write

More information

CSE 100: STREAM I/O, BITWISE OPERATIONS, BIT STREAM I/O

CSE 100: STREAM I/O, BITWISE OPERATIONS, BIT STREAM I/O CSE 100: STREAM I/O, BITWISE OPERATIONS, BIT STREAM I/O PA2: encoding/decoding ENCODING: 1.Scan text file to compute frequencies 2.Build Huffman Tree 3.Find code for every symbol (letter) 4.Create new

More information

Piqi-RPC. Exposing Erlang services via JSON, XML and Google Protocol Buffers over HTTP. Friday, March 25, 2011

Piqi-RPC. Exposing Erlang services via JSON, XML and Google Protocol Buffers over HTTP. Friday, March 25, 2011 Piqi-RPC Exposing Erlang services via JSON, XML and Google Protocol Buffers over HTTP 1 Anton Lavrik http://piqi.org http://www.alertlogic.com 2 Overview Call Erlang functions using HTTP POST : Server

More information

Fall 2017 CISC/CMPE320 9/27/2017

Fall 2017 CISC/CMPE320 9/27/2017 Notices: CISC/CMPE320 Today File I/O Text, Random and Binary. Assignment 1 due next Friday at 7pm. The rest of the assignments will also be moved ahead a week. Teamwork: Let me know who the team leader

More information

Operating Systems. 18. Remote Procedure Calls. Paul Krzyzanowski. Rutgers University. Spring /20/ Paul Krzyzanowski

Operating Systems. 18. Remote Procedure Calls. Paul Krzyzanowski. Rutgers University. Spring /20/ Paul Krzyzanowski Operating Systems 18. Remote Procedure Calls Paul Krzyzanowski Rutgers University Spring 2015 4/20/2015 2014-2015 Paul Krzyzanowski 1 Remote Procedure Calls 2 Problems with the sockets API The sockets

More information

Homework 5. Yuji Shimojo CMSC 330. Instructor: Prof. Reginald Y. Haseltine

Homework 5. Yuji Shimojo CMSC 330. Instructor: Prof. Reginald Y. Haseltine Homework 5 Yuji Shimojo CMSC 330 Instructor: Prof. Reginald Y. Haseltine July 13, 2013 Question 1 Consider the following Java definition of a mutable string class. class MutableString private char[] chars

More information

Input and Output File (Files and Stream )

Input and Output File (Files and Stream ) Input and Output File (Files and Stream ) BITE 1513 Computer Game Programming Week 14 Scope Describe the fundamentals of input & output files. Use data files for input & output purposes. Files Normally,

More information

Streams contd. Text: Chapter12, Big C++

Streams contd. Text: Chapter12, Big C++ Streams contd pm_jat@daiict.ac.in Text: Chapter12, Big C++ Streams Objects are Abstracted Wrapper around input/output source/destinations Steps in reading/writing streams- Open: Establish connection between

More information

Lecture 5 Files and Streams

Lecture 5 Files and Streams Lecture 5 Files and Streams Introduction C programs can store results & information permanently on disk using file handling functions These functions let you write either text or binary data to a file,

More information

Java Basic Syntax. Java vs C++ Wojciech Frohmberg / OOP Laboratory. Poznan University of Technology

Java Basic Syntax. Java vs C++ Wojciech Frohmberg / OOP Laboratory. Poznan University of Technology Java vs C++ 1 1 Department of Computer Science Poznan University of Technology 2012.10.07 / OOP Laboratory Outline 1 2 3 Outline 1 2 3 Outline 1 2 3 Tabular comparizon C++ Java Paradigm Procedural/Object-oriented

More information

Input/output. Remember std::ostream? std::istream std::ostream. std::ostream cin std::istream. namespace std { class ostream { /*...

Input/output. Remember std::ostream? std::istream std::ostream. std::ostream cin std::istream. namespace std { class ostream { /*... Input/output Remember std::ostream? namespace std { class ostream { /*... */ }; } extern istream cin; extern ostream cout; extern ostream cerr; extern ostream clog; 7 / 24 std::istream std::ostream std

More information

C++ Programming Lecture 10 File Processing

C++ Programming Lecture 10 File Processing C++ Programming Lecture 10 File Processing By Ghada Al-Mashaqbeh The Hashemite University Computer Engineering Department Outline Introduction. The Data Hierarchy. Files and Streams. Creating a Sequential

More information

Strings and Streams. Professor Hugh C. Lauer CS-2303, System Programming Concepts

Strings and Streams. Professor Hugh C. Lauer CS-2303, System Programming Concepts Strings and Streams Professor Hugh C. Lauer CS-2303, System Programming Concepts (Slides include materials from The C Programming Language, 2 nd edition, by Kernighan and Ritchie, Absolute C++, by Walter

More information

STL in Action: A Universal Streaming Service

STL in Action: A Universal Streaming Service STL in Action: A Universal Streaming Service By Graham Glass Introduction Now that STL is becoming an established part of every C++ toolkit, programmers are eager to add nonstandard extensions to STL that

More information

Fig: iostream class hierarchy

Fig: iostream class hierarchy Unit 6: C++ IO Systems ================== Streams: Θ A stream is a logical device that either produces or consumes information. Θ A stream is linked to a physical device by the I/O system. Θ All streams

More information

Advanced File Operations. Review of Files. Declaration Opening Using Closing. CS SJAllan Chapter 12 2

Advanced File Operations. Review of Files. Declaration Opening Using Closing. CS SJAllan Chapter 12 2 Chapter 12 Advanced File Operations Review of Files Declaration Opening Using Closing CS 1410 - SJAllan Chapter 12 2 1 Testing for Open Errors To see if the file is opened correctly, test as follows: in.open("cust.dat");

More information

Fortran Standards Toolkit (FST)

Fortran Standards Toolkit (FST) NCAR/SIParCS/CISL Fortran Standards Toolkit (FST) Frank Duffy SIParCS Dan Nagle, Davide Del Vento Bridgewater State University Bridgewater MA August 1, 2018 What is FST? An ongoing project Mission: provide

More information

Connecting with Computer Science, 2e. Chapter 15 Programming II

Connecting with Computer Science, 2e. Chapter 15 Programming II Connecting with Computer Science, 2e Chapter 15 Programming II Objectives In this chapter you will: Gain an understanding of the basics of high-level programming languages, using Java and C++ as examples

More information

Distributed Systems 8. Remote Procedure Calls

Distributed Systems 8. Remote Procedure Calls Distributed Systems 8. Remote Procedure Calls Paul Krzyzanowski pxk@cs.rutgers.edu 10/1/2012 1 Problems with the sockets API The sockets interface forces a read/write mechanism Programming is often easier

More information

File handling Basics. Lecture 7

File handling Basics. Lecture 7 File handling Basics Lecture 7 What is a File? A file is a collection of information, usually stored on a computer s disk. Information can be saved to files and then later reused. 2 File Names All files

More information

Elliotte Rusty Harold August From XML to Flat Buffers: Markup in the Twenty-teens

Elliotte Rusty Harold August From XML to Flat Buffers: Markup in the Twenty-teens Elliotte Rusty Harold elharo@ibiblio.org August 2018 From XML to Flat Buffers: Markup in the Twenty-teens Warning! The Contenders XML JSON YAML EXI Protobufs Flat Protobufs XML JSON YAML EXI Protobuf Flat

More information

C++ files and streams. Lec 28-31

C++ files and streams. Lec 28-31 C++ files and streams Lec 28-31 Introduction So far, we have been using the iostream standard library, which provides cin and cout methods for reading from standard input and writing to standard output

More information

COMP 117 Ping Demonstration Programs & Makefile / C++ Hints

COMP 117 Ping Demonstration Programs & Makefile / C++ Hints COMP 150-IDS: Internet Scale Distributed Systems (Spring 2017) COMP 117 Ping Demonstration Programs & Makefile / C++ Hints Noah Mendelsohn Tufts University Email: noah@cs.tufts.edu Web: http://www.cs.tufts.edu/~noah

More information

SPES 2020 Deliverable 1.4.B-21. Using Protocol Buffers for Resource-Constrained Distributed Embedded Systems

SPES 2020 Deliverable 1.4.B-21. Using Protocol Buffers for Resource-Constrained Distributed Embedded Systems TECHNISCHE UNIVERSITÄT MÜNCHEN FAKULTÄT FÜR INFORMATIK Software & Systems Engineering Prof. Dr. Dr. h.c. Manfred Broy SPES 2020 Deliverable 1.4.B-21 Using Protocol Buffers for Resource-Constrained Distributed

More information

COSC 6397 Big Data Analytics. Distributed File Systems (II) Edgar Gabriel Fall HDFS Basics

COSC 6397 Big Data Analytics. Distributed File Systems (II) Edgar Gabriel Fall HDFS Basics COSC 6397 Big Data Analytics Distributed File Systems (II) Edgar Gabriel Fall 2018 HDFS Basics An open-source implementation of Google File System Assume that node failure rate is high Assumes a small

More information

Lab 1: First Steps in C++ - Eclipse

Lab 1: First Steps in C++ - Eclipse Lab 1: First Steps in C++ - Eclipse Step Zero: Select workspace 1. Upon launching eclipse, we are ask to chose a workspace: 2. We select a new workspace directory (e.g., C:\Courses ): 3. We accept the

More information

IS 0020 Program Design and Software Tools

IS 0020 Program Design and Software Tools 1 IS 0020 Program Design and Software Tools Stack/Queue - File Processing Lecture 10 March 29, 2005 Introduction 2 Storage of data Arrays, variables are temporary Files are permanent Magnetic disk, optical

More information

COMP 117 Ping Demonstration Programs & Makefile / C++ Hints

COMP 117 Ping Demonstration Programs & Makefile / C++ Hints COMP 150-IDS: Internet Scale Distributed Systems (Spring 2018) COMP 117 Ping Demonstration Programs & Makefile / C++ Hints Noah Mendelsohn Tufts University Email: noah@cs.tufts.edu Web: http://www.cs.tufts.edu/~noah

More information

ECE 462 Exam 1. 6:30-7:30PM, September 22, 2010

ECE 462 Exam 1. 6:30-7:30PM, September 22, 2010 ECE 462 Exam 1 6:30-7:30PM, September 22, 2010 I will not receive nor provide aid to any other student for this exam. Signature: You must sign here. Otherwise, the exam is not graded. This exam is printed

More information

C++ Namespaces, Exceptions

C++ Namespaces, Exceptions C++ Namespaces, Exceptions CSci 588: Data Structures, Algorithms and Software Design http://www.cplusplus.com/doc/tutorial/namespaces/ http://www.cplusplus.com/doc/tutorial/exceptions/ http://www.cplusplus.com/doc/tutorial/typecasting/

More information

System Design and Programming II

System Design and Programming II System Design and Programming II CSCI 194 Section 01 CRN: 10968 Fall 2017 David L. Sylvester, Sr., Assistant Professor Chapter 12 Advanced File Operation File Operations A file is a collection of data

More information

C++ Basics. Brian A. Malloy. References Data Expressions Control Structures Functions. Slide 1 of 24. Go Back. Full Screen. Quit.

C++ Basics. Brian A. Malloy. References Data Expressions Control Structures Functions. Slide 1 of 24. Go Back. Full Screen. Quit. C++ Basics January 19, 2012 Brian A. Malloy Slide 1 of 24 1. Many find Deitel quintessentially readable; most find Stroustrup inscrutable and overbearing: Slide 2 of 24 1.1. Meyers Texts Two excellent

More information

Abstract Data Types. 2501ICT/7421ICTNathan. René Hexel. Semester 1, School of Information and Communication Technology Griffith University

Abstract Data Types. 2501ICT/7421ICTNathan. René Hexel. Semester 1, School of Information and Communication Technology Griffith University Collections 2501ICT/7421ICTNathan School of Information and Communication Technology Griffith University Semester 1, 2012 Outline Collections 1 Collections 2 Linear Collections Collections Collections

More information

BITG 1113: Files and Stream LECTURE 10

BITG 1113: Files and Stream LECTURE 10 BITG 1113: Files and Stream LECTURE 10 1 LEARNING OUTCOMES At the end of this lecture, you should be able to: 1. Describe the fundamentals of input & output files. 2. Use data files for input & output

More information

CSE 100: STREAM I/O, BITWISE OPERATIONS, BIT STREAM I/O

CSE 100: STREAM I/O, BITWISE OPERATIONS, BIT STREAM I/O CSE 100: STREAM I/O, BITWISE OPERATIONS, BIT STREAM I/O PA3: encoding/decoding ENCODING: 1.Scan text file to compute frequencies 2.Build Huffman Tree 3.Find code for every symbol (letter) 4.Create new

More information

COSC 6397 Big Data Analytics. Distributed File Systems (II) Edgar Gabriel Spring HDFS Basics

COSC 6397 Big Data Analytics. Distributed File Systems (II) Edgar Gabriel Spring HDFS Basics COSC 6397 Big Data Analytics Distributed File Systems (II) Edgar Gabriel Spring 2017 HDFS Basics An open-source implementation of Google File System Assume that node failure rate is high Assumes a small

More information

Fundamentals of Programming Session 25

Fundamentals of Programming Session 25 Fundamentals of Programming Session 25 Instructor: Reza Entezari-Maleki Email: entezari@ce.sharif.edu 1 Fall 2013 These slides have been created using Deitel s slides Sharif University of Technology Outlines

More information

Chapter 1: Why Program? Computers and Programming. Why Program?

Chapter 1: Why Program? Computers and Programming. Why Program? Chapter 1: Introduction to Computers and Programming 1.1 Why Program? Why Program? Computer programmable machine designed to follow instructions Program instructions in computer memory to make it do something

More information

Distributed Programming and Remote Procedure Calls (RPC): Apache Thrift. George Porter CSE 124 February 19, 2015

Distributed Programming and Remote Procedure Calls (RPC): Apache Thrift. George Porter CSE 124 February 19, 2015 Distributed Programming and Remote Procedure Calls (RPC): Apache Thrift George Porter CSE 124 February 19, 2015 End-to-end RPC protocol RPC Components Defines messages, message exchange behavior, Programming

More information

for (int i = 1; i <= 3; i++) { do { cout << "Enter a positive integer: "; cin >> n;

for (int i = 1; i <= 3; i++) { do { cout << Enter a positive integer: ; cin >> n; // Workshop 1 #include using namespace std; int main () int n, k; int sumdigits; for (int i = 1; i n; cin.clear (); cin.ignore (100,

More information

Programming. C++ Basics

Programming. C++ Basics Programming C++ Basics Introduction to C++ C is a programming language developed in the 1970s with the UNIX operating system C programs are efficient and portable across different hardware platforms C++

More information

Strings and Stream I/O

Strings and Stream I/O Strings and Stream I/O C Strings In addition to the string class, C++ also supports old-style C strings In C, strings are stored as null-terminated character arrays str1 char * str1 = "What is your name?

More information

Chapter 6: User-Defined Functions. Objectives (cont d.) Objectives. Introduction. Predefined Functions 12/2/2016

Chapter 6: User-Defined Functions. Objectives (cont d.) Objectives. Introduction. Predefined Functions 12/2/2016 Chapter 6: User-Defined Functions Objectives In this chapter, you will: Learn about standard (predefined) functions Learn about user-defined functions Examine value-returning functions Construct and use

More information

[CSE10200] Programming Basis ( 프로그래밍기초 ) Chapter 7. Seungkyu Lee. Assistant Professor, Dept. of Computer Engineering Kyung Hee University

[CSE10200] Programming Basis ( 프로그래밍기초 ) Chapter 7. Seungkyu Lee. Assistant Professor, Dept. of Computer Engineering Kyung Hee University [CSE10200] Programming Basis ( 프로그래밍기초 ) Chapter 7 Seungkyu Lee Assistant Professor, Dept. of Computer Engineering Kyung Hee University Input entities Keyboard, files Output entities Monitor, files Standard

More information

CS 103 Unit 14 - Streams

CS 103 Unit 14 - Streams CS 103 Unit 14 - Streams 1 2 I/O Streams '>>' operator reads from a stream (and convert to the desired type) Always skips leading whitespace ('\n', ' ', '\t') and stops at first trailing whitespace '

More information

C++ Binary File I/O. C++ file input and output are typically achieved by using an object of one of the following classes:

C++ Binary File I/O. C++ file input and output are typically achieved by using an object of one of the following classes: C++ Binary File I/O C++ file input and output are typically achieved by using an object of one of the following classes: ifstream for reading input only. ofstream for writing output only. fstream for reading

More information

File I/O Christian Schumacher, Info1 D-MAVT 2013

File I/O Christian Schumacher, Info1 D-MAVT 2013 File I/O Christian Schumacher, chschuma@inf.ethz.ch Info1 D-MAVT 2013 Input and Output in C++ Stream objects Formatted output Writing and reading files References General Remarks I/O operations are essential

More information

CE221 Programming in C++ Part 1 Introduction

CE221 Programming in C++ Part 1 Introduction CE221 Programming in C++ Part 1 Introduction 06/10/2017 CE221 Part 1 1 Module Schedule There are two lectures (Monday 13.00-13.50 and Tuesday 11.00-11.50) each week in the autumn term, and a 2-hour lab

More information

1- Write a single C++ statement that: A. Calculates the sum of the two integrates 11 and 12 and outputs the sum to the consol.

1- Write a single C++ statement that: A. Calculates the sum of the two integrates 11 and 12 and outputs the sum to the consol. 1- Write a single C++ statement that: A. Calculates the sum of the two integrates 11 and 12 and outputs the sum to the consol. B. Outputs to the console a floating point number f1 in scientific format

More information

Module Contact: Dr Anthony J. Bagnall, CMP Copyright of the University of East Anglia Version 2

Module Contact: Dr Anthony J. Bagnall, CMP Copyright of the University of East Anglia Version 2 UNIVERSITY OF EAST ANGLIA School of Computing Sciences Main Series UG Examination 2014/15 PROGRAMMING 2 CMP-5015Y Time allowed: 2 hours Answer four questions. All questions carry equal weight. Notes are

More information

CHAPTER 3 BASIC INSTRUCTION OF C++

CHAPTER 3 BASIC INSTRUCTION OF C++ CHAPTER 3 BASIC INSTRUCTION OF C++ MOHD HATTA BIN HJ MOHAMED ALI Computer programming (BFC 20802) Subtopics 2 Parts of a C++ Program Classes and Objects The #include Directive Variables and Literals Identifiers

More information

UEE1303(1070) S 12 Object-Oriented Programming in C++

UEE1303(1070) S 12 Object-Oriented Programming in C++ Computational Intelligence on Automation Lab @ NCTU Learning Objectives UEE1303(1070) S 12 Object-Oriented Programming in C++ Lecture 06: Streams and File Input/Output I/O stream istream and ostream member

More information

There are, of course, many other possible solutions and, if done correctly, those received full credit.

There are, of course, many other possible solutions and, if done correctly, those received full credit. Question 1. (20 points) STL. Complete the function ChangeWords below. This function has as inputs a vector of strings, and a map of key-value pairs. The function should return a new vector

More information

grpc - A solution for RPCs by Google Distributed Systems Seminar at Charles University in Prague, Nov 2016 Jan Tattermusch - grpc Software Engineer

grpc - A solution for RPCs by Google Distributed Systems Seminar at Charles University in Prague, Nov 2016 Jan Tattermusch - grpc Software Engineer grpc - A solution for RPCs by Google Distributed Systems Seminar at Charles University in Prague, Nov 2016 Jan Tattermusch - grpc Software Engineer About me Software Engineer at Google (since 2013) Working

More information

CSC 138 Structured Programming CHAPTER 4: TEXT FILE [PART 1]

CSC 138 Structured Programming CHAPTER 4: TEXT FILE [PART 1] CSC 138 Structured Programming CHAPTER 4: TEXT FILE [PART 1] LEARNING OBJECTIVES Upon completion, you should be able to: o define C++ text files o explain the benefits of using I/O file processing o explain

More information

Darshan Institute of Engineering & Technology for Diploma Studies

Darshan Institute of Engineering & Technology for Diploma Studies Streams A stream is a sequence of data. In Java a stream is composed of bytes. In java, 3 streams are created for us automatically. 1. System.out : standard output stream 2. System.in : standard input

More information

Lecture 9. Introduction

Lecture 9. Introduction Lecture 9 File Processing Streams Stream I/O template hierarchy Create, update, process files Sequential and random access Formatted and raw processing Namespaces Lec 9 Programming in C++ 1 Storage of

More information

Here is a hierarchy of classes to deal with Input and Output streams.

Here is a hierarchy of classes to deal with Input and Output streams. PART 25 25. Files and I/O 25.1 Reading and Writing Files A stream can be defined as a sequence of data. The InputStream is used to read data from a source and the OutputStream is used for writing data

More information

Selenium Testing Course Content

Selenium Testing Course Content Selenium Testing Course Content Introduction What is automation testing? What is the use of automation testing? What we need to Automate? What is Selenium? Advantages of Selenium What is the difference

More information

Unit-V File operations

Unit-V File operations Unit-V File operations What is stream? C++ IO are based on streams, which are sequence of bytes flowing in and out of the programs. A C++ stream is a flow of data into or out of a program, such as the

More information

CSE 333 Final Exam June 6, 2017 Sample Solution

CSE 333 Final Exam June 6, 2017 Sample Solution Question 1. (24 points) Some C and POSIX I/O programming. Given an int file descriptor returned by open(), write a C function ReadFile that reads the entire file designated by that file descriptor and

More information

Introduction. Lecture 5 Files and Streams FILE * FILE *

Introduction. Lecture 5 Files and Streams FILE * FILE * Introduction Lecture Files and Streams C programs can store results & information permanently on disk using file handling functions These functions let you write either text or binary data to a file, and

More information

Fundamentals of Programming Session 27

Fundamentals of Programming Session 27 Fundamentals of Programming Session 27 Instructor: Reza Entezari-Maleki Email: entezari@ce.sharif.edu 1 Fall 2013 These slides have been created using Deitel s slides Sharif University of Technology Outlines

More information

Apache Thrift Introduction & Tutorial

Apache Thrift Introduction & Tutorial Apache Thrift Introduction & Tutorial Marlon Pierce, Suresh Marru Q & A TIOBE Index Programming Language polyglotism Modern distributed applications are rarely composed of modules written in a single language.

More information

Programming with Haiku

Programming with Haiku Programming with Haiku Lesson 5 Written by DarkWyrm All material 2010 DarkWyrm Let's take some time to put together all of the different bits of code that we've been learning about. Since the first lesson,

More information