Guido van Rossum 9th LASER summer school, Sept. 2012

Size: px
Start display at page:

Download "Guido van Rossum 9th LASER summer school, Sept. 2012"

Transcription

1 Guido van Rossum 9th LASER summer school, Sept. 2012

2 Async I/O is as old as computers But programmers prefer synchronous I/O Easier to understand Easier to write code for Fewer chances of bugs Many languages don't have async I/O at all Or hide it deep down in advanced libraries Or recommend using threads

3 Probably an old Mac OS interface (sound?) The OS would call a C function when done Problem was to call a Python function safely Ended up using a queue, polled by interpreter loop There's still mention of this in ceval.c... Next were UNIX signal handlers Solution used the same mechanism Still in use

4 select() and ioct() syscalls make file descriptor non-blocking use select() to wait or poll for I/O also available on Windows (for sockets only) Later: poll(), epoll(), kqueue() these are all faster versions of select(), really How to write code for all this?

5 Threads "solve" the problem differently Just dedicate a separate thread to each socket Use synchronous I/O in the thread The OS will multiplex threads Problems: thread switching overhead memory allocations for thread stacks max #sockets way higher than max #threads locking bugs

6 Modules asyncore.py, asynchat.py Sam Rushing, 1996 dispatcher class wraps non-blocking socket all active dispatchers collected in a table event handler methods called when data ready e.g. handle_read(); user subclasses to define actions loop() calls select() and calls handlers as needed Sadly, considered inflexible and out of date also, async I/O is remarkably subtle

7 800 pound gorilla of Python's async I/O world Glyph Lefkowitz, 2002 flexible, robust, mature, maintained separation of protocols and transports deferreds and callbacks integration with foreign event loops e.g. GUI libraries integration with threads (via deferreds) inline callbacks (based on generators)

8 Web framework designed for speed FriendFeed/Facebook, 2009 Lowest level: async I/O using callbacks IOLoop class IOLoop.add_handler(fd, handler, events) Focus is web serving (that's a trend!)

9 Handles most requests/sec E.g. Tornado: 3k req/sec on 1 core (8k / 4 cores) No locks required (usually) Easy to handle timeouts Familiar to JavaScript programmers

10 Logic spread out over multiple functions Boilerplate overwhelms program logic Passing state around is hard Proliferation of higher order functions Error handling often broken Utilizes only a single core Hard to debug If you accidentally block, everyone hangs

11 Stackless Python (Christian Tismer, 1999) Avoid using C stack for Python-to-Python calls Microthreads, channels, scheduling Implemented as a fork of CPython Stackless 3.1 (2012) Greenlet (Armin Rigo, 2004) Derived from Stackless Microthreads called tasklets (just coroutines, really) Implemented as a C extension module May move the C stack around! Greenlet (2012)

12 from greenlet import greenlet def test1(x, y): z = gr2.switch(x+y) print z def test2(u): print u gr1.switch(42) gr1 = greenlet(test1) gr2 = greenlet(test2) gr1.switch("hello", " world") hello world 42

13 Both built on top of Greenlet Eventlet (Bob Ippolito, 2006) "Asynchronous I/O with a synchronous interface" Seems pretty dead now Gevent (Denis Bilenko, 2009) Successor to Eventlet Uses libev (an event loop C library) Monkey-patches networking modules in stdlib

14 import gevent def foo(): print('running in foo') gevent.sleep(0) print('explicit context switch to foo again') def bar(): print('explicit context to bar') gevent.sleep(0) print('implicit context switch back to bar') gevent.joinall([gevent.spawn(foo), gevent.spawn(bar)])

15 import gevent, gevent.monkey, json, urllib2 gevent.monkey.patch_socket() url = ' def fetch(pid): response = urllib2.urlopen(url) result = response.read() json_result = json.loads(result) datetime = json_result['datetime'] print 'Process ', pid, datetime return json_result['datetime'] for i in range(1, 10): fetch(i) # synchronous threads = [] for i in range(1, 10): threads.append(gevent.spawn(fetch, i)) gevent.joinall(threads) # asynchronous

16 No callbacks! Logic remains in one place Looks a lot like OS-level threads Synchronous and async code looks the same Still as fast as Twisted or Tornado In fact, Tornado+Gevent is even faster!

17 Monkey-patching may be brittle It's easy to forget where switching happens Synchronization primitives reimplemented Not pure Python (won't work on App Engine)

18 Explicit couroutines, using Python generators Typically built on top of event loop, callbacks Twisted inlinecallbacks Monocle (Greg+Steve Hazel, 2010) Works with asyncore, Twisted, Tornado NDB (GvR, 2011) For Google App Engine only

19 Not quite classic Knuth-style coroutines Python generator: function using yield Used to implement iterators One (!) stack frame is suspended by yield Resumed by.next() (Python 3:. next ()) Addition in PEP 342:.send(),.throw() g.send(x) makes yield return x g.throw(e) makes yield raise e A "trampoline" or scheduler can be written to implement nanothreads using generators

20 from monocle import def do_cmd(conn): cmd = yield conn.read_until("\n") if cmd.type == "get-address": user = yield db.query(cmd.username) yield conn.write(user.address) else: yield conn.write("unknown command")

21 from google.appengine.ext import ndb class Employee(ndb.Model): empid = def get_employee_async(empid): emp = yield Employee.get_by_id_async(empid) if emp is None: emp = Employee(empid=empid, id=empid) yield emp.put_async() raise ndb.return(emp) joe = get_employee_async('42').get_result()

22 No callbacks! Logic remains in one place Looks a lot like OS-level threads Synchronous and async code looks the same Still as fast as Twisted or Tornado It's a Comonad!!! Pure Python implementation possible User is always aware of switch points Less need for locking

23 Async and synchronous calls look different Only a single frame can be suspended It's easy to forget the yield keyword Mixing async and sync calls is hazardous

24 ???

1999 Stackless Python 2004 Greenlet 2006 Eventlet 2009 Gevent 0.x (libevent) 2011 Gevent 1.0dev (libev, c-ares)

1999 Stackless Python 2004 Greenlet 2006 Eventlet 2009 Gevent 0.x (libevent) 2011 Gevent 1.0dev (libev, c-ares) Denis Bilenko 1999 Stackless Python 2004 Greenlet 2006 Eventlet 2009 Gevent 0.x (libevent) 2011 Gevent 1.0dev (libev, c-ares) Replaced libevent with libev Replaced libevent-dns with c-ares Event loop is

More information

Why I still develop synchronous web in the asyncio era. April 7th, 2017 Giovanni Barillari - pycon otto - Firenze, Italy

Why I still develop synchronous web in the asyncio era. April 7th, 2017 Giovanni Barillari - pycon otto - Firenze, Italy Why I still develop synchronous web in the asyncio era April 7th, 2017 Giovanni Barillari - pycon otto - Firenze, Italy Who am I? I m Gio! pronounced as Joe trust me, I m a physicist :) code principally

More information

Introduction to Asynchronous Programming Fall 2014

Introduction to Asynchronous Programming Fall 2014 CS168 Computer Networks Fonseca Introduction to Asynchronous Programming Fall 2014 Contents 1 Introduction 1 2 The Models 1 3 The Motivation 3 4 Event-Driven Programming 4 5 select() to the rescue 5 1

More information

Django on Gevent. asynchronous i/o in a synchronous world. Tuesday, September 4, 12

Django on Gevent. asynchronous i/o in a synchronous world. Tuesday, September 4, 12 Django on Gevent asynchronous i/o in a synchronous world About Me 5 years professionally as web developer 4 years using Django 2 years at Lawrence Journal-World (birthplace of Django) Tech obsessions

More information

Guido van Rossum 9th LASER summer school, Sept. 2012

Guido van Rossum 9th LASER summer school, Sept. 2012 Guido van Rossum guido@python.org 9th LASER summer school, Sept. 2012 It took many steps to get to Py3k generators Interesting example of a "random walk" Origins go as far back as it gets ABC has datatypes

More information

December 8, epoll: asynchronous I/O on Linux. Pierre-Marie de Rodat. Synchronous/asynch. epoll vs select and poll

December 8, epoll: asynchronous I/O on Linux. Pierre-Marie de Rodat. Synchronous/asynch. epoll vs select and poll e:.. e: e vs select and December 8, 2011 Plan e:.1 ronous.2 e vs select and e vs select and.3 Synchronous I/O A system call for I/O blocks until something can be returned. Quite easy to use. But when you

More information

Exception-Less System Calls for Event-Driven Servers

Exception-Less System Calls for Event-Driven Servers Exception-Less System Calls for Event-Driven Servers Livio Soares and Michael Stumm University of Toronto Talk overview At OSDI'10: exception-less system calls Technique targeted at highly threaded servers

More information

Concurrent Python. Cody Soyland Austin Web Python User Group - Feb 28, Thursday, February 28, 13

Concurrent Python. Cody Soyland Austin Web Python User Group - Feb 28, Thursday, February 28, 13 Concurrent Python Cody Soyland Austin Web Python User Group - Feb 28, 2013 Getting started Notes on codysoyland.com Install gevent 1.0 (release candidate) Load my ipython Notebooks The Real-time Web Technologies

More information

nucleon Documentation

nucleon Documentation nucleon Documentation Release 0.1 Daniel Pope December 23, 2014 Contents 1 Getting started with Nucleon 3 1.1 An example application......................................... 3 1.2 Our first database app..........................................

More information

Asynchronous I/O: A Case Study in Python

Asynchronous I/O: A Case Study in Python Asynchronous I/O: A Case Study in Python SALEIL BHAT A library for performing await -style asynchronous socket I/O was written in Python. It provides an event loop, as well as a set of asynchronous functions

More information

Process Concepts. CSC400 - Operating Systems. 3. Process Concepts. J. Sumey

Process Concepts. CSC400 - Operating Systems. 3. Process Concepts. J. Sumey CSC400 - Operating Systems 3. Process Concepts J. Sumey Overview Concurrency Processes & Process States Process Accounting Interrupts & Interrupt Processing Interprocess Communication CSC400 - Process

More information

Python Twisted. Mahendra

Python Twisted. Mahendra Python Twisted Mahendra M @mahendra http://creativecommons.org/licenses/by-sa/3.0/ Methods of concurrency Workers Threads and processes Event driven Let us examine this with the case of a web server Worker

More information

State of the real-time web with Django

State of the real-time web with Django State of the real-time web with Django Aymeric Augustin - @aymericaugustin DjangoCon US - September 5th, 2013 1 What are we talking about? 2 Real-time 1. Systems responding within deadlines 2. Simulations

More information

Processes and Threads

Processes and Threads COS 318: Operating Systems Processes and Threads Kai Li and Andy Bavier Computer Science Department Princeton University http://www.cs.princeton.edu/courses/archive/fall13/cos318 Today s Topics u Concurrency

More information

Concurrency, Parallelism, Events, Asynchronicity, Oh My

Concurrency, Parallelism, Events, Asynchronicity, Oh My Concurrency, Parallelism, Events, Asynchronicity, Oh My threads, processes, queues & workers, multiprocessing, gevent, twisted, yo momma Tommi Virtanen / @tv / DreamHost SoCal Piggies 2011-11-10 One thing

More information

LECTURE 15. Web Servers

LECTURE 15. Web Servers LECTURE 15 Web Servers DEPLOYMENT So, we ve created a little web application which can let users search for information about a country they may be visiting. The steps we ve taken so far: 1. Writing the

More information

Python 3 10 years later

Python 3 10 years later Python 3 10 years later FOSDEM 2018, Brussels Victor Stinner vstinner@redhat.com Victor Stinner CPython core developer since 2010 Work on CPython and OpenStack for Red Hat Very happy user of Fedora and

More information

The Kernel Abstraction

The Kernel Abstraction The Kernel Abstraction Debugging as Engineering Much of your time in this course will be spent debugging In industry, 50% of software dev is debugging Even more for kernel development How do you reduce

More information

Processes and Non-Preemptive Scheduling. Otto J. Anshus

Processes and Non-Preemptive Scheduling. Otto J. Anshus Processes and Non-Preemptive Scheduling Otto J. Anshus Threads Processes Processes Kernel An aside on concurrency Timing and sequence of events are key concurrency issues We will study classical OS concurrency

More information

Introduction to Concurrent Software Systems. CSCI 5828: Foundations of Software Engineering Lecture 12 09/29/2016

Introduction to Concurrent Software Systems. CSCI 5828: Foundations of Software Engineering Lecture 12 09/29/2016 Introduction to Concurrent Software Systems CSCI 5828: Foundations of Software Engineering Lecture 12 09/29/2016 1 Goals Present an overview of concurrency in software systems Review the benefits and challenges

More information

NODE.JS SERVER SIDE JAVASCRIPT. Introduc)on Node.js

NODE.JS SERVER SIDE JAVASCRIPT. Introduc)on Node.js NODE.JS SERVER SIDE JAVASCRIPT Introduc)on Node.js Node.js was created by Ryan Dahl starting in 2009. For more information visit: http://www.nodejs.org 1 What about Node.js? 1. JavaScript used in client-side

More information

Threads Chapter 5 1 Chapter 5

Threads Chapter 5 1 Chapter 5 Threads Chapter 5 1 Chapter 5 Process Characteristics Concept of Process has two facets. A Process is: A Unit of resource ownership: a virtual address space for the process image control of some resources

More information

Last time: introduction. Networks and Operating Systems ( ) Chapter 2: Processes. This time. General OS structure. The kernel is a program!

Last time: introduction. Networks and Operating Systems ( ) Chapter 2: Processes. This time. General OS structure. The kernel is a program! ADRIAN PERRIG & TORSTEN HOEFLER Networks and Operating Systems (252-0062-00) Chapter 2: Processes Last time: introduction Introduction: Why? February 12, 2016 Roles of the OS Referee Illusionist Glue Structure

More information

Introduction to Concurrent Software Systems. CSCI 5828: Foundations of Software Engineering Lecture 08 09/17/2015

Introduction to Concurrent Software Systems. CSCI 5828: Foundations of Software Engineering Lecture 08 09/17/2015 Introduction to Concurrent Software Systems CSCI 5828: Foundations of Software Engineering Lecture 08 09/17/2015 1 Goals Present an overview of concurrency in software systems Review the benefits and challenges

More information

Core Development > PEP Index > PEP Addition of the multiprocessing package to the standard library

Core Development > PEP Index > PEP Addition of the multiprocessing package to the standard library Core Development > PEP Index > PEP 371 -- Addition of the multiprocessing package to the standard library PEP: 371 Title: Version: 70469 Addition of the multiprocessing package to the standard library

More information

EECS 482 Introduction to Operating Systems

EECS 482 Introduction to Operating Systems EECS 482 Introduction to Operating Systems Winter 2018 Harsha V. Madhyastha Monitors vs. Semaphores Monitors: Custom user-defined conditions Developer must control access to variables Semaphores: Access

More information

Guido van Rossum 9th LASER summer school, Sept. 2012

Guido van Rossum 9th LASER summer school, Sept. 2012 Guido van Rossum guido@python.org 9th LASER summer school, Sept. 2012 Static analysis for Python is hard There are no good current tools There are a number of lint-like tools Some IDEs have some refactoring

More information

Threads SPL/2010 SPL/20 1

Threads SPL/2010 SPL/20 1 Threads 1 Today Processes and Scheduling Threads Abstract Object Models Computation Models Java Support for Threads 2 Process vs. Program processes as the basic unit of execution managed by OS OS as any

More information

CS 160: Interactive Programming

CS 160: Interactive Programming CS 160: Interactive Programming Professor John Canny 3/8/2006 1 Outline Callbacks and Delegates Multi-threaded programming Model-view controller 3/8/2006 2 Callbacks Your code Myclass data method1 method2

More information

Cross-platform daemonization tools.

Cross-platform daemonization tools. Cross-platform daemonization tools. Release 0.1.0 Muterra, Inc Sep 14, 2017 Contents 1 What is Daemoniker? 1 1.1 Installing................................................. 1 1.2 Example usage..............................................

More information

PyPy - How to not write Virtual Machines for Dynamic Languages

PyPy - How to not write Virtual Machines for Dynamic Languages PyPy - How to not write Virtual Machines for Dynamic Languages Institut für Informatik Heinrich-Heine-Universität Düsseldorf ESUG 2007 Scope This talk is about: implementing dynamic languages (with a focus

More information

PROCESSES AND THREADS THREADING MODELS. CS124 Operating Systems Winter , Lecture 8

PROCESSES AND THREADS THREADING MODELS. CS124 Operating Systems Winter , Lecture 8 PROCESSES AND THREADS THREADING MODELS CS124 Operating Systems Winter 2016-2017, Lecture 8 2 Processes and Threads As previously described, processes have one sequential thread of execution Increasingly,

More information

A Scalable Event Dispatching Library for Linux Network Servers

A Scalable Event Dispatching Library for Linux Network Servers A Scalable Event Dispatching Library for Linux Network Servers Hao-Ran Liu and Tien-Fu Chen Dept. of CSIE National Chung Cheng University Traditional server: Multiple Process (MP) server A dedicated process

More information

PERL 6 and PARROT An Overview. Presentation for NYSA The New York System Administrators Association

PERL 6 and PARROT An Overview. Presentation for NYSA The New York System Administrators Association PERL 6 and PARROT An Overview Presentation for NYSA The New York System Administrators Association June 9 th, 2004 By Josh Rabinowitz http://skateboarddirectory.com Why Perl 6? Perl 5 codebase difficult

More information

3.1 Introduction. Computers perform operations concurrently

3.1 Introduction. Computers perform operations concurrently PROCESS CONCEPTS 1 3.1 Introduction Computers perform operations concurrently For example, compiling a program, sending a file to a printer, rendering a Web page, playing music and receiving e-mail Processes

More information

Yi Shi Fall 2017 Xi an Jiaotong University

Yi Shi Fall 2017 Xi an Jiaotong University Threads Yi Shi Fall 2017 Xi an Jiaotong University Goals for Today Case for Threads Thread details Case for Parallelism main() read_data() for(all data) compute(); write_data(); endfor main() read_data()

More information

BENCHMARKING LIBEVENT AGAINST LIBEV

BENCHMARKING LIBEVENT AGAINST LIBEV BENCHMARKING LIBEVENT AGAINST LIBEV Top 2011-01-11, Version 6 This document briefly describes the results of running the libevent benchmark program against both libevent and libev. Libevent Overview Libevent

More information

Python Asynchronous Programming with Salt Stack (tornado, asyncio) and RxPY

Python Asynchronous Programming with Salt Stack (tornado, asyncio) and RxPY Python Asynchronous Programming with Salt Stack (tornado, asyncio) and RxPY PyCon Korea 2017 Kim Sol kstreee@gmail.com Python Asynchronous Programming with Salt Stack (tornado, asyncio) and RxPY Kim Sol

More information

A Sense of Time for JavaScript and Node.js

A Sense of Time for JavaScript and Node.js A Sense of Time for JavaScript and Node.js First-Class Timeouts as a Cure for Event Handler Poisoning James C. Davis Eric R. Williamson Dongyoon Lee COMPUTER SCIENCE - 1 - Contributions Attack: Event Handler

More information

Python Implementation Strategies. Jeremy Hylton Python / Google

Python Implementation Strategies. Jeremy Hylton Python / Google Python Implementation Strategies Jeremy Hylton Python / Google Python language basics High-level language Untyped but safe First-class functions, classes, objects, &c. Garbage collected Simple module system

More information

Threads. Computer Systems. 5/12/2009 cse threads Perkins, DW Johnson and University of Washington 1

Threads. Computer Systems.   5/12/2009 cse threads Perkins, DW Johnson and University of Washington 1 Threads CSE 410, Spring 2009 Computer Systems http://www.cs.washington.edu/410 5/12/2009 cse410-20-threads 2006-09 Perkins, DW Johnson and University of Washington 1 Reading and References Reading» Read

More information

Process Scheduling Queues

Process Scheduling Queues Process Control Process Scheduling Queues Job queue set of all processes in the system. Ready queue set of all processes residing in main memory, ready and waiting to execute. Device queues set of processes

More information

Interoperation of tasks

Interoperation of tasks Operating systems (vimia219) Interoperation of tasks Tamás Kovácsházy, PhD 4 th topic, Implementation of tasks, processes and threads Budapest University of Technology and Economics Department of Measurement

More information

Processes. CS 416: Operating Systems Design, Spring 2011 Department of Computer Science Rutgers University

Processes. CS 416: Operating Systems Design, Spring 2011 Department of Computer Science Rutgers University Processes Design, Spring 2011 Department of Computer Science Von Neuman Model Both text (program) and data reside in memory Execution cycle Fetch instruction Decode instruction Execute instruction CPU

More information

Overview. Rationale Division of labour between script and C++ Choice of language(s) Interfacing to C++ Performance, memory

Overview. Rationale Division of labour between script and C++ Choice of language(s) Interfacing to C++ Performance, memory SCRIPTING Overview Rationale Division of labour between script and C++ Choice of language(s) Interfacing to C++ Reflection Bindings Serialization Performance, memory Rationale C++ isn't the best choice

More information

CSCE Introduction to Computer Systems Spring 2019

CSCE Introduction to Computer Systems Spring 2019 CSCE 313-200 Introduction to Computer Systems Spring 2019 Processes Dmitri Loguinov Texas A&M University January 24, 2019 1 Chapter 3: Roadmap 3.1 What is a process? 3.2 Process states 3.3 Process description

More information

Comet and WebSocket Web Applications How to Scale Server-Side Event-Driven Scenarios

Comet and WebSocket Web Applications How to Scale Server-Side Event-Driven Scenarios Comet and WebSocket Web Applications How to Scale Server-Side Event-Driven Scenarios Simone Bordet sbordet@intalio.com 1 Agenda What are Comet web applications? Impacts of Comet web applications WebSocket

More information

Capriccio : Scalable Threads for Internet Services

Capriccio : Scalable Threads for Internet Services Capriccio : Scalable Threads for Internet Services - Ron von Behren &et al - University of California, Berkeley. Presented By: Rajesh Subbiah Background Each incoming request is dispatched to a separate

More information

Intel Threading Tools

Intel Threading Tools Intel Threading Tools Paul Petersen, Intel -1- INFORMATION IN THIS DOCUMENT IS PROVIDED IN CONNECTION WITH INTEL PRODUCTS. EXCEPT AS PROVIDED IN INTEL'S TERMS AND CONDITIONS OF SALE FOR SUCH PRODUCTS,

More information

CloudI Integration Framework. Chicago Erlang User Group May 27, 2015

CloudI Integration Framework. Chicago Erlang User Group May 27, 2015 CloudI Integration Framework Chicago Erlang User Group May 27, 2015 Speaker Bio Bruce Kissinger is an Architect with Impact Software LLC. Linkedin: https://www.linkedin.com/pub/bruce-kissinger/1/6b1/38

More information

CSE : Python Programming

CSE : Python Programming CSE 399-004: Python Programming Lecture 2: Data, Classes, and Modules January 22, 2007 http://www.seas.upenn.edu/~cse39904/ Administrative things Teaching assistant Brian Summa (bsumma @ seas.upenn.edu)

More information

UI, Graphics & EFL. Carsten Haitzler Principal Engineer Samsung Electronics Korea Founder/Leader Enlightenment / EFL

UI, Graphics & EFL. Carsten Haitzler Principal Engineer Samsung Electronics Korea Founder/Leader Enlightenment / EFL UI, Graphics & EFL Carsten Haitzler Principal Engineer Samsung Electronics Korea c.haitzler@samsung.com Founder/Leader Enlightenment / EFL Display System Overview Graphics 4 Graphics Old-School FB 5 In

More information

Chapter 4: Multithreaded

Chapter 4: Multithreaded Chapter 4: Multithreaded Programming Chapter 4: Multithreaded Programming Overview Multithreading Models Thread Libraries Threading Issues Operating-System Examples 2009/10/19 2 4.1 Overview A thread is

More information

Lecture 21 Concurrent Programming

Lecture 21 Concurrent Programming Lecture 21 Concurrent Programming 13th November 2003 Leaders and followers pattern Problem: Clients periodically present service requests which requires a significant amount of work that along the way

More information

ELEC 377 Operating Systems. Week 1 Class 2

ELEC 377 Operating Systems. Week 1 Class 2 Operating Systems Week 1 Class 2 Labs vs. Assignments The only work to turn in are the labs. In some of the handouts I refer to the labs as assignments. There are no assignments separate from the labs.

More information

An introduction to checkpointing. for scientifc applications

An introduction to checkpointing. for scientifc applications damien.francois@uclouvain.be UCL/CISM An introduction to checkpointing for scientifc applications November 2016 CISM/CÉCI training session What is checkpointing? Without checkpointing: $./count 1 2 3^C

More information

CSCE 120: Learning To Code

CSCE 120: Learning To Code CSCE 120: Learning To Code Module 11.0: Consuming Data I Introduction to Ajax This module is designed to familiarize you with web services and web APIs and how to connect to such services and consume and

More information

Python for Earth Scientists

Python for Earth Scientists Python for Earth Scientists Andrew Walker andrew.walker@bris.ac.uk Python is: A dynamic, interpreted programming language. Python is: A dynamic, interpreted programming language. Data Source code Object

More information

CSE398: Network Systems Design

CSE398: Network Systems Design CSE398: Network Systems Design Instructor: Dr. Liang Cheng Department of Computer Science and Engineering P.C. Rossin College of Engineering & Applied Science Lehigh University February 23, 2005 Outline

More information

10/10/ Gribble, Lazowska, Levy, Zahorjan 2. 10/10/ Gribble, Lazowska, Levy, Zahorjan 4

10/10/ Gribble, Lazowska, Levy, Zahorjan 2. 10/10/ Gribble, Lazowska, Levy, Zahorjan 4 What s in a process? CSE 451: Operating Systems Autumn 2010 Module 5 Threads Ed Lazowska lazowska@cs.washington.edu Allen Center 570 A process consists of (at least): An, containing the code (instructions)

More information

The control of I/O devices is a major concern for OS designers

The control of I/O devices is a major concern for OS designers Lecture Overview I/O devices I/O hardware Interrupts Direct memory access Device dimensions Device drivers Kernel I/O subsystem Operating Systems - June 26, 2001 I/O Device Issues The control of I/O devices

More information

Table of Contents EVALUATION COPY

Table of Contents EVALUATION COPY Table of Contents Introduction... 1-2 A Brief History of Python... 1-3 Python Versions... 1-4 Installing Python... 1-5 Environment Variables... 1-6 Executing Python from the Command Line... 1-7 IDLE...

More information

Operating Systems: Internals and Design Principles. Chapter 4 Threads Seventh Edition By William Stallings

Operating Systems: Internals and Design Principles. Chapter 4 Threads Seventh Edition By William Stallings Operating Systems: Internals and Design Principles Chapter 4 Threads Seventh Edition By William Stallings Operating Systems: Internals and Design Principles The basic idea is that the several components

More information

What s in a process?

What s in a process? CSE 451: Operating Systems Winter 2015 Module 5 Threads Mark Zbikowski mzbik@cs.washington.edu Allen Center 476 2013 Gribble, Lazowska, Levy, Zahorjan What s in a process? A process consists of (at least):

More information

19: I/O. Mark Handley. Direct Memory Access (DMA)

19: I/O. Mark Handley. Direct Memory Access (DMA) 19: I/O Mark Handley Direct Memory Access (DMA) 1 Interrupts Revisited Connections between devices and interrupt controller actually use interrupt lines on the bus rather than dedicated wires. Interrupts

More information

Advanced Computer Networks. End Host Optimization

Advanced Computer Networks. End Host Optimization Oriana Riva, Department of Computer Science ETH Zürich 263 3501 00 End Host Optimization Patrick Stuedi Spring Semester 2017 1 Today End-host optimizations: NUMA-aware networking Kernel-bypass Remote Direct

More information

Python 3000 and You. Guido van Rossum EuroPython July 7, 2008

Python 3000 and You. Guido van Rossum EuroPython July 7, 2008 Python 3000 and You Guido van Rossum EuroPython July 7, 2008 Why Py3k Open source needs to move or die Matz (creator of Ruby) To fix early, sticky design mistakes e.g. classic classes, int division, print

More information

The addition of improved support for asynchronous I/O in Python 3

The addition of improved support for asynchronous I/O in Python 3 DAVID BEAZLEY David Beazley is an open source developer and author of the Python Essential Reference (4th Edition, Addison-Wesley, 2009). He is also known as the creator of Swig (http://www.swig.org) and

More information

Object-Oriented Thinking

Object-Oriented Thinking Chapter 9 Object-Oriented Thinking Smalltalk is one of the pure Object-Oriented (OO) languages. Unlike C++, which makes it very easy to write procedural code (ie, use C++ as a better C), Smalltalk makes

More information

HotPy (2) Binary Compatible High Performance VM for Python. Mark Shannon

HotPy (2) Binary Compatible High Performance VM for Python. Mark Shannon HotPy (2) Binary Compatible High Performance VM for Python Mark Shannon Who am I? Mark Shannon PhD thesis on building VMs for dynamic languages During my PhD I developed: GVMT. A virtual machine tool kit

More information

CHAPTER 3 - PROCESS CONCEPT

CHAPTER 3 - PROCESS CONCEPT CHAPTER 3 - PROCESS CONCEPT 1 OBJECTIVES Introduce a process a program in execution basis of all computation Describe features of processes: scheduling, creation, termination, communication Explore interprocess

More information

Threads. Raju Pandey Department of Computer Sciences University of California, Davis Spring 2011

Threads. Raju Pandey Department of Computer Sciences University of California, Davis Spring 2011 Threads Raju Pandey Department of Computer Sciences University of California, Davis Spring 2011 Threads Effectiveness of parallel computing depends on the performance of the primitives used to express

More information

Large Systems: Design + Implementation: Communication Coordination Replication. Image (c) Facebook

Large Systems: Design + Implementation: Communication Coordination Replication. Image (c) Facebook Large Systems: Design + Implementation: Image (c) Facebook Communication Coordination Replication Credits Slides largely based on Distributed Systems, 3rd Edition Maarten van Steen Andrew S. Tanenbaum

More information

High Performance Python Micha Gorelick and Ian Ozsvald

High Performance Python Micha Gorelick and Ian Ozsvald High Performance Python Micha Gorelick and Ian Ozsvald Beijing Cambridge Farnham Koln Sebastopol Tokyo O'REILLY 0 Table of Contents Preface ix 1. Understanding Performant Python 1 The Fundamental Computer

More information

Autosave for Research Where to Start with Checkpoint/Restart

Autosave for Research Where to Start with Checkpoint/Restart Autosave for Research Where to Start with Checkpoint/Restart Brandon Barker Computational Scientist Cornell University Center for Advanced Computing (CAC) brandon.barker@cornell.edu Workshop: High Performance

More information

Questions answered in this lecture: CS 537 Lecture 19 Threads and Cooperation. What s in a process? Organizing a Process

Questions answered in this lecture: CS 537 Lecture 19 Threads and Cooperation. What s in a process? Organizing a Process Questions answered in this lecture: CS 537 Lecture 19 Threads and Cooperation Why are threads useful? How does one use POSIX pthreads? Michael Swift 1 2 What s in a process? Organizing a Process A process

More information

For use by students enrolled in #71251 CSE430 Fall 2012 at Arizona State University. Do not use if not enrolled.

For use by students enrolled in #71251 CSE430 Fall 2012 at Arizona State University. Do not use if not enrolled. Operating Systems: Internals and Design Principles Chapter 4 Threads Seventh Edition By William Stallings Operating Systems: Internals and Design Principles The basic idea is that the several components

More information

UNIT -3 PROCESS AND OPERATING SYSTEMS 2marks 1. Define Process? Process is a computational unit that processes on a CPU under the control of a scheduling kernel of an OS. It has a process structure, called

More information

Deviations are things that modify a thread s normal flow of control. Unix has long had signals, and these must be dealt with in multithreaded

Deviations are things that modify a thread s normal flow of control. Unix has long had signals, and these must be dealt with in multithreaded Deviations are things that modify a thread s normal flow of control. Unix has long had signals, and these must be dealt with in multithreaded improvements to Unix. There are actually two fairly different

More information

Announcements. My office hours are today in Gates 160 from 1PM-3PM. Programming Project 3 checkpoint due tomorrow night at 11:59PM.

Announcements. My office hours are today in Gates 160 from 1PM-3PM. Programming Project 3 checkpoint due tomorrow night at 11:59PM. IR Generation Announcements My office hours are today in Gates 160 from 1PM-3PM. Programming Project 3 checkpoint due tomorrow night at 11:59PM. This is a hard deadline and no late submissions will be

More information

Threads and Continuations COS 320, David Walker

Threads and Continuations COS 320, David Walker Threads and Continuations COS 320, David Walker Concurrency Concurrency primitives are an important part of modern programming languages. Concurrency primitives allow programmers to avoid having to specify

More information

操作系统概念 13. I/O Systems

操作系统概念 13. I/O Systems OPERATING SYSTEM CONCEPTS 操作系统概念 13. I/O Systems 东南大学计算机学院 Baili Zhang/ Southeast 1 Objectives 13. I/O Systems Explore the structure of an operating system s I/O subsystem Discuss the principles of I/O

More information

Processes and Threads. Processes and Threads. Processes (2) Processes (1)

Processes and Threads. Processes and Threads. Processes (2) Processes (1) Processes and Threads (Topic 2-1) 2 홍성수 Processes and Threads Question: What is a process and why is it useful? Why? With many things happening at once in a system, need some way of separating them all

More information

Chapter 4: Threads. Operating System Concepts. Silberschatz, Galvin and Gagne

Chapter 4: Threads. Operating System Concepts. Silberschatz, Galvin and Gagne Chapter 4: Threads Silberschatz, Galvin and Gagne Chapter 4: Threads Overview Multithreading Models Thread Libraries Threading Issues Operating System Examples Linux Threads 4.2 Silberschatz, Galvin and

More information

Evolution of Virtual Machine Technologies for Portability and Application Capture. Bob Vandette Java Hotspot VM Engineering Sept 2004

Evolution of Virtual Machine Technologies for Portability and Application Capture. Bob Vandette Java Hotspot VM Engineering Sept 2004 Evolution of Virtual Machine Technologies for Portability and Application Capture Bob Vandette Java Hotspot VM Engineering Sept 2004 Topics Virtual Machine Evolution Timeline & Products Trends forcing

More information

2017 Pearson Educa2on, Inc., Hoboken, NJ. All rights reserved.

2017 Pearson Educa2on, Inc., Hoboken, NJ. All rights reserved. Operating Systems: Internals and Design Principles Chapter 4 Threads Ninth Edition By William Stallings Processes and Threads Resource Ownership Process includes a virtual address space to hold the process

More information

AJAX: Asynchronous Event Handling Sunnie Chung

AJAX: Asynchronous Event Handling Sunnie Chung AJAX: Asynchronous Event Handling Sunnie Chung http://adaptivepath.org/ideas/ajax-new-approach-web-applications/ http://stackoverflow.com/questions/598436/does-an-asynchronous-call-always-create-call-a-new-thread

More information

Chapter 3 Processes. Process Concept. Process Concept. Process Concept (Cont.) Process Concept (Cont.) Process Concept (Cont.)

Chapter 3 Processes. Process Concept. Process Concept. Process Concept (Cont.) Process Concept (Cont.) Process Concept (Cont.) Process Concept Chapter 3 Processes Computers can do several activities at a time Executing user programs, reading from disks writing to a printer, etc. In multiprogramming: CPU switches from program to

More information

1 Multiprocessors. 1.1 Kinds of Processes. COMP 242 Class Notes Section 9: Multiprocessor Operating Systems

1 Multiprocessors. 1.1 Kinds of Processes. COMP 242 Class Notes Section 9: Multiprocessor Operating Systems COMP 242 Class Notes Section 9: Multiprocessor Operating Systems 1 Multiprocessors As we saw earlier, a multiprocessor consists of several processors sharing a common memory. The memory is typically divided

More information

Chapter 4: Threads. Overview Multithreading Models Thread Libraries Threading Issues Operating System Examples Windows XP Threads Linux Threads

Chapter 4: Threads. Overview Multithreading Models Thread Libraries Threading Issues Operating System Examples Windows XP Threads Linux Threads Chapter 4: Threads Overview Multithreading Models Thread Libraries Threading Issues Operating System Examples Windows XP Threads Linux Threads Chapter 4: Threads Objectives To introduce the notion of a

More information

MRI Internals. Koichi Sasada.

MRI Internals. Koichi Sasada. MRI Internals Koichi Sasada ko1@heroku.com MRI Internals towards Ruby 3 Koichi Sasada ko1@heroku.com Today s talk Koichi is working on improving Ruby internals Introduce my ideas toward Ruby 3 Koichi Sasada

More information

Distributed Systems Operation System Support

Distributed Systems Operation System Support Hajussüsteemid MTAT.08.009 Distributed Systems Operation System Support slides are adopted from: lecture: Operating System(OS) support (years 2016, 2017) book: Distributed Systems: Concepts and Design,

More information

Lecture 4: Process Management

Lecture 4: Process Management Lecture 4: Process Management (Chapters 2-3) Process: execution context of running program. A process does not equal a program! Process is an instance of a program Many copies of same program can be running

More information

W4118 Operating Systems. Junfeng Yang

W4118 Operating Systems. Junfeng Yang W4118 Operating Systems Junfeng Yang What is a process? Outline Process dispatching Common process operations Inter-process Communication What is a process Program in execution virtual CPU Process: an

More information

Chapter 13: I/O Systems

Chapter 13: I/O Systems Chapter 13: I/O Systems I/O Hardware Application I/O Interface Kernel I/O Subsystem Transforming I/O Requests to Hardware Operations Streams Performance Objectives Explore the structure of an operating

More information

How async and await ended up in Python. EuroPython 2018, Edinburgh

How async and await ended up in Python. EuroPython 2018, Edinburgh How async and await ended up in Python EuroPython 2018, Edinburgh Hello (: https://www.hacksoft.io Django React Twitter: @pgergov_ Python is synchronous Request Response Threads Asynchronous asyncio

More information

Introduction and Overview Socket Programming Higher-level interfaces Final thoughts. Network Programming. Samuli Sorvakko/Nixu Oy

Introduction and Overview Socket Programming Higher-level interfaces Final thoughts. Network Programming. Samuli Sorvakko/Nixu Oy Network Programming Samuli Sorvakko/Nixu Oy Telecommunications software and Multimedia Laboratory T-110.4100 Computer Networks October 16, 2008 Agenda 1 Introduction and Overview Introduction 2 Socket

More information

Native POSIX Thread Library (NPTL) CSE 506 Don Porter

Native POSIX Thread Library (NPTL) CSE 506 Don Porter Native POSIX Thread Library (NPTL) CSE 506 Don Porter Logical Diagram Binary Memory Threads Formats Allocators Today s Lecture Scheduling System Calls threads RCU File System Networking Sync User Kernel

More information

Introduction and Overview Socket Programming Lower-level stuff Higher-level interfaces Security. Network Programming. Samuli Sorvakko/Nixu Oy

Introduction and Overview Socket Programming Lower-level stuff Higher-level interfaces Security. Network Programming. Samuli Sorvakko/Nixu Oy Network Programming Samuli Sorvakko/Nixu Oy Telecommunications software and Multimedia Laboratory T-110.4100 Computer Networks October 5, 2009 Agenda 1 Introduction and Overview 2 Socket Programming 3

More information

Jython Python for the Java Platform. Jim Baker, Committer twitter.com/jimbaker zyasoft.com/pythoneering

Jython Python for the Java Platform. Jim Baker, Committer twitter.com/jimbaker zyasoft.com/pythoneering Jython Python for the Java Platform Jim Baker, Committer jbaker@zyasoft.com twitter.com/jimbaker zyasoft.com/pythoneering Jython Milestones 9 years ago - Released JPython 1.1 8 years ago - Released 2.0

More information