Lecture 8. Introduction to Python! Lecture 8

Transcription

1 Lecture 8 Introduction to Python Lecture 8

2 Summary Python exceptions Processes and Threads Programming with Threads

3 Python Exceptions In Python, there are two distinguishable kinds of errors: syntax errors and exceptions. Syntax errors are usually called parsing errors. Exceptions are events that can modify the flow of control through a program. Python triggers exceptions automatically on errors, after that they can be intercepted and processed by the program, or can be thrown up to the calling methods. Exceptions allow jumping directly to a routine/handler which can process the unexpected occurred event

4 Exception roles Error handling Event notification Special case handling Termination actions Unusual flow control

5 Statements processing exceptions try/except Catch and recover from exceptions raised by Python, or by you. try/finally Perform cleanup actions, whether exceptions occur or not. raise Trigger an exception manually in your code. assert Conditionally trigger an exception in your code. with/as Implement context managers in Python 2.6 and 3.0 (optional in 2.5)

6 The general format of the try statement try: <statements> except <name1>: <statements> except (name2, name3): <statements> except <name4> as <data>: <statements> except: <statements> else: <statements> # Run this main action first # Run if name1 is raised during try block # Run if any of these exceptions occur # Run if name4 is raised, and get instance raised # Run for all (other) exceptions raised # Run if no exception was raised during try block

7 try statement clauses Clause Comments except: except name: except name as value: except (name1, name2): except (name1, name2) as value: else: finally: Catch all (or all other) exception types Catch a specific exception only Catch the listed exception and its instance Catch any of the listed exceptions Catch any listed exception and its instance Run if no exceptions are raised Always perform this block

8 Catching an exception def getatindex(str, idx): return str[idx] print(getatindex("python", 7)) print("statement after catching exception...") Output: Traceback (most recent call last): File "exception-1.py", line 5, in <module> print(getatindex("python", 7)) File "exception-1.py", line 2, in getatindex return str[idx] IndexError: string index out of range Notice that the statement after the function call is never executed Catching the exception def getatindex(str, idx): return str[idx] try: print(getatindex("python", 7)) except IndexError: print("exception occurred ) print("statement after catching exception...") Output: Exception occurred Statement after catching exception... After handling the exception, the execution of the program is resumed

9 Raising an exception def getatindex(str, idx): if idx > len(str): raise IndexError return str[idx] try: print(getatindex("python", 7)) except IndexError: print("exception occurred ) print("statement after catching exception...") Output: Exception occurred Statement after catching exception... One can raise exceptions using the assert directive too: def assert_fct(x): assert x = 0 print(x) assert_fct(0) Output: Traceback (most recent call last): File "exception-5.py", line 5, in <module> assert_fct(0) File "exception-5.py", line 2, in assert_fct assert x = 0 AssertionError

10 Creating a custom exception class CustomException(Exception): def str (self): return "CustomException representation..." def getatindex(str, idx): if idx > len(str): raise CustomException return str[idx] try: print(getatindex("python", 7)) except CustomException as e: print("exception occurred {}".format(e)) Output: Exception occurred CustomException representation...

11 Custom exceptions The recommended mode for creating custom exceptions is by using OOP, using classes and classes hierarchies. The most important advantages of using class-based exceptions are: They can be organized in categories (adding future categories will not require changes in the try block) They have state information attached They support inheritance

12 Termination Actions Let s suppose we need to deal with a file, open it, do some processing, then close it class MyException(Exception): pass def process_file(file): raise MyException try: f = open("../lab3/morse.txt", "r") process_file(f) print("after file processing...") finally: f.close() print("finally statement reached...") Output: finally statement reached... File "exception-4.py", line 9, in <module> process_file(f) File "exception-4.py", line 5, in process_file raise MyException main.myexception The try/finally construction assures that the statements included in the finally block are executed regardless of what happens in the try block This is extremely useful when we involve various resources in our program and we have to release them regardless of what events could occur during the execution of the program flow

13 Unified try/except/finally class MyException(Exception): def str (self): return "MyException" def process_file(file): raise MyException try: f = open("../lab3/morse.txt", "r") process_file(f) print("after file processing...") except MyException as me: print("exception occurred {}".format(me)) finally: f.close() print("finally statement reached...") Output: Exception occurred MyException finally statement reached... In this case, the exception raised in process_file() function is caught and processed in the exception handler defined in the except block, then the code defined in finally section is executed. In this way we assure ourselves that we release all the resources that we used during the program (file handlers, connections to databases, sockets, etc )

14 C vs. Python error handling C style Python style dostuff() { if (dofirstthing() == ERROR) return ERROR; if (donextthing() == ERROR) return ERROR;... return dolastthing(); } main() { if (dostuff() == ERROR) badending(); else goodending(); } def dostuff(): dofirstthing() donextthing()... dolastthing() if name == ' main ': try: dostuff() except: badending() else: goodending()

15 Processes and Threads Processes and Threads

16 Processes context switching How multiple processes share a CPU (image from 2 states for processes: Run Sleep PCB - Program Context Block

17 Relation between threads and processes [image taken from

18 Python Threads A Thread or a Thread of Execution is defined in computer science as the smallest unit that can be scheduled in an operating system. There are two different kind of threads: Kernel threads User-space Threads or user threads (image from

19 Threads Every process has at least one thread, i.e. the process itself. A process can start multiple threads. The operating system executes these threads like parallel "processes". On a single processor machine, this parallelism is achieved by thread scheduling or timeslicing. Advantages of Threading: Multithreaded programs can run faster on computer systems with multiple CPUs, because theses threads can be executed truly concurrent. A program can remain responsive to input. This is true both on single and on multiple CPU Threads of a process can share the memory of global variables. If a global variable is changed in one thread, this change is valid for all threads. A thread can have local variables.

20 Threads in Python Python has a Global Interpreter Lock (GIL) It imposes various restrictions on threads, one of the most important is that it cannot utilize multiple CPUs Consider the following piece of code: def count(n): while n > 0: n -= 1 Two scenarios: Run it twice in series: count( ) count( ) Now, run it in parallel in two threads t1 = Thread(target=count,args=( ,)) t1.start() t2 = Thread(target=count,args=( ,)) t2.start() t1.join(); t2.join() Some performance results on a Dual-Core MacBook? Sequential : 24.6s Threaded : 45.5s (1.8X slower) And if disabled one of the CPU cores, why does the threaded performance get better? Threaded : 38.0s

21 Python Threads Thread-Specific State Each thread has its own interpreter specific data structure (PyThreadState) Current stack frame (for Python code) Current recursion depth Thread ID Some per-thread exception information Optional tracing/profiling/debugging hooks It's a small C structure (<100 bytes) The interpreter has a global variable that simply points to the ThreadState structure of the currently running thread

22 Python Threads Only one Python thread can execute in the interpreter at once There is a "global interpreter lock" that carefully controls thread execution The GIL ensures that sure each thread get exclusive access to the interpreter internals when it's running (and that call-outs to C extensions play nice)

23 Using _thread package import _thread def print_chars(thread_name, char): for i in range(5): print("{} {}".format(thread_name, i * char)) print("{} successfully ended".format(thread_name)) try: print("starting main program...") _thread.start_new_thread(print_chars, ("Thread1 ", "*")) _thread.start_new_thread(print_chars, ("Thread2 ", "-")) _thread.start_new_thread(print_chars, ("Thread3 ", "=")) print("ending main program...") except: print("threads are unable to start") while 1: pass Starting main program... Ending main program... Thread2 Thread2 - Thread2 -- Thread2 --- Thread Thread2 successfully ended Thread3 Thread3 = Thread3 == Thread3 === Thread1 Thread1 * Thread1 ** Thread1 *** Thread1 **** Thread1 successfully ended Thread3 ==== Thread3 successfully ended _thread is a deprecated module, acting at low-level. _thread.start_new_thread method was used here to start a new thread. One can notice here that we have a while loop in the main program, looping forever. This is to avoid the situation in which the main program starts the three threads, but it exists before the threads finish their execution, and the buffers are not flushed (the output is buffered by default) See the program output and notice that the threads output is displayed after the main program end its execution.

24 First threading program import threading class ThreadExample(threading.Thread): def init (self, name, char): threading.thread. init (self) self.name = name self.char = char def run(self): print("starting thread {} ".format(self.name)) print_chars(self.name, self.char) print("ending thread {} ".format(self.name)) def print_chars(thread_name, char): for i in range(5): print("{} {}".format(thread_name, i * char)) if name == " main ": thread_one = ThreadExample("Thread1", "*") thread_two = ThreadExample("Thread2", "-") thread_three = ThreadExample("Thread3", "=") thread_one.start() thread_two.start() thread_three.start() print("exiting the main program...") Starting thread Thread1 Thread1 Thread1 * Thread1 ** Thread1 *** Thread1 **** Ending thread Thread1 Starting thread Thread2 Thread2 Thread2 - Thread2 -- Thread2 --- Thread Ending thread Thread2 Starting thread Thread3 Exiting the main program... Thread3 Thread3 = Thread3 == Thread3 === Thread3 ==== Ending thread Thread3? What happened here?

25 Using join() import threading class ThreadExample(threading.Thread): def init (self, name, char): threading.thread. init (self) self.name = name self.char = char def run(self): print("starting thread {} ".format(self.name)) print_chars(self.name, self.char) print("ending thread {} ".format(self.name)) def print_chars(thread_name, char): for i in range(5): print("{} {}".format(thread_name, i * char)) if name == " main ": thread_one = ThreadExample("Thread1", "*") thread_two = ThreadExample("Thread2", "-") thread_three = ThreadExample("Thread3", "=") thread_one.start() thread_two.start() thread_three.start() thread_one.join() thread_two.join() thread_three.join() print("exiting the main program...") Starting thread Thread1 Thread1 Thread1 * Thread1 ** Thread1 *** Thread1 **** Ending thread Thread1 Starting thread Thread2 Thread2 Thread2 - Thread2 -- Thread2 --- Thread Ending thread Thread2 Starting thread Thread3 Thread3 Thread3 = Thread3 == Thread3 === Thread3 ==== Ending thread Thread3 Exiting the main program... Note that now the main program ends AFTER all the threads finish

26 Determining current thread import threading import random class ThreadExample(threading.Thread): def init (self, name = None, char = '+'): threading.thread. init (self) if name is None: name = 'Thread' + str(random.randint(1, 100)) else: self.name = name self.char = char def run(self): print("starting thread {} ".format(self.name)) print_chars(threading.current_thread().getname(), self.char) print("ending thread {} ".format(self.name)) def print_chars(thread_name, char): for i in range(5): print("{} {}".format(thread_name, i * char)) if name == " main ": thread_one = ThreadExample(char = "*") thread_two = ThreadExample(char = "-") thread_three = ThreadExample(char = "=") thread_one.start() thread_two.start() thread_three.start() thread_one.join() thread_two.join() thread_three.join() print("exiting the main program...") Starting thread Thread-1 Thread-1 Thread-1 * Thread-1 ** Thread-1 *** Thread-1 **** Ending thread Thread-1 Starting thread Thread-2 Thread-2 Thread-2 - Thread-2 -- Thread Thread Ending thread Thread-2 Starting thread Thread-3 Thread-3 Thread-3 = Thread-3 == Thread-3 === Thread-3 ==== Ending thread Thread-3 Exiting the main program...

27 Thread synchronization Thread synchronization is defined as a mechanism which ensures that two or more concurrent processes or threads do not simultaneously execute some particular program segment known as critical section.

28 Synchronizing threads import threading import time class mythread (threading.thread): def init (self, threadid, name, counter): threading.thread. init (self) self.threadid = threadid self.name = name self.counter = counter def run(self): print ("Starting " + self.name) print_time(self.name, self.counter, 3) def print_time(threadname, delay, counter): while counter: time.sleep(delay) print ("%s: %s" % (threadname, time.ctime(time.time()))) counter -= 1 threads = [] # Create new threads thread1 = mythread(1, "Thread-1", 1) thread2 = mythread(2, "Thread-2", 2) # Start new Threads thread1.start() thread2.start() # Add threads to thread list threads.append(thread1) threads.append(thread2) # Wait for all threads to complete for t in threads: t.join() print ("Exiting Main Thread") We want to print some info about the running threads, first the info about one thread, then the info about a second thread. But, wait What happens with the output? The print result is scrambled between threads Not really what we wanted, right? Output: Starting Thread-1 Starting Thread-2 Thread-1: Thu Nov 24 18:20: Thread-2: Thu Nov 24 18:20: Thread-1: Thu Nov 24 18:20: Thread-1: Thu Nov 24 18:20: Thread-2: Thu Nov 24 18:20: Thread-2: Thu Nov 24 18:20: Exiting Main Thread

29 Synchronizing threads import threading import time class mythread (threading.thread): def init (self, threadid, name, counter): threading.thread. init (self) self.threadid = threadid self.name = name self.counter = counter def run(self): print ("Starting " + self.name) # Get lock to synchronize threads threadlock.acquire() print_time(self.name, self.counter, 3) # Free lock to release next thread threadlock.release() def print_time(threadname, delay, counter): while counter: time.sleep(delay) print ("%s: %s" % (threadname, time.ctime(time.time()))) counter -= 1 threadlock = threading.lock() threads = [] # Create new threads thread1 = mythread(1, "Thread-1", 1) thread2 = mythread(2, "Thread-2", 2) # Start new Threads thread1.start() thread2.start() # Add threads to thread list threads.append(thread1) threads.append(thread2) # Wait for all threads to complete for t in threads: t.join() print ("Exiting Main Thread") This time one can see that the output looks much better. critical section Output: Starting Thread-1 Starting Thread-2 Thread-1: Thu Nov 24 18:26: Thread-1: Thu Nov 24 18:27: Thread-1: Thu Nov 24 18:27: Thread-2: Thu Nov 24 18:27: Thread-2: Thu Nov 24 18:27: Thread-2: Thu Nov 24 18:27: Exiting Main Thread

30 Queue data structure A queue is an abstract data structure, open at both ends. One of the ends is always used to insert data in the structure, while the other end is used to extract data fro the structure. The data insertion operation is called enqueueing, while the extraction of the data from the queue is called dequeueing.

31 Multithread Queues in Python The Queue module provides a FIFO implementation suitable for multithreaded programming. It can be used to pass messages or other data between producer and consumer threads safely. Locking is handled for the caller, so it is simple to have as many threads as you want working with the same Queue instance. get(): The get() removes and returns an item from the queue. put(): The put() adds item to a queue. qsize() : The qsize() returns the number of items that are currently in the queue. empty(): The empty() returns True if queue is empty; otherwise, False. full(): the full() returns True if queue is full; otherwise, False.

32 Multithread Queues import queue import threading import time exitflag = 0 class mythread (threading.thread): def init (self, threadid, name, q): threading.thread. init (self) self.threadid = threadid self.name = name self.q = q def run(self): print ("Starting " + self.name) process_data(self.name, self.q) print ("Exiting " + self.name) def process_data(threadname, q): while not exitflag: queuelock.acquire() if not workqueue.empty(): data = q.get() queuelock.release() print ("%s processing %s" % (threadname, data)) else: queuelock.release() time.sleep(1) threadlist = ["Thread-1", "Thread-2", "Thread-3"] namelist = ["One", "Two", "Three", "Four", "Five"] queuelock = threading.lock() workqueue = queue.queue(10) threads = [] threadid = 1 # Create new threads for tname in threadlist: thread = mythread(threadid, tname, workqueue) thread.start() threads.append(thread) threadid += 1 # Fill the queue queuelock.acquire() for word in namelist: workqueue.put(word) queuelock.release() # Wait for queue to empty while not workqueue.empty(): pass # Notify threads it's time to exit exitflag = 1 # Wait for all threads to complete for t in threads: t.join() print ("Exiting Main Thread") Starting Thread-1 Starting Thread-2 Starting Thread-3 Thread-1 processing One Thread-2 processing Two Thread-3 processing Three Thread-1 processing Four Thread-3 processing Five Exiting Thread-1 Exiting Thread-2 Exiting Thread-3 Exiting Main Thread

33 References Mark Lutz - Learning Python, O Reilly Queue A thread-safe FIFO implementationhttps://pymotw.com/2/ Queue/ Python 3 - Multithreaded programming Inside the Python GIL - Operating System Tutorial - operating_system/os_process_scheduling.html