【原】python threading模块进行多线程编程

生信修炼手册 2020-05-24

展开全文

提高程序运行效率的常见方法包括多进程和多线程两种，前面已经介绍了python中的多进程编程，今天来看下多线程在python中的实现。

在使用python的多线程之前，首先要理解GIL这个概念。GIL是Global Interpreter Lock的缩写，称之为全局解释器锁，是python在开发之初为了保证数据安全而设计的，每一个python进程只有一个GIL，同一时刻，只有拿到GIL的线程可以运行，这就使得python中的多线程无法实现真正意义上的并发。所以多线程在python中的应用场景受到了限制，只适用于处理文件IO，网络IO密集型的任务。

在python中，通过内置模块threading实现多线程处理，基本用法和多进程类似，示意如下

import threadingimport urllib.request

def download_html(pathway): print('Start download kgml') url = 'http://rest./get/{}/kgml'.format(pathway) out = './{}.kgml'.format(pathway) f = urllib.request.urlopen(url) with open(out, 'w') as fp: fp.write(f.read().decode('utf8'))

if __name__ == '__main__': pathway = 'hsa00010' p = threading.Thread(target = download_html, args = (pathway, )) p.start() p.join() print('Finish download kgml')

通过Thread类来定义一个线程，start方法用于启动线程，join方法用于阻塞线程。上述代码展示了用一个单独的线程来下载pathway对应的kgml文件。如果有多个pathway对应的文件要下载，用多线程的写法如下

pathways = ['hsa00010', 'hsa00020', 'hsa00030', 'hsa00040', 'hsa00051', 'hsa00052', 'hsa00053']thread_list = []for pathway in pathways: p = threading.Thread(target = download_html, args = (pathway, )) thread_list.append(p) p.start()

for thread in thread_list: thread.join()

尽管多线程并不是真正意义上的并发，但是也有对应的方法来控制同时运行的最大线程数，代码如下

import threadingimport urllib.request

def download_html(pathway, semaphore): semaphore.acquire() print('Start download kgml') url = 'http://rest./get/{}/kgml'.format(pathway) out = './{}.kgml'.format(pathway) f = urllib.request.urlopen(url) with open(out, 'w') as fp: fp.write(f.read().decode('utf8')) semaphore.release()

if __name__ == '__main__': pathways = ['hsa00010', 'hsa00020', 'hsa00030', 'hsa00040', 'hsa00051', 'hsa00052', 'hsa00053']

thread_list = [] semaphore = threading.BoundedSemaphore(3) for pathway in pathways: p = threading.Thread(target = download_html, args = (pathway, semaphore )) p.start() thread_list.append(p) for thread in thread_list: thread.join()

print('Finish download kgml')

多线程中变量是共享的，如果每个子进程都对同一个变量进行修改，就会出现预期之外的错误，专业点的说法叫做产生了脏数据，示例如下

import threadingimport urllib.request

# 存钱def append_money(): global total for cnt in range(1000000): total += cnt print('total money : {}'.format(total))

# 花钱def remove_money(): global total for cnt in range(1000000): total -= cnt print('total money : {}'.format(total))

if __name__ == '__main__': total = 100 print('total money : {}'.format(total))

p1 = threading.Thread(target = append_money, args = ()) p1.start()

p2 = threading.Thread(target = remove_money, args = ()) p2.start()

p1.join() p2.join()

print('total money : {}'.format(total))

多次运行上述代码，每次的结果会不一样

C:\Users\Administrator\Desktop>python test.pytotal money : 100total money : 340552501975total money : -33525835564total money : -33525835564

C:\Users\Administrator\Desktop>python test.pytotal money : 100total money : -55696821900total money : -197689058903total money : -197689058903

C:\Users\Administrator\Desktop>python test.pytotal money : 100total money : -260664176670total money : -245691977911total money : -245691977911

多个进程同时对一个变量进行修改，就是会存在脏数据的隐患，为此，我们需要对线程加锁，保证每次只有一个线程对变量进行修改，代码如下

import threadingimport urllib.request

def append_money(lock): lock.acquire() global total for cnt in range(1000000): total += cnt print('total money : {}'.format(total)) lock.release()

def remove_money(lock): lock.acquire() global total for cnt in range(1000000): total -= cnt print('total money : {}'.format(total)) lock.release()

if __name__ == '__main__': total = 100 print('total money : {}'.format(total)) lock = threading.Lock() p1 = threading.Thread(target = append_money, args = (lock, )) p1.start()

p2 = threading.Thread(target = remove_money, args = (lock, )) p2.start()

p1.join() p2.join()

print('total money : {}'.format(total))

添加了锁之后，就可以保证多次运行的结果都和预期保持一致了