在Django中,如何在启动时间较慢的情况下调用子进程

假设您在Linux上运行Django,并且您有一个视图,并且您希望该视图从名为cmd的子进程返回数据,该子进程对视图创建的文件进行操作,例如：likeo：

 def call_subprocess(request):
     response = HttpResponse()

     with tempfile.NamedTemporaryFile("W") as f:
         f.write(request.GET['data']) # i.e. some data

     # cmd operates on fname and returns output
     p = subprocess.Popen(["cmd", f.name], 
                   stdout=subprocess.PIPE, 
                   stderr=subprocess.PIPE)

     out, err = p.communicate()

     response.write(p.out) # would be text/plain...
     return response

现在,假设cmd的启动时间非常慢,但运行时间非常快,并且它本身没有守护进程模式.我想改善这种观点的响应时间.

我想通过在工作池中启动一些cmd实例,让它们等待输入,并让call_process请求其中一个工作池进程处理数据,使整个系统运行得更快.

这实际上是两个部分：

第1部分.调用cmd和cmd的函数等待输入.这可以通过管道完成,即

def _run_subcmd():
    p = subprocess.Popen(["cmd", fname], 
        stdout=subprocess.PIPE, stderr=subprocess.PIPE)

    out, err = p.communicate()
    # write 'out' to a tmp file
    o = open("out.txt", "W")
    o.write(out)
    o.close()
    p.close()
    exit()

def _run_cmd(data):
    f = tempfile.NamedTemporaryFile("W")
    pipe = os.mkfifo(f.name)

    if os.fork() == 0:
        _run_subcmd(fname)
    else:
        f.write(data)

    r = open("out.txt", "r")
    out = r.read()
    # read 'out' from a tmp file
    return out

def call_process(request):
    response = HttpResponse()

    out = _run_cmd(request.GET['data'])

    response.write(out) # would be text/plain...
    return response

第2部分.在后台运行的一组正在等待数据的工作者.即,我们希望扩展上述内容,以便子进程已经运行,例如当Django实例初始化时,或者首次调用此call_process时,会创建一组这些worker

WORKER_COUNT = 6
WORKERS = []

class Worker(object):
    def __init__(index):
        self.tmp_file = tempfile.NamedTemporaryFile("W") # get a tmp file name
        os.mkfifo(self.tmp_file.name)
        self.p = subprocess.Popen(["cmd", self.tmp_file], 
            stdout=subprocess.PIPE, stderr=subprocess.PIPE)
        self.index = index

    def run(out_filename, data):
        WORKERS[self.index] = Null # qua-mutex??
        self.tmp_file.write(data)
        if (os.fork() == 0): # does the child have access to self.p??
            out, err = self.p.communicate()
            o = open(out_filename, "w")
            o.write(out)
            exit()

        self.p.close()
        self.o.close()
        self.tmp_file.close()
        WORKERS[self.index] = Worker(index) # replace this one
        return out_file

    @classmethod
    def get_worker() # get the next worker
    # ... static, incrementing index 

应该在某处对工作人员进行一些初始化,如下所示：

def init_workers(): # create WORKERS_COUNT workers
    for i in xrange(0, WORKERS_COUNT):
        tmp_file = tempfile.NamedTemporaryFile()
        WORKERS.push(Worker(i))

现在,我上面的内容变成了以下内容：

def _run_cmd(data):
     Worker.get_worker() # this needs to be atomic & lock worker at Worker.index

     fifo = open(tempfile.NamedTemporaryFile("r")) # this stores output of cmd

     Worker.run(fifo.name, data)
     # please ignore the fact that everything will be
     # appended to out.txt ... these will be tmp files, too, but named elsewhere.

     out = fifo.read()
     # read 'out' from a tmp file
     return out


def call_process(request):
     response = HttpResponse()

     out = _run_cmd(request.GET['data'])

     response.write(out) # would be text/plain...
     return response

现在,问题：

>这会有用吗？ (我只是把它从头顶输入StackOverflow,所以我确定存在问题,但从概念上讲,它会起作用)
>要寻找的问题是什么？
>有更好的替代品吗？例如线程是否也能正常运行(它是Debian Lenny Linux)？是否有任何库可以处理这样的并行流程工作池？
>我应该注意与Django的交互吗？

谢谢阅读！我希望你觉得这和我一样有趣.

布赖恩

解决方法:

看起来我正在推销这款产品,因为这是我第二次回复推荐.

但似乎您需要一个Message Queing服务,特别是一个分布式消息队列.

它是如何工作的：

>您的Django应用程序请求CMD
> CMD被添加到队列中
> CMD被推到了几个作品
>它被执行并且结果返回上游

大多数代码都存在,您不必去构建自己的系统.

看看最初用Django构建的Celery.

http://www./
http:///blog/2009/09/10/rabbitmq-celery-and-django/

来源：http://www./content-3-227951.html