解决python rq获取返回结果和异常的问题

首先说明下python rq的一个错误提示  Functions from the __main__ module cannot be processed ,  这个提示是在enqueue的时候发生的,他会检测你真正干活的模块是否跟rq.enqueue是在同一个模块。 RQ 暂时是不支持的在一起的。 

文章后期会有所更新,这里标注下原文链接,http://xiaorui.cc/?p=2326

下面是发生错误的上下文场景:

#xiaorui.cc
import requests

def count_words_at_url(url):
    resp = requests.get(url)
    return len(resp.text.split())

from rq import Connection, Queue
from redis import Redis
redis_conn = Redis()
q = Queue(connection=redis_conn)
job = q.enqueue(count_words_at_url, 'http://xiaorui.cc')

当你enqueue的时候,提示了下面的错误….

Traceback (most recent call last):
File “<stdin>”, line 1, in <module>
File “/Library/Python/2.7/site-packages/rq/queue.py”, line 159, in enqueue
‘Functions from the __main__ module cannot be processed ‘
>>> ValueError: Functions from the __main__ module cannot be processed by workers.

至于原因上面有所描述,就是任务发起的不能跟要你的函数在一个文件里面。又看了下rq的源代码 ,解决方法是,只要把函数的名字字符串及参数塞入队列就可以了。

job = q.enqueue(‘count_words_at_url’, ‘http://xiaorui.cc’)

Traceback (most recent call last):
  File “/Library/Python/2.7/site-packages/rq/worker.py”, line 558, in perform_job
    rv = job.perform()
  File “/Library/Python/2.7/site-packages/rq/job.py”, line 495, in perform
    self._result = self.func(*self.args, **self.kwargs)
  File “/Library/Python/2.7/site-packages/rq/job.py”, line 206, in func
    return import_attribute(self.func_name)
  File “/Library/Python/2.7/site-packages/rq/utils.py”, line 148, in import_attribute
    module_name, attribute = name.rsplit(‘.’, 1)
ValueError: need more than 1 value to unpack
Traceback (most recent call last):
  File “/Library/Python/2.7/site-packages/rq/worker.py”, line 558, in perform_job
    rv = job.perform()
  File “/Library/Python/2.7/site-packages/rq/job.py”, line 495, in perform
    self._result = self.func(*self.args, **self.kwargs)
  File “/Library/Python/2.7/site-packages/rq/job.py”, line 206, in func
    return import_attribute(self.func_name)
  File “/Library/Python/2.7/site-packages/rq/utils.py”, line 148, in import_attribute
    module_name, attribute = name.rsplit(‘.’, 1)
ValueError: need more than 1 value to unpack

23:32:27 Moving job to failed queue.
23:32:27
23:32:27 *** Listening on default…

结果还是报错…  因为rq rqworker不仅需要函数名字,而且需要函数所属的模块,也就是具体的py文件。

那么怎么解决? 在塞入func.func_name函数名字的时候要把模块也一起塞入进去。

job = q.enqueue('task.count_words_at_url', 'http://xiaorui.cc')

另外在使用RQ的时候需要给用户返回job_id ,然后用户再使用job id取出任务的相关信息(结果,状态,完成时间,创建时间)。 

job = q.enqueue('task.count_words_at_url', 'http://xiaorui.cc')
return job.key

获取结果第一个方法:

import rq
job_id=job.key.split(':')[2]
job = rq.job.Job.fetch(job_id,redis_conn) #fetch Job from redis
if job.is_finished:
    ret = job.return_value
elif job.is_queued:
    ret = {'status':'in-queue'}
elif job.is_started:
    ret = {'status':'waiting'}
elif job.is_failed:
    ret = {'status': 'failed'}

获取结果第二个方法:

return redis_conn.hgetall(job_id)

END…


大家觉得文章对你有些作用! 如果想赏钱,可以用微信扫描下面的二维码,感谢!
另外再次标注博客原地址  xiaorui.cc

发表评论

邮箱地址不会被公开。 必填项已用*标注