分析requests源码解决headers无法json问题

今又在丰富mock http接口了,由于我的开发速度太快,同事们的开发速度太慢,哈哈别打我, 他们在一定程度上拖慢了哥的大腿.  但又因为哥的接口要大量耦合调用他们的rest api,又为了kpi稳定的推进.  哥直接用tornado自己写一套完整的mock server…  

文章写的不是很严谨,欢迎来喷,另外该文后续有更新的,请到原文地址查看更新。

http://xiaorui.cc/2016/01/25/%E5%88%86%E6%9E%90requests%E6%BA%90%E7%A0%81%E8%A7%A3%E5%86%B3headers%E6%97%A0%E6%B3%95json%E9%97%AE%E9%A2%98/


这个是其中遇到的问题,以前都是用msgpack来序列化数据,但因为很多时候不好调试数据,你用json序列化数据还是能看懂的.  下面是我序列化headers头部信息报错.

#blog: xiaorui.cc

In [92]: import requests

In [93]: r = requests.get('http://www.xiaorui.cc')

In [94]: import json

In [95]: json.dumps(r.headers)
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-95-435fbd17ffc1> in <module>()
----> 1 json.dumps(r.headers)

/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/__init__.pyc in dumps(obj, skipkeys, ensure_ascii, check_circular, allow_nan, cls, indent, separators, encoding, default, sort_keys, **kw)
    241         cls is None and indent is None and separators is None and
    242         encoding == 'utf-8' and default is None and not sort_keys and not kw):
--> 243         return _default_encoder.encode(obj)
    244     if cls is None:
    245         cls = JSONEncoder

/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/encoder.pyc in encode(self, o)
    205         # exceptions aren't as detailed.  The list call should be roughly
    206         # equivalent to the PySequence_Fast that ''.join() would do.
--> 207         chunks = self.iterencode(o, _one_shot=True)
    208         if not isinstance(chunks, (list, tuple)):
    209             chunks = list(chunks)

/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/encoder.pyc in iterencode(self, o, _one_shot)
    268                 self.key_separator, self.item_separator, self.sort_keys,
    269                 self.skipkeys, _one_shot)
--> 270         return _iterencode(o, 0)
    271
    272 def _make_iterencode(markers, _default, _encoder, _indent, _floatstr,

/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/encoder.pyc in default(self, o)
    182
    183         """
--> 184         raise TypeError(repr(o) + " is not JSON serializable")
    185
    186     def encode(self, o):

TypeError: {'x-powered-by': 'PHP/5.3.3', 'transfer-encoding': 'chunked', 'content-encoding': 'gzip', 'server': 'nginx/1.8.0', 'connection': 'keep-alive', 'date': 'Mon, 25 Jan 2016 09:56:54 GMT', 'content-type': 'text/html; charset=UTF-8', 'x-pingback': 'http://xiaorui.cc/xmlrpc.php'} is not JSON serializabled

raise TypeError(repr(o) + ” is not JSON serializable”) ,还真没看到过着问题.  一顿狂搜出来的结果跟我遇到的问题都不一样,最后通过看requests structures类解决的.

我们看看headers的type,是CaseInsensitiveDict类.

In [99]: type(r.headers)
Out[99]: requests.structures.CaseInsensitiveDict

这是解决方法.

#blog: xiaorui.cc

In [96]: json.dumps(dict(r.headers))
Out[96]: '{"x-powered-by": "PHP/5.3.3", "transfer-encoding": "chunked", "content-encoding": "gzip", "server": "nginx/1.8.0", "connection": "keep-alive", "date": "Mon, 25 Jan 2016 09:56:54 GMT", "content-type": "text/html; charset=UTF-8", "x-pingback": "http://xiaorui.cc/xmlrpc.php"}'

In [97]: json.loads(json.dumps(dict(r.headers)))
Out[97]:
{u'connection': u'keep-alive',
 u'content-encoding': u'gzip',
 u'content-type': u'text/html; charset=UTF-8',
 u'date': u'Mon, 25 Jan 2016 09:56:54 GMT',
 u'server': u'nginx/1.8.0',
 u'transfer-encoding': u'chunked',
 u'x-pingback': u'http://xiaorui.cc/xmlrpc.php',
 u'x-powered-by': u'PHP/5.3.3'}

我们为什么要调用dict函数呢? 因为CaseInsensitiveDict类不能直接使用,他跟原装的dict字典微微有些差异的.

#blog: xiaorui.cc

In [100]: r.headers.items()
Out[100]:
[('content-encoding', 'gzip'),
 ('transfer-encoding', 'chunked'),
 ('x-powered-by', 'PHP/5.3.3'),
 ('server', 'nginx/1.8.0'),
 ('connection', 'keep-alive'),
 ('date', 'Mon, 25 Jan 2016 09:56:54 GMT'),
 ('content-type', 'text/html; charset=UTF-8'),
 ('x-pingback', 'http://xiaorui.cc/xmlrpc.php')]

In [101]: dict(r.headers.items())
Out[101]:
{'connection': 'keep-alive',
 'content-encoding': 'gzip',
 'content-type': 'text/html; charset=UTF-8',
 'date': 'Mon, 25 Jan 2016 09:56:54 GMT',
 'server': 'nginx/1.8.0',
 'transfer-encoding': 'chunked',
 'x-pingback': 'http://xiaorui.cc/xmlrpc.php',
 'x-powered-by': 'PHP/5.3.3'}

既然搜不出解决方法,那就直接看下requests的源码,看看为什么会有出现那问题… 

vim /Library/Python/2.7/site-packages/requests/structures.py

#blog: xiaorui.cc
继承了collections父类.

class CaseInsensitiveDict(collections.MutableMapping):
   
     """ CaseInsensitiveDict里用了各种的魔法函数,他的目的是为了抽象自带的dict字典."""
    def __init__(self, data=None, **kwargs):
        self._store = dict()
        if data is None:
            data = {}
        self.update(data, **kwargs)

    def __setitem__(self, key, value):
        # Use the lowercased key for lookups, but store the actual
        # key alongside the value.
        self._store[key.lower()] = (key, value)

    def __getitem__(self, key):
        return self._store[key.lower()][1]

    def __delitem__(self, key):
        del self._store[key.lower()]

    def __iter__(self):
        return (casedkey for casedkey, mappedvalue in self._store.

    def __len__(self):
        return len(self._store)

    def lower_items(self):
        """Like iteritems(), but with all lowercase keys."""
        return (
            (lowerkey, keyval[1])
            for (lowerkey, keyval)
            in self._store.items()
        )
    def __eq__(self, other):
        if isinstance(other, collections.Mapping):
            other = CaseInsensitiveDict(other)
        else:
            return NotImplemented
        return dict(self.lower_items()) == dict(other.lower_items())

    # Copy is required
    def copy(self):
        return CaseInsensitiveDict(self._store.values())

    #repr是返回一个可以用来表示对象的可打印字符串,简单说,str()出来的值是给人看的 ...  repr()出来的值是给python看的,可以通过eval()重新变回一个Python对象...
    def __repr__(self):
        return str(dict(self.items()))

通过看了requests的相关源码,我们还可以这么理解,所谓是一通百通 …   repr跟str的区别就在于此.

In [111]: str(r.headers)
Out[111]: “{‘x-powered-by’: ‘PHP/5.3.3’, ‘transfer-encoding’: ‘chunked’, ‘content-encoding’: ‘gzip’, ‘server’: ‘nginx/1.8.0’, ‘connection’: ‘keep-alive’, ‘date’: ‘Mon, 25 Jan 2016 09:56:54 GMT’, ‘content-type’: ‘text/html; charset=UTF-8’, ‘x-pingback’: ‘http://xiaorui.cc/xmlrpc.php’}”

In [112]: eval(str(r.headers))
Out[112]:
{‘connection’: ‘keep-alive’,
 ‘content-encoding’: ‘gzip’,
 ‘content-type’: ‘text/html; charset=UTF-8’,
 ‘date’: ‘Mon, 25 Jan 2016 09:56:54 GMT’,
 ‘server’: ‘nginx/1.8.0’,
 ‘transfer-encoding’: ‘chunked’,
 ‘x-pingback’: ‘http://xiaorui.cc/xmlrpc.php’,
 ‘x-powered-by’: ‘PHP/5.3.3’}

遇到这个python requests不是坑的坑. 现在也不是很理解为什么要那么设计.  就这样吧,本来不大的问题,扯这么多… 见谅…


大家觉得文章对你有些作用! 如果想赏钱,可以用微信扫描下面的二维码,感谢!
另外再次标注博客原地址  xiaorui.cc

发表评论

邮箱地址不会被公开。 必填项已用*标注