前些日子朋友让我过去坐坐，顺便做个关于python的技术分享。话题有他们来挑选的，出了几个主题，最后定了python内存管理话题。 这次ppt做的简而易懂，图多字少，尽量少输出高大上的概念，多拓扑一些实用的概念。 我相信大家有些cpython源码的基础是可以看懂ppt的。 如果有问题，大家可以跟我联系，邮箱 rfyiamcool@163.com 。
ppt下载地址:
在线地址:
https://www.slideshare.net/rfyiamcool/python-76939304
python高级内存管理 from rfyiamcool
1. Python⾼级内存管理 - xiaorui.cc
2. Object-specific allocators _____ ______ ______ ________ [ int ] [ dict ] [ list ] ... [ string ] Python core | +3 | <----- Object-specific memory -----> | <-- Non-object memory --> | _______________________________ | | [ Python's object allocator ] | | +2 | ####### Object memory ####### | <------ Internal buffers ------> | ______________________________________________________________ | [ Python's raw memory allocator (PyMem_ API) ] | +1 | <----- Python memory (under PyMem manager's control) ------> | | __________________________________________________________________ [ Underlying general-purpose allocator (ex: C library malloc) ] 0 | <------ Virtual memory allocated for the python process -------> | ========================================================================= _______________________________________________________________________ [ OS-specific Virtual Memory Manager (VMM) ] -1 | <--- Kernel dynamic storage allocation & management (page-based) ---> | __________________________________ __________________________________ [ ] [ ] -2 | <-- Physical memory: ROM/RAM --> | | <-- Secondary storage (swap) --> |
3. * Request in bytes Size of allocated block Size class idx * ---------------------------------------------------------------- * 1-8 8 0 * 9-16 16 1 * 17-24 24 2 * 25-32 32 3 * 33-40 40 4 * 41-48 48 5 * 49-56 56 6 * 57-64 64 7 * ... ... ... * 497-504 504 62 * 505-512 512 63 * * */
4. 名词解释 process heap Arenas Pool UsedPools FreePools
5. method posix malloc python memory pool object buffer pool
6. Arena FeeePool Pool Pool Pool Pool Headers No BLock Arena malloc heap & pool Process stack heap bss init data text UserPool 1-8 … … 249 - 256 Pool Free Block Free Block Use Block
7. userdpool design UserdPools 1-8 9-16 17-24 … … 249-256 Pool Header Free Block Free Block Header 分配 回收 … Pool Free Block Free Block Use Block 同⼀个Pool下Block⼀样长 单Pool为4kb Block及Pool都为单链表
8. free pool desgin FeeePool Pool Pool Pool … Pool Headers No BLock Pool Headers No BLock Pool为4kb⼤小 Pool清理Headers
9. where store variable ? run-time Stack list dict int heap [1 ,2, 3] {“n”: “1”} 1
10. why ? In [1]: a = 123 In [2]: b = 123 In [3]: a is b Out[3]: True In [4]: a = 1000 In [5]: b = 1000 In [6]: a is b Out[6]: False In [7]: a = 'n' In [8]: b = 'n' In [9]: a is b Out[9]: True In [10]: a = "python" In [11]: b = "python" In [12]: a is b Out[12]: True
11. why ? In [1]: def go(var): ...: print id(var) …: In [2]: id(a) Out[2]: 4401335072 In [3]: go(a) 4401335072 In [10]: a = b = 'nima' In [11]: b = a In [12]: a is b Out[12]: True In [13]: b = 'hehe' In [14]: a is b Out[14]: False 只有引用 ?
12. python objects stored in memory? names names object Python Has Names, Not Variables ! ! !
13. 整数对象池 -5 -4 … 0 … 256 257 小整数 ⼤整数 … … -5 -4 … … 257 … … var_1 var_2 … … -5 -4 … … 257 … … var_3 var_4 the same addr ! not the same addr ! 28 bytes 解释器初始化
14. 整数对象池 Block List Free List PyIntBlock PyIntBlock PyIntBlock PyIntBlock 不会归还给 Arena和os ! ! !
15. 字符对象池 a b c d … … … var_1 var_2 the same addr ! 单个字符38 bytes 由解释器初始化
16. 字符串对象池 aa en cao oh woyao buyao kuai feile var_1 0 1 2 3 … var_2 hash存储变量 共用地址 记录引用计数 ref
17. ref count x = 300 y = x z = [x, y] X ref += 1 300 y Z ref += 1 ref += 2 References -> 4 ！
18. What does del do? x = 300 y = x del x X ref -= 1 300 y References -> 1！The del statement doesn’t delete objects. • removes that name as a reference to that object • reduces the ref count by 1
19. ref count case def go(): w = 300 go() a = “fuc . ” del a b = “en, a” b = None ref count +1 w is out of scope; ref count -1 del a; ref count -1 重新赋值; ref count -1
20. cyclical refclass Node: def __init__(self, va): self.va = va def next(self, next): self.next = next mid = Node(‘root’) left = Node(‘left’) right = Node(‘right’) mid(left) left.next(right) right.next(left) Mid rightleft if del mid node: how ?
21. mark & sweep gc root b a w c K G R
22. 分代回收 node node node node node node node node node node node node node node node node node node node
23. 可变 vs 不可变 (obj) string int tuple list dict
24. container objects a = [10, 10, 11] b = a PyListObject Type list rc 1 items size … … PyObject Type integer rc 2 value 10 PyObject Type integer rc 1 value 11 10 10 11
25. copy.copy a = [10, 10, [10, 11] ] b = copy.copy(a) PyListObject Type list rc 1 items size … … PyObject Type integer rc 2 value 10 PyObject Type integer rc 1 value 11 10 10 ref PyListObject Type list rc 1 items size … … 10 10 ref PyListObject 10 11
26. copy.deepcopy a = [10, [ 10, 11 ] ] b = copy.deep(a) PyListObject Type list rc 1 items size … … PyObject Type integer rc 2 value 10 PyObject Type integer rc 1 value 11 10 ref PyListObject Type list rc 1 items size … … 10 ref PyListObject 10 11 PyListObject 10 11
27. diy gc import gc import sys gc.set_debug(gc.DEBUG_STATS|gc.DEBUG_LEAK) a=[] b=[] a.append(b) print 'a refcount:',sys.getrefcount(a) # 2 print 'b refcount:',sys.getrefcount(b) # 3 del a del b print gc.collect() # 0
28. Garbage Collector Optimize memory bound 可以降低threshold来时间换空间 cpu bound 提⾼threshold来空间换时间 暂停gc, 引⼊master worker设计
29. 引用计数 跟 gil 的影响 ? gc 是否是原⼦ ? gc的 stop the world现象 ? … Q & A
30. “ END ” – xiaorui.cc
END
