Profiling Django object size and memory usage with Pympler
Check out the new site at https://rkblog.dev.
7 July 2009
Comments
Pympler is a set of Python modules that can assist in profiling memory usage, or debugging memory leaks. It's a good tool for testing multi threaded network script-services, but you also can use it to profile Django views, especially if they use complex Python modules (generate PDFs, images, other). To profile a Django view we can use a middleware like this one:
from pympler.muppy import muppy
from pympler.muppy import summary
from pympler.muppy import refbrowser
from pympler.asizeof import asizeof
def output_function(o):
return str(type(o))
class memoryMiddleware(object):
"""
Measure memory taken by requested view, and response
"""
def process_request(self, request):
req = request.META['PATH_INFO']
if req.find('site_media') == -1:
self.start_objects = muppy.get_objects()
def process_response(self, request, response):
req = request.META['PATH_INFO']
if req.find('site_media') == -1:
print req
self.end_objects = muppy.get_objects()
sum_start = summary.summarize(self.start_objects)
sum_end = summary.summarize(self.end_objects)
diff = summary.get_diff(sum_start, sum_end)
summary.print_(diff)
#print '~~~~~~~~~'
#cb = refbrowser.ConsoleBrowser(response, maxdepth=2, str_func=output_function)
#cb.print_tree()
print '~~~~~~~~~'
a = asizeof(response)
print 'Total size of response object in kB: %s' % str(a/1024.0)
print '~~~~~~~~~'
a = asizeof(self.end_objects)
print 'Total size of end_objects in MB: %s' % str(a/1048576.0)
b = asizeof(self.start_objects)
print 'Total size of start_objects in MB: %s' % str(b/1048576.0)
print '~~~~~~~~~'
return response
types | # objects | total size
================================================= | =========== | ============
list | 232 | 4.31 MB
dict | 859 | 469.63 KB
tuple | 2067 | 157.79 KB
str | 1422 | 104.87 KB
function (__wrapper__) | 561 | 65.74 KB
type | 51 | 45.02 KB
code | 308 | 36.09 KB
cell | 606 | 33.14 KB
<class 'django.utils.safestring.SafeUnicode | 1 | 26.14 KB
<class 'django.forms.widgets.MediaDefiningClass | 14 | 12.36 KB
unicode | 55 | 10.63 KB
<class 'django.utils.datastructures.SortedDict | 25 | 9.09 KB
weakref | 102 | 8.77 KB
<class 'django.utils.functional.__proxy__ | 85 | 5.31 KB
function (__init__) | 44 | 5.16 KB
~~~~~~~~~
Total size of response object in kB: 27.9375
~~~~~~~~~
Total size of end_objects in MB: 27.5232620239
Total size of start_objects in MB: 27.5429458618
~~~~~~~~~
RkBlog
Check out the new site at https://rkblog.dev.
Comment article