Profiling Django object size and memory usage with Pympler

Pympler is a set of Python modules that can assist in profiling memory usage, or debugging memory leaks. It's a good tool for testing multi threaded network script-services, but you also can use it to profile Django views, especially if they use complex Python modules (generate PDFs, images, other). To profile a Django view we can use a middleware like this one:

from pympler.muppy import muppy
from pympler.muppy import summary
from pympler.muppy import refbrowser
from pympler.asizeof import asizeof

def output_function(o):
	return str(type(o))

class memoryMiddleware(object):
	"""
	Measure memory taken by requested view, and response
	"""
	def process_request(self, request):
		req = request.META['PATH_INFO']
		if req.find('site_media') == -1:
			self.start_objects = muppy.get_objects()
	def process_response(self, request, response):
		req = request.META['PATH_INFO']
		if req.find('site_media') == -1:
			print req
			self.end_objects = muppy.get_objects()
			sum_start = summary.summarize(self.start_objects)
			sum_end = summary.summarize(self.end_objects)
			diff = summary.get_diff(sum_start, sum_end)
			summary.print_(diff)
			#print '~~~~~~~~~'
			#cb = refbrowser.ConsoleBrowser(response, maxdepth=2, str_func=output_function)
			#cb.print_tree()
			print '~~~~~~~~~'
			a = asizeof(response)
			print 'Total size of response object in kB: %s' % str(a/1024.0)
			print '~~~~~~~~~'
			a = asizeof(self.end_objects)
			print 'Total size of end_objects in MB: %s' % str(a/1048576.0)
			b = asizeof(self.start_objects)
			print 'Total size of start_objects in MB: %s' % str(b/1048576.0)
			print '~~~~~~~~~'
		return response

In settings.py we add this middleware to MIDDLEWARE_CLASSES and start the development server (if you have media files not in site_media - change the name in the middleware code). This middleware will GREATLY increase request processing. Here is an example result for my forum:

                                           types |   # objects |   total size
================================================= | =========== | ============
                                             list |         232 |      4.31 MB
                                             dict |         859 |    469.63 KB
                                            tuple |        2067 |    157.79 KB
                                              str |        1422 |    104.87 KB
                           function (__wrapper__) |         561 |     65.74 KB
                                             type |          51 |     45.02 KB
                                             code |         308 |     36.09 KB
                                             cell |         606 |     33.14 KB
      <class 'django.utils.safestring.SafeUnicode |           1 |     26.14 KB
  <class 'django.forms.widgets.MediaDefiningClass |          14 |     12.36 KB
                                          unicode |          55 |     10.63 KB
   <class 'django.utils.datastructures.SortedDict |          25 |      9.09 KB
                                          weakref |         102 |      8.77 KB
        <class 'django.utils.functional.__proxy__ |          85 |      5.31 KB
                              function (__init__) |          44 |      5.16 KB
~~~~~~~~~
Total size of response object in kB: 27.9375
~~~~~~~~~
Total size of end_objects in MB: 27.5232620239
Total size of start_objects in MB: 27.5429458618
~~~~~~~~~

If your view code leaks memory or is inefficient you will be able to determine that with asizeof, or heapmonitor (why I doesn't have this in my Pympler package?).