Why Django ORM Sucks : It takes a hell lot of memory in processing.

For a project i wanted to keep one of my table in memcache. I tried to do it using Django ORM but it was taking a hell lot of memory.

Let me tell u some details.

Table Structure : This table has almost 1,300,000 rows

id movie_id user_id rating
1 1 1 5.0
2 1 2 4.5
3 1 3 4.5
4 2 1 3.5
5 3 3 2.0

All I wanted was to keep them in cache(memcached), with following format:
memcache Key : movie-id
memcache Data : [ (user1:rating1), (user2:rating2) ]

First Approach : Directly from DB

for r in FlixsterUserRating.objects.all(): 
	ratingDict = cache.get(r.movie.id) 
	if ratingDict is None:  
		cache.set(r.movie.id, [(r.user.id, r.rating)], 86400) 
	else:  
		ratingDict.append((r.user.id, r.rating))  
		cache.set(r.movie.id, ratingDict, 86400) 
	del ratingDict

In this approach for processing every 1000 it was taking some 5.1 secs.

Read More …