The Code Dreamer : Nitin Hayaran » Django

Django Template Tag to protect the E-mail address

nitin — Tue, 02 Feb 2010 10:26:51 +0000

Here is a Django Template tag which you can use to protect the E-mail address on your website against bots or spiders that index or harvest E-mail addresses. It uses a substitution cipher with a different key for every page load.

This template tag encrypts the email address and generates a equivalent Javascript code that can decrypt it. This Javascript produce an email link when it runs on browser. As most bots and spider don't executes Javascript they wouldn't be able to extract mail addresses. While a visitor of your web page will not notice that you used this script as long as he/she has javascript enabled.

A normal visitor with a javascript enabled browser will find no difference. The visitor with non-javascript browser will see “[javascript protected email address]” in stead of the E-mail address.

Template Tag Code

class EncryptEmail(template.Node):
    def __init__(self, context_var):
        self.context_var = template.Variable(context_var)# context_var
    def render(self, context):
        import random
        email_address = self.context_var.resolve(context)
        character_set = '+-.0123456789@ABCDEFGHIJKLMNOPQRSTUVWXYZ_abcdefghijklmnopqrstuvwxyz'
        char_list = list(character_set)
        random.shuffle(char_list)
 
        key = ''.join(char_list)
 
        cipher_text = ''
        id = 'e' + str(random.randrange(1,999999999))
 
        for a in email_address:
            cipher_text += key[ character_set.find(a) ]
 
        script = 'var a="'+key+'";var b=a.split("").sort().join("");var c="'+cipher_text+'";var d="";'
        script += 'for(var e=0;e
        script += 'document.getElementById("'+id+'").innerHTML="\\"mailto:"+d+"\\">"+d+""'
 
 
        script = "eval(\""+ script.replace("\\","\\\\").replace('"','\\"') + "\")"
        script = ''
 
        return 'id + '">[javascript protected email address]'+ script
 
 
def  encrypt_email(parser, token):
    """
        {% encrypt_email user.email %}
    """
 
    tokens = token.contents.split()
    if len(tokens)!=2:
        raise template.TemplateSyntaxError("%r tag accept two argument" % tokens[0])
    return EncryptEmail(tokens[1])
 
register.tag('encrypt_email', encrypt_email)

Usage

{% encrypt_email user.email %}

The javascript code it generates is something like this:

<span id="e857606962">[javascript protected email address]</span>
<script type="text/javascript">
/*[CDATA[*/eval("var a=\"cnel8gMuoCjaYRDkv7p+ErHI5qL2x-O3Q@GdtbWKUwT9VfZSN_J04.P1hiFAXz6Bmys\";var b=a.split(\"\").sort().join(\"\");var c=\"AXzVfPXCRgMAhw9fAe91.\";var d=\"\";for(var e=0;ee857606962\").innerHTML=\"<a href=\\\"mailto:\"+d+\"\\\">\"+d+\"</a>\"")/*]]>*/
</script>

Demo

For demo visit :

Demo

Hope this template tag will help you to secure your e-mail addresses from bots and spider.

Lets have a spam free web.

Implementing Search in Django Site using HayStack and Xapian / Whoosh

Implementing Search in Django Site using HayStack and Xapian / Whoosh

nitin — Fri, 09 Oct 2009 12:08:34 +0000

For one of my personal project OnlyAds.in , I wanted to implement search. The requirements I had were :

Easy & Fast to Install / Setup
Can perform reasonably complex searches

But until we have anything like django.contrib.search, we'll have to work with alternate applications.

The options we have are :

For this particular project I went ahead with HayStack Search. The decision for choosing HayStack was great, the implementation was very easy, and the performance of search is also fair enough.

Implementation

In this section i'll guide you through implementation procedure of haystack using Xapian / Whoosh.
For this tutorial i'll be using my Video Model which is in myApp

class Video(models.Model):
	title = models.CharField(max_length=255)
	desc = models.TextField()
	thumbnail = models.CharField(max_length=255)
	tags = TagField()
	creation_date = models.DateTimeField(auto_now_add=True)

HayStack basically provide option of choosing search backend from the following options:

You need to choose one of the search backend and install it in your system. I would recommend Xapian for its simplicity and ease to setup. However if you find it difficult to install, you can go ahead with Whoosh. For install instructions please visit : Installing Search Engines

After Installing search backend you need to install haystack

Now you are ready to go ahead with actual implementation.

Step 1

In settings.py, add haystack to INSTALLED_APPS.

Step 2

Create a file search_sites.py in your project directory. In my case the project name was myproject, so I created myproject/search_sites.py. The content of search_sites.py are like this.

import haystack
haystack.autodiscover()

Step 3

Enter SearchBackend info in settings.py

HAYSTACK_SITECONF = 'myproject.search_sites'
#If you choose whoosh as search backend uncomment following 2 lines and comment last 2 lines
#HAYSTACK_SEARCH_ENGINE = 'whoosh'
#HAYSTACK_WHOOSH_PATH = '/home/xyz/django_projects/index'
HAYSTACK_SEARCH_ENGINE = 'xapian'
HAYSTACK_XAPIAN_PATH = '/home/xyz/django_projects/xapian-index'

You'll need to create respective directories. The search backend will store index files in these directories.

Step 4

Now you'll have to create a SearchIndexes file for all the apps where you want search. For e.g in my case I need to search Video of myApp, so i'll create a search_indexes.py file in myApp.
The file is like this:

import datetime
from haystack import indexes
from haystack import site
from myproject.ads.models import Video
 
class VideoIndex(indexes.SearchIndex):
    text = indexes.CharField(use_template=True, document=True)
    title = indexes.CharField(model_attr='title')
    desc = indexes.CharField(model_attr='desc')
    tags = indexes.CharField(model_attr='tags')
 
    def get_queryset(self):
        "Used when the entire index for model is updated."
        return Video.objects.filter(creation_date__lte=datetime.datetime.now())
 
site.register(Video, VideoIndex)

Step 5

Create a template for the field for which we have kept use_template=True i.e., text (see the code above, with first field "text"). The path of this template should be search/indexes/{app_name}/{modelClass}_{fieldname}.txt.
In my case the file was in templates/search/indexes/myApp/video_text.txt. The content of this file is like this.

{{ object.title }}
{{ object.desc }}
{{ object.tags }}

Step 6

Run manage.py reindex
This will index the objects, and index files will be created in the directories you mentioned in settings.py

Step 7

In this step you need to actually write a view and template to search your content. An example of it is given below

myApp/views.py

def search_videos(request):
	try:
		sTerm = request.GET['q']
	except:
		return HttpResponseRedirect('/')
 
	results = SearchQuerySet().auto_query(sTerm)
	videos = []
	for r in results:
		videos.append(r.object)
	#videos list contains all the videos those match the search criteria
	return HttpResponse(videos)

Hurrry !!! Thats all

This is all we need to do for implementing search. The auto_query(sTerm) function is very impressive as it also support complex search queries like:
india "cute kid" -young

Long Live Django Community

I was really impressed by HayStack. One reason for this was its power / ease of use, but the bigger reason was Daniel Lindsley [1]. He is the creator of HayStack. He is very helpful person, always available in IRC Chat for help developer like me.

Thanks Daniel.

Debugging Django : Best Stuff on Web

nitin — Sun, 22 Mar 2009 20:22:34 +0000

Continuing my previous post Debugging Django Application, in this post I am trying to aggregate best of web tutorial which will definitely help you while debugging any django application.

Eric Holscher Video ScreenCasts

Simon Willison Presentation

Debugging Django

View more presentations from simon.

Here is my list please do share any link that is not listed here.

Debugging Django on Windows : Developement Setup

nitin — Sun, 22 Mar 2009 20:07:59 +0000

I remember the days when I used to struggle while debugging django web applications. But after working for more than a year now with django, I am more or less settled with tools I use for debugging Django Application. My Development System is on Windows XP, so this post is specifically for Debugging django on windows.

Debugging Django on Windows XP

The best tool I found on Windows environment for Django/ Python Debugging is : WinPDB, Just download and install.

Steps to Debug a application

Launch WinPDB

You'll find following screen after opening WinPDB.

Start App in WinPDB

Launch an the application using File -> Launch. If For e.g. my application is in C:\projectHome\manage.py
I will write launch application as:

C:\projectHome\manage.py runserver --noreload

Click on "GO" button on top toolbar.

Put BreakPoints

Open the affected file(file which needs debugging e.g. views.py), File->Open Source. Now put breakpoint in the source code by pressing F9. You are all set for debugging. Isn't that simple.
Now go to browser and hit the URL which is causing problem. The debugger will stop execution on breakpoint.

Now What

Now you are free to do anything for debugging, you can go for step execution if you think there is some error in login/code. You can also see inmemory objects. etc, etc.

For me WinPDB has solved everything, hope it will help you as well.

Django ORM : What to do when the memory requirement goes high?

nitin — Tue, 24 Feb 2009 15:06:19 +0000

As the title suggest this post is regarding "Memory Requirement of django ORM".
Actually some days back I myself faced this problem and wrote "Why Django ORM Sucks : It takes a hell lot of memory in processing". From various comments and a thread which started on google groups, I came up with few points one should check while operating on large amount of data.

For showing points i'll use following example:

for r in UserRating.objects.all():
	ratingDict = cache.get(r.movie.id)
	if ratingDict is None:
		cache.set(r.movie.id, [(r.user.id, r.rating)], 86400)
	else:
		ratingDict.append((r.user.id, r.rating))
		cache.set(r.movie.id, ratingDict, 86400)

So here are the mistakes in previous code. Make sure such mistakes are not with you code.

Reduce DB Interaction

Use of r.movie.id, r.user.id : As you can see that here ".id" is used just to get id of movie and user object. In place of this r.user_id and r.movie_id should be used.

Always use paginator

While handling large amount of data you should always use paginator. This will keep memory size limited.
Here is the sample code for using paginator.

p = Paginator(UserRating.objects.all(), 50000)
for i in p.page_range:
	page = p.page(i)
	for r in page.object_list:
		...

So here the memory requirement will be limited to 50000 rows.

Use Iterator()

If your dataset is not really very huge, and you don't want to use paginator, use Iterator().
QuerySet.__iter__() : this is the default method of accessing query set.
If we do UserRating.objects.all() it means UserRating.objects.all().__iter__()
QuerySet.Iterator() :
we should do UserRating.objects.all().iterator().
Both of these iteration method do chunked reads from the DB. Here __iter__ also caches results , so if you reiterate you don't do a second db query, whereas iterator() doesn't cache them.

As iterator() doesn't caches them the memory consumption is kept low.

Hope these points will help you as well.
Keep Coding … and keep Djangoing

Django Example: FileField and ImageField

nitin — Mon, 23 Feb 2009 08:18:32 +0000

The syntax to add it in Model is:

class MyModel(models.Model):
	video = FileField(upload_to=None[, max_length=100, **options])

Here upload_to is local filesystem path that will be appended to your MEDIA_ROOT setting to determine the value of the url(mymodel.video.url) attribute.

Setting MEDIA_ROOT and upload_to

If,
MEDIA_ROOT = ‘/home/myname/files/
upload_to=’videos’
file: abc.flv
results in: /home/myname/files/videos/abc.flv

where as if,
MEDIA_ROOT = '/home/myname/files/'
upload_to='/videos'
and file : abc.flv
result will be saved in : '/videos/abc.flv' (i.e., root directory of filesystem)

You can also set upload_to='videos/%Y/%m/%d'. The '%Y/%m/%d' part of upload_to is strftime formatting; '%Y' is the four-digit year, '%m' is the two-digit month and '%d' is the two-digit day. If you upload a file on Jan. 15, 2007, it will be saved in the directory /home/myname/files/videos/2007/01/15.

Now here's how would you save the actual data.

from django.core.files.base import ContentFile
def save_file(request):
	mymodel = MyModel.objects.get(id=1)
	file_content = ContentFile(request.FILES['video'].read())
	mymodel.video.save(request.FILES['video'].name, file_content)

Hope the code is self explanatory. For further help do drop comment.

Why Django ORM Sucks : It takes a hell lot of memory in processing.

nitin — Tue, 17 Feb 2009 06:50:00 +0000

For a project i wanted to keep one of my table in memcache. I tried to do it using Django ORM but it was taking a hell lot of memory.

Let me tell u some details.

Table Structure : This table has almost 1,300,000 rows

id	movie_id	user_id	rating
1	1	1	5.0
2	1	2	4.5
3	1	3	4.5
4	2	1	3.5
5	3	3	2.0

All I wanted was to keep them in cache(memcached), with following format:
memcache Key : movie-id
memcache Data : [ (user1:rating1), (user2:rating2) ]

First Approach : Directly from DB

for r in FlixsterUserRating.objects.all():
	ratingDict = cache.get(r.movie.id)
	if ratingDict is None:
		cache.set(r.movie.id, [(r.user.id, r.rating)], 86400)
	else:
		ratingDict.append((r.user.id, r.rating))
		cache.set(r.movie.id, ratingDict, 86400)
	del ratingDict

In this approach for processing every 1000 it was taking some 5.1 secs. But here Memory Requirement was growing like hell.
In windows this program memory requirement was growing at rate of 34 MB / 10K of records. Don't ask about using linux. In linux it was taking some 900MB for just processing 50K records.
So to process full data it would need :
(34 MB/10,000)*(1,300,000) = 4420 MB = almost 4Gigs…. No way I can't afford this approach.

Second Approach : From database in Chunks

for m in FlixsterMovie.objects.all():
	d = []
	for r in FlixsterUserRating.objects.filter(movie=m):
		d.append((r.user.id, r.rating))
		cache.set(m.id, d)

In this approach for processing every 1000 it was taking some 2.0 secs.
In windows this program memory requirement was growing at rate of 8 MB / 10K of records.
So to process full data it would need :
(8 MB/10,000)*(1,300,000) = 1040 MB = almost 1Gigs…. Still it is a lot.. My Development system has only 1.5 GB RAM.

Third Approach : Fuck Database Use CSV Dump

Finally I left using Database. Dumped my data in a CSV. And used following program.

for s in csv.reader(open(csv_filename)):
	movie_id = int(s[1])
	user_id = int(s[2])
	rating = float(s[3])
	ratingDict = cache.get(movie_id)
	if ratingDict is None:
		cache.set(movie_id, [(user_id, rating)], 86400)
	else:
		ratingDict.append((user_id, rating))
		cache.set(movie_id, ratingDict, 86400)

In this approach memory requirement was just : 17-18MB and it wasn't growing. Some relief. But, As expected this approach was very slow. For processing 1K records it was taking some 55 Secs.
So to perform full task it will take some : 55 * 1,300 ~ 20Hrs.

Fuck, What should I do now

I ran out of options.
I am going ahead with 2nd approach. But suggestions from you guys are most welcome.

Update

Final Version:

p = Paginator(FlixsterUserRating.objects.all(), 50000)
for i in p.page_range:
	page = p.page(i)
	for r in page.object_list:
		ratingDict = cache.get(r.movie_id)
		if ratingDict is None:
			cache.set(r.movie_id, {r.user_id: r.rating}, 86400)
		else:
			ratingDict[r.user_id] =  r.rating
			cache.set(r.movie_id, ratingDict, 86400)

This approach kept the memory level below 100 MB, and whole process was completed in ~ 1hr.

Thanks you all…
Moreover, I didn't put stat for Iterator version because there wasn't any significant improvement. Memory consumption was growing even in this case.

Installing Memcached for Django Application on Windows XP

nitin — Tue, 17 Feb 2009 04:18:00 +0000

For a project I needed Memcached. But the problem was that I have my development system on WindowsXP.

When I Google I got various links to install Memcached on windows, but there wasn't any link that tell all about Memcached with Django on Windows

So Here we go:

Download memcached 1.2.4 Win32.

Unzip the binaries in your desired directory (eg. c:\memcached)

Install the service using the command: 'c:\memcached\memcached.exe -d install' from either the command line

Start the server from the Microsoft Management Console or by running the following command: 'c:\memcached\memcached.exe -d start'

Use the server, by default listening to port 11211

Now we have to get something so that python can talk with our memcached server. For this we have various options, but the easiest is python-memcache

Grap latest file from python-memcached-latest.tar.gz

Extract the compressed files in a directory say C:\python-memcache\

Now Run "python setup.py install"

So till this point the installation part is done. Now the question comes how to use it in Django application.

from django.core.cache import cache
 
def test():
	cache_key = 'myID'
	result = cache.get(qstr_key)
	if result is None:
		"""Not Found in Cache"""
		data = 'hello'
		cache.set(cache_key, data, 86400)

So Basically we have two functions here :
cache.set
cache.get

Go and enjoy the benefit of memcached