Implementing Search in Django Site using HayStack and Xapian / Whoosh

For one of my personal project OnlyAds.in , I wanted to implement search. The requirements I had were :

• Easy & Fast to Install / Setup
• Can perform reasonably complex searches

But until we have anything like django.contrib.search, we'll have to work with alternate applications.

The options we have are :

• django haystack search
• django-sphinx
• http://code.google.com/p/djapian/
• http://code.google.com/p/django-search-lucene/
• http://www.sai.msu.su/~megera/postgres/gist/tsearch/V2/

For this particular project I went ahead with HayStack Search. The decision for choosing HayStack was great, the implementation was very easy, and the performance of search is also fair enough.

Implementation

In this section i'll guide you through implementation procedure of haystack using Xapian / Whoosh.
For this tutorial i'll be using my Video Model which is in myApp

class Video(models.Model):
	title = models.CharField(max_length=255)
	desc = models.TextField()
	thumbnail = models.CharField(max_length=255)
	tags = TagField()
	creation_date = models.DateTimeField(auto_now_add=True)

HayStack basically provide option of choosing search backend from the following options:

• Whoosh
• Xapian
• Solr

You need to choose one of the search backend and install it in your system. I would recommend Xapian for its simplicity and ease to setup. However if you find it difficult to install, you can go ahead with Whoosh. For install instructions please visit : Installing Search Engines

After Installing search backend you need to install haystack

Now you are ready to go ahead with actual implementation.

Step 1

In settings.py, add haystack to INSTALLED_APPS.

Step 2

Create a file search_sites.py in your project directory. In my case the project name was myproject, so I created myproject/search_sites.py. The content of search_sites.py are like this.

import haystack
haystack.autodiscover()

Step 3

Enter SearchBackend info in settings.py

HAYSTACK_SITECONF = 'myproject.search_sites'
#If you choose whoosh as search backend uncomment following 2 lines and comment last 2 lines
#HAYSTACK_SEARCH_ENGINE = 'whoosh'
#HAYSTACK_WHOOSH_PATH = '/home/xyz/django_projects/index'
HAYSTACK_SEARCH_ENGINE = 'xapian'
HAYSTACK_XAPIAN_PATH = '/home/xyz/django_projects/xapian-index'

You'll need to create respective directories. The search backend will store index files in these directories.

Step 4

Now you'll have to create a SearchIndexes file for all the apps where you want search. For e.g in my case I need to search Video of myApp, so i'll create a search_indexes.py file in myApp.
The file is like this:

import datetime
from haystack import indexes
from haystack import site
from myproject.ads.models import Video
 
class VideoIndex(indexes.SearchIndex):
    text = indexes.CharField(use_template=True, document=True)
    title = indexes.CharField(model_attr='title')
    desc = indexes.CharField(model_attr='desc')
    tags = indexes.CharField(model_attr='tags')
 
    def get_queryset(self):
        "Used when the entire index for model is updated."
        return Video.objects.filter(creation_date__lte=datetime.datetime.now())
 
site.register(Video, VideoIndex)

Step 5

Create a template for the field for which we have kept use_template=True i.e., text (see the code above, with first field "text"). The path of this template should be search/indexes/{app_name}/{modelClass}_{fieldname}.txt.
In my case the file was in templates/search/indexes/myApp/video_text.txt. The content of this file is like this.

{{ object.title }}
{{ object.desc }}
{{ object.tags }}

Step 6

Run manage.py reindex
This will index the objects, and index files will be created in the directories you mentioned in settings.py

Step 7

In this step you need to actually write a view and template to search your content. An example of it is given below

myApp/views.py

def search_videos(request):
	try:
		sTerm = request.GET['q']
	except:
		return HttpResponseRedirect('/')
 
	results = SearchQuerySet().auto_query(sTerm)
	videos = []
	for r in results:
		videos.append(r.object)
	#videos list contains all the videos those match the search criteria
	return HttpResponse(videos)

Hurrry !!! Thats all

This is all we need to do for implementing search. The auto_query(sTerm) function is very impressive as it also support complex search queries like:
india "cute kid" -young

Long Live Django Community

I was really impressed by HayStack. One reason for this was its power / ease of use, but the bigger reason was Daniel Lindsley [1]. He is the creator of HayStack. He is very helpful person, always available in IRC Chat for help developer like me.

Thanks Daniel.

If you enjoyed this post, please consider leaving a comment or subscribing to the RSS feed to have future articles delivered to your feed reader.

Other Posts which you may like:

1. Django Example: FileField and ImageField
2. Debugging Django on Windows : Developement Setup
3. Installing Memcached for Django Application on Windows XP
4. Debugging Django : Best Stuff on Web
5. Chat Client for Naukari Job Search