... or how to make user editable 404 page that stays in the pages tree of the CMS

Basics: Yes you need it! You need 404 page cause you never know what may happen to a link: bad link paste, obsolete or deleted article, someone just playing with your URLs etc. It is better for both you and your website visitors to have a beauty page that follows the website design instead of the webserver default one that usually contains server information which is possible security issue. With Django this is easy, just make a HTML template with file name 404.html, place it in you root template directory and voilà - you are ready. You will also automatically have a request_path variable defined in the context which caries the URL that was not found.

Problem: sometimes clients require to be able to edit their 404 pages. Or other times you need to use some custom context or you want to integrate plug-ins and be able to modify them easy trough the CMS administration. For example: you want do display your brand new awesome "Sitemap Plug-in" on this 404 page.

Solution: Django allows you to specify custom 404 handler view so you just need to define one, set it in urls.py and make it to render the wanted page:

# in urls.py
handler404 = 'site_utils.handler404' 

# in site_utils.py
from cms.views import details 
def handler404(request):
    return details(request, '404-page-url')

Where '404-page-url' is the URL of the page you want to show for 404 errors. So everything seems fine and here is the pitfall. If you use it this way your web page will return "200 OK" instead of "404 Not Found". This could kill your SEO(except if you want your 404 page as first result for your website). So you just need to add a 404 header to the response:

def handler404(request):
    response = details(request, 'novini')
    response.status_code = 404
    return response

Final words: Why are these HTTP status codes so important. The reason is that they tells the search engines and other auto crawling services what is the page status. Is it normal page, redirect, not found, error or something else. Providing incorrect status codes may/will have a negative effect on your website SEO so try to keep them correct, especially when it is easy to achieve as in the example above.

Note: If the code above is not working for you, please check Allan's solution in the comments

... or how to limit dynamic foreign key choices in both edit drop-down and admin filter.

The problem: Having a foreign key between models in django is really simple. For example:

class Question(models.Model):
    # some fields here

class Answer(models.Model):
    # some fields here
    question = models.ForeignKey(Question)

Unfortunately in the real live the choices allowed for the connection are frequently limited by some application logic e.g. you may add answers only to questions created by you. In some simple(rare) cases this can be easy achieved using choices or limit_choices_to attributes in the models.ForeignKey call. In the case of choices you just have to pass list/tuple each element of which contains the value to be stored and human readable name for the choice. Unfortunately this one is computed on server run and is not updated with new items during run-time. If you use limit_choices_to you may pass to it some kind of filter expression e.g. limit_choices_to = {'pub_date__lte': datetime.now} but this not always can do the job. Speciality: So if you want dynamic choices in the admin drop down you have to write a method that will return list with options and bind it to the form(as shown in Django forms ChoiceField with dynamic values…) which is used by admin. This will work great but if you decided to add this column in admin`s list_filter you will see all element from the connected model in filter. How to limit them to the same list used for the form choices? Solution: The simplest solutions is to extend RelatedFilterSpec, overwrite its default choices and add a single row to the model:

from django.contrib.admin.filterspecs import FilterSpec, RelatedFilterSpec

class CustomFilterSpec(RelatedFilterSpec):
    def __init__(self, *args, **kwargs):
        super(CustomFilterSpec, self).__init__(*args, **kwargs)        
        self.lookup_choices = get_questions() #this method returns the dynamic list

FilterSpec.filter_specs.insert(0, (lambda f: bool(f.rel and hasattr(f, 'custom_filter_spec')), CustomFilterSpec))

class Answer(models.Model):
    # some fields here
    question = models.ForeignKey(Question)
    question.custom_filter_spec = True # this is used to identify the fields which use the custom filter

Final word: The solutions is pretty simple and really easy to implement but should be used carefully. If your filtering method(in my case get_questions) is slow/resource consuming it may bring you troubles. Here is the place where and you should think about caching it. This is a place where a application cache can be used. Hope this will help you as much as it helped me.

...Why, Where, What and How of caching web sites

Basics of web page load: Every time when you open an webpage this results in a series of requests and responses between the client(your browser) and the web server that hosts the requested sites(most of the times this includes more the one servers). Each request tells the server that the client wants specific resource(CSS/JavaScript/image/etc.) and each response carries the resource content.

The main purposes of caching is to decrease bandwidth and to improve website performance.

Where are resources cached: According to the image above there are 2 places where content can be cached: on the client(browser cache) and on the server(server cache). The first one(browser cache) is more suitable when the client often requests single recourse again and again. As an example for this can be pointed CSS-files/JavaScripts/Images and other resources that form the page layout(these are requested on every page load). The second one(server cache) is hold on the server. It also can be split in two major pieces web server cache(when entire resource is cached) and application cache(when data chunks inside the application are cached).

How browser cache works: In order to tell the browser to cache such resources you have to supply each response with the following headers:

• Last-Modified - datetime indication of when the content was last modified/updated on the server
• Expires - the datetime after which the response will be considered "out of date"
• Cache-Control - a directives about whether the resource must be cached and its maximal age of validity
• Date - origin servers datetime

When the client request the resource for a first time the response contains both the resource data and the headers mentioned above. This tells the browser to store the response body(data) into its own cache. On the next request for the same resource the browser sends "If-Modified-Since" header with datetime value identical to the "Last-Modified" received from the server on the previous request. If the cache is still valid the web server returns HTTP Status code "304 Not Modified" which tell the browser to retrieve the content from its cache. Otherwise if the resource is modified between the two requests the response contains the new data along with the corresponding date headers.
This one is extremely useful to decrease websites bandwidth and speed up their loading because the common resources are read directly from the client and not pulled again and again from the server over the network.
For static resources this can be set in the webserver configuration. For dynamic ones(generated images or resources pulled from the database) these headers must be provided and checked from the application itself.

How server cache works: As I stated above the server cache can can be split into web server cache and application cache. Both are used when a single resource is requested by multiple of clients. For example the pages HTML is requested by every site visitor. If the content is the same for all of them then the HTML computation(for dynamically generated pages) can be done only for the first request and the latter ones can get the precomputed one from cache. For this you will need a caching proxy. The most common applications that are used for this purpose are Squid, Varnish and NginX

The process is similar to the one on the first figure but here the client communicates with the proxy instead of directly to the web server. If the proxy does not has a copy of the requested resource it gets if from the web server and then serves it to the client. Otherwise it pulls the content from its cache and return it directly.

Application cache: Sometimes caching full page is not an option. For example if you have a news website with two columns: one that is same for all users and one that is customizable by user preferences you can not serve every one with the same page. But you can pre-compute first column HTML, store it in the application cache and retrieve it when need instead of computing it for every user. The most common tool for this is Memcached. Its implementation is application specific but in pseudo code it works the following way:

key = 'cached_item_unique_key'
result = cache.get(key)
if not result:
   # do some computation here
   result = computed_result
   cache.set(key, result, is_valid_period)
return result

Each cached item is represented in the cache with unique key. When requested if there is such data in the cache it is pulled directly from there and server. Otherwise the date is computed, stored in the cache and then served. Server cache will decrease server load and allow your pages to be computed faster.

Final words: Caching is a double-edge razor. Using if carefully and with comprehension will make your life easier. Playing with it without knowing what you do may ruin your application. Do not cache rarely requested content. Do not try to cache everything, cache also has its limits. Use it wise.
If there is something not well explained, or missed, or wrong feel free to ask/comment. Your participation will be appreciated. Also if you find this article helpful share it and help other too.

... or how "invalid" HTML can break your JavaScript without a trace

The problem: A few days ago I had the following case. A script that I wrote was working everywhere except in the Internet Explorer(IE) 8 and 7. To be honest I don`t remember the exact error message but when I use the built-in debugger it point me to a row X in jQuery.min.js(those of you who have ever opened a minified JavaScript file know that it is practically unreadable) and not only this, but also provide me with an useless trace-back totally unrelated to the function called. So I decided to start executing the code that break line by line(good for me that I was activated on a "click" so I knew at least where to start).

Speciality: The extra hardness here came from the fact that the problem code was loaded as a result from an AJAX request so the debugger was powerless. So I started with the simplest approach I could imagine - alerting a text/value after every row to see where the error appears. The problem was in the following code:
[sourcecode type="javascript"] $('#my_element').html(data) Obviously there was something wrong during the execution of the html() method so I started exploring the results from the jQuery selectors. [sourcecode type="javascript"] alert($('container')) // result: [object Object] alert($('container').html) // result: function(a){...} alert($('container').html()) // result: blank (empty string)

I was getting close. The only thing left was to find why the html() method was returning an empty string when I was sure that there was something inside this element(I ran a quick check in FireFox) but I was stuck again.

Investigation: After a few minutes walking in the room later an old issue came to my mind. I ran into it while I was working on a social network project last year. Javascript/jQuery(I didn`t find which one or both) used to act strange when loading invalid HTML with AJAX inside a page under IE.
"Ok, probably a badly closed tag or similar or other fix in a second thing" was my first thought but... unfortunately it turned to be more complex.

Speciality 2: Did I mention it was an HTML5 page? Yeah, I now that IE7&8 does not support HTML5 so we used modernizr but how to recall it on an HTML fragment.

Solution: My favorite StackOverflow come in hand with reloading-modernizr-after-ajax. Following the answer I visited innerShiv, downloaded the plugin and tried again. This time no error but also no reaction on the click - It looked like I was skipping something. Although this it is not explicitly said it looks like that if you don`t pass "false" as second argument of the innerShiv call as in the innershiv jQuery notes the JavaScript code is not executed. I added it and the problem was fixed.

Conclusion: Should I say that I hate IE? Yes, No, Maybe. In the matter of fact I doubt that there is a web developer that uses IE as a primary browser but I have to confess that this time we were both guilty(our developers team and the IE). We used an "invalid code", IE breaks our JavaScript - "eye for an eye" or whatever it is called. The important here is not whose fault it is but to learn from our mistakes. Blaming a two versions old browser is not beneficial to anyone. But this was a good experience and I am sure that I`ll meet this problem again.

... or how to select items that have one or more categories matching with other item categories

The problem: having a Many To Many relation(ManyToManyField) is really useful when you have to link one object to many other. For example one artist may play in several styles, or a teacher can teach more than one subjects. I`ll use the latter example for the post. If you want to get all teachers that teach (for example) "music" you just have to do something like:

Teacher.objects.filter(subjects = music_obj)

The real problem comes when you want to select all teachers that have subject matching one or more of another teacher subjects. For example "teacher_a" teaches several subject and we want to find all his colleges that teach at least one of his. Solution: to achieve this we have to use the Q object(imported from django.db.models) in a way similar to the making of "OR" query.

from django.db.models import Q
x = Q()
for subject in teacher_a.subjects.all():
    x = x | Q(subjects = subject) 
colegues = Teacher.objects.filter(x).distinct()

Lets see what happens line by line:

1) import the Q class
2) create empty Q object
3) iterate over every subject taught by the specified teacher
4) on every iteration we add an "OR"-clause to the object corresponding to the current subject
5) we use the generated query as a filter

Have in mind that after using the Q object you have to sort your result(manually add order_by clause), because when you use Q the default ordering is not considered.

Final words: The speed of this method is not tested but to be honest I suspect that this can bring performance problems on big tables. In that case maybe you can speed the things up with custom SQL query. So the choice is yours - database independence or speed. Both ways have their pros and cons and its case specific which one you will choose.
If you have found a better way or want to add something feel free to comment.