... or what is the cost of using regular expressions for simple tasks

Regular expressions are one of the most powerful tools in computing I have ever seen. My previous post about Django compressor and image preloading is a good example how useful they might be. The only limit of their use is your imagination. But "with great power, comes great responsibility" or in this case a great cost. Even the simplest expressions can be quite heavy compared with other methods.

The reason to write about this is a question recently asked in a python group. It was about how to get the elements of a list that match specific string. My proposal was to use comprehension list and simple string comparison while other member proposed using a regular expression. I was pretty sure that the regular expression is slower but not sure exactly how much slower so I made a simple test to find out.

import re
import timeit

my_list = ['abc-123', 'def-456', 'ghi-789', 'abc456', 'abc', 'abd']

def re_check():
    return [i for i in my_list if re.match('^abc$', i)]

t = timeit.Timer(re_check)
print 're_check result >>', re_check()
print "%.2f usec/pass" % (1000000 * t.timeit(number=100000)/100000)

def simple_check():
    return [i for i in my_list if i=='abc']

t = timeit.Timer(simple_check)
print 'simple_check result >>', simple_check()
print "%.2f usec/pass" % (1000000 * t.timeit(number=100000)/100000)

The results was 23.99 vs 1.41 usec/pass respectively for regular expression vs direct comparison i.e. the regexp was 17 times slower. The difference from the example above may be OK in some cases but it rises with the size of the list. This is a simple example how something really quick on local version may take significant time on production and even to broke your application.

So, should you learn and use regular expressions?
Yes! Absolutely!

They are powerful and useful. They will open your mind and allow you to do things you haven't done before. But remember that they are a double-edge razor and should be used cautiously. If you can avoid it with other comparison(one or more) just run a quick test to see whether it will be faster. Of course if you can not avoid it you can also think about caching the results.

Preface: Have you noticed how on some websites when you click on a link that opens a lightbox or any overlay for first time it takes some time to display the border/background/button images. Not quite fancy, right?
This is because the load of this images starts at the moment the overlay is rendered on the screen. If this is your first load and these images are not in your browser cache it will take some time for the browser to retrieve them from the server.

Solution: The solution for this is to preload the images i.e. to force the browser to request them from the server before they are actually used. With a simple javascript function and a list of the images URLs this is a piece of cake:

$.preLoadImages = function() {
    var args_len = arguments.length;
    for (var i=0; i < args_len; i++) {
        var cacheImage = document.createElement('img');
        cacheImage.src = arguments[i];
    }
}

$.preLoadImages('/img/img1.png', '/img/img2.png')

Please have in mind that the code above uses the jQuery library. Specialty: Pretty easy, but you have to hardcode the URLs of all images. Also if you are using Django compressor then probably you are aware that it adds extra hash to the URLs of the images in the compressed CSS files. The hash depends from the COMPRESS_CSS_HASHING_METHOD settings and can not be avoided. It is pretty useful cause it forces the client browser to reload the images every time when something has been changed. unfortunately our hardcoded list of URLs does not have this hash. So wouldn't it be much simpler if instead of hardcoding URLs we just read them from the CSS files? Solution 2:

$.preLoadImages = function() {
    $.get($('link[rel="stylesheet"]')[0].href, function(data){
        r = /url\(['|"]?(\S+\.(gif|jpg|jpeg|png)[^'(]*)['|"]?\)/ig;
        while (match = r.exec(data)){
                var cacheImage = document.createElement('img');
                cacheImage.src = match[1];
            }
        });
}

$.preLoadImages()

Now with the help of regular expressions we can read the image URLs directly from the CSS file together with the hash part. Please note the zero index in the css file selector, if your main CSS is not the first declared style-sheet then you will have to change the index according to its position.

I hope you will find this solutions simple and useful. As always feel free to comment, share and propose code improvements.

... neither is any other language or framework

This post was inspired by the serial discussion on the topic "Python vs other language"(in the specific case the other one was PHP, and the question was asked in a Python group so you may guess whether there are any answers in favor of PHP). It is very simple, I believe that every Python developer will tell you that Python is the greatest language ever build, how easy is to learn it, how readable and flexible it is, how much fun it is to work with it and so on. They will tell you that you can do everything with it: web and desktop development, testing, automation, scientific simulations etc. But what most of them will forgot to tell you is that it is not a Panacea.

In the matter of fact you can also build "ugly" and unstable applications in Python too. Most problems come not from the language or framework used, but from bad coding practices and bad understanding of the environment. Python will force you to write readable code but it wont solve all your problems. It is hard to make a complete list of what exactly you must know before starting to build application, big part of the knowledge comes with the experience but here is a small list of some essential things.

• Write clean with meaningful variable/class names.
• Exceptions are raised, learn how to handle them.
• Learn OOP(Object Oriented Programming)
• Use functions to granulate functionality and make code reusable
• DRY(Don't Repeat Youtself)
• If you are going do develop web application learn about the Client-Server relation
• Use "layers" to seprate the different parts of your application - database methods, business logic, output etc. MVC is a nice example of such separation
• Never store passwords in plain text. Even hashed password are not completely safe, check what Rainbow Tables are.
• Comment/Document your code
• Write unit test and learn TDD.
• Learn how to use version control.
• There is a client waiting on the other side - don't make him wait too long.
• Learn functional programming.

I hope the above does not sounds as an anti Python talk. This is not its idea. Firstly because there are things that are more important than the language itself(the list above) and secondly because... Python is awesome )))
There are languages that will help you learn the things above faster, Python is one of them - built in documentation features, easy to learn and try and extremely useful. My advice is not to start with PHP as your first programming language it will make you think that mixing variables with different types is OK. It may be fast for some things but most of the times it is not safe so you should better start with more type strict language where you can learn casting, escaping user output etc.

Probably I have missed a few(or more) pointa but I hope I've covered the basics. If you think that anything important is missing, just add it in the comments and I will update the post.

Or how automate the creation of new projects with simple script

Preface: Do you remember all this tiny little steps that you have to perform every time when you start new project - create virtual environment, install packages, start and setup Django project? Kind of annoying repetition, isn't it? How about to automate it a bit.

Solution: Recently I started learning Fabric and thought "What better way to test it in practice than automating a simple, repetitive task?". So, lets mark the tasks that I want the script to perform:

1. Create virtual environment with the project name
2. Activate the virtual environment
3. Download list of packages and install them
4. Make 'src' directory where the project source will reside
5. Create new Django project in source directory
6. Update the settings

Thanks to the local command the first one was easy. The problem was with the second one. Obviously each local command is run autonomously so I had to find some way have activated virtual environment for each task after this. Fortunately the prefix context manager works like a charm. I had some issues making it read and write in the paths I wants and voilà it was working exactly as I want.

The script is too long to place it here but is publicly available at https://gist.github.com/2818562
It is quite simple to use, you only need python, fabric and virtual environment. Then just use the following code.

fab start_project:my_mew_project

To Do: Here are few things that can be improved:

• Read the packages from a file
• Update urls.py to enable admin
• Generate Nginx server block file

So this is my first try with Fabric, I hope that you will like it and find it useful. As always any comments, questions and/or improvement ideas are welcome.

During the development of Simple Site Checker I realised that it would be useful for test purposes if there is a website returning all possible HTTP status codes. Thanks to Google App Engine and webapp2 framework building such website was a piece of cake.

The site can be found at http://httpstatuscodes.appspot.com.

The home page provides a list of all HTTP status codes and their names and if you want to get an HTTP response with a specific status code just add the code after the slash, example:
http://httpstatuscodes.appspot.com/200 - returns 200 OK
http://httpstatuscodes.appspot.com/500 - returns 500 Internal Server Error
Also at the end of each page is located the URL of the HTTP protocol Status Codes Definitions with detailed explanation for each one of them.

The website code is publicly available in github at HTTP Status Codes Site.

If you find it useful feel free to comment and/or share it.