Here is my presentation part of the in company Python course.

View the slides on SlideShare

The last slide - "Problem to solve" is something like a simple homework. Sample solutions will be uploaded later this week.

Preface: Nine months ago(I can't believe it was that long) I created a script called Simple Site Checker to ease the check of sitemaps for broken links. The script code if publicly available at Github. Yesterday(now when I finally found time to finish this post it must be "A few weeks ago") I decided to run it again on this website and nothing happened - no errors, no warning, nothing. Setting the output level to DEBUG showed the following message "Loading sitemap ..." and exited.
Here the fault was mine, I have missed a corner case in the error catching mechanism i.e. when the sitemap URL returns something different from "200 OK" or "500 internal server error". Just a few second and the mistake was fix.

Problem and Solution: I ran the script again and what a surprise the sitemap URL was returning "403 Forbidden". At the same time the sitemap was perfectly accessible via my browser. After some thinking I remembered about that some security plugins block the access to the website if there is not User-Agent header supplied. The reason for this is to block the access of simple script. In my case even an empty User-Agent did the trick to delude the plugin.

urllib2.urlopen(urllib2.Request(url,
                                headers={'User-Agent': USER_AGENT}))

Final words: As a result of the issue mention above one bug in simple site checker was found fixed. At the same time another issue about missing status and progress was raised, more details can be found at Github but in a few words an info message was added to each processed URL to indicate the progress.

If you have any ideas for improvement or anything else feel free to comment, create issues and/or fork the script.

As a follow up post of Automated deployment with Ubuntu, Fabric and Django here are the slides from my presentation on topic "Automation, Fabric and Django". Unfortunately there is no audio record but if there is interest I can add some comments about each slide as part of this post.

If there is anything that need explanation feel free to ask.

As I promised in Automated deployment with Ubuntu, Fabric and Django I will use this post to explain the file structure that I use for my Django projects and what I benefit from it. So here is my project directory tree.

The structure

~/workspace/<project_name>/
|-- bin
|-- include
|-- lib
|-- local
|-- src
   |-- .git
   |-- .gitignore
   |-- required_packages.txt
   |-- media
   |-- static
   |-- <project_name>
   |   |-- <project_name>
   |   |   |-- __init__.py
   |   |   |-- settings
   |   |   |   |-- __init__.py
   |   |   |   |-- <environment_type>.py
   |   |   |   |-- local.py
   |   |   |-- templates
   |   |   |-- urls.py
   |   |   |-- views.py
   |   |-- manage.py
   |   |-- wsgi.py
   |-- <project_name>.development.nginx.local.conf
   |-- <project_name>.< environment_type>.nginx.uwsgi.conf
   |-- <project_name>.< environment_type>.uwsgi.conf

Explanation

At the top I have a directory named as the project and virtual environment inside of it. The benefit from it is complete isolation of the project from the surrounding projects and python packages installed at OS level and ability to install packages without administrator permissions. It also provides an easy way to transfer the project from one system to another using a requirements file.
The src folder is where I keep everything that is going to enter the version control project source, requirements files, web server configuration etc.
My default .gitignore is made to skip the pyc-files, the PyDev files and everything in the static and media directories.
The media directory is where the MEDIA_ROOT settings point to, respectively static is for the STATIC_ROOT.
All required packages with their version are placed in required_packages.txt so we can install/update them with a single command in the virtual environment.
In a directory with the name of the project is where the python code resides. Inside it the project structure partly follows the new project layout introduced in Django 1.4.
The big difference here is the settings part. It is moved as a separate module where all common/general settings are place in __init__.py. We have one file for each possible environment(development, staging, production) with the environment specific settings like DEBUG and TEMPLATE_DEBUG. In local.py are placed the settings specific for the machine where the project is running. This file should not go into the repository as this is where the access database password and API keys should reside.
I having one local Nginx configuration file because I use the webserver to serve the static files when working locally this is the <project_name>.development.nginx.local.conf.
For each environment there is also a couple of configuration files - one for the web server(<project_name>.< environment_type>.nginx.uwsgi.conf) and one for the uwsgi(<project_name>.< environment_type>.uwsgi.conf). I make symbolic links pointing to these files so any changes made are automatically pushed/pulled via version control, I only have to reload the configuration. There is no option to change something in the configuration at one station and to forget to transfer it to the rest.

Benefits

• Complete isolation from other projects using virtual environment
• Easy to transfer and update packages on other machines thanks to the pip requirements file
• Media/static files outside the source directory for higher security
• Web server/uWSGI configuration as part of the repository for easier and error proof synchronization

Probably it have some downside(I can not think any of these now but you never know) so if you think that this can be improved feel free to share your thoughts. Also if there is anything not clear enough, just ask me, I will be happy to clear it.

A few months ago I started to play with Fabric, and the result was a simple script that automates the creation of a new Django project. In the last months I continued my experiments and extended the script to a full stack for creation and deployment of Django projects.

As the details behind the script like the project structure that I use and the server setup are a bit long, I will keep this post only on the script usage. I will write a follow-up one describing the project structure and server.

So in brief, the setup that I use consists of Ubuntu, Nginx as web server and uWSGI as application server. Upstart controls the last one. The script is available for download at GitHub.
In a wait for more detailed documentation, here is a short description of the main tasks and what they do:

startproject:<project_name>

• Creates a new virtual environment
• Installs the predefined packages (uWSGI, Django and South))
• Creates a new Django project from the predefined template
• Creates different configuration files for development and production environment
• Initializes a new git repository
• Prompts the user to choose a database type. Then installs database required packages, creates a new database and user with the name of the project and generates the local settings file
• Runs syncdb(if database is selected) and collectstatic

setup_server

• installs the required packages such as Nginx, PIP, GCC, etc.
• prompts for a database type and install it
• reboots the server as some packages may require it

Once you have a ready server and a project, just use this the following task to deploy it to the server. Please have in mind that it should be used only for initial deployment.

deploy_project:<project_name>,<env_type>,<project_repository>

• Creates a new virtual environment
• Clones the project from the repository
• Installs the required packages
• Creates symbolic links to the nginx/uwsgi configuration files
• Asks for a database engine, creates a new DB / User with the project name and updates the settings file
• Calls the update_project task

update_project:<project_name>,<env_type>

• Updates the source code
• Installs the required packages(if there are new)
• Runs syncdb, migrate and collect static
• Restart the Upstart job that runs the uWSGI and the Nginx server

The script is still in development, so use it at your own risk. Also, it reflects my own idea of server/application (I am planning to describe it deeper in a follow-up post) setup, but I would really like if you try it and give feedback. Feel free to fork, comment and advice.