Django

Using Arecibo and Django, the easy way

Mon, 21 Jun 2010 11:16:07 -0700

So you've got your Django project on the production server. Want an easy way to track the errors in Django? The existing Django package is pretty easy to use, but it doesn't get much simpler than this: add in your Arecibo server to your ADMINS in settings.py.

When Django emails out those error emails, it will send one to Arecibo. Arecibo will parse it and add it your site. The email address for that is: django-YOUR_PUBLIC_ARECIBO_KEY@yourserver.appspotmail.com. You can try this out right now, the email address for the test server is: django-sw3tqw35ywq45ws4kqa4ia6yw5q45serws23w351245lk6y@test-areciboapp.appspotmail.com

Becomes:

To do this we are parsing the email down, which is less accurate than just doing a HTTP post via the API, but its pretty easy to setup.

Note: the key for the email address for the test server might change, you can find it on the setup pages on the test server. All errors are deleted after one hour on the test server.

Taskqueues in App Engine

Tue, 08 Jun 2010 12:46:36 -0700

App Engine has task queues as an experimental feature. There's nothing fancy here, it's just a way of queueing something up to be run later.

We've probably all written a few queues in our time, the task queue is just a continuation of that. Add a URL to be executed a certain point and a scheduler will come along and try all the urls in the queue. For Arecibo we use this to do the post processing on an error. When an error comes into Arecibo we write it to the datastore quickly, we then add in one task queue that will: figure out the grouping, figure out any notifications, figure out the user agent (surprisingly a bit expensive). That post processing will be done async via the task.

First thing that I noticed about the queue is that the development server doesn't run the task queue automatically meaning that all my unit tests failed. So here's a way to do it immediately if you are not in production:

if os.environ.get('SERVER_SOFTWARE', '').startswith('Dev'): # send the signal, otherwise we have to clicking buttons # to process the queue send_signal(None, self.id)
else: # enqueue the send notification # if development taskqueue.add(url=reverse("error-created", args=[self.id,]))

In Django we've got a URL "error-created" that maps to send_signal. That's a method that first figures out if this object has had send_signal run before. We tell this by the use of a Boolean property on the object. It's a bit annoying to have a property for that, but it could be sent by the task queue multiple times. There might be a cleaner way to cope with that.

def send_signal(request, pk): error = Error.get(pk) if not error.create_signal_sent: error.create_signal_sent = True error.save() error_created.send(sender=error.__class__, instance=error) return HttpResponse("Signal sent") return HttpResponse("Signal not sent")

It then sends off the custom Django signal, when then triggers all the signal processing we've assigned in Django to that error. Because the view is only going to be executed by the App Engine scheduler, we can just return a simple string.

The end result is that the unit tests work and it works in production and keeps the load on that initial write nice and low.

Avoiding some generic views

Wed, 21 Apr 2010 04:05:23 -0700

As I wrote this, I realised I seemed to be on a roll with negative blog posts. Sorry, I will get positive again soon, I'm sure.

If you follow through the Django tutorial, you'll get to stage 4 which encourages you to move to generic views. Generic views are a neat way of removing the view part of the equation. Instead of a standard URL > view > template, you have URL > template.

There's documentation on all the generic views and there's quite a few.

The problem

Here's the conversation that goes on in #django IRC about once every day or so. Abbreviated, its normally more long winded than this.

djangodood: can I pass parameters to a query in templates?

others: no

djangodood: well how do I do this then? I tried doing it in urls.py and that didn't work

others: do it in your view

...usually a break of a few minutes.

djangodood: but I want to use my generic views

others bang heads on table

Logic for a template should be done mostly in the view. If its very model specific, doing it in the model might make sense. But the key is keep the template as simple as possible. The templating language reinforces this by not allowing parameters.

Using generic views for things like object_list gives you a problem, you need to put the logic somewhere, so you either try pushing it up into urls.py or down into the template. We all know why the template is the wrong place (hopefully). So then urls.py starts to get really complicated. If you urls.py is full of logic for the view, wouldn't it just be better to push that back into a view?

In practice I've found using generic views that a few things will happen: 1) I write a pile of generic views, 2) my urls.py gets complicated and then 3) as the site gets more complicated I have to start removing generic views. In a non-trivial site, the generic views quickly get removed.

What does make sense

The direct_to_template or the redirect_to make sense to me. These generic views don't involve passing any new data to the templates, different querysets or parsed data - that means most of the time they work just fine.

Writing a wrapper that uses the generic view, as James Bennet shows.

In the tutorial, the generic views seem to say "hey skip writing this bit of Python". Chances are you'll be better off just writing that bit of Python as you'll need to go back to it. And it will stop you trying to do too much work in the template.

Running Django admin separately

Tue, 30 Mar 2010 11:58:11 -0700

Django admin is great for administrating a site, but chances are you won't want to run that on your "big serious production site" for a few reasons.

Setting up

This is pretty easy to do, all your apps will be stored in source control (or something more exotic) so pull those apps down. Leave out the apps you don't need. Perhaps you've got a public facing specific app that pulls in public specific things, sets up context processor or specific auth. backends, all things you don't need in the admin site.

This could be set up on the same server, or on a different server as long as it can reach the database.

Security

There's no limit to the number of password attempts on the admin site. So you could dictionary attack the admin site if you really wanted to get in. Moving it to a separate instance isn't necessarily more secure, unless you set up an appropriate policy.

For example: all requests to admin.yoursite.com are internal only, or restricted by IP. Those sorts of policies are very easy for system admins to setup and maintain.

Knowing that pretty much no matter what the end user does, they can't access django-admin is a big comfort.

Performance

Admin can be a bit slow if some queries aren't optimized, you'd be really annoyed if someone browsing the admin site took down your public site. Likewise the public facing site might have options that are unnecessary on the admin (such as the afore mentioned context processors).

At DjangoSki I was chatting to Jakob about this and found we've both also run admin on ready-only database slaves. So it's not really an admin, just a data browsing site.

Configuration

For a big site you will likely have quite a few things in front of a site. Your media is likely on a CDN. You've likely got a load balancer and a caching server in there. All things that make your site go nice and fast. You don't need any of those for the admin. Again, route the requests differently.

Different Apps

There's apps that you might want to run on one instance and not the other. At one time I ran a SQL logging app on the public site over a few hours to track down some performance, that was a public specific app. It was easy to turn that on the public site and not have to worry about logging admin queries.

You don't even have to limit yourself to just one admin. If you really, really, wanted to, you could have multiple admins, each accessing different apps. Django admin will only allow you to admin apps you have INSTALLED_APPS, so if you really wanted to you could have one admin for group X that only accessed app X. However if you creating lots of admins, you'll probably find better ways to do that.

Setting up admin to run separately is pretty easy and I'd fully recommend it for any project of a serious size.

Django-lawnchair

Thu, 25 Mar 2010 05:49:37 -0700

Lawnchair is a library that provides a client side JSON store and that's really cool. Since I'm going to be using this quite a bit in some of our applications, I started to build out some of the common uses into a library. This makes working with django models on the client side a little bit easier.

At the moment django-lawnchair contains a few utilities to do some work on the server, but to be honest, not to much. I'm hoping I can integrate django-piston a bit to make writing that easier.

There's a bit more on the client side of things, using django-lawnchair.js you can instantiate a database like this:

var todos = new djangochair('/todo/list');

Grabbing all the objects for a model (in this case Todo objects), populating the local database (or whatever backend store you choose) is as simple as:

todos.remote_get(add_row);

...as each Todo is added to the DB, the callback add_row will be called.

Users can interact with the objects. Each time you change a model in the local DB, we change the state. You can the push all the changed objects to Django:

todos.updated(function(r) { todos.remote_save(r); });

This will iterate through each changed object and push them to the server, where they will be saved. Got deleted objects?

todos.deleted(function(r) { todos.remote_delete(r, function(r) { todos.purge(r.key); });
});

...and so on. There's a lot to do in this area, handling errors, handling syncing problems and so on. The library doesn't do much of that yet, it's trying to set a base line for this stuff. As we build this out for some applications, we'll be fleshing these cases out.

This is in a "works on my machine" state at: http://github.com/andymckay/django-lawnchair.

Creating a ModelForm dynamically

Wed, 24 Mar 2010 10:39:54 -0700

A simple short snippet for creating a ModelForm for an object dynamically. This is a useful library function to validate data and save changes to a model without having to declare a form:

from django import forms def model_to_modelform(model): meta = type('Meta', (), { "model":model, }) modelform_class = type('modelform', (forms.ModelForm,), {"Meta": meta}) return modelform_class

Based on "So you want a dynamic form".

Django query analyzer

Thu, 18 Mar 2010 11:35:11 -0700

One of the things I wanted to work on at the DjangoSki sprints was a django query analyzer.

The basic premise is that I want to create an easy way for users to experiment with and play with the ORM to produce the queries that they want. If you are new to Django, figuring out the ORM is one the first things you will need to tackle and it can be a little opaque. There's some ways to play with it at the shell, or use the debug toolbar, but these can be a little limited and unhelpful to newbies.

The main interface would ideally be a series of options introspected from the model. Choose a model, choose a query manager, then add in multiple filter queries. This would not be able to capture everything and you'd always need an opt out clause of a text area where you can enter anything you want.

This is pretty similar to the GUI's you'll get with most SQL servers, its just the Django ORM layer on top of it.

Sadly I didn't get much chance apart from some time on Thursday afternoon. But a whole bunch of other people did including Stefan Tjarks, Rob Hudson and Nick Fitzgerald. These guys did some great work on the project despite my vague handwaving about what the project should be. Fortunately after the conference I got a chance to do a few hours works and got to this stage:

The introspection code is there, just need to get around to pulling it altogether on the front end so that we've got a nice way. But I think for a newbie this can be a very useful way to play with the ORM and see what comes out and how those little changes to the code can make substantial changes to the SQL.

The code is very rough and has been messed around by me from that original hard work by Rob, Stefan and Nick - but it's at: http://github.com/andymckay/django-query-analyzer

DjangoSki checklist

Fri, 19 Feb 2010 04:20:37 -0800

Sadly Matt Berg won't be able to make it DjangoSki. Far too busy at the moment in Uganda working on the next version of Child Count. That's a shame, we'll get him next year when he's had time to catch up on his skiing a bit ;).

Very fortunate to have Brian Leroux, a self described "software hack" from Nitobi coming to speak in Matt's place. Brian works on mobile apps and things like PhoneGap, XUI and Lawnchair.

Checklist for DjangoSki:

Laptop
Passport (for any non-Canadians that is)
Skis / snowboards and gear
Hot tub gear

We have a few sprint topics planned on the wiki, but if you've got your own, please come ready with those ideas.

Faster or lazier pagination

Fri, 05 Feb 2010 02:00:00 -0800

Or how to avoid slow counts in Postgres

New DjangoSki speakers

Tue, 02 Feb 2010 16:36:59 -0800

Pleased to say that DjangoSki picked up a couple of good speakers.

Andy Smith is a developer at Google, where he works on Google App Engine. He'll be speaking on Google App Engine and how to develop apps for it. Also he'll be talking about the status of non-relational backends for App Engine. A handy person to be sprinting with.

Michael Richardson is one of the brains behind Urban Airship and Bac'n.com. He'll talk about django-piston and building an API in Django which Urban Airship use to do cool things for the iPhone.

Also pleased to note that Rob Hudson is making his way up from Portland. Rob is the writer and maintainer of Django Debug Toolbar. Possibly the most useful debug tool you'll ever use in Django. As a connoisseur of fine Portland beer, I hope you'll be buying him a beer.

Django Documentation iPhone app updated

Mon, 11 Jan 2010 15:54:23 -0800

A new version of the Django documentation iPhone app has been pushed to iTunes. Updates are:

Updated to 1.2 alpha 1 documentation
App now remembers the page you were last on and re-opens it at the start.
Code samples when page is rotated widen.
Some copyright notices added.

The power of Q

Fri, 08 Jan 2010 05:01:50 -0800

Q objects are a great tool in Django

Send someone from a charity to DjangoSki

Wed, 06 Jan 2010 03:31:06 -0800

Thanks to sponsorship from the Python Software Foundation we've got some free tickets to give away to people from charities or non-profits.

Do you know someone who uses Django in a charity? Or are you at a charity? Do you use Django or want to learn about it? Like skiing, snowboarding or just hanging out in Whistler? Then nominate someone, or yourself. We've got a limited number of tickets and time is short - so hop to it.

~~Fill out the form here~~.

Note: the image isn't relevant, but was one of the first hits on Flickr for ticket. It's much colder in Whistler.

Update: people chosen and contacted.

DjangoSki change of hotel

Sat, 02 Jan 2010 14:31:57 -0800

Over the Christmas holidays the DjangoSki changed location from one hotel in Whistler to another. This was the reason for the registration and hotel information going offline over the Christmas holidays. We've got a new hotel sorted out and the conference will be at the new location, a few minutes down the road.

So we went for plan B, one of the other hotels. It's a good hotel with other benefits for example: they don't mind external food coming in, a kitchenette in the room, washer and dryer etc. In fact this hotel is directly on the Creekside chair. You can go directly from the conference room, walk to the hot tub, past the barbeque to the chair lift (or some combination of that).

The reason? During the process of sorting out the contract with the original hotel the contract changed quite dramatically against us. We proceeded hoping that the pre-registration numbers would solve that problem. As it turned out it wasn't enough and the amount of money that we were putting up for the conference was so large to be a really, really big concern for us. A conference is a risk, but we are a small company and the risk was too large.

We are sorry about the inconvenience and there's no more changes now, it's full on to the finish line of a great conference in Whistler. We think we've contacted everyone involved who's booked a room, but if this does affect you, then please contact us immediately.

Details are up on the DjangoSki website of the Legends hotel.

DjangoSki Registration is now open

Thu, 03 Dec 2009 14:56:47 -0800

Registration is now open for DjangoSki. Early bird tickets are on a simple first come, first served basis and registration is here.

With our three keynotes set up and all the hairy contracts signed away, all that's left for us to do is plan the rest of the conference. And let it snow, snow and snow so that we've got a good base of snow ready for the conference.

The next date to worry about is in about 3 weeks time, which is the deadline for any scheduled talks. We've got a couple of interesting case study talks lined up and still looking for some good introductory talks eg: Introduction to Django, Pinax, Django-CMS and so on.

Been putting quite a few posts about DjangoSki up, sorry about that. It's very exciting for me and been quite a bit of work. Normal business of ignoring the blog and going back to Twitter will resume shortly.

Matt Berg keynote, call for speakers for DjangoSki

Wed, 02 Dec 2009 15:53:42 -0800

For our final keynote I'm pleased to say that Matt Berg will be speaking at DjangoSki.

Matt Berg is the ICT Coordinator on the Millenium Villages Project at Columbia University. As part of that project he's been building technical systems in developing countries. He's involved with RapidSMS and projects include work in Kenya, South Senegal, Malawi and Rwanda that help children with malaria and malnutrition. And it's all using open source software and Django.

Not only does Matt do interesting work that is helping people right now, but he passionately believes that he can use Django and other technologies to make a difference. His passion and dedication is infectious, as you'll find out.

We have room for other speakers at the conference, so we'd like to call for other speakers to submit proposals to speak at the conference. Please apply here or see details here.

Jacob Kaplan-Moss keynote at DjangoSki

Mon, 30 Nov 2009 15:31:45 -0800

We are excited to announce that Jacob Kaplan-Moss will be giving one of the keynote talks at DjangoSki.

It's almost impossible to be around Django and not know who Jacob is. He's one of the original founders of Django, is on the foundation board, is the co-BDFL of Django and is the co-author of The Definitive Guide to Django. Let's face it, if it's got something to do with Django, Jacob has had some influence on it. Currently Jacob works at Revolution Systems where he gets to do lots more Django.

After missing Jacob at DjangoCon in Portland, it will be certainly good to have him around in Whistler. Perhaps even get some sprinting on Django 1.2 (which will be out a few days after DjangoSki). Jacob writes a great blog and is a great speaker.

Djangopeople JSON parser

Fri, 27 Nov 2009 16:59:22 -0800

I bet you've always wanted to parse your djangopeople.net account into JSONP so you can use it somewhere else on the web. Well your wait is over: djangopeople-jsonp.appspot.com will do just. Some examples of some famous and not so famous Django people. (and of course mr nobody).

David Ascher keynote at DjangoSki

Fri, 27 Nov 2009 09:10:18 -0800

We are pleased to announce the first keynote speaker for DjangoSki is David Ascher.

David Ascher is the CEO of Mozilla Messaging. He's been involved with Python for many years and has dabbled around in Django too. He co-authored Learning Python and worked on Komodo whilst at ActiveState (where he became the CTO). Most recently Mozilla Messaging released the rather cool Raindrop, which is written in Python.

David has done some great talks at Python conferences in the past, providing thought provoking and entertaining talks. We think he can ski too, but we'll find that out.

We would also like to thank our first sponsor for DjangoSki, OpenRoad Communications.

If you would like to sponsor DjangoSki and get your company in front of key Djangoers whilst supporting a great conference, contact ClearWind. For more information on the DjangoSki conference, go to djangoski.com.

Django Documentation iPhone App

Tue, 24 Nov 2009 14:54:29 -0800

We've released iPhone app that contains all the Django documentation. Basically we took the documentation, reformatted it a bit to try and fit on the iPhone and then pushed it all into PhoneGap. Unfortunately there was then a 5 week wait as Apple thought about approving it.

The app has just now been approved and is on the app store. There's more information on the Clearwind site, here.

There's a few things we'd like to improve like: reducing the memory footprint, remembering the last page and indexing in spotlight. I'm not sure how many of those things will be possible, but we'll keep the app to date with the latest docs.

And the best thing is, 30 cents out of every app sale goes to the Django Software Foundation.

Javascript inline forms outside of contrib.admin

Mon, 23 Nov 2009 13:50:22 -0800

This recipe follows on from the last two and shows how to do the addition of inline fields to a model form, outside of contrib.admin.

Announcing DjangoSki Conference

Thu, 19 Nov 2009 15:44:58 -0800

I am very excited to announce the DjangoSki Conference in Whistler, March 2nd-4th 2010.

This is a conference that combines my favourite web application framework with one of my favourite sports, skiing. If you are going to run a conference, may as well do it somewhere with style right? The conference is slightly unusual in that it's styled a bit after an unconference. There are several keynote talks we are lining up and some introductory talks, but the rest of the conference is up to the attendees.

Each day will have sprints or ad-hoc talks created by the attendees. The real goal is to focus on the things I find enjoyable about a conference - meeting people and learning. Sprints are the ideal way to learn from other people. I'm not alone in this.

Details will follow as we get registration and keynotes organized.

So if you are planning a ski holiday this year and do Django, the choice has just become a whole lot easier. Come to Whistler and have a damn good time.

Adding Javascript to contrib.admin inline fields

Mon, 16 Nov 2009 12:08:10 -0800

This recipe follows on from the last by adding Javascript to the inline fields of a model in the contrib.admin.

Setting up inline forms in contrib.admin

Sun, 08 Nov 2009 02:47:14 -0800

This recipe covers how to set up inline forms in contrib admin and how to show foreign keys in those inline forms.

Announcing Django Training

Sun, 25 Oct 2009 05:46:50 -0700

I'm pleased to announce that in January 2010 I'll be running Django Training in Vancouver. The four day training covers all the things you need to know about getting Django up and running. The training is very hands-on - a bit of talking and then a bit of writing code.

Since it's in Vancouver in the winter, we also thought we'd throw in a nights skiing up Cypress Mountain (where the Olympic free-style skiing and snowboarding will be held). After many years of skiing I still consider myself quite an amateur, so that will be another opportunity for me to learn from you.

After several years of Plone training, I'm really looking forward to doing some Django training. Hope to see you here.

Tuning the count method on a queryset

Sat, 17 Oct 2009 16:18:35 -0700

Some alternate ways to manipulate the pagination of Django query sets.

First things to do on a Django project...

Thu, 03 Sep 2009 14:12:49 -0700

Call them best practices if you like, but here's the things I do when starting a new Django project.

Serving multiple templates

Tue, 28 Jul 2009 13:21:03 -0700

The lack of magic in Django

Sat, 25 Jul 2009 01:12:05 -0700

...or why Rails and Zope 2 were too magical.

Forms, forms, forms

Tue, 21 Jul 2009 15:42:37 -0700

Some questions answered about Django forms.

Features of a good Django plugin

Wed, 08 Jul 2009 14:44:49 -0700

Well, my opinion on them anyway.

Read only fields in models

Thu, 18 Jun 2009 15:43:12 -0700

Supposing you have a field in your model and you don't want it to be changed, ever.

djapp.org

Mon, 08 Jun 2009 15:56:25 -0700

Another site listing django plugins

Class based views

Sun, 31 May 2009 14:24:30 -0700

We discussed these at a recent Vancouver Django (May 2009) meetup and it was also a pattern I used on a recent project.

Useful Django API's

Mon, 18 May 2009 17:18:56 -0700

Two useful Django API's that I've found today.

Job board added

Fri, 08 May 2009 10:55:29 -0700

As promised the site now has a job board

Caching and signals

Tue, 05 May 2009 04:00:30 -0700

Adding in cached objects on signals

Django, Malnutrition, SMS and Kenya

Tue, 28 Apr 2009 15:59:45 -0700

How I ended up doing one of most interesting and important projects I've been involved with.

Turning off django signals

Sat, 11 Apr 2009 01:36:57 -0700

While I'm on the subject, how to turn off signals that you don't want on.

Signal vs overriding save

Tue, 07 Apr 2009 04:33:09 -0700

I'm a big fan of signals and wonder why people are so adverse to using them, preferring overriding save instead.