I wanted a function that 'crawled' all the users in a django queryset so the whole problem was the old, how to work on a whole list but in small chunks. I needed this for performance reasons because it was expensive to try to generate all the user's thumbnails at once and expecting that nothing would go wrong was insane. Apparently facebook servers are not as stable as you think.
So less bla bla y more code:
def fanfunders_subset(self,start=None,end=None):
return UserProfile.objects.fanfunders()[start:end]
So i would call this function like this in a loop
fanfunders_subset(0,10)
# and in the next iteration
fanfunders_subset(10,20)
At this moment i was thankful python lists end in -1, so slicing [0,10] will get you the first number the decond and up to the ninth, if it didnt ended in -1 i would have to do something akward like [0,10] [11,20] [21,30] and that looks ugly.
So what was the trick? oh yeah, you can slice with None! in this context
>>> list[0,None] == list[0,-1]
True
And then
>>> list[None,-1] == list[0,-1]
True
So you can only give on the two arguments to the function and it will return the correct slice, yeah this trick is more for that, to get this flexible interface, sometimes you want a chunk from the end of the list and sometimes you want it from the begining and somethings you want in the middle, lots of cases are covered safely here it just looks a little weird if you see this at first, thats why i wrote this post :P So more people know it and you don't judge me for using weird tricks in Python.
sábado, 10 de noviembre de 2012
domingo, 4 de noviembre de 2012
Connection pooling in heroku with Django
You should use this:
So Postgresql is your thing on heroku with django, and you think to your self, how can i shave miliseconds of my response time, while doing practically nothing.
Your answer, database connectiong pooling.
In your settings.py
import urlparse
url = urlparse.urlparse(os.environ['HEROKU_POSTGRESQL_GOLD_URL'])
path = url.path[1:]
path = path.split('?', 2)[0]
DATABASES = {
'default': {
'ENGINE': 'dbpool.db.backends.postgresql_psycopg2',
'OPTIONS': {'max_conns': 1},
'HOST': url.hostname,
'NAME': path,
'OPTIONS': {},
'PASSWORD': url.password,
'PORT': url.port,
'TEST_CHARSET': None,
'TEST_COLLATION': None,
'TEST_MIRROR': None,
'TEST_NAME': None,
'TIME_ZONE': 'America/Mexico_City',
'USER': url.username
}
}
#If you use south migrations
SOUTH_DATABASE_ADAPTERS = {
'default': 'south.db.postgresql_psycopg2',
}
And in your requirements.txt
django-db-pool==0.0.7
So, numbers, i'll just leave my new relic graph here.
This is a graph of 7 days with response time on the x axis and time on the y axis.
That bump in response time, that orangeish growth is the database taking 50% more time answering because django has to setup the connection to the database on each request.
It is pretty clear with connection pooling > 200 ms response time, after > 300 ms. thats a 50% increase in your app performance for basically changing a setting.
Notice:
Heroku limits the 9 dls database plan to 20 connections, and because the connections are persistent thanks to the pool, youll be permanently using more(the exact number i don't know) but be careful when running out connections, if you have a lot of workers writing to the database and you restart the web app it might run out of connections and you'll site will go down(experience talking here).
You'll have to migrate to crane the 50 dls plan to avoid the connection limit. So be careful out there, it happened to me with a cronjob using the connections leaving my main app connection thirsthy and really down.
Update:
I was totally wrong, as it turns out, i changed the pooling setting at the same time heroku migrated to the new databases, and they are the guilty ones for this increase in response time. tzk tzk tzk heroku.
So Postgresql is your thing on heroku with django, and you think to your self, how can i shave miliseconds of my response time, while doing practically nothing.
Your answer, database connectiong pooling.
In your settings.py
import urlparse
url = urlparse.urlparse(os.environ['HEROKU_POSTGRESQL_GOLD_URL'])
path = url.path[1:]
path = path.split('?', 2)[0]
DATABASES = {
'default': {
'ENGINE': 'dbpool.db.backends.postgresql_psycopg2',
'OPTIONS': {'max_conns': 1},
'HOST': url.hostname,
'NAME': path,
'OPTIONS': {},
'PASSWORD': url.password,
'PORT': url.port,
'TEST_CHARSET': None,
'TEST_COLLATION': None,
'TEST_MIRROR': None,
'TEST_NAME': None,
'TIME_ZONE': 'America/Mexico_City',
'USER': url.username
}
}
#If you use south migrations
SOUTH_DATABASE_ADAPTERS = {
'default': 'south.db.postgresql_psycopg2',
}
And in your requirements.txt
django-db-pool==0.0.7
So, numbers, i'll just leave my new relic graph here.
This is a graph of 7 days with response time on the x axis and time on the y axis.
That bump in response time, that orangeish growth is the database taking 50% more time answering because django has to setup the connection to the database on each request.
It is pretty clear with connection pooling > 200 ms response time, after > 300 ms. thats a 50% increase in your app performance for basically changing a setting.
Notice:
Heroku limits the 9 dls database plan to 20 connections, and because the connections are persistent thanks to the pool, youll be permanently using more(the exact number i don't know) but be careful when running out connections, if you have a lot of workers writing to the database and you restart the web app it might run out of connections and you'll site will go down(experience talking here).
You'll have to migrate to crane the 50 dls plan to avoid the connection limit. So be careful out there, it happened to me with a cronjob using the connections leaving my main app connection thirsthy and really down.
Update:
I was totally wrong, as it turns out, i changed the pooling setting at the same time heroku migrated to the new databases, and they are the guilty ones for this increase in response time. tzk tzk tzk heroku.
Suscribirse a:
Entradas (Atom)