Playing with Mongo

So, since I got the JunaTracker working in pythonanywhere.com I started thinking about the performance. Which is not that great at the moment, although this will never really be a public application.

Regardless, I started playing with the MongoDB and after some head scratching I managed to get the data import working. The root cause for head scratching was I that wanted to create a composite index based on the version of the timetable, train number and departure date. This is much faster than trying to figure out the updated fields and actually update them. So all fun and sun shine, right?

However the rata API seems to resend some of the old data even when querying with version number (the idea is that it should send data AFTER that version) and obvisously that caused issues with duplicates. So I did what any Python newbie would do and tried to tackle that with

try:
db.insert(data)
except pymongo.errors.DuplicateKeyError,e:
pass

Obviously that only works up until the first error and then the exception causes the program to exit.

But with some thinking and a bit of help from SO I figured to encapsulate the try block into a for loop. So the exception would only exit the current loop iteration and continue to the next one.


for item in data:
try:
collection.insert(item)
except pymongo.errors.DuplicateKeyError,e:
pass

So live and learn. I learned more about exceptions in Python and also was able to control the data flow (insert and query). So perhaps the JunaTracker will be migrated into MongoDB in future.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s