So, since I got the JunaTracker working in pythonanywhere.com I started thinking about the performance. Which is not that great at the moment, although this will never really be a public application.
Regardless, I started playing with the MongoDB and after some head scratching I managed to get the data import working. The root cause for head scratching was I that wanted to create a composite index based on the version of the timetable, train number and departure date. This is much faster than trying to figure out the updated fields and actually update them. So all fun and sun shine, right?
However the rata API seems to resend some of the old data even when querying with version number (the idea is that it should send data AFTER that version) and obvisously that caused issues with duplicates. So I did what any Python newbie would do and tried to tackle that with
Obviously that only works up until the first error and then the exception causes the program to exit.
But with some thinking and a bit of help from SO I figured to encapsulate the try block into a for loop. So the exception would only exit the current loop iteration and continue to the next one.
for item in data:
So live and learn. I learned more about exceptions in Python and also was able to control the data flow (insert and query). So perhaps the JunaTracker will be migrated into MongoDB in future.