A couple steps back, none forward

I haven’t had time nor real interest to work with the train app. Not that I have abonded it, I just haven’t had the energy lately. Nor the nerves to come home at 7pm and start banging the code. Gotta socialise with the family too 😀

In any case, the code works as perfectly as I would imagine on the Vagrant but I just can’t deploy it properly to the RPI. So gotta start looking into deployment tools, which is not that bad of an idea as the whole idea is to learn new tools, techniques and processes.

Let’s see.

edit. The root cause was an old version of the MySQL on the Vagrant image (5.5x) compared to the 5.7x on the RPI. The timestamp has changed and that caused all the issues. I have fixed it but it’s rather quick fix instead of sofisticated one. So still some work to do. At least the app works on the RPI.

Basic skills – are what?

As this lifelong project advances I will pick up new skills which some people might find to be basic skills. But what are the basic skills and are there IT-wide basic skills or are the basic skills related to the field? The main driver for this post is that I still consider myself a n00b13 in many cases but I get asked “why would you do/learn that?”. I just think some skills are basic skills. But to what extend?

I really don’t know

I had started going through some of the basic skills but that makes no sense. It would just be an uneducated guess. Let’s just settle with the following: You know how the computers work  and you know how to make them work . And you can explain that to others.

So you need to understand the hardware, have a command of (at least one) language and be reasonably good in communication. There’s one exception to the first requirement: Nobody understand printers 😀

 

 

 

 

 

RPI Hadoop cluster (brief summary)

The JunaTracker app is on hold for a while until I figure out the nginx configuration. Although, I have done some tinkering on the templates and the backend. But that’s not the topic here.

So with lots of time on hand during the weekend I decided to configure two RPI 3’s to work as a Hadoop cluster. Two because I only had those two available. However, I just ordered two more so the cluster will grow soon 😉 . Just to get them working I opted not to compile the source and just used the 2.7.3 binaries. I’ll redo the steps and document the process and will most likely also compile so a) I get rid off the warnings of non-nativeness and b) to benchmark to see if that makes a considerable difference in the performance.

It took some time as some of the tutorials are in fact missing some bits and pieces or are actually plain wrong (wrong config item names) but I finally made it and the two RPI 3’s are clustered as you can see:pi_cluster

I’ll start working and testing this week and will add two more Pi’s as they arrive. And in order to ease my life I will also document the whole process here.

It’s good to succeed in something every once in a while 🙂

django+uwsgi+nginx == frustration

Just a short post so I can vent 😀

I’ve been trying to migrate the django app into uwsgi+ningx stack with no real success. There are multiple tutorials and I’m sure they are good and decent but they basically start by creating a skeleton django app. And that’s not what I have. So not sure if this is the reason for the failures but it just does not compute.

As soon as I get them to work I’ll post detailed instructions how to do it and will also do it multiple times.

Django & JunaTracker

As hinted in the previous JunaTracker post, I have started creating a GUI for the project. Again took some time to learn new stuff (Django, bootstrap) but it’s going pretty well. I have a running application in the dev environment and at the moment I’m figuring out what other useful information I can extract from the API on top of the running times.

The next steps will be to expand the application to cover all the stations and also learn nginx or other web server configuration as the django server is only for dev. Not that I’m sure I will publish this but it never harms to learn how to setup a webserver in production. Perhaps try pentesting and other security related topics 🙂

The setup

Originally the program was running in a way that it requested a json from the API, saved it on the fs and then parsed and printed it on the screen. It has now been modified and the fs part has been scrapped. The queried json is now parsed on the fly and loaded into a MySQL database. Django comes with SqlLite but I figured it’s better to use MySQL for the job.

As I’m still a novice, I’m not even going to try to cover what Django is or is not other than: It’s a web applications framework and fairly easy to understand. And it enables separated development of models, views and templates. So at this point I’m mainly focusing on the models as I’m figuring out how the data should actually be modelled. I have of course views and templates to show the data and I even applied bootstrap on top of the site but the content is more important than the look and feel at this point.

There are a plenty of great tutorials on how to get MySQL to work with the Django. You basically change the settings.py’s database part and setup the MySQL. At later point I will config the db settings in a separate file and just reference that.

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.mysql',
        'NAME': 'TRAINDB',
        'USER': <username>,
        'PASSWORD': <password>,
        'HOST': '192.168.68.8',
        'PORT': '',
        'OPTIONS': {'init_command': "SET sql_mode='STRICT_TRANS_TABLES'"},
    }
}

I think I have struggled more on finding python-MySQL connector for the Raspberry. Oh yes, it’s already running on my Raspi 🙂 I’ll cover that later. The actual loading of the parsed JSON is easy as you can see. The data is inserted as string at the moment but the types will be corrected at later point. I just wanted to get things going in.

junatracker_db

The look and feel

As stated, I haven’t spent too much time on the html/css part and that will be another interesting area to dig into. But as you can see, I have managed to make it bearable at least junatrackerThe train number is a link to the details and perhaps an OpenStreetMap with the location. To be seen though.

All in all, at the moment I’m more focused on the back-end than front-end. Should I publish the application then the front-end requires some serious work and testing but that’s really not my ultimate goal. Although the idea is tempting and would push me to “learn more” so to speak.

 

Vagrant

One major reason I actually started this journey is Vagrant. I wanted to practice my Linux skills and was trying to figure out how to install and maintain an image. Cue in Vagrant and I was sold. I instantly realised that I could be testing various things without compromising the laptop’s integrity.

After learning the basics I started to configure the vagrant file in order to make it more suitable for my needs. It’s still a very very simple file but I have couple of provisions, setup scripts for MySQL, ngingx and mongo and the needed port forwards and IP bindings. So in the case I need to drop the image I can have useful dev environment up and running in short time. I still need to figure out the needed Python lib installs (requirement file would be sufficient I suppose) but those are minor nuisances at the moment.

I have also checked the Docker but haven’t really figured out a real use case so I’ll leave it for now.

Learning – how?

The biggest hurdles for me so far have been lack of clear vision. I’m not planning on creating the next wazoo! app (or something hip, along those those lines) as I’d rather develop my skills in order to match the realities of big data and move from traditional data warehouses.

But obviously the vision is not the only thing you need when learning new skills. One obviously needs some tutorials or guides when learning something completely new and the net has abundance of resources. A tricky question is which ones to use and focus on?

M(O)OCs

The first MOOC I came across was Udacity (for profit nowadays though) short after it was launched. I still check it occasionally and have multiple courses on my todo list but the early courses just feel, well old. And with that I mean the interface, not the content itself (even though that might be the case, not sure).

So, lately I have found myself logging into Code School. Surely it’s still growing the course base but for novices it has fairly good selection of modules and complete courses. And even though I really hate the intro songs (really annoiyng) the actual platform just seems fit to me. Plus the content, which is still the main driver here.

There are others too but I will stick with these two for now. I’m only subscribed to the Code School as Udacity has a reasonable amount of free (mini) courses. But perhaps during the next year I’ll try to achieve one of their nanodegrees. I just feel I need to build up my skills in order to decide which one(s) to pursue.

Other online resources

I have found myself drawn to Stack overflow whenever I need information as so far I haven’t had a question which wasn’t asked. And with some searching you will also find the right questions being answered to as well 🙂

Online documentation of Python, JavaScript or other languages is a no brainer and those should be bookmarked.

And for the rest, just google and you shall find. I don’t really follow any specific forums or blogs, mostly because I would spend my time there instead of coding. Perhaps this blog will force me to see what and how others are tackling the issues. Let’s see.

Books

I have purchased couple of Kindle books but with so much information readily available I haven’t had the need to spend too much on books. Plus also I have access to Books24x7 via the company I work for it makes no sense to spend too much on curated literature.