Monitoring

[theaskou]

As described in #128 we need to decide a solution for monitoring.
Aleksander is suggesting we use Grafana with Prometheus, but maybe we should see what other solutions there are.
We should also discuss what we want to monitor.

Ideally, we will be able to start setting up the monitoring next week, according to Anders' plan.

[Aleksandergreg]

I have looked into some other options, just to see if anything caught the eye.

ELK

Another option that is used industry wide is ELK stack, which Anders also has an individual assignment for. It consists of consists of ElasticSearch, Logstash, and Kibana, which seems to be the industry standard, and Kibana, the visualisation tool looks very proffesional:

Especially ElasticSearch is really ressource intensive, from my understanding, so I think even with a seperate VM for our monitor, we might run out of CPU/RAM with our current Azure Student setup.

Datadog

The big pro here would be that it's a PAAS, so the setup should be a lot easier to setup. The big con on the other hand would be that they only offer a 14 day free trial, which I think isn't enough for us, especially when we should have it up and running for the exam. The cost pr month from there is 15$ a month, which I really don't think we should put into it.

Why go with Prometheus and Grafana?

From my point, the biggest pro of this is that there is a wide community for Prometheus, especially if/when we start organizing our containers in Kubernetes, this seems to be the way to go for logging and monitoring, and there is a lot of community support if we run into problems.

Also Anders' made it very clear that this is a viable option, and he has made a guide for it, that should be pretty plug-n-play, with just adjusting what we want to monitor.

What should we be monitoring?

Infrastructure Metrics:

• CPU Usage
• RAM usage
• Ressource user pr container

APM (Application metrics)

• Request Rate/Throughput: How many requests per second is the Ruby application handling?
• Request Latency/Duration: How long do requests take (average, 95th percentile, 99th percentile)?

In regards to the request Latency/Duration, I have setup a very simple logging mechanism for #124 that currently logs plain text inside a volume of our container, and it could be fun to see how it would affect our system, if anything, if we changed this to JSON, which from my understanding is the best way to handle logging data.