Install Prometheus on Ubuntu 16.04

Prometheus

Prometheus could be a powerful and open-source, ASCII text file observance system that collects metrics from your services and stores them in an exceedingly time-series information.

It proposes a multi-dimensional data model, a versatile query language, and numerous visualization image prospects through tools like Grafana.

Prometheus solely exports metrics regarding itself (e.g. the number of requests it’s received, its memory consumption, etc.). But, you’ll greatly expand Prometheus by installing exporters, optional gratia programs that generate further metrics.

Exporters—both the official ones that the Prometheus team maintains further because the community-contributed ones—provide info regarding everything from infrastructure, databases, and web servers to electronic communication systems, APIs, and more.

Some of the foremost common selections include:

• node_exporter – This produces metrics regarding infrastructure, as well as the present CPU, memory, and disk usage, further as I/O and network statistics, like the quantity of bytes scan from a disk or a server’s average load.
• blackbox_exporter – This generates metrics derived from inquisitory protocols like protocol HTTP and HTTPS to see end convenience, time interval, and more.
• mysqld_exporter – This gathers metrics associated with a MySQL server, like the quantity of executed queries, average response time interval, and cluster replication standing.
• rabbitmq_exporter – This outputs metrics regarding the RabbitMQ electronic communication system, as well as the number of messages printed, the number of messages able to be delivered, and also the size of all the messages within the queue.
• nginx-vts-exporter – This provides metrics regarding associate Nginx web server victimization the Nginx VTS module, as well as the number of open connections, the quantity of sent responses (grouped by response codes), and also the total size of sent or received requests in bytes.

You will notice a lot of complete list of each official and community-contributed exporters on Prometheus’ web site.

You’ll install, configure, and secure Prometheus and Node Exporter to come up with metrics that may build it easier to observe your server’s performance.

Prerequisites

Before following this tutorial check that you have:

• One Ubuntu 16.04 Droplet, established by following the Initial Server Setup with Ubuntu sixteen.04 tutorial, as well as a sudo non-root user and a firewall.
• Nginx installed by following the primary 2 steps of the way to Install Nginx on Ubuntu 16.04 tutorial.

Step 1 — Create Service Users

For the safety functions, we’ll begin by making 2 new user accounts, prometheus and node_exporter. We’ll use these accounts throughout the tutorial to isolate the possession on Prometheus’ core files and directories.

Create these 2 users, and use the –no-create-home and –shell /bin/false choices in order that these users cannot log into the server.

$ sudo useradd --no-create-home --shell /bin/false prometheus
$ sudo useradd --no-create-home --shell /bin/false node_exporter

Before downloading the Prometheus binaries, create the essential directories to store Prometheus’ files and data.

The upcoming standard Linux conventions, we’ll generate a directory in /etc for Prometheus’ configuration files and a directory in /var/lib for its data.

$ sudo mkdir /etc/prometheus
$ sudo mkdir /var/lib/prometheus

Set user and group ownership for new directories of prometheus user.

$ sudo chown prometheus:prometheus /etc/prometheus
$ sudo chown prometheus:prometheus /var/lib/prometheus

With our users in place, we will currently download prometheus so produce the specified configuration file to run for 1st time.

Step 2 — Download Prometheus

Download and take the present stable version of Prometheus into your home directory. You’ll notice the newest binaries together with their checksums on the Prometheus transfer page.

$ cd ~
$ curl -LO https://github.com/prometheus/prometheus/releases/download/v2.0.0/prometheus-2.0.0.linux-amd64.tar.gz

Next, use the sha256sum command to come up with a checksum of the downloaded file:

$ sha256sum prometheus-2.0.0.linux-amd64.tar.gz

Compare the output from this command with the checksum on the Prometheus download page to confirm that your file is each real and not corrupted.

Output

e12917b25b32980daee0e9cf879d9ec197e2893924bd1574604eb0f550034d46  prometheus-2.0.0.linux-amd64.tar.gz

If the checksums do not match, take away the downloaded file and repeat the preceding steps to re-download the file.

Now, take out the downloaded archive.

$ tar xvf prometheus-2.0.0.linux-amd64.tar.gz

This can produce a directory known as prometheus-1.7.1.linux-amd64 containing 2 binary files (prometheus and promtool), consoles, console_libraries, and several other example files.

Copy the 2 binaries to the /usr/local/bin directory.

$ sudo cp prometheus-2.0.0.linux-amd64/prometheus /usr/local/bin/
$ sudo cp prometheus-2.0.0.linux-amd64/promtool /usr/local/bin/

Set the user and cluster possession on the binaries to the Prometheus user created in Step 1.

$ sudo chown prometheus:prometheus /usr/local/bin/prometheus
$ sudo chown prometheus:prometheus /usr/local/bin/promtool

Copy the consoles and console_libraries directories to /etc/prometheus.

$ sudo cp -r prometheus-2.0.0.linux-amd64/consoles /etc/prometheus
$ sudo cp -r prometheus-2.0.0.linux-amd64/console_libraries /etc/prometheus

Set the user and cluster ownership on the directories to the prometheus user. Using the -R flag will confirm that ownership is set on the files inside the directory as well.

$ sudo chown -R prometheus:prometheus /etc/prometheus/consoles
$ sudo chown -R prometheus:prometheus /etc/prometheus/console_libraries

Lastly, take away the leftover files from your home directory as they’re not required.

$ rm -rf prometheus-2.0.0.linux-amd64.tar.gz prometheus-2.0.0.linux-amd64

Now Prometheus is installed, we’ll produce its configuration and service files in preparation of its initial run.

Step 3 — Configure Prometheus

In the /etc/prometheus directory, use nano or your favorite text editor to make a configuration file named Prometheus.yml. For now, this file can contain barely enough info to run Prometheus for the primary time.

$ sudo nano /etc/prometheus/prometheus.yml

Warning: Prometheus’ configuration file uses the YAML format, that strictly forbids tabs and needs 2 areas for indentation. Prometheus can fail to start out if the configuration file is incorrectly formatted.

In the global settings, outline the default interval for scraping metrics. Note that Prometheus can apply these settings to each exporter unless a personal exporter’s own settings override the globals.

global:
  scrape_interval: 15s

This scrape_interval value tells Prometheus to gather metrics from its exporters every fifteen seconds, that is long enough for many exporters.

Now, add Prometheus itself to the list of exporters to scrape from with the subsequent scrape_configs directive:

...
scrape_configs:
  - job_name: 'prometheus'
    scrape_interval: 5s
    static_configs:
      - targets: ['localhost:9090']

Prometheus uses the job_name to label exporters in queries and on graphs, therefore make certain to choose one thing descriptive here.

And, as Prometheus exports necessary information regarding itself that you simply will use for observance performance and debugging, we’ve overridden the global scrape_interval directive from 15 seconds to 5 seconds for a lot of frequent updates.

Lastly, Prometheus uses the static_configs and targets directives to see wherever exporters square measure running. Since this explicit exporter is running on an identical server as Prometheus itself, we will use localhost rather than IP address together with the default port, 9090.

Your configuration file will to currently seem like this:

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'prometheus'
    scrape_interval: 5s
    static_configs:
      - targets: ['localhost:9090']

Save the file and exit your text editor.

Now, set the user and cluster possession on the configuration file to the Prometheus user created in Step 1.

$ sudo chown prometheus:prometheus /etc/prometheus/prometheus.yml

With the configuration complete, we’re able to take a look at prometheus by running it for the primary time.

Step 4 — Running Prometheus

Startup Prometheus because the Prometheus user, providing the trail to each the configuration file and also the data directory.

$ sudo -u prometheus /usr/local/bin/prometheus \
$    --config.file /etc/prometheus/prometheus.yml \
$   --storage.tsdb.path /var/lib/prometheus/ \
$    --web.console.templates=/etc/prometheus/consoles \
$    --web.console.libraries=/etc/prometheus/console_libraries

The output contains info regarding Prometheus’ loading progress, configuration file, and connected services. It conjointly confirms that Prometheus is listening on port 9090.

Output

level=info ts=2017-11-17T18:37:27.474530094Z caller=main.go:215 msg="Starting Prometheus" version="(version=2.0.0, branch=HEAD, re
vision=0a74f98628a0463dddc90528220c94de5032d1a0)"
level=info ts=2017-11-17T18:37:27.474758404Z caller=main.go:216 build_context="(go=go1.9.2, user=root@615b82cb36b6, date=20171108-
07:11:59)"
level=info ts=2017-11-17T18:37:27.474883982Z caller=main.go:217 host_details="(Linux 4.4.0-98-generic #121-Ubuntu SMP Tue Oct 10 1
4:24:03 UTC 2017 x86_64 prometheus-update (none))"
level=info ts=2017-11-17T18:37:27.483661837Z caller=web.go:380 component=web msg="Start listening for connections" address=0.0.0.0
:9090
level=info ts=2017-11-17T18:37:27.489730138Z caller=main.go:314 msg="Starting TSDB"
level=info ts=2017-11-17T18:37:27.516050288Z caller=targetmanager.go:71 component="target manager" msg="Starting target manager...
"
level=info ts=2017-11-17T18:37:27.537629169Z caller=main.go:326 msg="TSDB started"
level=info ts=2017-11-17T18:37:27.537896721Z caller=main.go:394 msg="Loading configuration file" filename=/etc/prometheus/promethe
us.yml
level=info ts=2017-11-17T18:37:27.53890004Z caller=main.go:371 msg="Server is ready to receive requests."

If an error message comes, ensure that you’ve got used YAML syntax in your configuration file so follow the on-screen directions to resolve the matter.

Now, halt Prometheus by pressing CTRL+C, so open a brand new systemd service file.

$ sudo nano /etc/systemd/system/prometheus.service

The service file tells systemd to run Prometheus because the Prometheus user, with the configuration file situated within the /etc/prometheus/prometheus.yml directory and to store its information within the /var/lib/prometheus directory. (The details of systemd service files square measure on the far side the scope of this tutorial)

Copy the subsequent content into the file:

[Unit]
Description=Prometheus
Wants=network-online.target
After=network-online.target

[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/prometheus \
    --config.file /etc/prometheus/prometheus.yml \
    --storage.tsdb.path /var/lib/prometheus/ \
    --web.console.templates=/etc/prometheus/consoles \
    --web.console.libraries=/etc/prometheus/console_libraries

[Install]
WantedBy=multi-user.target

Finally, save the file and shut your text editor.

To use the recently created service, reload systemd.

$ sudo systemctl daemon-reload

You will currently begin Prometheus victimization the subsequent command:

$ sudo systemctl start prometheus

To check that Prometheus is running, check the service’s status.

$ sudo systemctl status prometheus

The output tells you Prometheus’ status, main method symbol (PID), memory use, and more.

If the service’s status is not active, follow the on-screen directions and re-trace the preceding steps to resolve the matter before continued the tutorial.

Output

● prometheus.service - Prometheus
   Loaded: loaded (/etc/systemd/system/prometheus.service; disabled; vendor preset: enabled)
   Active: active (running) since Fri 2017-07-21 11:40:40 UTC; 3s ago
 Main PID: 2104 (prometheus)
    Tasks: 7
   Memory: 13.8M
      CPU: 470ms
   CGroup: /system.slice/prometheus.service
...

When you are able to move, press Q to quit the status command.

Lastly, change the service to start on boot.

$ sudo systemctl enable prometheus

Now that Prometheus is up and running, we will install an extra exporter to come up with metrics regarding our server’s resources.

Step 5 — Downloading Node Exporter

To expand Prometheus on the far side metrics regarding itself solely, we’ll install an extra exporter known as Node exporter. Node exporter provides careful info regarding the system, as well as CPU, disk, and memory usage.

First, download the present stable version of Node Exporter into your home directory. You’ll notice the newest binaries together with their checksums on Prometheus’ download page.

$ cd ~
$ curl -LO https://github.com/prometheus/node_exporter/releases/download/v0.15.1/node_exporter-0.15.1.linux-amd64.tar.gz

Use the sha256sum command to create a checksum of the downloaded file:

$ sha256sum node_exporter-0.15.1.linux-amd64.tar.gz

Now, verify the downloaded file’s integrity by comparing its checksum with the one on the download page.

Output

7ffb3773abb71dd2b2119c5f6a7a0dbca0cff34b24b2ced9e01d9897df61a127  node_exporter-0.15.1.linux-amd64.tar.gz

If the checksums do not match, take away the downloaded file and repeat the preceding steps.

Now, unload downloaded archive.

$ tar xvf node_exporter-0.15.1.linux-amd64.tar.gz

This can produce a directory known as node_exporter-0.14.0.linux-amd64 containing a computer file named node_exporter, a license, and a notice.

Copy the binary to the /usr/local/bin directory and set the user and cluster possession to the node_exporter user that you simply created in Step 1.

$ sudo cp node_exporter-0.15.1.linux-amd64/node_exporter /usr/local/bin
$ sudo chown node_exporter:node_exporter /usr/local/bin/node_exporter

Lastly, take away the leftover files from your home directory as they’re not required.

$ rm -rf node_exporter-0.15.1.linux-amd64.tar.gz node_exporter-0.15.1.linux-amd64

Now that you’ve got put in Node Exporter, you’ll take a look at it out by running it so we will produce a service file for it in order that it starts on boot.

Step 6 — Running Node Exporter

The steps for running Node Exporter is the same as those for running Prometheus itself. Begin by making the Systemd service file for Node Exporter.

$ sudo nano /etc/systemd/system/node_exporter.service

This service file tells your system to run Node Exporter because the node_exporter user with the default set of collectors enabled.

Copy the subsequent content into the service file:

[Unit]
Description=Node Exporter
Wants=network-online.target
After=network-online.target

[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter

[Install]
WantedBy=multi-user.target

Collectors outline that metrics Node Exporter can generate. You’ll see Node Exporter’s complete list of collectors—including enabled by default and that deprecated—in the Node Exporter README file.

If you ever have to be compelled to override the default list of collectors, you’ll use the –collectors.enabled flag, like:

...
ExecStart=/usr/local/bin/node_exporter --collectors.enabled meminfo,loadavg,filesystem
...

The preceding example would tell Node Exporter to create metrics victimization solely the meminfo, loadavg, and filesystem collectors. You’ll limit the collectors to but few or several you would like, however, note that there are not any blank areas before or when the commas.

Save the file and shut your text editor.

Finally, reload systemd to use the recently created service.

$ sudo systemctl daemon-reload

At the end of the scrape_configs block, add a brand new entry known as node_exporter.

...
  - job_name: 'node_exporter'
    scrape_interval: 5s
    static_configs:
      - targets: ['localhost:9100']

Because this Exporter is additionally running on an identical server as Prometheus itself. We will use localhost rather than IP address once more together with Node Exporter’s default port, 9100.

Your whole configuration file ought to seem like this:

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'prometheus'
    scrape_interval: 5s
    static_configs:
      - targets: ['localhost:9090']
  - job_name: 'node_exporter'
    scrape_interval: 5s
    static_configs:
      - targets: ['localhost:9100']   

Save the file and exit your text editor once you are able to continue.

Finally, restart Prometheus to place the changes into impact.

$ sudo systemctl restart prometheus

Once again, verify that everything is running properly with the status command.

$ sudo systemctl status prometheus

If the service’s status is not set to active, follow the on-screen directions and re-trace your previous steps before moving on.

Output

prometheus.service - Prometheus
   Loaded: loaded (/etc/systemd/system/prometheus.service; disabled; vendor preset: enabled)
   Active: active (running) since Fri 2017-07-21 11:46:39 UTC; 6s ago
 Main PID: 2219 (prometheus)
    Tasks: 6
   Memory: 19.9M
      CPU: 433ms
   CGroup: /system.slice/prometheus.service

We currently have Prometheus and Node Exporter installed, configured, and running. As a final precaution before connecting to the online interface, we’ll enhance our installation’s security with basic protocol HTTP authentication to confirm that unauthorized users cannot access our metrics.

Step 8 — Securing Prometheus

Prometheus doesn’t embrace in-built authentication or the other general purpose security mechanism. On the one hand, this suggests you are obtaining an extremely versatile system with fewer configuration restraints; on the opposite hand, it means that it’s up to you to confirm that your metrics and overall setup square measure sufficiently secure.

For simplicity’s sake, we’ll use Nginx to feature basic HTTP protocol authentication to our installation, that each Prometheus and its most popular data visualization tool, Grafana, totally support.

Start by putting in apache2-utils, which is able to offer you access to the htpasswd utility for generating password files.

$ sudo apt-get update
$ sudo apt-get install apache2-utils

Now, generate a password file by telling htpasswd wherever you wish to store the file and that username you want to use for authentication.

Note: htpasswd can prompt you to enter and re-confirm the password you want to go together with this user. Also, build note of each the username and password you enter here, as you will need them to log into Prometheus in Step 9.

The results of this command could be a newly-created file known as .htpasswd, situated within the /etc/nginx directory, containing the username and a hashed version of the password you entered.

Next, set up Nginx to use the newly-created passwords.

First, build a Prometheus-specific copy of the default Nginx configuration file in order that you’ll revert back to the defaults later if you run into a haul.

$ sudo cp /etc/nginx/sites-available/default /etc/nginx/sites-available/prometheus

Then, open the new configuration file.

$ sudo nano /etc/nginx/sites-available/prometheus

Locate the location/ block below the server block. It must look like:

...
    location / {
        try_files $uri $uri/ =404;
    }
...

As we are going to be forwarding all traffic to Prometheus, replace the try_files directive with the subsequent content:

...
    location / {
        auth_basic "Prometheus server authentication";
        auth_basic_user_file /etc/nginx/.htpasswd;
        proxy_pass http://localhost:9090;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection 'upgrade';
        proxy_set_header Host $host;
        proxy_cache_bypass $http_upgrade;
    }
...

These settings make sure that users can demonstrate at the beginning of every new session. In addition, the reverse proxy can direct all requests handled by this block to Prometheus.

When you are finished creating changes, save the file and shut your text editor.

Now, deactivate the default Nginx configuration file by removing the link thereto within the /etc/nginx/sites-enabled directory, and activate the new configuration file by generating a link to it.

$ sudo rm /etc/nginx/sites-enabled/default
$ sudo ln -s /etc/nginx/sites-available/prometheus /etc/nginx/sites-enabled/

Before restarting Nginx, check the configuration for errors victimization the subsequent command:

$ sudo nginx -t

The output must indicate that the syntax is ok and also the test is successful. If you receive an error message, follow the on-screen directions to repair the matter before continuing to the following step.

An output of Nginx configuration tests

nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful

Then, reload Nginx to include all of the changes.

$ sudo systemctl reload nginx

Verify that Nginx is up and running.

$ sudo systemctl status nginx

If your output does not indicate that the service’s status is active, follow the on-screen messages and re-trace the preceding steps to resolve the difficulty before continued.

● nginx.service - A high performance web server and a reverse proxy server
   Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: en
   Active: active (running) since Mon 2017-07-31 21:20:57 UTC; 12min ago
  Process: 4302 ExecReload=/usr/sbin/nginx -g daemon on; master_process on; -s r
 Main PID: 3053 (nginx)
    Tasks: 2
   Memory: 3.6M
      CPU: 56ms
   CGroup: /system.slice/nginx.service

At this time, we have a fully-functional and secured Prometheus server. Therefore we will log into the web interface to start observing metrics.

Step 9 — Testing Prometheus

Prometheus provides a basic web interface for observance the status of itself and its exporters, queries, and generating graphs. But, because of the interface’s simplicity, the Prometheus team recommends to install and use Grafana for all the difficult than testing and debugging.

In this tutorial, we’ll use the inbuilt web interface to confirm that Prometheus and Node Exporter are up and running, and we’ll conjointly take a glance at straightforward queries and graphs.

To begin, purpose your web browser to http://your_server_ip.

In the HTTP protocol authentication dialogue box, enter the username and password you selected in Step 8.

Once logged in, you will see the Expression Browser, wherever you’ll execute and visualize custom queries.

Before performing any expressions, verify the status of each Prometheus and Node Explorer by clicking initial on the status menu at the highest of the screen so on the Targets menu possibility. As we’ve organized Prometheus to scrape each itself and Node Exporter, you must see each target listed within the UP state.

If either Exporter is missing or displays an error message, check the service’s status with the subsequent commands:

$ sudo systemctl status prometheus
$ sudo systemctl status node_exporter

The output for each service must report a status of Active: active (running). If a service either is not active in the least or is active however still not operating properly. Follow the on-screen directions and re-trace the previous steps before continuing.

Next, to form certain that the exporters are operating properly, we’ll execute some expressions against Node Exporter.

First, click on the Graph menu at the highest of the screen to come back to the Expression Browser.

In the Expression field, type node_memory_MemAvailable and press the Execute button to update the Console tab with the quantity of memory your server has.

By default, Node Exporter reports this quantity in bytes. To convert to megabytes, we’ll use math operators to divide by 1024 double time.

In the Expression field, enter node_memory_MemAvailable/1024/1024 so press the Execute button.

The Console tab can currently show the ends up in megabytes.

If you wish to verify the results, execute the free command from your terminal. (The -h flag tells free to report back in an exceedingly human-readable format, giving a quantity in megabytes.)

$ free -h

This output contains details regarding memory usage, as well as on the available memory displayed within the on the available column.

Output

              total        used        free      shared  buff/cache   available
Mem:           488M        144M         17M        3.7M        326M        324M
Swap:            0B          0B          0B

In addition to basic operators, the Prometheus query language conjointly provides several functions for aggregating results.

In the Expression field, type avg_over_time(node_memory_MemAvailable[5m])/1024/1024 and click on on the Execute button. The result is going to be common on the available memory over the last five minutes in megabytes.

Now, click on the Graph tab to show the executed expression as a graph rather than as text.

Finally, whereas still on this tab, hover your mouse over the graph for extra details regarding any specific purpose on the graph’s X and Y axes.

If you want to find out a lot of regarding making expressions in Prometheus’ intrinsic web interface, see the Querying Prometheus portion of the official documentation.

Conclusion

• In this tutorial we have a tendency to downloaded, configured, secured, and tested a whole Prometheus installation with one further Exporter.
• If you want to find out a lot of regarding, however, Prometheus works below the hood, take a glance at How to Query Prometheus on Ubuntu fourteen.04. (Since you have already got Prometheus put in, you’ll skip the primary step.)
• To see what else Prometheus will do, visit the official Prometheus documentation.
• And, to learn more about extending Prometheus, check out the list of available exporters as well as the official Grafana website.