Monitoring standalone ESXI hosts using Prometheus and Grafana

In this article, I will explain how to monitor the ESXI hosts using Prometheus and displaying the metrics on Grafana. ESXI hosts metrics are monitored by Prometheus-VMware-Exporter [Git] and it’s pulled by Prometheus server and finally Grafana is used for visualization.

The architecture looks like below,

All the services (Prometheus server, Prometheus-VMware-Exporter and Grafana) are running in the Docker containers that are hosted in a Single VM that is placed in one of the ESXI hosts. To monitor each ESXI host a separate Prometheus-VMware-Exporter container should be spawned. Let’s dive deep into the process.

The VM is installed with OS Centos-8 and latest Docker version. To start with,

1. Install latest Docker version in Centos 8.
   
2. If you encounter the below error, then issue the command as shown beneath the error diagram. Then continue with the installation of Docker.
   
   
3. Start and enable the Docker services as given below
   
4. Create following folders to place configuration files
   exporter -> Used to store the files related to Prometheus-Exporter downloaded from Git
   grafana -> Used to store grafana configuration file
   Prometheus -> Used to store the files related to Prometheus server downloaded from Git
   Prometheus_vol -> To map as a volume to Prometheus Docker Server
   
5. Change the directory into “exporter” and download the files from following git repo
   
6. Add the following line to /etc/docker/daemon.json, if the file exists or else create a new file and the entry as shown below. Purpose is to make Docker daemon to utilize DNS server when accessing links inside a Docker container. If not error might be pop up during Docker build saying github.com is inaccessible
   
7. Then change the directory into subfolder and Build the Docker file of “Prometheus-VMware-Exporter”
   
   
8. Create and Run the “Prometheus-VMware-Exporter” for each ESXI hosts to be monitored. Following parameters needs to be passed as a part of the command,
   –name -> Name of the container
   –ESX_HOST -> ESXI IP address
   –ESX_USERNAME -> Username to connect to ESXI
   –ESX_PASSWORD -> Password to connect to ESXI
   –ESX_LOG -> Logging level
   –ESX_LISTEN -> Port of ESXI to listen [9512]
   Command can be found here
   
   
9. You can able to view the scraped parameters by accessing <Docker IP>:9512/metrics
   
10. Create the Prometheus configuration file as here under the folder Prometheus_vol. Name the file as prometheus.yaml [ Change the IP address and port as required]
11. Create and run the Prometheus server using the below command
    -p -> To map the localhost port to container port
    -d -> Run the container in detached mode
    -v -> To map the Prometheus configuration file into the container
    –config.file-> Prometheus configuration file location inside the container
    Command can be found here
    
12. Access the Prometheus server in the following location
    http://localhost:9090
    
13. Then in that page, Click on “Status” [Top bar] -> Targets. You should see all 3 ESXI Hosts and local server status as up
    
    
14. Now its time to configure the Grafana. Run the below command to run the Grafana server as a Docker application. The container ran as “user 1000”, if not the container will fail to startup because of permission issue (Owning the container as same user to start the container) or else instead own the volume as “grafana” user whose id is “472”
    command can be found here
    
15. Access the Grafana portal on http://localhost:<port> [http://localhost:3050]. To login use, username as “admin” and password as “HelloWorld” as configured in the previous command. This is depending on how you executed the previous command
    
16. Click on “Add data source” -> Select Prometheus -> Provide the values as shown below in the diagram. Make note that for “URL” you need to give the path of your Prometheus server which is itself a container running on the same host [To view the IP address of your containers, you can use this script or else issue the command “docker inspect <container name>]. Finally click on “Save and Test”. It should return a message saying “Data source is working”
    
    
    
17. I have used pre-existing Grafana template to display the collected metrics from Prometheus. It is available here. You can either download the template or copy the json contents from it or else just copy the link only
18. Click on “+” icon and select the option import as shown below
    
19. Click on “Upload .json file” option if you want to upload the downloaded JSON file, or else just paste the URL link under “Grafana.com Dashboard”, or else paste the contents directly under “paste JSON” and click on Load
    
20. For datasource option, select the name of the Prometheus datasource that you have created in the previous step. Then select Import
    
21. Here we go, the Grafana dashboard will be displayed like below,
    

Git Repo