Apple MAC, Programming

Setting Up Grafana and Prometheus on MacBook Pro (Intel Version) via Docker and Kubernetes

This guide will walk you through setting up Grafana and Prometheus on a MacBook Pro (Intel version) using Docker Desktop’s Kubernetes service. You’ll learn how to monitor your system metrics and set up persistent data storage for Grafana and Prometheus.

Prerequisites

  • MacBook Pro 2019 (Intel version)
  • Docker Desktop installed with Kubernetes enabled
  • Basic knowledge of Kubernetes and command-line operations
  • tmux (optional but recommended for managing multiple terminal sessions)
  • stern (for log monitoring)
  • yq (for YAML processing)

Pre-Steps for All Methods

1. Start Docker Desktop and Kubernetes Service

Ensure Docker Desktop is running and Kubernetes is enabled:

kubectl get nodes

Expected output:

NAME             STATUS   ROLES           AGE     VERSION
docker-desktop   Ready    control-plane   3h28m   v1.30.2

Check for any existing pods:

kubectl get pods

Expected output:

No resources found in default namespace.

2. Create Necessary Directories

Set up your working directories:

mkdir -p ~/tmp/mac-k8s-monitoring
cd ~/tmp/mac-k8s-monitoring

3. Set Up Node Exporter

In a separate terminal, download and run the Node Exporter to gather system metrics:

Download Node Exporter

For Intel Macs:

wget https://github.com/prometheus/node_exporter/releases/download/v1.8.2/node_exporter-1.8.2.darwin-amd64.tar.gz
tar xf node_exporter-1.8.2.darwin-amd64.tar.gz

For ARM MACs (M series): Download the darwin-arm64 version. This would be the only difference in the steps as long as the other prereqs work.

Run Node Exporter

./node_exporter-1.8.2.darwin-amd64/node_exporter

You should see output similar to:

ts=2024-10-27T02:30:14.522Z caller=tls_config.go:313 level=info msg="Listening on" address=[::]:9100
ts=2024-10-27T02:30:14.522Z caller=tls_config.go:316 level=info msg="TLS is disabled." http2=false address=[::]:9100

4. Obtain Your Mac’s IP Address

You’ll need your Mac’s IP address for configuration:

ifconfig | grep inet

Look for the IP address under your main network interface. For example:

inet 192.168.86.222 netmask 0xffffff00 broadcast 192.168.86.255

This IP (192.168.86.222 in this example) will be used later in the configuration files.

Method 1: Non-Persistent Method

This method sets up Grafana and Prometheus without persistent storage, suitable for testing purposes.

1. Create the Kubernetes Manifest

Create a file named grafana-prometheus.yml and replace your Mac’s IP Address in place of mine (192.168.86.222):

cat > grafana-prometheus.yml << 'EOF'
apiVersion: v1
kind: Namespace
metadata:
  name: monitoring
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: prometheus-config
  namespace: monitoring
data:
  prometheus.yml: |
    global:
      scrape_interval: 15s
    scrape_configs:
      - job_name: 'prometheus'
        static_configs:
          - targets: ['localhost:9090']
      - job_name: 'host-metrics'
        static_configs:
          - targets: ['YOUR_MAC_IP:9100']  # Replace with your Mac's IP address
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: prometheus
  namespace: monitoring
spec:
  replicas: 1
  selector:
    matchLabels:
      app: prometheus
  template:
    metadata:
      labels:
        app: prometheus
    spec:
      containers:
      - name: prometheus
        image: prom/prometheus:latest
        args:
          - "--config.file=/etc/prometheus/prometheus.yml"
        ports:
          - containerPort: 9090
        volumeMounts:
          - name: config-volume
            mountPath: /etc/prometheus/
          - name: proc
            mountPath: /host/proc
            readOnly: true
          - name: sys
            mountPath: /host/sys
            readOnly: true
      volumes:
        - name: config-volume
          configMap:
            name: prometheus-config
        - name: proc
          hostPath:
            path: /proc
        - name: sys
          hostPath:
            path: /sys
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: grafana
  namespace: monitoring
spec:
  replicas: 1
  selector:
    matchLabels:
      app: grafana
  template:
    metadata:
      labels:
        app: grafana
    spec:
      containers:
      - name: grafana
        image: grafana/grafana:latest
        ports:
          - containerPort: 3000
        env:
          - name: GF_SECURITY_ADMIN_PASSWORD
            value: "foobar"
---
apiVersion: v1
kind: Service
metadata:
  name: prometheus
  namespace: monitoring
spec:
  type: NodePort
  ports:
    - port: 9090
      targetPort: 9090
      nodePort: 30001
      protocol: TCP
  selector:
    app: prometheus
---
apiVersion: v1
kind: Service
metadata:
  name: grafana
  namespace: monitoring
spec:
  type: NodePort
  ports:
    - port: 3000
      targetPort: 3000
      nodePort: 30002
      protocol: TCP
  selector:
    app: grafana
EOF

2. Apply the Kubernetes Manifest

Deploy Grafana and Prometheus:

kubectl apply -f grafana-prometheus.yml

3. Monitor Deployment Status

Check the status of the pods and services:

kubectl get all -n monitoring

Expected output:

NAME                             READY   STATUS    RESTARTS   AGE
pod/grafana-xxxxxxxxx-xxxxx      1/1     Running   0          XXm
pod/prometheus-xxxxxxxxx-xxxxx   1/1     Running   0          XXm

NAME                 TYPE       CLUSTER-IP       EXTERNAL-IP   PORT(S)          AGE
service/grafana      NodePort   10.xxx.xxx.xxx   <none>        3000:30002/TCP   XXm
service/prometheus   NodePort   10.xxx.xxx.xxx   <none>        9090:30001/TCP   XXm

4. Access Grafana and Prometheus

Grafana

Open your browser and navigate to http://localhost:30002/.

  • Login Credentials:
    • Username: admin
    • Password: foobar

You’ll be prompted to change the password upon first login.

Prometheus

Open your browser and navigate to http://localhost:30001/.

To check the targets, go to http://localhost:30001/targets. You should see all targets as “Up”.

5. Monitor Logs with Stern (Optional)

Install stern if you haven’t already:

brew install stern

Use stern to monitor logs:

stern -n monitoring .

You can filter logs for specific components:

stern -n monitoring prom
stern -n monitoring graf

6. Adjusting Configurations

If you need to update configurations (e.g., the IP address):

  • Edit grafana-prometheus.yml.
  • Reapply the manifest: 
    kubectl apply -f grafana-prometheus.yml
  • Restart Prometheus to load new configurations: 
    kubectl get pod -n monitoring      # get your PROMETHEUS_POD_NAME
kubectl exec <pod/PROMETHEUS_POD_NAME> -n monitoring -- kill -1 1

    Or use the one-liner to find the pod name automatically:

    # naive with kubectl, grep and awk
kubectl exec `kubectl get pods -n monitoring | grep prometheus | awk '{print $1}'` -n monitoring -- kill -1 1

# precise with only kubectl
kubectl exec $(kubectl get pods -n monitoring -l app=prometheus -o jsonpath="{.items[0].metadata.name}") -n monitoring -- kill -1 1

7. Set Up Grafana Dashboard

Import Dashboard

  • In Grafana, click on Dashboards > Import.
  • Enter dashboard ID 1860 (Node Exporter Full).
  • Select your Prometheus data source.
  • Click Import.

Note: Explore all of grafana public dashboards online. The numerical ID of the dashboard is the only requirement.

Method 2: Persistent Data for Prometheus and Grafana

This method sets up persistent storage for Grafana and Prometheus, ensuring data and configurations are retained across restarts.

1. Clean Up Previous Deployments

If you have previous deployments:

kubectl delete ns monitoring

Optionally, remove old storage directories:

rm -rf ~/tmp/k8s-mon/grafana-storage ~/tmp/k8s-mon/prometheus-storage

2. Create Persistent Storage Directories

Set up directories for persistent storage:

mkdir -p ~/tmp/k8s-mon/grafana-storage
mkdir -p ~/tmp/k8s-mon/prometheus-storage

Note: Keep these directories outside of synced folders (like Dropbox) to avoid file locking issues. While the documented locations should be fine, be cautious if choosing alternative directory locations.

3. Create the Kubernetes Manifest with HostPath Volumes

Create a file named grafana-prometheus-hostpath.yml:

cat > grafana-prometheus-hostpath.yml << 'EOF'
apiVersion: v1
kind: Namespace
metadata:
  name: monitoring
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: prometheus-config
  namespace: monitoring
data:
  prometheus.yml: |
    global:
      scrape_interval: 15s
    scrape_configs:
      - job_name: 'prometheus'
        static_configs:
          - targets: ['localhost:9090']
      - job_name: 'host-metrics'
        static_configs:
          - targets: ['YOUR_MAC_IP:9100']  # Replace with your Mac's IP address
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: prometheus
  namespace: monitoring
spec:
  replicas: 1
  selector:
    matchLabels:
      app: prometheus
  template:
    metadata:
      labels:
        app: prometheus
    spec:
      containers:
      - name: prometheus
        image: prom/prometheus:latest
        args:
          - "--config.file=/etc/prometheus/prometheus.yml"
          - "--storage.tsdb.path=/prometheus"  # Directory within the container for data storage
        ports:
          - containerPort: 9090
        volumeMounts:
          - name: config-volume
            mountPath: /etc/prometheus/
          - name: prometheus-storage  # Persistent storage mount
            mountPath: /prometheus
          - name: proc
            mountPath: /host/proc
            readOnly: true
          - name: sys
            mountPath: /host/sys
            readOnly: true
      volumes:
        - name: config-volume
          configMap:
            name: prometheus-config
        - name: prometheus-storage
          hostPath:
            path: /Users/USERNAME/tmp/k8s-mon/prometheus-storage  # Replace USERNAME with your actual username
        - name: proc
          hostPath:
            path: /proc
        - name: sys
          hostPath:
            path: /sys
---
apiVersion: v1
kind: Service
metadata:
  name: prometheus
  namespace: monitoring
spec:
  type: NodePort
  ports:
    - port: 9090
      targetPort: 9090
      nodePort: 30001
      protocol: TCP
  selector:
    app: prometheus
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: grafana
  namespace: monitoring
spec:
  replicas: 1
  selector:
    matchLabels:
      app: grafana
  template:
    metadata:
      labels:
        app: grafana
    spec:
      containers:
      - name: grafana
        image: grafana/grafana:latest
        ports:
          - containerPort: 3000
        env:
          - name: GF_SECURITY_ADMIN_PASSWORD
            value: "foobar"  # Set a secure password for production
        volumeMounts:
          - name: grafana-storage
            mountPath: /var/lib/grafana  # Grafana data will be stored here
      volumes:
        - name: grafana-storage
          hostPath:
            path: /Users/USERNAME/tmp/k8s-mon/grafana-storage  # Replace USERNAME with your actual username
---
apiVersion: v1
kind: Service
metadata:
  name: grafana
  namespace: monitoring
spec:
  type: NodePort
  ports:
    - port: 3000
      targetPort: 3000
      nodePort: 30002
      protocol: TCP
  selector:
    app: grafana
EOF

Important:

  • Replace YOUR_MAC_IP with your actual Mac IP address.
  • Replace USERNAME with your actual macOS username in the hostPath entries.

4. Apply the Kubernetes Manifest

Deploy Grafana and Prometheus with persistent storage:

kubectl apply -f grafana-prometheus-hostpath.yml

5. Configure Grafana’s Prometheus Data Source via REST API

Copy & paste the subshell below into your terminal to configure the data source:

(
  # Variables
  PROMETHEUS_URL=http://prometheus.monitoring.svc.cluster.local:9090/
  GRAFANA_URL=http://localhost:30002
  GRAFANA_USERNAME=admin
  GRAFANA_PASSWORD=foobar

  # Create a Prometheus data source in Grafana
  curl -XPOST -u $GRAFANA_USERNAME:$GRAFANA_PASSWORD 
    -H 'Content-Type: application/json' 
    -H 'Accept: application/json' 
    $GRAFANA_URL/api/datasources 
    -d "{"name": "prometheus", "type": "prometheus", "url": "$PROMETHEUS_URL", "access": "proxy", "basicAuth": false}"
  echo
)

6. Verify Data Source Creation

Check if the data source has been created by inspecting Grafana’s SQLite database:

sqlite3 ~/tmp/k8s-mon/grafana-storage/grafana.db 'select * from data_source;'

Expected output:

1|1|2|prometheus|prometheus|proxy|http://prometheus.monitoring.svc.cluster.local:9090/||||0|||1|{"httpMethod":"POST"}|2024-10-27 03:23:03|2024-10-27 03:23:24|0|{}|0|ce23k4ra97mrkb|0|

7. Set Up Grafana Dashboard

Import Dashboard

  • In Grafana, click on Dashboards > Import.
  • Enter dashboard ID 1860 (Node Exporter Full).
  • Select the prometheus data source.
  • Click Import.

Note: Explore all of grafana public dashboards online. The numerical ID of the dashboard is the only requirement.

Testing Deletion and Recreation

To ensure that persistent storage is working, you can delete and recreate the namespace:

kubectl delete ns monitoring
kubectl apply -f grafana-prometheus-hostpath.yml

Your dashboards and data sources should persist across redeployments due to the persistent volumes.

Tips and Additional Notes

Monitoring Logs with Stern

Install stern for log monitoring:

brew install stern

Use stern to monitor logs:

stern -n monitoring .

Using tmux for Session Management

tmux can help you manage multiple terminal sessions efficiently. You can have separate panes for:

  • Running stern for log monitoring
  • Running node_exporter
  • Editing YAML files
  • Executing bash commands (kubectl, etc.)

For more information on running tmux, check out my TMUX cheatsheet article.

Checking YAML Files with yq

Before applying Kubernetes manifests, validate and view them with yq:

brew install yq
yq grafana-prometheus-hostpath.yml

This command outputs syntax-highlighted YAML and can help catch any syntax errors.

Conclusion

By following this guide, you’ve successfully set up Grafana and Prometheus on your MacBook Pro (Intel version) using Docker Desktop and Kubernetes. You now have a monitoring stack capable of collecting and visualizing system metrics with persistent data storage.

Feel free to explore more dashboards and customize your monitoring setup according to your needs. Happy monitoring!

Leave a Reply

Your email address will not be published. Required fields are marked *