Introduction #

Thanos is a monitoring solution built on top of Prometheus that adds centralized querying and long‑term storage capabilities to Prometheus. With Thanos, you can sync metrics collected by one or more Prometheus instances to object storage (such as S3‑compatible object storage), and then run global PromQL queries by connecting to that object storage. Thanos consists of multiple microservice components; here we will use Thanos Sidecar, Thanos Store, and Thanos Query:

Thanos Sidecar: Runs on the same host as Prometheus, connects to object storage, and automatically uploads each TSDB block generated by Prometheus to object storage.
Thanos Store: Connects to object storage and serves queries against the data stored there.
Thanos Query: Aggregates multiple Thanos components as data sources and executes queries over the combined data.
Thanos Query Frontend: Improves Query performance.
Thanos Compactor: Compacts and downsamples data in object storage.
Thanos Ruler: Manages and evaluates alerting rules.
Thanos Receive: Receives data from Prometheus Remote Write.

Each Thanos component exposes both a PromQL HTTP API and a Thanos StoreAPI gRPC interface. This means you can either cascade Thanos components via Thanos Query, or connect any individual Thanos component instance directly to Grafana.

Deployment#

[Note] Starting from the release-0.11 version, NetworkPolicy manifests were added and by default only allow intra‑component access. If you understand NetworkPolicy, you can adjust the default rules (see ls manifests/*networkPolicy*). If you leave them as is, changing Services to NodePort will still not make them reachable. If you are not familiar with NetworkPolicy, you can simply delete those manifests.

Change image registry mirror#

1
# Search
2
grep -rn 'quay.io' *
3
# Batch replace
4
sed -i 's/quay.io/quay.mirrors.ustc.edu.cn/g' `grep "quay.io" -rl *`
5
# Verify
6
grep -rn 'quay.io' *
7
grep -rn 'image: ' *

Modify the Prometheus Service#

1
# Expose it via NodePort by adding the following two lines; full config shown below.
2
# type: NodePort
3
# nodePort: 30090
4

5
vi manifests/prometheus-service.yaml
6

7
apiVersion: v1
8
kind: Service
9
metadata:
10
  labels:
11
    app.kubernetes.io/component: prometheus
12
    app.kubernetes.io/instance: k8s
13
    app.kubernetes.io/name: prometheus
14
    app.kubernetes.io/part-of: kube-prometheus
15
    app.kubernetes.io/version: 2.53.0
16
  name: prometheus-k8s
17
  namespace: monitoring
18
spec:
19
  type: NodePort
20
  ports:
21
  - name: web
22
    port: 9090
23
    targetPort: web
24
    nodePort: 30090
25
  - name: reloader-web
26
    port: 8080
27
    targetPort: reloader-web
28
  selector:
29
    app.kubernetes.io/component: prometheus
30
    app.kubernetes.io/instance: k8s
31
    app.kubernetes.io/name: prometheus
32
    app.kubernetes.io/part-of: kube-prometheus
33
  sessionAffinity: ClientIP

Modify the Grafana Service#

1
# Expose it via NodePort by adding the following two lines; full config shown below.
2
# type: NodePort
3
# nodePort: 30300
4
vi manifests/grafana-service.yaml
5

6
apiVersion: v1
7
kind: Service
8
metadata:
9
  labels:
10
    app.kubernetes.io/component: grafana
11
    app.kubernetes.io/name: grafana
12
    app.kubernetes.io/part-of: kube-prometheus
13
    app.kubernetes.io/version: 11.1.0
14
  name: grafana
15
  namespace: monitoring
16
spec:
17
  type: NodePort
18
  ports:
19
  - name: http
20
    port: 3000
21
    targetPort: http
22
    nodePort: 30300
23
  selector:
24
    app.kubernetes.io/component: grafana
25
    app.kubernetes.io/name: grafana
26
    app.kubernetes.io/part-of: kube-prometheus

Modify the Alertmanager Service#

1
apiVersion: v1
2
kind: Service
3
metadata:
4
  labels:
5
    app.kubernetes.io/component: alert-router
6
    app.kubernetes.io/instance: main
7
    app.kubernetes.io/name: alertmanager
8
    app.kubernetes.io/part-of: kube-prometheus
9
    app.kubernetes.io/version: 0.27.0
10
  name: alertmanager-main
11
  namespace: monitoring
12
spec:
13
  type: NodePort
14
  ports:
15
  - name: web
16
    port: 9093
17
    targetPort: web
18
    nodePort: 30093
19
  - name: reloader-web
20
    port: 8080
21
    targetPort: reloader-web
22
  selector:
23
    app.kubernetes.io/component: alert-router
24
    app.kubernetes.io/instance: main
25
    app.kubernetes.io/name: alertmanager
26
    app.kubernetes.io/part-of: kube-prometheus
27
  sessionAffinity: ClientIP

To reduce costs, we use a self‑hosted MinIO instance as object storage instead of S3/OSS services. The MinIO deployment itself is not covered here.

1
git clone https://github.com/Yuri-NagaSaki/Prometheus-Operator-Thanos.git
2
cd Prometheus-Operator-Thanos

Create CRDs#

1
kubectl create -f setup/

Create all base components#

1
kubectl apply -f .

After creation, the following new Kinds will be available:

Alertmanager
PodMonitor
Probe
Prometheus
PrometheusRule
ServiceMonitor
ThanosRuler

Update Prometheus configuration#

1
kubectl create -f thanos/

Create Thanos modules (choose the Thanos mode you want to use)#

1
kubectl create -f thanos/thanos-sidecar
2
kubectl create -f thanos/thanos-receive

Create rules as needed#

1
kubectl create -f thanos/rules

Notes#

reading meta file failed, will override it” err=”failed to read /prometheus/thanos.shipper.json: open /prometheus/thanos.shipper.json: no such file or directory”

Solution:
When using the Thanos Sidecar mode, the Sidecar reads the thanos.shipper.json file under the Prometheus data directory. This file is used to track what has been uploaded to object storage; the Sidecar updates it whenever it uploads data.
The error above usually occurs because thanos.shipper.json is owned by root, so the Sidecar does not have sufficient permissions to read or write it. Since the Sidecar itself does not retry reading, simply chown‑ing the file inside the container will not help. You need to persist Prometheus’s data directory, grant 777 permissions to this file on the host, and then restart the entire Prometheus Pod.

Using Thanos to Manage Multiple Prometheus Data Sources for High Availability