This commit is contained in:
2026-04-06 21:36:15 +07:00
commit 35874bf520
29 changed files with 1625 additions and 0 deletions

2
.gitignore vendored Normal file
View File

@@ -0,0 +1,2 @@
.claude
.playwright-mcp

318
README.md Normal file
View File

@@ -0,0 +1,318 @@
# Loki 3 + Vector + Grafana — Test Stand
Test environment for centralized log collection using Grafana Loki in Simple Scalable mode with Vector as the log collector and MinIO as S3-compatible object storage.
Two deployment options: **Docker Compose** (local testing with log-generator) and **Helm chart** (Kubernetes with DaemonSet log collection from all pods).
## Architecture
![Architecture Diagram](docs/architecture.excalidraw.png)
## Components
| Service | Image | Role |
|---|---|---|
| **minio** | `minio/minio:latest` | S3-compatible object storage for chunks and index |
| **loki-write** | `grafana/loki:3.4.2` | Ingester — accepts and stores log streams |
| **loki-read** | `grafana/loki:3.4.2` | Querier — executes LogQL queries |
| **loki-backend** | `grafana/loki:3.4.2` | Compactor, ruler, index gateway |
| **gateway** | `nginx:1.27-alpine` | Reverse proxy — Loki write/read/ruler + Grafana UI |
| **vector** | `timberio/vector:0.44.0-alpine` | Log collector — Loki sink |
| **grafana** | `grafana/grafana:12.4` | Visualization — Loki datasource pre-provisioned |
## Data Flow
1. **Vector** collects container logs (docker_logs / kubernetes_logs source)
2. Vector transforms logs (extracts metadata, removes raw labels) and pushes to **gateway**
3. **Gateway** (nginx) routes `/loki/api/v1/push` to **loki-write**, all other `/loki/api/*` to **loki-read**
4. **loki-write** ingests logs and stores chunks in **MinIO** (`loki-chunks` bucket)
5. **loki-backend** runs compactor on a 10-minute interval
6. **Grafana** queries logs via gateway → loki-read
## Loki Configuration
- **Deployment mode**: Simple Scalable (write / read / backend targets)
- **Schema**: v13 with TSDB store
- **Storage**: S3 (MinIO) — bucket `loki-chunks`
- **Service discovery**: memberlist (port 7946)
- **Replication factor**: 1 (test stand)
- **Chunk encoding**: snappy
- **Retention**: enabled via compactor, old samples rejected after 168h
---
## Option 1: Docker Compose
Local test stand with a built-in log-generator container (`flog`). Vector collects logs from containers with label `vector.collect=true` via Docker socket.
### Quick Start
```bash
docker compose up -d
```
Wait for all services to become healthy:
```bash
docker compose ps
```
### Access Points
| Service | URL | Credentials |
|---|---|---|
| Grafana (via gateway) | http://localhost:3000 | admin / admin (anonymous access enabled) |
| Loki API (via gateway) | http://localhost:3100 | — |
| MinIO Console | http://localhost:9001 | loki / supersecret |
| MinIO API | http://localhost:9000 | loki / supersecret |
### Network
All containers run on a static network `192.168.97.0/24`:
| Container | IP |
|---|---|
| `loki-minio` | 192.168.97.10 |
| `loki-write` | 192.168.97.11 |
| `loki-read` | 192.168.97.12 |
| `loki-backend` | 192.168.97.13 |
| `loki-gateway` | 192.168.97.14 |
| `loki-vector` | 192.168.97.15 |
| `loki-grafana` | 192.168.97.16 |
| `loki-log-generator` | 192.168.97.17 |
### Adding Your Own Containers
To collect logs from any container, add the label:
```yaml
services:
my-app:
image: my-app:latest
labels:
vector.collect: "true"
```
Logs will appear in Grafana with label `{container="my-app"}`.
### Useful Commands
```bash
# Start
docker compose up -d
# Stop
docker compose down
# Stop and remove volumes
docker compose down -v
# View Vector logs
docker compose logs vector -f
# View Loki write logs
docker compose logs loki-write -f
# Check Loki readiness
curl -s http://localhost:3100/ready
# Query logs via API
curl -s http://localhost:3100/loki/api/v1/query_range \
--data-urlencode 'query={container="log-generator"}' \
--data-urlencode 'limit=10' | jq .
# Check MinIO bucket
docker compose exec minio mc ls local/loki-chunks/
```
---
## Option 2: Helm Chart (Kubernetes)
Production-oriented deployment. Vector runs as a **DaemonSet** collecting logs from all pods in the cluster via `kubernetes_logs` source. No log-generator — real workload logs are collected.
### Prerequisites
- Kubernetes cluster (1.24+)
- Helm 3
- `kubectl` configured
### Install
```bash
helm install loki-stack ./helm/loki-stack -n loki --create-namespace
```
### Install with custom values
```bash
helm install loki-stack ./helm/loki-stack -n loki --create-namespace \
-f my-values.yaml
```
### Upgrade
```bash
helm upgrade loki-stack ./helm/loki-stack -n loki
```
### Uninstall
```bash
helm uninstall loki-stack -n loki
```
### Key Differences from Docker Compose
| Feature | Docker Compose | Helm |
|---|---|---|
| Vector source | `docker_logs` (label filter) | `kubernetes_logs` (all pods) |
| Vector mode | container | DaemonSet (one per node) |
| Log labels | `container`, `project` | `namespace`, `pod`, `container`, `node` |
| Loki targets | Deployment | StatefulSet with PVC |
| Service discovery | static IPs + memberlist | headless Service + memberlist |
| Gateway (nginx) | yes — proxies Loki + Grafana | no — direct Service routing |
| Ingress | — | Grafana ingress (disabled by default) |
### Configuration (values.yaml)
Key parameters:
```yaml
# Scale Loki components
loki:
write:
replicas: 1 # Increase for HA
read:
replicas: 1 # Increase for query throughput
backend:
replicas: 1
# MinIO storage
minio:
rootUser: loki
rootPassword: supersecret
storage:
size: 10Gi
# Vector tolerations (to run on tainted nodes)
vector:
tolerations: []
# Grafana
grafana:
adminUser: admin
adminPassword: admin
anonymousAccess: true
service:
type: ClusterIP
ingress:
enabled: false
className: nginx
hosts:
- host: grafana.example.com
paths:
- path: /
pathType: Prefix
tls: []
```
### Access in Kubernetes
Port-forward to services:
```bash
# Grafana
kubectl port-forward -n loki svc/loki-stack-grafana 3000:3000
# Loki API (read)
kubectl port-forward -n loki svc/loki-stack-loki-read 3100:3100
# Loki API (write / push)
kubectl port-forward -n loki svc/loki-stack-loki-write 3100:3100
# MinIO Console
kubectl port-forward -n loki svc/loki-stack-minio 9001:9001
```
### Viewing Logs
Open Grafana and query:
```logql
# All logs from a namespace
{namespace="default"}
# Specific pod
{pod="my-app-7d4b8c6f5-x2k9p"}
# By container name across all namespaces
{container="nginx"}
# By node
{node="worker-01"}
```
### Helm Chart Structure
```
helm/loki-stack/
├── Chart.yaml
├── values.yaml
└── templates/
├── _helpers.tpl
├── minio-deployment.yaml
├── minio-pvc.yaml
├── minio-service.yaml
├── loki-configmap.yaml
├── loki-write-statefulset.yaml
├── loki-read-statefulset.yaml
├── loki-backend-statefulset.yaml
├── loki-services.yaml
├── vector-rbac.yaml
├── vector-configmap.yaml
├── vector-daemonset.yaml
├── grafana-configmap.yaml
├── grafana-deployment.yaml
├── grafana-ingress.yaml
├── grafana-pvc.yaml
└── grafana-service.yaml
```
### Kubernetes Resources Created
| Kind | Count | Description |
|---|---|---|
| StatefulSet | 3 | loki-write, loki-read, loki-backend |
| Deployment | 2 | minio, grafana |
| DaemonSet | 1 | vector (one per node) |
| Service | 9 | ClusterIP + headless for each Loki target + memberlist + minio + grafana |
| ConfigMap | 3 | loki config, vector config, grafana datasource |
| PVC | 2 + 3 | minio + grafana (static) + loki targets (via volumeClaimTemplates) |
| ServiceAccount | 1 | vector |
| ClusterRole | 1 | vector (pods, namespaces, nodes read access) |
| ClusterRoleBinding | 1 | vector |
| Ingress | 0/1 | grafana (disabled by default) |
---
## File Structure
```
.
├── docker-compose.yaml # Docker Compose deployment
├── loki-config.yaml # Loki configuration (docker-compose)
├── nginx.conf # Gateway routing rules (docker-compose)
├── vector.yaml # Vector pipeline (docker-compose)
├── provisioning/
│ └── datasources/
│ └── loki.yaml # Grafana datasource provisioning (docker-compose)
├── docs/
│ └── architecture.excalidraw.png # Architecture diagram
├── helm/
│ └── loki-stack/ # Helm chart for Kubernetes
│ ├── Chart.yaml
│ ├── values.yaml
│ └── templates/
└── README.md
```

159
docker-compose.yaml Normal file
View File

@@ -0,0 +1,159 @@
services:
# --- MinIO ---
minio:
container_name: loki-minio
image: minio/minio:latest
entrypoint:
- sh
- -euc
- |
mkdir -p /data/loki-chunks && \
minio server /data --console-address ":9001"
environment:
MINIO_ROOT_USER: loki
MINIO_ROOT_PASSWORD: supersecret
ports:
- "9000:9000"
- "9001:9001"
volumes:
- minio-data:/data
healthcheck:
test: ["CMD", "mc", "ready", "local"]
interval: 5s
timeout: 5s
retries: 10
networks:
loki-net:
ipv4_address: 192.168.97.10
# --- Loki Write ---
loki-write:
container_name: loki-write
image: grafana/loki:3.4.2
hostname: loki-write
command: -config.file=/etc/loki/loki-config.yaml -config.expand-env=true -target=write
volumes:
- ./loki-config.yaml:/etc/loki/loki-config.yaml:ro
- loki-write-data:/loki
depends_on:
minio:
condition: service_healthy
networks:
loki-net:
ipv4_address: 192.168.97.11
# --- Loki Read ---
loki-read:
container_name: loki-read
image: grafana/loki:3.4.2
hostname: loki-read
command: -config.file=/etc/loki/loki-config.yaml -config.expand-env=true -target=read
volumes:
- ./loki-config.yaml:/etc/loki/loki-config.yaml:ro
- loki-read-data:/loki
depends_on:
minio:
condition: service_healthy
networks:
loki-net:
ipv4_address: 192.168.97.12
# --- Loki Backend ---
loki-backend:
container_name: loki-backend
image: grafana/loki:3.4.2
hostname: loki-backend
command: -config.file=/etc/loki/loki-config.yaml -config.expand-env=true -target=backend
volumes:
- ./loki-config.yaml:/etc/loki/loki-config.yaml:ro
- loki-backend-data:/loki
depends_on:
minio:
condition: service_healthy
networks:
loki-net:
ipv4_address: 192.168.97.13
# --- Gateway (nginx) ---
gateway:
container_name: loki-gateway
image: nginx:1.27-alpine
volumes:
- ./nginx.conf:/etc/nginx/nginx.conf:ro
ports:
- "3100:3100"
- "3000:3000"
depends_on:
- loki-write
- loki-read
- loki-backend
- grafana
healthcheck:
test: ["CMD-SHELL", "wget --quiet --tries=1 --output-document=- http://192.168.97.14:3100/ready | grep -q 'ready'"]
interval: 10s
timeout: 5s
retries: 15
networks:
loki-net:
ipv4_address: 192.168.97.14
# --- Vector ---
vector:
container_name: loki-vector
image: timberio/vector:0.44.0-alpine
volumes:
- ./vector.yaml:/etc/vector/vector.yaml:ro
- /var/run/docker.sock:/var/run/docker.sock:ro
depends_on:
gateway:
condition: service_healthy
networks:
loki-net:
ipv4_address: 192.168.97.15
# --- Grafana ---
grafana:
container_name: loki-grafana
image: grafana/grafana:12.4
environment:
GF_SECURITY_ADMIN_USER: admin
GF_SECURITY_ADMIN_PASSWORD: admin
GF_AUTH_ANONYMOUS_ENABLED: "true"
GF_AUTH_ANONYMOUS_ORG_ROLE: Admin
volumes:
- ./provisioning:/etc/grafana/provisioning:ro
- grafana-data:/var/lib/grafana
depends_on:
loki-read:
condition: service_started
networks:
loki-net:
ipv4_address: 192.168.97.16
# --- Log Generator ---
log-generator:
container_name: loki-log-generator
image: mingrammer/flog:0.4.3
command: -f json -d 2s -l
labels:
vector.collect: "true"
depends_on:
- vector
networks:
loki-net:
ipv4_address: 192.168.97.17
networks:
loki-net:
driver: bridge
ipam:
config:
- subnet: 192.168.97.0/24
gateway: 192.168.97.1
volumes:
minio-data:
loki-write-data:
loki-read-data:
loki-backend-data:
grafana-data:

Binary file not shown.

After

Width:  |  Height:  |  Size: 96 KiB

View File

@@ -0,0 +1,6 @@
apiVersion: v2
name: loki-stack
description: Loki 3 Simple Scalable + Vector DaemonSet + Grafana + MinIO
version: 0.1.0
appVersion: "3.4.2"
type: application

View File

@@ -0,0 +1,40 @@
{{/*
Expand the name of the chart.
*/}}
{{- define "loki-stack.name" -}}
{{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" }}
{{- end }}
{{/*
Create a default fully qualified app name.
*/}}
{{- define "loki-stack.fullname" -}}
{{- if .Values.fullnameOverride }}
{{- .Values.fullnameOverride | trunc 63 | trimSuffix "-" }}
{{- else }}
{{- $name := default .Chart.Name .Values.nameOverride }}
{{- if contains $name .Release.Name }}
{{- .Release.Name | trunc 63 | trimSuffix "-" }}
{{- else }}
{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" }}
{{- end }}
{{- end }}
{{- end }}
{{/*
Common labels
*/}}
{{- define "loki-stack.labels" -}}
helm.sh/chart: {{ include "loki-stack.name" . }}
app.kubernetes.io/managed-by: {{ .Release.Service }}
app.kubernetes.io/version: {{ .Chart.AppVersion | quote }}
app.kubernetes.io/part-of: {{ include "loki-stack.name" . }}
{{- end }}
{{/*
Selector labels for a component
*/}}
{{- define "loki-stack.selectorLabels" -}}
app.kubernetes.io/name: {{ include "loki-stack.name" . }}
app.kubernetes.io/instance: {{ .Release.Name }}
{{- end }}

View File

@@ -0,0 +1,17 @@
apiVersion: v1
kind: ConfigMap
metadata:
name: {{ include "loki-stack.fullname" . }}-grafana-datasources
labels:
{{- include "loki-stack.labels" . | nindent 4 }}
app.kubernetes.io/component: grafana
data:
loki.yaml: |
apiVersion: 1
datasources:
- name: Loki
type: loki
access: proxy
url: http://{{ include "loki-stack.fullname" . }}-loki-read:3100
isDefault: true
editable: true

View File

@@ -0,0 +1,56 @@
apiVersion: apps/v1
kind: Deployment
metadata:
name: {{ include "loki-stack.fullname" . }}-grafana
labels:
{{- include "loki-stack.labels" . | nindent 4 }}
app.kubernetes.io/component: grafana
spec:
replicas: {{ .Values.grafana.replicas }}
selector:
matchLabels:
{{- include "loki-stack.selectorLabels" . | nindent 6 }}
app.kubernetes.io/component: grafana
template:
metadata:
labels:
{{- include "loki-stack.selectorLabels" . | nindent 8 }}
app.kubernetes.io/component: grafana
spec:
securityContext:
fsGroup: 472
containers:
- name: grafana
image: {{ .Values.grafana.image.repository }}:{{ .Values.grafana.image.tag }}
env:
- name: GF_SECURITY_ADMIN_USER
value: {{ .Values.grafana.adminUser }}
- name: GF_SECURITY_ADMIN_PASSWORD
value: {{ .Values.grafana.adminPassword }}
- name: GF_AUTH_ANONYMOUS_ENABLED
value: {{ .Values.grafana.anonymousAccess | quote }}
- name: GF_AUTH_ANONYMOUS_ORG_ROLE
value: Admin
ports:
- name: http
containerPort: 3000
readinessProbe:
httpGet:
path: /api/health
port: http
initialDelaySeconds: 10
periodSeconds: 10
volumeMounts:
- name: datasources
mountPath: /etc/grafana/provisioning/datasources
- name: data
mountPath: /var/lib/grafana
resources:
{{- toYaml .Values.grafana.resources | nindent 12 }}
volumes:
- name: datasources
configMap:
name: {{ include "loki-stack.fullname" . }}-grafana-datasources
- name: data
persistentVolumeClaim:
claimName: {{ include "loki-stack.fullname" . }}-grafana

View File

@@ -0,0 +1,36 @@
{{- if .Values.grafana.ingress.enabled }}
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: {{ include "loki-stack.fullname" . }}-grafana
labels:
{{- include "loki-stack.labels" . | nindent 4 }}
app.kubernetes.io/component: grafana
{{- with .Values.grafana.ingress.annotations }}
annotations:
{{- toYaml . | nindent 4 }}
{{- end }}
spec:
{{- if .Values.grafana.ingress.className }}
ingressClassName: {{ .Values.grafana.ingress.className }}
{{- end }}
{{- if .Values.grafana.ingress.tls }}
tls:
{{- toYaml .Values.grafana.ingress.tls | nindent 4 }}
{{- end }}
rules:
{{- range .Values.grafana.ingress.hosts }}
- host: {{ .host | quote }}
http:
paths:
{{- range .paths }}
- path: {{ .path }}
pathType: {{ .pathType }}
backend:
service:
name: {{ include "loki-stack.fullname" $ }}-grafana
port:
name: http
{{- end }}
{{- end }}
{{- end }}

View File

@@ -0,0 +1,16 @@
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: {{ include "loki-stack.fullname" . }}-grafana
labels:
{{- include "loki-stack.labels" . | nindent 4 }}
app.kubernetes.io/component: grafana
spec:
accessModes:
- ReadWriteOnce
{{- if .Values.grafana.storage.storageClassName }}
storageClassName: {{ .Values.grafana.storage.storageClassName }}
{{- end }}
resources:
requests:
storage: {{ .Values.grafana.storage.size }}

View File

@@ -0,0 +1,16 @@
apiVersion: v1
kind: Service
metadata:
name: {{ include "loki-stack.fullname" . }}-grafana
labels:
{{- include "loki-stack.labels" . | nindent 4 }}
app.kubernetes.io/component: grafana
spec:
type: ClusterIP
ports:
- name: http
port: 3000
targetPort: http
selector:
{{- include "loki-stack.selectorLabels" . | nindent 4 }}
app.kubernetes.io/component: grafana

View File

@@ -0,0 +1,69 @@
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: {{ include "loki-stack.fullname" . }}-loki-backend
labels:
{{- include "loki-stack.labels" . | nindent 4 }}
app.kubernetes.io/component: loki-backend
spec:
replicas: {{ .Values.loki.backend.replicas }}
serviceName: {{ include "loki-stack.fullname" . }}-loki-backend-headless
selector:
matchLabels:
{{- include "loki-stack.selectorLabels" . | nindent 6 }}
app.kubernetes.io/component: loki-backend
template:
metadata:
labels:
{{- include "loki-stack.selectorLabels" . | nindent 8 }}
app.kubernetes.io/component: loki-backend
loki.grafana.com/memberlist: "true"
spec:
containers:
- name: loki
image: {{ .Values.loki.image.repository }}:{{ .Values.loki.image.tag }}
args:
- -config.file=/etc/loki/loki-config.yaml
- -config.expand-env=true
- -target=backend
env:
- name: POD_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
ports:
- name: http
containerPort: 3100
- name: grpc
containerPort: 9095
- name: memberlist
containerPort: 7946
readinessProbe:
httpGet:
path: /ready
port: http
initialDelaySeconds: 15
periodSeconds: 10
volumeMounts:
- name: config
mountPath: /etc/loki
- name: data
mountPath: /loki
resources:
{{- toYaml .Values.loki.backend.resources | nindent 12 }}
volumes:
- name: config
configMap:
name: {{ include "loki-stack.fullname" . }}-loki
volumeClaimTemplates:
- metadata:
name: data
spec:
accessModes:
- ReadWriteOnce
{{- if .Values.loki.backend.storage.storageClassName }}
storageClassName: {{ .Values.loki.backend.storage.storageClassName }}
{{- end }}
resources:
requests:
storage: {{ .Values.loki.backend.storage.size }}

View File

@@ -0,0 +1,74 @@
apiVersion: v1
kind: ConfigMap
metadata:
name: {{ include "loki-stack.fullname" . }}-loki
labels:
{{- include "loki-stack.labels" . | nindent 4 }}
app.kubernetes.io/component: loki
data:
loki-config.yaml: |
auth_enabled: false
server:
http_listen_port: 3100
grpc_listen_port: 9095
log_level: info
common:
compactor_address: http://{{ include "loki-stack.fullname" . }}-loki-backend:3100
ring:
instance_addr: ${POD_IP}
kvstore:
store: memberlist
replication_factor: 1
path_prefix: /loki
memberlist:
join_members:
- {{ include "loki-stack.fullname" . }}-loki-memberlist:7946
schema_config:
configs:
- from: "2024-01-01"
store: tsdb
object_store: s3
schema: v13
index:
prefix: index_
period: 24h
storage_config:
aws:
endpoint: {{ include "loki-stack.fullname" . }}-minio:9000
insecure: true
bucketnames: {{ .Values.minio.bucketName }}
access_key_id: {{ .Values.minio.rootUser }}
secret_access_key: {{ .Values.minio.rootPassword }}
s3forcepathstyle: true
tsdb_shipper:
active_index_directory: /loki/index
cache_location: /loki/index_cache
ingester:
chunk_encoding: snappy
querier:
max_concurrent: 4
frontend_worker:
frontend_address: 127.0.0.1:9095
limits_config:
reject_old_samples: true
reject_old_samples_max_age: 168h
allow_structured_metadata: true
volume_enabled: true
compactor:
working_directory: /loki/compactor
compaction_interval: 10m
retention_enabled: true
delete_request_store: s3
pattern_ingester:
enabled: true

View File

@@ -0,0 +1,69 @@
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: {{ include "loki-stack.fullname" . }}-loki-read
labels:
{{- include "loki-stack.labels" . | nindent 4 }}
app.kubernetes.io/component: loki-read
spec:
replicas: {{ .Values.loki.read.replicas }}
serviceName: {{ include "loki-stack.fullname" . }}-loki-read-headless
selector:
matchLabels:
{{- include "loki-stack.selectorLabels" . | nindent 6 }}
app.kubernetes.io/component: loki-read
template:
metadata:
labels:
{{- include "loki-stack.selectorLabels" . | nindent 8 }}
app.kubernetes.io/component: loki-read
loki.grafana.com/memberlist: "true"
spec:
containers:
- name: loki
image: {{ .Values.loki.image.repository }}:{{ .Values.loki.image.tag }}
args:
- -config.file=/etc/loki/loki-config.yaml
- -config.expand-env=true
- -target=read
env:
- name: POD_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
ports:
- name: http
containerPort: 3100
- name: grpc
containerPort: 9095
- name: memberlist
containerPort: 7946
readinessProbe:
httpGet:
path: /ready
port: http
initialDelaySeconds: 15
periodSeconds: 10
volumeMounts:
- name: config
mountPath: /etc/loki
- name: data
mountPath: /loki
resources:
{{- toYaml .Values.loki.read.resources | nindent 12 }}
volumes:
- name: config
configMap:
name: {{ include "loki-stack.fullname" . }}-loki
volumeClaimTemplates:
- metadata:
name: data
spec:
accessModes:
- ReadWriteOnce
{{- if .Values.loki.read.storage.storageClassName }}
storageClassName: {{ .Values.loki.read.storage.storageClassName }}
{{- end }}
resources:
requests:
storage: {{ .Values.loki.read.storage.size }}

View File

@@ -0,0 +1,133 @@
apiVersion: v1
kind: Service
metadata:
name: {{ include "loki-stack.fullname" . }}-loki-write
labels:
{{- include "loki-stack.labels" . | nindent 4 }}
app.kubernetes.io/component: loki-write
spec:
type: ClusterIP
ports:
- name: http
port: 3100
targetPort: http
- name: grpc
port: 9095
targetPort: grpc
selector:
{{- include "loki-stack.selectorLabels" . | nindent 4 }}
app.kubernetes.io/component: loki-write
---
apiVersion: v1
kind: Service
metadata:
name: {{ include "loki-stack.fullname" . }}-loki-write-headless
labels:
{{- include "loki-stack.labels" . | nindent 4 }}
app.kubernetes.io/component: loki-write
spec:
type: ClusterIP
clusterIP: None
ports:
- name: http
port: 3100
targetPort: http
selector:
{{- include "loki-stack.selectorLabels" . | nindent 4 }}
app.kubernetes.io/component: loki-write
---
apiVersion: v1
kind: Service
metadata:
name: {{ include "loki-stack.fullname" . }}-loki-read
labels:
{{- include "loki-stack.labels" . | nindent 4 }}
app.kubernetes.io/component: loki-read
spec:
type: ClusterIP
ports:
- name: http
port: 3100
targetPort: http
- name: grpc
port: 9095
targetPort: grpc
selector:
{{- include "loki-stack.selectorLabels" . | nindent 4 }}
app.kubernetes.io/component: loki-read
---
apiVersion: v1
kind: Service
metadata:
name: {{ include "loki-stack.fullname" . }}-loki-read-headless
labels:
{{- include "loki-stack.labels" . | nindent 4 }}
app.kubernetes.io/component: loki-read
spec:
type: ClusterIP
clusterIP: None
ports:
- name: http
port: 3100
targetPort: http
selector:
{{- include "loki-stack.selectorLabels" . | nindent 4 }}
app.kubernetes.io/component: loki-read
---
apiVersion: v1
kind: Service
metadata:
name: {{ include "loki-stack.fullname" . }}-loki-backend
labels:
{{- include "loki-stack.labels" . | nindent 4 }}
app.kubernetes.io/component: loki-backend
spec:
type: ClusterIP
ports:
- name: http
port: 3100
targetPort: http
- name: grpc
port: 9095
targetPort: grpc
selector:
{{- include "loki-stack.selectorLabels" . | nindent 4 }}
app.kubernetes.io/component: loki-backend
---
apiVersion: v1
kind: Service
metadata:
name: {{ include "loki-stack.fullname" . }}-loki-backend-headless
labels:
{{- include "loki-stack.labels" . | nindent 4 }}
app.kubernetes.io/component: loki-backend
spec:
type: ClusterIP
clusterIP: None
ports:
- name: http
port: 3100
targetPort: http
selector:
{{- include "loki-stack.selectorLabels" . | nindent 4 }}
app.kubernetes.io/component: loki-backend
---
apiVersion: v1
kind: Service
metadata:
name: {{ include "loki-stack.fullname" . }}-loki-memberlist
labels:
{{- include "loki-stack.labels" . | nindent 4 }}
app.kubernetes.io/component: loki-memberlist
spec:
type: ClusterIP
clusterIP: None
publishNotReadyAddresses: true
ports:
- name: memberlist
port: 7946
targetPort: memberlist
protocol: TCP
selector:
{{- include "loki-stack.selectorLabels" . | nindent 4 }}
loki.grafana.com/memberlist: "true"

View File

@@ -0,0 +1,69 @@
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: {{ include "loki-stack.fullname" . }}-loki-write
labels:
{{- include "loki-stack.labels" . | nindent 4 }}
app.kubernetes.io/component: loki-write
spec:
replicas: {{ .Values.loki.write.replicas }}
serviceName: {{ include "loki-stack.fullname" . }}-loki-write-headless
selector:
matchLabels:
{{- include "loki-stack.selectorLabels" . | nindent 6 }}
app.kubernetes.io/component: loki-write
template:
metadata:
labels:
{{- include "loki-stack.selectorLabels" . | nindent 8 }}
app.kubernetes.io/component: loki-write
loki.grafana.com/memberlist: "true"
spec:
containers:
- name: loki
image: {{ .Values.loki.image.repository }}:{{ .Values.loki.image.tag }}
args:
- -config.file=/etc/loki/loki-config.yaml
- -config.expand-env=true
- -target=write
env:
- name: POD_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
ports:
- name: http
containerPort: 3100
- name: grpc
containerPort: 9095
- name: memberlist
containerPort: 7946
readinessProbe:
httpGet:
path: /ready
port: http
initialDelaySeconds: 15
periodSeconds: 10
volumeMounts:
- name: config
mountPath: /etc/loki
- name: data
mountPath: /loki
resources:
{{- toYaml .Values.loki.write.resources | nindent 12 }}
volumes:
- name: config
configMap:
name: {{ include "loki-stack.fullname" . }}-loki
volumeClaimTemplates:
- metadata:
name: data
spec:
accessModes:
- ReadWriteOnce
{{- if .Values.loki.write.storage.storageClassName }}
storageClassName: {{ .Values.loki.write.storage.storageClassName }}
{{- end }}
resources:
requests:
storage: {{ .Values.loki.write.storage.size }}

View File

@@ -0,0 +1,52 @@
apiVersion: apps/v1
kind: Deployment
metadata:
name: {{ include "loki-stack.fullname" . }}-minio
labels:
{{- include "loki-stack.labels" . | nindent 4 }}
app.kubernetes.io/component: minio
spec:
replicas: 1
selector:
matchLabels:
{{- include "loki-stack.selectorLabels" . | nindent 6 }}
app.kubernetes.io/component: minio
template:
metadata:
labels:
{{- include "loki-stack.selectorLabels" . | nindent 8 }}
app.kubernetes.io/component: minio
spec:
containers:
- name: minio
image: {{ .Values.minio.image.repository }}:{{ .Values.minio.image.tag }}
command:
- sh
- -euc
- |
mkdir -p /data/{{ .Values.minio.bucketName }} && \
minio server /data --console-address ":9001"
env:
- name: MINIO_ROOT_USER
value: {{ .Values.minio.rootUser }}
- name: MINIO_ROOT_PASSWORD
value: {{ .Values.minio.rootPassword }}
ports:
- name: api
containerPort: 9000
- name: console
containerPort: 9001
readinessProbe:
exec:
command: ["mc", "ready", "local"]
initialDelaySeconds: 5
periodSeconds: 5
volumeMounts:
- name: data
mountPath: /data
resources:
{{- toYaml .Values.minio.resources | nindent 12 }}
volumes:
- name: data
persistentVolumeClaim:
claimName: {{ include "loki-stack.fullname" . }}-minio

View File

@@ -0,0 +1,16 @@
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: {{ include "loki-stack.fullname" . }}-minio
labels:
{{- include "loki-stack.labels" . | nindent 4 }}
app.kubernetes.io/component: minio
spec:
accessModes:
- ReadWriteOnce
{{- if .Values.minio.storage.storageClassName }}
storageClassName: {{ .Values.minio.storage.storageClassName }}
{{- end }}
resources:
requests:
storage: {{ .Values.minio.storage.size }}

View File

@@ -0,0 +1,19 @@
apiVersion: v1
kind: Service
metadata:
name: {{ include "loki-stack.fullname" . }}-minio
labels:
{{- include "loki-stack.labels" . | nindent 4 }}
app.kubernetes.io/component: minio
spec:
type: ClusterIP
ports:
- name: api
port: 9000
targetPort: api
- name: console
port: 9001
targetPort: console
selector:
{{- include "loki-stack.selectorLabels" . | nindent 4 }}
app.kubernetes.io/component: minio

View File

@@ -0,0 +1,10 @@
{{- if .Values.createNamespace }}
apiVersion: v1
kind: Namespace
metadata:
name: {{ .Release.Namespace }}
labels:
pod-security.kubernetes.io/audit: privileged
pod-security.kubernetes.io/enforce: privileged
pod-security.kubernetes.io/warn: privileged
{{- end }}

View File

@@ -0,0 +1,33 @@
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: {{ include "loki-stack.fullname" . }}-allow-internal
labels:
{{- include "loki-stack.labels" . | nindent 4 }}
spec:
podSelector:
matchLabels:
{{- include "loki-stack.selectorLabels" . | nindent 6 }}
policyTypes:
- Ingress
- Egress
ingress:
# Allow all traffic within the namespace
- from:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: {{ .Release.Namespace }}
egress:
# Allow all traffic within the namespace
- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: {{ .Release.Namespace }}
# Allow DNS
- to:
- namespaceSelector: {}
ports:
- protocol: UDP
port: 53
- protocol: TCP
port: 53

View File

@@ -0,0 +1,43 @@
apiVersion: v1
kind: ConfigMap
metadata:
name: {{ include "loki-stack.fullname" . }}-vector
labels:
{{- include "loki-stack.labels" . | nindent 4 }}
app.kubernetes.io/component: vector
data:
vector.yaml: |
sources:
kubernetes_logs:
type: kubernetes_logs
self_node_name: ${VECTOR_SELF_NODE_NAME}
transforms:
parse_logs:
type: remap
inputs:
- kubernetes_logs
source: |
.namespace = .kubernetes.pod_namespace
.pod = .kubernetes.pod_name
.container = .kubernetes.container_name
.node = .kubernetes.pod_node_name
del(.kubernetes)
del(.file)
del(.source_type)
sinks:
loki:
type: loki
inputs:
- parse_logs
endpoint: http://{{ include "loki-stack.fullname" . }}-loki-write:3100
encoding:
codec: text
labels:
source: vector
namespace: "{{`{{ namespace }}`}}"
pod: "{{`{{ pod }}`}}"
container: "{{`{{ container }}`}}"
node: "{{`{{ node }}`}}"
remove_label_fields: true

View File

@@ -0,0 +1,60 @@
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: {{ include "loki-stack.fullname" . }}-vector
labels:
{{- include "loki-stack.labels" . | nindent 4 }}
app.kubernetes.io/component: vector
spec:
selector:
matchLabels:
{{- include "loki-stack.selectorLabels" . | nindent 6 }}
app.kubernetes.io/component: vector
template:
metadata:
labels:
{{- include "loki-stack.selectorLabels" . | nindent 8 }}
app.kubernetes.io/component: vector
spec:
serviceAccountName: {{ include "loki-stack.fullname" . }}-vector
containers:
- name: vector
image: {{ .Values.vector.image.repository }}:{{ .Values.vector.image.tag }}
securityContext:
privileged: true
env:
- name: VECTOR_SELF_NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
volumeMounts:
- name: config
mountPath: /etc/vector/vector.yaml
subPath: vector.yaml
- name: var-log
mountPath: /var/log
readOnly: true
- name: var-lib-containers
mountPath: /var/log/pods
readOnly: true
- name: data
mountPath: /vector-data-dir
resources:
{{- toYaml .Values.vector.resources | nindent 12 }}
{{- with .Values.vector.tolerations }}
tolerations:
{{- toYaml . | nindent 8 }}
{{- end }}
volumes:
- name: config
configMap:
name: {{ include "loki-stack.fullname" . }}-vector
- name: var-log
hostPath:
path: /var/log
- name: var-lib-containers
hostPath:
path: /var/log/pods
- name: data
hostPath:
path: /var/lib/vector

View File

@@ -0,0 +1,35 @@
apiVersion: v1
kind: ServiceAccount
metadata:
name: {{ include "loki-stack.fullname" . }}-vector
labels:
{{- include "loki-stack.labels" . | nindent 4 }}
app.kubernetes.io/component: vector
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: {{ include "loki-stack.fullname" . }}-vector
labels:
{{- include "loki-stack.labels" . | nindent 4 }}
app.kubernetes.io/component: vector
rules:
- apiGroups: [""]
resources: ["namespaces", "nodes", "pods"]
verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: {{ include "loki-stack.fullname" . }}-vector
labels:
{{- include "loki-stack.labels" . | nindent 4 }}
app.kubernetes.io/component: vector
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: {{ include "loki-stack.fullname" . }}-vector
subjects:
- kind: ServiceAccount
name: {{ include "loki-stack.fullname" . }}-vector
namespace: {{ .Release.Namespace }}

112
helm/loki-stack/values.yaml Normal file
View File

@@ -0,0 +1,112 @@
nameOverride: ""
fullnameOverride: ""
# Create namespace with PSA labels (privileged) for Talos
createNamespace: false
# --- MinIO ---
minio:
image:
repository: minio/minio
tag: latest
rootUser: loki
rootPassword: supersecret
bucketName: loki-chunks
storage:
size: 10Gi
storageClassName: ""
resources:
requests:
cpu: 100m
memory: 256Mi
limits:
cpu: 500m
memory: 512Mi
# --- Loki ---
loki:
image:
repository: grafana/loki
tag: "3.4.2"
write:
replicas: 1
resources:
requests:
cpu: 100m
memory: 256Mi
limits:
cpu: 500m
memory: 512Mi
storage:
size: 10Gi
storageClassName: ""
read:
replicas: 1
resources:
requests:
cpu: 100m
memory: 256Mi
limits:
cpu: 500m
memory: 512Mi
storage:
size: 10Gi
storageClassName: ""
backend:
replicas: 1
resources:
requests:
cpu: 100m
memory: 256Mi
limits:
cpu: 500m
memory: 512Mi
storage:
size: 10Gi
storageClassName: ""
# --- Vector ---
vector:
image:
repository: timberio/vector
tag: 0.44.0-alpine
resources:
requests:
cpu: 50m
memory: 64Mi
limits:
cpu: 200m
memory: 256Mi
tolerations: []
# --- Grafana ---
grafana:
image:
repository: grafana/grafana
tag: "12.4"
adminUser: admin
adminPassword: admin
anonymousAccess: true
replicas: 1
resources:
requests:
cpu: 100m
memory: 128Mi
limits:
cpu: 500m
memory: 512Mi
service:
type: ClusterIP
storage:
size: 5Gi
storageClassName: ""
ingress:
enabled: false
className: ""
annotations: {}
hosts:
- host: grafana.local
paths:
- path: /
pathType: Prefix
tls: []

67
loki-config.yaml Normal file
View File

@@ -0,0 +1,67 @@
auth_enabled: false
server:
http_listen_port: 3100
grpc_listen_port: 9095
log_level: info
common:
compactor_address: http://loki-backend:3100
ring:
instance_addr: ${HOSTNAME}
kvstore:
store: memberlist
replication_factor: 1
path_prefix: /loki
memberlist:
join_members:
- loki-write:7946
- loki-read:7946
- loki-backend:7946
schema_config:
configs:
- from: "2024-01-01"
store: tsdb
object_store: s3
schema: v13
index:
prefix: index_
period: 24h
storage_config:
aws:
endpoint: minio:9000
insecure: true
bucketnames: loki-chunks
access_key_id: loki
secret_access_key: supersecret
s3forcepathstyle: true
tsdb_shipper:
active_index_directory: /loki/index
cache_location: /loki/index_cache
ingester:
chunk_encoding: snappy
querier:
max_concurrent: 4
frontend_worker:
frontend_address: loki-read:9095
limits_config:
reject_old_samples: true
reject_old_samples_max_age: 168h
allow_structured_metadata: true
volume_enabled: true
compactor:
working_directory: /loki/compactor
compaction_interval: 10m
retention_enabled: true
delete_request_store: s3
pattern_ingester:
enabled: true

59
nginx.conf Normal file
View File

@@ -0,0 +1,59 @@
worker_processes 1;
events {
worker_connections 1024;
}
http {
resolver 127.0.0.11 valid=10s;
# Loki gateway
server {
listen 3100;
location = /ready {
proxy_pass http://loki-read:3100$request_uri;
}
# Write path
location = /loki/api/v1/push {
proxy_pass http://loki-write:3100$request_uri;
}
# Read path
location ~ /loki/api/.* {
proxy_pass http://loki-read:3100$request_uri;
}
# Ruler
location ~ /api/prom/rules.* {
proxy_pass http://loki-backend:3100$request_uri;
}
location ~ /prometheus/api/v1/rules.* {
proxy_pass http://loki-backend:3100$request_uri;
}
}
# Grafana proxy
server {
listen 3000;
location / {
proxy_pass http://grafana:3000;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
}
# Grafana WebSocket (live features)
location /api/live/ {
proxy_pass http://grafana:3000;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_set_header Host $host;
}
}
}

View File

@@ -0,0 +1,9 @@
apiVersion: 1
datasources:
- name: Loki
type: loki
access: proxy
url: http://gateway:3100
isDefault: true
editable: true

30
vector.yaml Normal file
View File

@@ -0,0 +1,30 @@
sources:
docker_logs:
type: docker_logs
include_labels:
- vector.collect=true
transforms:
parse_logs:
type: remap
inputs:
- docker_logs
source: |
.container_name = del(.label."com.docker.compose.service")
.compose_project = del(.label."com.docker.compose.project")
# Remove unnecessary labels
del(.label)
sinks:
loki:
type: loki
inputs:
- parse_logs
endpoint: http://gateway:3100
encoding:
codec: text
labels:
source: vector
container: "{{ container_name }}"
project: "{{ compose_project }}"
remove_label_fields: true