init
This commit is contained in:
318
README.md
Normal file
318
README.md
Normal file
@@ -0,0 +1,318 @@
|
||||
# Loki 3 + Vector + Grafana — Test Stand
|
||||
|
||||
Test environment for centralized log collection using Grafana Loki in Simple Scalable mode with Vector as the log collector and MinIO as S3-compatible object storage.
|
||||
|
||||
Two deployment options: **Docker Compose** (local testing with log-generator) and **Helm chart** (Kubernetes with DaemonSet log collection from all pods).
|
||||
|
||||
## Architecture
|
||||
|
||||
|
||||
|
||||
## Components
|
||||
|
||||
| Service | Image | Role |
|
||||
|---|---|---|
|
||||
| **minio** | `minio/minio:latest` | S3-compatible object storage for chunks and index |
|
||||
| **loki-write** | `grafana/loki:3.4.2` | Ingester — accepts and stores log streams |
|
||||
| **loki-read** | `grafana/loki:3.4.2` | Querier — executes LogQL queries |
|
||||
| **loki-backend** | `grafana/loki:3.4.2` | Compactor, ruler, index gateway |
|
||||
| **gateway** | `nginx:1.27-alpine` | Reverse proxy — Loki write/read/ruler + Grafana UI |
|
||||
| **vector** | `timberio/vector:0.44.0-alpine` | Log collector — Loki sink |
|
||||
| **grafana** | `grafana/grafana:12.4` | Visualization — Loki datasource pre-provisioned |
|
||||
|
||||
## Data Flow
|
||||
|
||||
1. **Vector** collects container logs (docker_logs / kubernetes_logs source)
|
||||
2. Vector transforms logs (extracts metadata, removes raw labels) and pushes to **gateway**
|
||||
3. **Gateway** (nginx) routes `/loki/api/v1/push` to **loki-write**, all other `/loki/api/*` to **loki-read**
|
||||
4. **loki-write** ingests logs and stores chunks in **MinIO** (`loki-chunks` bucket)
|
||||
5. **loki-backend** runs compactor on a 10-minute interval
|
||||
6. **Grafana** queries logs via gateway → loki-read
|
||||
|
||||
## Loki Configuration
|
||||
|
||||
- **Deployment mode**: Simple Scalable (write / read / backend targets)
|
||||
- **Schema**: v13 with TSDB store
|
||||
- **Storage**: S3 (MinIO) — bucket `loki-chunks`
|
||||
- **Service discovery**: memberlist (port 7946)
|
||||
- **Replication factor**: 1 (test stand)
|
||||
- **Chunk encoding**: snappy
|
||||
- **Retention**: enabled via compactor, old samples rejected after 168h
|
||||
|
||||
---
|
||||
|
||||
## Option 1: Docker Compose
|
||||
|
||||
Local test stand with a built-in log-generator container (`flog`). Vector collects logs from containers with label `vector.collect=true` via Docker socket.
|
||||
|
||||
### Quick Start
|
||||
|
||||
```bash
|
||||
docker compose up -d
|
||||
```
|
||||
|
||||
Wait for all services to become healthy:
|
||||
|
||||
```bash
|
||||
docker compose ps
|
||||
```
|
||||
|
||||
### Access Points
|
||||
|
||||
| Service | URL | Credentials |
|
||||
|---|---|---|
|
||||
| Grafana (via gateway) | http://localhost:3000 | admin / admin (anonymous access enabled) |
|
||||
| Loki API (via gateway) | http://localhost:3100 | — |
|
||||
| MinIO Console | http://localhost:9001 | loki / supersecret |
|
||||
| MinIO API | http://localhost:9000 | loki / supersecret |
|
||||
|
||||
### Network
|
||||
|
||||
All containers run on a static network `192.168.97.0/24`:
|
||||
|
||||
| Container | IP |
|
||||
|---|---|
|
||||
| `loki-minio` | 192.168.97.10 |
|
||||
| `loki-write` | 192.168.97.11 |
|
||||
| `loki-read` | 192.168.97.12 |
|
||||
| `loki-backend` | 192.168.97.13 |
|
||||
| `loki-gateway` | 192.168.97.14 |
|
||||
| `loki-vector` | 192.168.97.15 |
|
||||
| `loki-grafana` | 192.168.97.16 |
|
||||
| `loki-log-generator` | 192.168.97.17 |
|
||||
|
||||
### Adding Your Own Containers
|
||||
|
||||
To collect logs from any container, add the label:
|
||||
|
||||
```yaml
|
||||
services:
|
||||
my-app:
|
||||
image: my-app:latest
|
||||
labels:
|
||||
vector.collect: "true"
|
||||
```
|
||||
|
||||
Logs will appear in Grafana with label `{container="my-app"}`.
|
||||
|
||||
### Useful Commands
|
||||
|
||||
```bash
|
||||
# Start
|
||||
docker compose up -d
|
||||
|
||||
# Stop
|
||||
docker compose down
|
||||
|
||||
# Stop and remove volumes
|
||||
docker compose down -v
|
||||
|
||||
# View Vector logs
|
||||
docker compose logs vector -f
|
||||
|
||||
# View Loki write logs
|
||||
docker compose logs loki-write -f
|
||||
|
||||
# Check Loki readiness
|
||||
curl -s http://localhost:3100/ready
|
||||
|
||||
# Query logs via API
|
||||
curl -s http://localhost:3100/loki/api/v1/query_range \
|
||||
--data-urlencode 'query={container="log-generator"}' \
|
||||
--data-urlencode 'limit=10' | jq .
|
||||
|
||||
# Check MinIO bucket
|
||||
docker compose exec minio mc ls local/loki-chunks/
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Option 2: Helm Chart (Kubernetes)
|
||||
|
||||
Production-oriented deployment. Vector runs as a **DaemonSet** collecting logs from all pods in the cluster via `kubernetes_logs` source. No log-generator — real workload logs are collected.
|
||||
|
||||
### Prerequisites
|
||||
|
||||
- Kubernetes cluster (1.24+)
|
||||
- Helm 3
|
||||
- `kubectl` configured
|
||||
|
||||
### Install
|
||||
|
||||
```bash
|
||||
helm install loki-stack ./helm/loki-stack -n loki --create-namespace
|
||||
```
|
||||
|
||||
### Install with custom values
|
||||
|
||||
```bash
|
||||
helm install loki-stack ./helm/loki-stack -n loki --create-namespace \
|
||||
-f my-values.yaml
|
||||
```
|
||||
|
||||
### Upgrade
|
||||
|
||||
```bash
|
||||
helm upgrade loki-stack ./helm/loki-stack -n loki
|
||||
```
|
||||
|
||||
### Uninstall
|
||||
|
||||
```bash
|
||||
helm uninstall loki-stack -n loki
|
||||
```
|
||||
|
||||
### Key Differences from Docker Compose
|
||||
|
||||
| Feature | Docker Compose | Helm |
|
||||
|---|---|---|
|
||||
| Vector source | `docker_logs` (label filter) | `kubernetes_logs` (all pods) |
|
||||
| Vector mode | container | DaemonSet (one per node) |
|
||||
| Log labels | `container`, `project` | `namespace`, `pod`, `container`, `node` |
|
||||
| Loki targets | Deployment | StatefulSet with PVC |
|
||||
| Service discovery | static IPs + memberlist | headless Service + memberlist |
|
||||
| Gateway (nginx) | yes — proxies Loki + Grafana | no — direct Service routing |
|
||||
| Ingress | — | Grafana ingress (disabled by default) |
|
||||
|
||||
### Configuration (values.yaml)
|
||||
|
||||
Key parameters:
|
||||
|
||||
```yaml
|
||||
# Scale Loki components
|
||||
loki:
|
||||
write:
|
||||
replicas: 1 # Increase for HA
|
||||
read:
|
||||
replicas: 1 # Increase for query throughput
|
||||
backend:
|
||||
replicas: 1
|
||||
|
||||
# MinIO storage
|
||||
minio:
|
||||
rootUser: loki
|
||||
rootPassword: supersecret
|
||||
storage:
|
||||
size: 10Gi
|
||||
|
||||
# Vector tolerations (to run on tainted nodes)
|
||||
vector:
|
||||
tolerations: []
|
||||
|
||||
# Grafana
|
||||
grafana:
|
||||
adminUser: admin
|
||||
adminPassword: admin
|
||||
anonymousAccess: true
|
||||
service:
|
||||
type: ClusterIP
|
||||
ingress:
|
||||
enabled: false
|
||||
className: nginx
|
||||
hosts:
|
||||
- host: grafana.example.com
|
||||
paths:
|
||||
- path: /
|
||||
pathType: Prefix
|
||||
tls: []
|
||||
```
|
||||
|
||||
### Access in Kubernetes
|
||||
|
||||
Port-forward to services:
|
||||
|
||||
```bash
|
||||
# Grafana
|
||||
kubectl port-forward -n loki svc/loki-stack-grafana 3000:3000
|
||||
|
||||
# Loki API (read)
|
||||
kubectl port-forward -n loki svc/loki-stack-loki-read 3100:3100
|
||||
|
||||
# Loki API (write / push)
|
||||
kubectl port-forward -n loki svc/loki-stack-loki-write 3100:3100
|
||||
|
||||
# MinIO Console
|
||||
kubectl port-forward -n loki svc/loki-stack-minio 9001:9001
|
||||
```
|
||||
|
||||
### Viewing Logs
|
||||
|
||||
Open Grafana and query:
|
||||
|
||||
```logql
|
||||
# All logs from a namespace
|
||||
{namespace="default"}
|
||||
|
||||
# Specific pod
|
||||
{pod="my-app-7d4b8c6f5-x2k9p"}
|
||||
|
||||
# By container name across all namespaces
|
||||
{container="nginx"}
|
||||
|
||||
# By node
|
||||
{node="worker-01"}
|
||||
```
|
||||
|
||||
### Helm Chart Structure
|
||||
|
||||
```
|
||||
helm/loki-stack/
|
||||
├── Chart.yaml
|
||||
├── values.yaml
|
||||
└── templates/
|
||||
├── _helpers.tpl
|
||||
├── minio-deployment.yaml
|
||||
├── minio-pvc.yaml
|
||||
├── minio-service.yaml
|
||||
├── loki-configmap.yaml
|
||||
├── loki-write-statefulset.yaml
|
||||
├── loki-read-statefulset.yaml
|
||||
├── loki-backend-statefulset.yaml
|
||||
├── loki-services.yaml
|
||||
├── vector-rbac.yaml
|
||||
├── vector-configmap.yaml
|
||||
├── vector-daemonset.yaml
|
||||
├── grafana-configmap.yaml
|
||||
├── grafana-deployment.yaml
|
||||
├── grafana-ingress.yaml
|
||||
├── grafana-pvc.yaml
|
||||
└── grafana-service.yaml
|
||||
```
|
||||
|
||||
### Kubernetes Resources Created
|
||||
|
||||
| Kind | Count | Description |
|
||||
|---|---|---|
|
||||
| StatefulSet | 3 | loki-write, loki-read, loki-backend |
|
||||
| Deployment | 2 | minio, grafana |
|
||||
| DaemonSet | 1 | vector (one per node) |
|
||||
| Service | 9 | ClusterIP + headless for each Loki target + memberlist + minio + grafana |
|
||||
| ConfigMap | 3 | loki config, vector config, grafana datasource |
|
||||
| PVC | 2 + 3 | minio + grafana (static) + loki targets (via volumeClaimTemplates) |
|
||||
| ServiceAccount | 1 | vector |
|
||||
| ClusterRole | 1 | vector (pods, namespaces, nodes read access) |
|
||||
| ClusterRoleBinding | 1 | vector |
|
||||
| Ingress | 0/1 | grafana (disabled by default) |
|
||||
|
||||
---
|
||||
|
||||
## File Structure
|
||||
|
||||
```
|
||||
.
|
||||
├── docker-compose.yaml # Docker Compose deployment
|
||||
├── loki-config.yaml # Loki configuration (docker-compose)
|
||||
├── nginx.conf # Gateway routing rules (docker-compose)
|
||||
├── vector.yaml # Vector pipeline (docker-compose)
|
||||
├── provisioning/
|
||||
│ └── datasources/
|
||||
│ └── loki.yaml # Grafana datasource provisioning (docker-compose)
|
||||
├── docs/
|
||||
│ └── architecture.excalidraw.png # Architecture diagram
|
||||
├── helm/
|
||||
│ └── loki-stack/ # Helm chart for Kubernetes
|
||||
│ ├── Chart.yaml
|
||||
│ ├── values.yaml
|
||||
│ └── templates/
|
||||
└── README.md
|
||||
```
|
||||
Reference in New Issue
Block a user