# Loki 3 + Vector + Grafana — Test Stand

Test environment for centralized log collection using Grafana Loki in Simple Scalable mode with Vector as the log collector and MinIO as S3-compatible object storage.

Two deployment options: **Docker Compose** (local testing with log-generator) and **Helm chart** (Kubernetes with DaemonSet log collection from all pods).

## Architecture

![Architecture Diagram](docs/architecture.excalidraw.png)

## Components

| Service | Image | Role |
|---|---|---|
| **minio** | `minio/minio:latest` | S3-compatible object storage for chunks and index |
| **loki-write** | `grafana/loki:3.4.2` | Ingester — accepts and stores log streams |
| **loki-read** | `grafana/loki:3.4.2` | Querier — executes LogQL queries |
| **loki-backend** | `grafana/loki:3.4.2` | Compactor, ruler, index gateway |
| **gateway** | `nginx:1.27-alpine` | Reverse proxy — Loki write/read/ruler + Grafana UI |
| **vector** | `timberio/vector:0.44.0-alpine` | Log collector — Loki sink |
| **grafana** | `grafana/grafana:12.4` | Visualization — Loki datasource pre-provisioned |

## Data Flow

1. **Vector** collects container logs (docker_logs / kubernetes_logs source)
2. Vector transforms logs (extracts metadata, removes raw labels) and pushes to **gateway**
3. **Gateway** (nginx) routes `/loki/api/v1/push` to **loki-write**, all other `/loki/api/*` to **loki-read**
4. **loki-write** ingests logs and stores chunks in **MinIO** (`loki-chunks` bucket)
5. **loki-backend** runs compactor on a 10-minute interval
6. **Grafana** queries logs via gateway → loki-read

## Loki Configuration

- **Deployment mode**: Simple Scalable (write / read / backend targets)
- **Schema**: v13 with TSDB store
- **Storage**: S3 (MinIO) — bucket `loki-chunks`
- **Service discovery**: memberlist (port 7946)
- **Replication factor**: 1 (test stand)
- **Chunk encoding**: snappy
- **Retention**: enabled via compactor, old samples rejected after 168h

---

## Option 1: Docker Compose

Local test stand with a built-in log-generator container (`flog`). Vector collects logs from containers with label `vector.collect=true` via Docker socket.

### Quick Start

```bash
docker compose up -d
```

Wait for all services to become healthy:

```bash
docker compose ps
```

### Access Points

| Service | URL | Credentials |
|---|---|---|
| Grafana (via gateway) | http://localhost:3000 | admin / admin (anonymous access enabled) |
| Loki API (via gateway) | http://localhost:3100 | — |
| MinIO Console | http://localhost:9001 | loki / supersecret |
| MinIO API | http://localhost:9000 | loki / supersecret |

### Network

All containers run on a static network `192.168.97.0/24`:

| Container | IP |
|---|---|
| `loki-minio` | 192.168.97.10 |
| `loki-write` | 192.168.97.11 |
| `loki-read` | 192.168.97.12 |
| `loki-backend` | 192.168.97.13 |
| `loki-gateway` | 192.168.97.14 |
| `loki-vector` | 192.168.97.15 |
| `loki-grafana` | 192.168.97.16 |
| `loki-log-generator` | 192.168.97.17 |

### Adding Your Own Containers

To collect logs from any container, add the label:

```yaml
services:
  my-app:
    image: my-app:latest
    labels:
      vector.collect: "true"
```

Logs will appear in Grafana with label `{container="my-app"}`.

### Useful Commands

```bash
# Start
docker compose up -d

# Stop
docker compose down

# Stop and remove volumes
docker compose down -v

# View Vector logs
docker compose logs vector -f

# View Loki write logs
docker compose logs loki-write -f

# Check Loki readiness
curl -s http://localhost:3100/ready

# Query logs via API
curl -s http://localhost:3100/loki/api/v1/query_range \
  --data-urlencode 'query={container="log-generator"}' \
  --data-urlencode 'limit=10' | jq .

# Check MinIO bucket
docker compose exec minio mc ls local/loki-chunks/
```

---

## Option 2: Helm Chart (Kubernetes)

Production-oriented deployment. Vector runs as a **DaemonSet** collecting logs from all pods in the cluster via `kubernetes_logs` source. No log-generator — real workload logs are collected.

### Prerequisites

- Kubernetes cluster (1.24+)
- Helm 3
- `kubectl` configured

### Install

```bash
helm install loki-stack ./helm/loki-stack -n loki --create-namespace
```

### Install with custom values

```bash
helm install loki-stack ./helm/loki-stack -n loki --create-namespace \
  -f my-values.yaml
```

### Upgrade

```bash
helm upgrade loki-stack ./helm/loki-stack -n loki
```

### Uninstall

```bash
helm uninstall loki-stack -n loki
```

### Key Differences from Docker Compose

| Feature | Docker Compose | Helm |
|---|---|---|
| Vector source | `docker_logs` (label filter) | `kubernetes_logs` (all pods) |
| Vector mode | container | DaemonSet (one per node) |
| Log labels | `container`, `project` | `namespace`, `pod`, `container`, `node` |
| Loki targets | Deployment | StatefulSet with PVC |
| Service discovery | static IPs + memberlist | headless Service + memberlist |
| Gateway (nginx) | yes — proxies Loki + Grafana | no — direct Service routing |
| Ingress | — | Grafana ingress (disabled by default) |

### Configuration (values.yaml)

Key parameters:

```yaml
# Scale Loki components
loki:
  write:
    replicas: 1        # Increase for HA
  read:
    replicas: 1        # Increase for query throughput
  backend:
    replicas: 1

# MinIO storage
minio:
  rootUser: loki
  rootPassword: supersecret
  storage:
    size: 10Gi

# Vector tolerations (to run on tainted nodes)
vector:
  tolerations: []

# Grafana
grafana:
  adminUser: admin
  adminPassword: admin
  anonymousAccess: true
  service:
    type: ClusterIP
  ingress:
    enabled: false
    className: nginx
    hosts:
      - host: grafana.example.com
        paths:
          - path: /
            pathType: Prefix
    tls: []
```

### Access in Kubernetes

Port-forward to services:

```bash
# Grafana
kubectl port-forward -n loki svc/loki-stack-grafana 3000:3000

# Loki API (read)
kubectl port-forward -n loki svc/loki-stack-loki-read 3100:3100

# Loki API (write / push)
kubectl port-forward -n loki svc/loki-stack-loki-write 3100:3100

# MinIO Console
kubectl port-forward -n loki svc/loki-stack-minio 9001:9001
```

### Viewing Logs

Open Grafana and query:

```logql
# All logs from a namespace
{namespace="default"}

# Specific pod
{pod="my-app-7d4b8c6f5-x2k9p"}

# By container name across all namespaces
{container="nginx"}

# By node
{node="worker-01"}
```

### Helm Chart Structure

```
helm/loki-stack/
├── Chart.yaml
├── values.yaml
└── templates/
    ├── _helpers.tpl
    ├── minio-deployment.yaml
    ├── minio-pvc.yaml
    ├── minio-service.yaml
    ├── loki-configmap.yaml
    ├── loki-write-statefulset.yaml
    ├── loki-read-statefulset.yaml
    ├── loki-backend-statefulset.yaml
    ├── loki-services.yaml
    ├── vector-rbac.yaml
    ├── vector-configmap.yaml
    ├── vector-daemonset.yaml
    ├── grafana-configmap.yaml
    ├── grafana-deployment.yaml
    ├── grafana-ingress.yaml
    ├── grafana-pvc.yaml
    └── grafana-service.yaml
```

### Kubernetes Resources Created

| Kind | Count | Description |
|---|---|---|
| StatefulSet | 3 | loki-write, loki-read, loki-backend |
| Deployment | 2 | minio, grafana |
| DaemonSet | 1 | vector (one per node) |
| Service | 9 | ClusterIP + headless for each Loki target + memberlist + minio + grafana |
| ConfigMap | 3 | loki config, vector config, grafana datasource |
| PVC | 2 + 3 | minio + grafana (static) + loki targets (via volumeClaimTemplates) |
| ServiceAccount | 1 | vector |
| ClusterRole | 1 | vector (pods, namespaces, nodes read access) |
| ClusterRoleBinding | 1 | vector |
| Ingress | 0/1 | grafana (disabled by default) |

---

## File Structure

```
.
├── docker-compose.yaml              # Docker Compose deployment
├── loki-config.yaml                 # Loki configuration (docker-compose)
├── nginx.conf                       # Gateway routing rules (docker-compose)
├── vector.yaml                      # Vector pipeline (docker-compose)
├── provisioning/
│   └── datasources/
│       └── loki.yaml                # Grafana datasource provisioning (docker-compose)
├── docs/
│   └── architecture.excalidraw.png  # Architecture diagram
├── helm/
│   └── loki-stack/                  # Helm chart for Kubernetes
│       ├── Chart.yaml
│       ├── values.yaml
│       └── templates/
└── README.md
```