> For the complete documentation index, see [llms.txt](https://atomoh.gitbook.io/kubernetes/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://atomoh.gitbook.io/kubernetes/jp/ke-guan-ce-xing/observability/logging/01-loki.md). # Grafana Loki > **対応バージョン**: Loki 3.x **最終更新**: February 20, 2026 Grafana Loki は、Prometheus に着想を得た水平スケーラブルなログ集約システムです。ログ内容ではなく Label のみをインデックス化することで、コスト効率の高いログ保存とクエリを実現します。 ## 目次 1. [概要](#overview) 2. [アーキテクチャ](#architecture) 3. [デプロイモード](#deployment-modes) 4. [Helm インストール](#helm-installation) 5. [S3 バックエンド設定](#s3-backend-configuration) 6. [LogQL クエリ](#logql-queries) 7. [Label 設計](#label-design) 8. [パフォーマンスチューニング](#performance-tuning) 9. [保持ポリシー](#retention-policies) 10. [トラブルシューティング](#troubleshooting) *** ## 概要 ### Loki の基本理念 Loki は「ログを Prometheus のように扱う」という理念で設計されています。 * **Label ベースのインデックス作成**: ログ内容ではなくメタデータ（Label）のみをインデックス化 * **コスト効率**: Elasticsearch と比べて運用コストを 10 倍以上削減 * **シンプルさ**: 全文検索エンジンの複雑さを排除 * **Grafana 統合**: ログ、メトリクス、トレースを統合分析 ### 主な機能 | 機能 | 説明 | | --------------- | --------------------------------- | | **水平スケーリング** | 各コンポーネントを個別にスケール可能 | | **マルチテナンシー** | テナント単位のデータ分離をサポート | | **オブジェクトストレージ** | S3、GCS、Azure Blob などの低コストストレージを活用 | | **LogQL** | 直感的な PromQL スタイルのクエリ言語 | | **高可用性** | 組み込みのレプリケーションとフェイルオーバー | ### Loki と Elasticsearch の比較 ``` +---------------------+------------------+------------------+ | Item | Loki | Elasticsearch | +---------------------+------------------+------------------+ | Indexing method | Labels only | Full-text | | Storage cost | Low (object) | High (SSD rec.) | | Query complexity | Simple (LogQL) | Complex (Lucene) | | Full-text search | Limited | Excellent | | Operational complex.| Low | High | | Memory requirements | Low | High | | Grafana integration | Native | Plugin | +---------------------+------------------+------------------+ ``` *** ## アーキテクチャ ### コンポーネントの概要 ```mermaid flowchart TB subgraph Clients["Clients"] PROMTAIL[Promtail] FLUENTBIT[FluentBit] ALLOY[Grafana Alloy] end subgraph Write["Write Path"] DIST[Distributor] ING[Ingester] end subgraph Read["Read Path"] QF[Query Frontend] QS[Query Scheduler] QUERIER[Querier] end subgraph Backend["Backend"] COMP[Compactor] S3[(S3 Storage)] CACHE[(Redis/Memcached)] end subgraph Viz["Visualization"] GRAFANA[Grafana] end PROMTAIL --> DIST FLUENTBIT --> DIST ALLOY --> DIST DIST --> ING ING --> S3 GRAFANA --> QF QF --> QS QS --> QUERIER QUERIER --> ING QUERIER --> S3 QUERIER --> CACHE COMP --> S3 classDef client fill:#4CAF50,stroke:#333,color:white classDef write fill:#2196F3,stroke:#333,color:white classDef read fill:#FF9800,stroke:#333,color:white classDef backend fill:#9C27B0,stroke:#333,color:white classDef viz fill:#F44336,stroke:#333,color:white class PROMTAIL,FLUENTBIT,ALLOY client class DIST,ING write class QF,QS,QUERIER read class COMP,S3,CACHE backend class GRAFANA viz ``` ### コンポーネントの詳細 #### 1. Distributor クライアントからログストリームを最初に受信するコンポーネントです。 **責務:** * ログストリームの検証 * Label の正規化 * レート制限 * 一貫性ハッシュによる Ingester へのルーティング ```yaml # Distributor configuration example distributor: ring: kvstore: store: memberlist rate_limit_strategy: local rate_limit: enabled: true # Max streams per second per tenant ingestion_rate_limit_mb: 4 ingestion_burst_size_mb: 6 ``` #### 2. Ingester ログデータをメモリにバッファリングし、長期ストレージに書き込みます。 **責務:** * ログデータチャンクの作成 * WAL（Write-Ahead Log）の管理 * チャンクのストレージへのフラッシュ * リアルタイムクエリの提供 ```yaml # Ingester configuration example ingester: lifecycler: ring: replication_factor: 3 kvstore: store: memberlist heartbeat_period: 5s chunk_idle_period: 30m chunk_block_size: 262144 chunk_retain_period: 1m max_transfer_retries: 0 wal: enabled: true dir: /var/loki/wal ``` #### 3. Querier LogQL クエリを実行して結果を返します。 **責務:** * Ingester からリアルタイムデータをクエリ * 長期ストレージから履歴データをクエリ * 結果のマージと重複排除 ```yaml # Querier configuration example querier: max_concurrent: 10 query_timeout: 5m engine: timeout: 5m max_look_back_period: 30d ``` #### 4. Query Frontend クエリの最適化とキャッシュを処理します。 **責務:** * 大規模なクエリの分割 * 結果のキャッシュ * クエリキューの管理 * リトライの処理 ```yaml # Query Frontend configuration example query_frontend: max_outstanding_per_tenant: 2048 compress_responses: true log_queries_longer_than: 5s query_stats_enabled: true ``` #### 5. Compactor 保存済みデータを最適化します。 **責務:** * 小さなチャンクを大きなチャンクにマージ * インデックスの最適化 * 保持ポリシー（データ削除）の適用 ```yaml # Compactor configuration example compactor: working_directory: /var/loki/compactor shared_store: s3 compaction_interval: 10m retention_enabled: true retention_delete_delay: 2h retention_delete_worker_count: 150 ``` *** ## デプロイモード Loki には 3 つのデプロイモードがあります。 ### 1. モノリシックモード すべてのコンポーネントが単一のプロセスで実行されます。 ```yaml # values-monolithic.yaml deploymentMode: SingleBinary singleBinary: replicas: 1 resources: limits: cpu: 2 memory: 4Gi requests: cpu: 1 memory: 2Gi loki: auth_enabled: false commonConfig: replication_factor: 1 ``` **最適な用途:** * 開発／テスト環境 * 1 日のログ量が 100GB 未満 * 迅速なプロトタイピング ### 2. シンプルスケーラブルモード（推奨） 読み取り／書き込みパスを分離してスケーラビリティを実現します。 ```yaml # values-simple-scalable.yaml deploymentMode: SimpleScalable read: replicas: 3 resources: limits: cpu: 2 memory: 4Gi requests: cpu: 1 memory: 2Gi write: replicas: 3 resources: limits: cpu: 2 memory: 4Gi requests: cpu: 1 memory: 2Gi backend: replicas: 2 resources: limits: cpu: 1 memory: 2Gi requests: cpu: 500m memory: 1Gi ``` **最適な用途:** * 本番環境 * 1 日のログ量が 100GB ～ 10TB * ほとんどの EKS クラスター ### 3. マイクロサービスモード 各コンポーネントを独立してデプロイします。 ```yaml # values-microservices.yaml deploymentMode: Distributed distributor: replicas: 3 autoscaling: enabled: true minReplicas: 3 maxReplicas: 10 ingester: replicas: 3 autoscaling: enabled: true minReplicas: 3 maxReplicas: 20 persistence: enabled: true size: 50Gi querier: replicas: 3 autoscaling: enabled: true minReplicas: 3 maxReplicas: 15 queryFrontend: replicas: 2 autoscaling: enabled: true minReplicas: 2 maxReplicas: 5 compactor: replicas: 1 ``` **最適な用途:** * 大規模な本番環境 * 1 日のログ量が 10TB 超 * コンポーネント単位のきめ細かなリソース管理 *** ## Helm インストール ### 前提条件 ```bash # Add Helm repository helm repo add grafana https://grafana.github.io/helm-charts helm repo update # Create namespace kubectl create namespace loki ``` ### シンプルスケーラブルモードのインストール（EKS 推奨） ```yaml # values-eks-production.yaml deploymentMode: SimpleScalable loki: auth_enabled: false schemaConfig: configs: - from: "2024-01-01" store: tsdb object_store: s3 schema: v13 index: prefix: loki_index_ period: 24h storage: type: s3 bucketNames: chunks: my-loki-chunks ruler: my-loki-ruler admin: my-loki-admin s3: region: ap-northeast-2 # endpoint auto-configured when using IRSA commonConfig: replication_factor: 3 limits_config: retention_period: 744h # 31 days max_query_length: 721h max_query_parallelism: 32 ingestion_rate_mb: 10 ingestion_burst_size_mb: 20 per_stream_rate_limit: 5MB per_stream_rate_limit_burst: 15MB rulerConfig: storage: type: s3 s3: bucketnames: my-loki-ruler # Read path read: replicas: 3 resources: limits: cpu: 2 memory: 4Gi requests: cpu: 1 memory: 2Gi affinity: podAntiAffinity: preferredDuringSchedulingIgnoredDuringExecution: - weight: 100 podAffinityTerm: labelSelector: matchLabels: app.kubernetes.io/component: read topologyKey: topology.kubernetes.io/zone # Write path write: replicas: 3 resources: limits: cpu: 2 memory: 4Gi requests: cpu: 1 memory: 2Gi persistence: enabled: true size: 50Gi storageClass: gp3 affinity: podAntiAffinity: preferredDuringSchedulingIgnoredDuringExecution: - weight: 100 podAffinityTerm: labelSelector: matchLabels: app.kubernetes.io/component: write topologyKey: topology.kubernetes.io/zone # Backend backend: replicas: 2 resources: limits: cpu: 1 memory: 2Gi requests: cpu: 500m memory: 1Gi persistence: enabled: true size: 20Gi storageClass: gp3 # Gateway gateway: enabled: true replicas: 2 resources: limits: cpu: 500m memory: 512Mi requests: cpu: 100m memory: 128Mi ingress: enabled: true ingressClassName: alb annotations: alb.ingress.kubernetes.io/scheme: internal alb.ingress.kubernetes.io/target-type: ip hosts: - host: loki.internal.example.com paths: - path: / pathType: Prefix # Results caching resultsCache: enabled: true defaultValidity: 12h # External Redis recommended for production # host: redis.example.com:6379 # Chunks caching chunksCache: enabled: true defaultValidity: 12h # Monitoring monitoring: serviceMonitor: enabled: true labels: release: prometheus selfMonitoring: enabled: true grafanaAgent: installOperator: false # Disable tests test: enabled: false ``` ### インストールの実行 ```bash # Install helm install loki grafana/loki \ --namespace loki \ --values values-eks-production.yaml \ --version 6.x.x # Upgrade helm upgrade loki grafana/loki \ --namespace loki \ --values values-eks-production.yaml # Check status kubectl get pods -n loki kubectl get svc -n loki ``` *** ## S3 バックエンド設定 ### IRSA（IAM Roles for Service Accounts）の設定 ```bash # 1. Create IAM policy cat > loki-s3-policy.json << 'EOF' { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "s3:ListBucket", "s3:GetBucketLocation" ], "Resource": [ "arn:aws:s3:::my-loki-chunks", "arn:aws:s3:::my-loki-ruler", "arn:aws:s3:::my-loki-admin" ] }, { "Effect": "Allow", "Action": [ "s3:PutObject", "s3:GetObject", "s3:DeleteObject" ], "Resource": [ "arn:aws:s3:::my-loki-chunks/*", "arn:aws:s3:::my-loki-ruler/*", "arn:aws:s3:::my-loki-admin/*" ] } ] } EOF aws iam create-policy \ --policy-name LokiS3Policy \ --policy-document file://loki-s3-policy.json # 2. Setup IRSA eksctl create iamserviceaccount \ --cluster=my-cluster \ --namespace=loki \ --name=loki \ --attach-policy-arn=arn:aws:iam::123456789012:policy/LokiS3Policy \ --approve ``` ### S3 Bucket の作成（Terraform） ```hcl # s3.tf resource "aws_s3_bucket" "loki_chunks" { bucket = "my-loki-chunks" tags = { Name = "Loki Chunks" Environment = "production" } } resource "aws_s3_bucket" "loki_ruler" { bucket = "my-loki-ruler" tags = { Name = "Loki Ruler" Environment = "production" } } resource "aws_s3_bucket_versioning" "loki_chunks" { bucket = aws_s3_bucket.loki_chunks.id versioning_configuration { status = "Disabled" } } resource "aws_s3_bucket_lifecycle_configuration" "loki_chunks" { bucket = aws_s3_bucket.loki_chunks.id rule { id = "transition-to-ia" status = "Enabled" transition { days = 30 storage_class = "STANDARD_IA" } transition { days = 90 storage_class = "GLACIER" } expiration { days = 365 } } } resource "aws_s3_bucket_server_side_encryption_configuration" "loki_chunks" { bucket = aws_s3_bucket.loki_chunks.id rule { apply_server_side_encryption_by_default { sse_algorithm = "AES256" } } } resource "aws_s3_bucket_public_access_block" "loki_chunks" { bucket = aws_s3_bucket.loki_chunks.id block_public_acls = true block_public_policy = true ignore_public_acls = true restrict_public_buckets = true } ``` ### Loki ストレージ設定 ```yaml # loki-config.yaml storage_config: tsdb_shipper: active_index_directory: /var/loki/tsdb-index cache_location: /var/loki/tsdb-cache shared_store: s3 aws: s3: s3://ap-northeast-2/my-loki-chunks bucketnames: my-loki-chunks region: ap-northeast-2 # access_key_id and secret_access_key not needed with IRSA s3forcepathstyle: false insecure: false sse_encryption: true boltdb_shipper: active_index_directory: /var/loki/boltdb-index cache_location: /var/loki/boltdb-cache shared_store: s3 ``` *** ## LogQL クエリ ### 基本構文 LogQL は 2 種類のクエリをサポートしています。 1. **ログクエリ**: ログ行を返す 2. **メトリクスクエリ**: ログから計算した値を返す ### ストリームセレクター ```logql # Basic stream selection {namespace="production"} # Multiple label combinations {namespace="production", app="nginx"} # Label matching operators {namespace="production", app=~"nginx|apache"} # Regex match {namespace!="kube-system"} # Negation {app!~"test.*"} # Regex negation ``` ### 行フィルター ```logql # Contains {app="nginx"} |= "error" # Does not contain {app="nginx"} != "healthcheck" # Regex match {app="nginx"} |~ "status=[45][0-9]{2}" # Regex does not match {app="nginx"} !~ "GET /health" # Chaining {app="nginx"} |= "error" != "timeout" |~ "user_id=\\d+" ``` ### パーサー ```logql # JSON parser {app="api"} | json # Extract specific fields only {app="api"} | json level, message, user_id # Logfmt parser {app="api"} | logfmt # Regex parser {app="nginx"} | regexp `(?P[\d.]+) - - \[(?P[^\]]+)\]` # Pattern parser (faster) {app="nginx"} | pattern ` - - [<_>] " <_>" ` # Unpack (Promtail pack stage result) {app="api"} | unpack ``` ### Label フィルター ```logql # Filter after JSON parsing {app="api"} | json | level="error" # Numeric comparison {app="api"} | json | response_time > 1000 # Multiple conditions {app="api"} | json | level="error" and user_id!="" # IP filtering {app="nginx"} | pattern ` - -` | ip != "10.0.0.1" ``` ### 行フォーマット ```logql # Reconstruct log line {app="api"} | json | line_format "{{.level}}: {{.message}}" # Conditional format {app="api"} | json | line_format `{{ if eq .level "error" }}ERROR: {{ end }}{{.message}}` # Template functions {app="api"} | json | line_format `{{ .timestamp | toDate "2006-01-02T15:04:05Z07:00" | date "15:04:05" }}` ``` ### メトリクスクエリ ```logql # Log lines per second rate({app="nginx"}[5m]) # Error ratio sum(rate({app="nginx"} |= "error" [5m])) / sum(rate({app="nginx"}[5m])) # Response time percentiles quantile_over_time(0.99, {app="api"} | json | unwrap response_time [5m] ) by (endpoint) # Top 10 errors topk(10, sum by (error_type) ( count_over_time({app="api"} | json | level="error" [1h]) )) # Average response size avg_over_time( {app="nginx"} | pattern `<_> <_> ` | unwrap size [5m] ) by (path) # Error count aggregation sum(count_over_time({namespace="production"} |= "error" [1h])) by (app) # Absent log detection absent_over_time({app="critical-service"}[5m]) ``` ### 実践的なクエリ例 ```logql # Analyze Kubernetes pod restart causes {namespace="production"} |= "OOMKilled" or |= "CrashLoopBackOff" # Find slow API requests {app="api"} | json | response_time > 5000 | line_format `{{.method}} {{.path}}: {{.response_time}}ms` # Track specific user activity {app="api"} | json | user_id="user-12345" | line_format `{{.timestamp}} {{.action}}` # HTTP 5xx error analysis {app="nginx"} | pattern `<_> " <_>" ` | status >= 500 # Error patterns by time sum by (hour) ( count_over_time({app="api"} |= "error" [1h]) | label_format hour="{{ __timestamp__ | date \"15\" }}" ) # Detect error spike after deployment sum(increase( count_over_time({app="api"} |= "error" [5m]) )) > 100 ``` *** ## Label 設計 ### Label 設計の原則 適切な Label 設計は Loki のパフォーマンスの鍵です。 #### 推奨する Label ```yaml # Good labels (low cardinality) labels: - namespace # ~10-50 values - app # ~50-200 values - environment # dev, staging, production - component # api, worker, scheduler - log_level # debug, info, warn, error ``` #### 避けるべき Label ```yaml # Bad labels (high cardinality) labels: - pod_name # Thousands of unique values - request_id # Unique per request - user_id # Millions of users - timestamp # Never use as label - ip_address # Very high cardinality ``` ### カーディナリティ管理 ```mermaid graph LR A[Number of Labels] --> B{Total Streams} C[Label Value Types] --> B B --> D[Index Size] B --> E[Query Performance] B --> F[Memory Usage] style B fill:#FF9800,stroke:#333 ``` **ストリーム数の計算:** ``` Total streams = namespace values x app values x component values x ... ``` **推奨事項:** * クラスターあたりの総ストリーム数: 100,000 未満 * テナントあたりのアクティブストリーム数: 10,000 未満 * Label あたりの一意な値: 1,000 未満 ### Promtail Label 設定 ```yaml # promtail-config.yaml scrape_configs: - job_name: kubernetes-pods kubernetes_sd_configs: - role: pod relabel_configs: # Namespace label - source_labels: [__meta_kubernetes_namespace] target_label: namespace # App label (from Kubernetes labels) - source_labels: [__meta_kubernetes_pod_label_app] target_label: app # Component label - source_labels: [__meta_kubernetes_pod_label_component] target_label: component # Container name - source_labels: [__meta_kubernetes_pod_container_name] target_label: container # Do not add pod_name as label (high cardinality) # Include in log line instead pipeline_stages: - json: expressions: level: level - labels: level: ``` ### 動的 Label 付与 ```yaml # Extract labels from log content pipeline_stages: - json: expressions: level: level service: service - labels: level: service: # High cardinality values as structured metadata - structured_metadata: user_id: request_id: ``` *** ## パフォーマンスチューニング ### Ingester チューニング ```yaml ingester: # Chunk settings chunk_idle_period: 30m # Wait time before flushing idle stream chunk_block_size: 262144 # Chunk block size (256KB) chunk_target_size: 1572864 # Target chunk size (1.5MB) chunk_retain_period: 1m # Memory retention time after flush # Concurrency max_chunk_age: 2h # Maximum chunk age concurrent_flushes: 32 # Concurrent flush count # WAL wal: enabled: true dir: /var/loki/wal flush_on_shutdown: true replay_memory_ceiling: 4GB ``` ### Querier チューニング ```yaml querier: max_concurrent: 16 # Concurrent queries query_timeout: 5m # Query timeout engine: timeout: 5m max_look_back_period: 30d query_range: align_queries_with_step: true cache_results: true max_retries: 5 parallelise_shardable_queries: true results_cache: cache: embedded_cache: enabled: true max_size_mb: 500 ``` ### Frontend チューニング ```yaml query_frontend: max_outstanding_per_tenant: 4096 compress_responses: true log_queries_longer_than: 10s # Query splitting split_queries_by_interval: 30m query_scheduler: max_outstanding_requests_per_tenant: 2048 grpc_client_config: max_recv_msg_size: 104857600 # 100MB ``` ### リソースのガイドライン ```yaml # Small (daily < 100GB) write: replicas: 2 resources: requests: cpu: 500m memory: 1Gi limits: cpu: 1 memory: 2Gi read: replicas: 2 resources: requests: cpu: 500m memory: 1Gi limits: cpu: 1 memory: 2Gi --- # Medium (daily 100GB - 1TB) write: replicas: 3 resources: requests: cpu: 1 memory: 2Gi limits: cpu: 2 memory: 4Gi read: replicas: 3 resources: requests: cpu: 1 memory: 2Gi limits: cpu: 2 memory: 4Gi --- # Large (daily > 1TB) write: replicas: 5 autoscaling: enabled: true minReplicas: 5 maxReplicas: 20 resources: requests: cpu: 2 memory: 4Gi limits: cpu: 4 memory: 8Gi read: replicas: 5 autoscaling: enabled: true minReplicas: 5 maxReplicas: 15 resources: requests: cpu: 2 memory: 4Gi limits: cpu: 4 memory: 8Gi ``` *** ## 保持ポリシー ### グローバル保持ポリシー ```yaml # loki-config.yaml limits_config: retention_period: 744h # 31 days (default) compactor: working_directory: /var/loki/compactor shared_store: s3 retention_enabled: true retention_delete_delay: 2h retention_delete_worker_count: 150 delete_request_store: s3 ``` ### テナント別保持ポリシー ```yaml # runtime-config.yaml overrides: tenant-production: retention_period: 2160h # 90 days tenant-development: retention_period: 168h # 7 days tenant-compliance: retention_period: 8760h # 365 days ``` ### ストリーム別保持ポリシー ```yaml limits_config: retention_stream: - selector: '{namespace="production", level="error"}' priority: 1 period: 2160h # 90 days - production errors - selector: '{namespace="development"}' priority: 2 period: 72h # 3 days - development - selector: '{app="audit-log"}' priority: 1 period: 8760h # 365 days - audit logs ``` *** ## トラブルシューティング ### よくある問題と解決策 #### 1. "too many outstanding requests" ```yaml # Symptom: Query failures, 503 errors # Cause: Frontend/scheduler overload # Solution query_frontend: max_outstanding_per_tenant: 4096 # Increase from default 2048 query_scheduler: max_outstanding_requests_per_tenant: 2048 # Or increase querier replicas querier: replicas: 5 # From 3 to 5 ``` #### 2. "rate limit exceeded" ```yaml # Symptom: Log collection failures, 429 errors # Cause: Ingestion rate limit exceeded # Solution limits_config: ingestion_rate_mb: 20 # Increase from default 4 ingestion_burst_size_mb: 30 # Increase from default 6 per_stream_rate_limit: 10MB # Per-stream limit per_stream_rate_limit_burst: 30MB ``` #### 3. "max streams limit exceeded" ```yaml # Symptom: New stream creation fails # Cause: High cardinality labels # Solution 1: Increase limit (temporary) limits_config: max_streams_per_user: 20000 # Default 10000 # Solution 2: Reduce label cardinality (recommended) # Remove high cardinality labels in promtail config ``` #### 4. クエリパフォーマンスの低下 ```bash # Diagnostics # 1. Check query stats curl -s "http://loki:3100/loki/api/v1/query_range" \ -G --data-urlencode 'query={app="nginx"}' \ --data-urlencode 'start=1h' | jq '.data.stats' # 2. Check stream count curl -s "http://loki:3100/loki/api/v1/series" \ -G --data-urlencode 'match[]={namespace="production"}' | jq '.data | length' ``` ```yaml # Solution query_range: parallelise_shardable_queries: true split_queries_by_interval: 15m # From 30m to 15m limits_config: max_query_parallelism: 64 # From 32 to 64 ``` #### 5. Ingester OOM ```yaml # Symptom: Ingester pod restarts, OOM Killed # Cause: Insufficient memory settings or chunk configuration issues # Solution 1: Increase memory ingester: resources: limits: memory: 8Gi # Increase from 4Gi requests: memory: 4Gi # Solution 2: Adjust chunk settings ingester: chunk_idle_period: 15m # Decrease from 30m chunk_target_size: 1048576 # Smaller chunks max_chunk_age: 1h # Decrease from 2h ``` ### 便利な診断コマンド ```bash # Check Loki status kubectl exec -it loki-read-0 -n loki -- wget -qO- http://localhost:3100/ready # Check ring membership kubectl exec -it loki-write-0 -n loki -- wget -qO- http://localhost:3100/ring # Check flush status kubectl exec -it loki-write-0 -n loki -- wget -qO- http://localhost:3100/flush # Check metrics kubectl exec -it loki-write-0 -n loki -- wget -qO- http://localhost:3100/metrics | grep loki_ingester # Check configuration kubectl exec -it loki-read-0 -n loki -- wget -qO- http://localhost:3100/config ``` ### Grafana Dashboard の設定 ```json { "annotations": { "list": [] }, "panels": [ { "title": "Ingestion Rate", "targets": [ { "expr": "sum(rate(loki_distributor_bytes_received_total[5m]))", "legendFormat": "bytes/s" } ] }, { "title": "Active Streams", "targets": [ { "expr": "sum(loki_ingester_memory_streams)", "legendFormat": "streams" } ] }, { "title": "Query Latency", "targets": [ { "expr": "histogram_quantile(0.99, sum(rate(loki_request_duration_seconds_bucket{route=~\"loki_api_v1_query.*\"}[5m])) by (le))", "legendFormat": "p99" } ] } ] } ``` *** ## ベストプラクティスのまとめ ### 推奨事項 1. **Label を最小限に保つ**: namespace、app、component、level のみを使用 2. **JSON ロギングを採用する**: 構造化ログでパースのオーバーヘッドを削減 3. **S3 ライフサイクルを設定する**: コスト最適化のために階層化を設定 4. **IRSA を使用する**: Access Key ではなく IAM Role を使用 5. **キャッシュを有効にする**: クエリ結果とチャンクのキャッシュでパフォーマンスを改善 6. **監視を設定する**: Loki 自身のメトリクスを収集し、アラートを設定 ### 避けるべき事項 1. **高カーディナリティの Label を避ける**: pod\_name、request\_id など 2. **無制限のクエリ範囲を避ける**: 時間範囲の制限は必須 3. **単一ノードのデプロイを避ける**: 本番環境では最低 3 レプリカ 4. **WAL を無効にしない**: データ損失の防止に不可欠 5. **リソース制限なしでデプロイしない**: OOM を防止 *** ## クイズ [Loki クイズ](/kubernetes/jp/kuizu/ke-guan-ce-xing/rogu/01-loki-quiz.md)で理解度を確認しましょう。