Elasticsearch¶

Overview¶

Elasticsearch serves as the log storage and search engine for the ELK stack, providing full-text search and analytics capabilities.

Architecture¶

graph TB
    subgraph Cluster
        N1[Node 1]
        N2[Node 2]
        N3[Node 3]
    end

    subgraph Indices
        I1[logs-*]
        I2[metrics-*]
        I3[.fleet-*]
    end

    Agents[Elastic Agents] --> N1
    N1 --> I1
    N2 --> I2
    N3 --> I3

    Kibana --> N1

Deployment¶

Kubernetes Configuration¶

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: elasticsearch
  namespace: monitoring
spec:
  serviceName: elasticsearch
  replicas: 1
  template:
    spec:
      containers:
      - name: elasticsearch
        image: docker.elastic.co/elasticsearch/elasticsearch:8.17.0
        env:
        - name: discovery.type
          value: single-node
        - name: xpack.security.enabled
          value: "true"

Index Management¶

Index Patterns¶

Pattern	Data Type	Retention
`logs-*`	Application logs	30 days
`metrics-*`	System metrics	15 days
`.fleet-*`	Fleet management	7 days
`filebeat-*`	Filebeat logs	30 days

Index Lifecycle Management¶

{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": {
            "max_age": "1d",
            "max_size": "50gb"
          }
        }
      },
      "delete": {
        "min_age": "30d",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}

Data Streams¶

Log Data Stream¶

Name: logs-generic-default
Index pattern: logs-generic-default-*
Lifecycle: Managed by ILM

Metrics Data Stream¶

Name: metrics-system.cpu-default
Index pattern: metrics-system.cpu-default-*

Ingest Pipelines¶

Log Processing Pipeline¶

{
  "description": "Parse log messages",
  "processors": [
    {
      "grok": {
        "field": "message",
        "patterns": ["%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:level} %{GREEDYDATA:message}"]
      }
    },
    {
      "date": {
        "field": "timestamp",
        "formats": ["ISO8601"]
      }
    }
  ]
}

Queries¶

Search Query¶

GET /logs-*/_search
{
  "query": {
    "bool": {
      "must": [
        {"match": {"kubernetes.namespace": "hub"}},
        {"range": {"@timestamp": {"gte": "now-1h"}}}
      ]
    }
  }
}

Aggregation Query¶

GET /logs-*/_search
{
  "size": 0,
  "aggs": {
    "logs_by_level": {
      "terms": {"field": "log.level"}
    }
  }
}

API Endpoints¶

Endpoint	Description
`/_cluster/health`	Cluster health
`/_cat/indices`	List indices
`/_cat/nodes`	List nodes
`/_search`	Search documents

Monitoring¶

Key Metrics¶

Metric	Description	Alert Threshold
Cluster status	Health color	Yellow/Red
JVM heap	Memory usage	>85%
Disk usage	Storage used	>80%
Indexing rate	Docs/sec	N/A

Health Check¶

curl -X GET "http://elasticsearch:9200/_cluster/health?pretty"

Security¶

Authentication¶

Built-in users: elastic, kibana_system
API keys: For service authentication

Roles¶

Role	Permissions
superuser	Full access
kibana_system	Kibana access
monitoring_user	Read monitoring

Backup¶

Snapshot Repository¶

PUT /_snapshot/backup
{
  "type": "fs",
  "settings": {
    "location": "/backup/elasticsearch"
  }
}

Create Snapshot¶

PUT /_snapshot/backup/snapshot_1
{
  "indices": "logs-*,metrics-*"
}

Troubleshooting¶

Common Issues¶

Issue	Cause	Resolution
Yellow status	Unassigned replicas	Single node - expected
High heap	Large queries	Increase memory limit
Slow queries	Large indices	Add more shards