Differences between revisions 1 and 2
Revision 1 as of 2019-07-02 20:55:13
Size: 804
Editor: twotwo
Comment:
Revision 2 as of 2019-07-22 20:13:40
Size: 6828
Editor: twotwo
Comment:
Deletions are marked like this. Additions are marked like this.
Line 6: Line 6:
[[https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-getting-started.html||target="_blank"]] [[https://www.elastic.co/guide/en/beats/filebeat/7.x/filebeat-getting-started.html||target="_blank"]] Guide for v7.x
Line 8: Line 8:
== filebeat -> elasticsearch -> kibana ==
{{attachment:filebeat-es-kibana.png}}

=== Setting up and running Filebeat ===
==== Install on Host ====
[[https://www.elastic.co/guide/en/beats/filebeat/7.x/filebeat-installation.html||target="_blank"]]

{{{#!highlight bash numbers=disable
# Debian
curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.2.0-amd64.deb
sudo dpkg -i filebeat-7.2.0-amd64.deb
# Homebrew
brew tap elastic/tap
brew install elastic/tap/filebeat-full
}}}

[[https://www.elastic.co/guide/en/beats/filebeat/current/command-line-options.html||target="_blank"]] Filebeat command reference

[[https://www.elastic.co/guide/en/beats/filebeat/current/directory-layout.html||target="_blank"]] Directory layout

[[https://www.elastic.co/guide/en/beats/filebeat/master/regexp-support.html||target="_blank"]] Regular expression support

{{{#!highlight yaml numbers=disable
filebeat.config:
  modules:
    path: ${path.config}/modules.d/*.yml
    reload.enabled: false

filebeat.autodiscover:
  providers:
    - type: docker
      hints.enabled: false`

filebeat.inputs:
- type: container
  enabled: true
  stream: stdout
  tags: ["json"]
  include_lines: ['.*HTTP/1.1.*', '.*HosCode.*']
  exclude_lines: ['.*monitoring.*']
  paths:
    - '/var/lib/docker/containers/*/*.log'

output.console:
  enabled: true
  # pretty: true

}}}

{{{#!highlight bash numbers=disable
filebeat --path.config `pwd` --path.data /tmp/beat/data --path.logs /tmp/beat/log -c <filebeat.yml>
}}}

==== Running on Docker ====
[[https://www.elastic.co/guide/en/beats/filebeat/7.x/running-on-docker.html||target="_blank"]]

{{{#!highlight bash numbers=disable
# Pulling the image
docker pull docker.elastic.co/beats/filebeat:7.2.0
# Download this example configuration file
curl -L -O https://raw.githubusercontent.com/elastic/beats/7.2/deploy/docker/filebeat.docker.yml
# Run
docker run -d \
  --name=filebeat \
  --user=root \
  --volume="$(pwd)/filebeat.docker.yml:/usr/share/filebeat/filebeat.yml:ro" \
  --volume="/var/lib/docker/containers:/var/lib/docker/containers:ro" \
  --volume="/var/run/docker.sock:/var/run/docker.sock:ro" \
  docker.elastic.co/beats/filebeat:7.2.0 filebeat -e -strict.perms=false \
  -E output.elasticsearch.hosts=["elasticsearch:9200"]
}}}

{{{#!highlight yaml numbers=disable
  filebeat:
    container_name: filebeat
    hostname: filebeat
    #To read the docker socket
    user: root
    image: docker.elastic.co/beats/filebeat:7.2.0
    environment:
      - ELASTICSEARCH_HOSTS=elasticsearch
    volumes:
      # config of filebeat
      - ./filebeat.docker.yml:/usr/share/filebeat/filebeat.yml:ro
      # needed to persist filebeat tracking data :
      - "filebeat_data:/usr/share/filebeat/data:rw"
      # needed to access all docker logs (read only) :
      - /var/lib/docker/containers:/var/lib/docker/containers:ro
      # needed to access additional informations about containers:
      - /var/run/docker.sock:/var/run/docker.sock:ro
    # command: filebeat -e -strict.perms=false
    command: bash -c "filebeat -e -strict.perms=false >> logs/console.log 2>&1"
    restart: always
}}}

==== Running on Kubernetes ====
[[https://www.elastic.co/guide/en/beats/filebeat/7.x/running-on-docker.html||target="_blank"]]

=== Configure Filebeat ===
[[https://www.elastic.co/guide/en/beats/filebeat/7.x/running-on-kubernetes.html||target="_blank"]] `filebeat-kubernetes.yaml`

`curl -L -O https://raw.githubusercontent.com/elastic/beats/7.x/deploy/kubernetes/filebeat-kubernetes.yaml`

==== Define Inputs ====
[[https://www.elastic.co/guide/en/beats/filebeat/master/filebeat-input-container.html||target="_blank"]] `Container input` <== 读取 docker std out/err

{{{#!highlight yaml
filebeat.inputs:
- type: container
  enabled: true
  include_lines: ['sometext']
  paths:
    - '/var/lib/docker/containers/*/*.log'
}}}

==== Define Outputs ====
[[https://www.elastic.co/guide/en/beats/filebeat/7.x/configuring-output.html||target="_blank"]]

[[https://www.elastic.co/guide/en/beats/filebeat/7.x/console-output.html||target="_blank"]] Configure the Console output

{{{#!highlight yaml
output.elasticsearch:
  hosts: ["myEShost:9200"]
}}}

==== Setup to Kibana ====
[[https://www.elastic.co/guide/en/beats/filebeat/7.x/load-kibana-dashboards.html||target="_blank"]]

{{{#!highlight yaml
setup.kibana:
  host: "mykibanahost:5601"
  username: "my_kibana_user"
  password: "YOUR_PASSWORD"
}}}

=== Load ES index template ===
[[https://www.elastic.co/guide/en/beats/filebeat/7.x/filebeat-template.html||target="_blank"]]

Change the index name

{{{#!highlight yaml
output.elasticsearch.index: "customname-%{[agent.version]}-%{+yyyy.MM.dd}"
setup.template.name: "customname"
setup.template.pattern: "customname-*"

setup.dashboards.index: "customname-*"
}}}

=== Set up the Kibana dashboards ===
[[https://www.elastic.co/guide/en/beats/filebeat/7.x/load-kibana-dashboards.html||target="_blank"]]

=== View the sample Kibana dashboards ===
[[https://www.elastic.co/guide/en/beats/filebeat/7.x/view-kibana-dashboards.html||target="_blank"]]

{{{#!highlight yaml
output.elasticsearch.index: "customname-%{[agent.version]}-%{+yyyy.MM.dd}"
setup.template.name: "customname"
setup.template.pattern: "customname-*"

setup.dashboards.index: "customname-*"
}}}

== Modules ==
[[https://www.elastic.co/guide/en/beats/filebeat/7.x/filebeat-modules.html||target="_blank"]]

=== RabbitMQ module ===
Read logs of RabbitMQ

== Use Cases ==
=== filebeat -> elasticsearch -> kibana ===
[[https://medium.com/@sanjay.rajak/design-centralize-logging-architecture-using-filebeat-elasticsearch-kibana-6704fe01b7a9||target="_blank"]] Design centralize logging architecture using Filebeat →ElasticSearch → Kibana
 . {{attachment:filebeat-es-kibana.png}}

=== Docker JSON File Logging Driver with Filebeat as a docker container ===
[[https://medium.com/@bcoste/powerful-logging-with-docker-filebeat-and-elasticsearch-8ad021aecd87||target="_blank"]] Powerful logging with Docker, Filebeat and Elasticsearch

[[https://docs.docker.com/config/containers/logging/json-file/||target="_blank"]]
 . {{attachment:docker-filebeat.png}}
Line 13: Line 187:
 * [[https://github.com/elastic/beats||target="_blank"]] 
 * [[https://medium.com/@sanjay.rajak/design-centralize-logging-architecture-using-filebeat-elasticsearch-kibana-6704fe01b7a9||target="_blank"]] Design centralize logging architecture using Filebeat →ElasticSearch → Kibana
 * [[https://github.com/elastic/beats||target="_blank"]]
 * [[https://github.com/twotwo/efk-startup/tree/master/filebeat||target="_blank"]]

Back to Elastic Stack

See Also Centralized Logging ArchitectureKibanaElasticsearchLogstash

FileBeat

1. Getting Started

https://www.elastic.co/guide/en/beats/filebeat/7.x/filebeat-getting-started.html Guide for v7.x

1.1. Setting up and running Filebeat

1.1.1. Install on Host

https://www.elastic.co/guide/en/beats/filebeat/7.x/filebeat-installation.html

# Debian
curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.2.0-amd64.deb
sudo dpkg -i filebeat-7.2.0-amd64.deb
# Homebrew
brew tap elastic/tap
brew install elastic/tap/filebeat-full

https://www.elastic.co/guide/en/beats/filebeat/current/command-line-options.html Filebeat command reference

https://www.elastic.co/guide/en/beats/filebeat/current/directory-layout.html Directory layout

https://www.elastic.co/guide/en/beats/filebeat/master/regexp-support.html Regular expression support

filebeat.config:
  modules:
    path: ${path.config}/modules.d/*.yml
    reload.enabled: false

filebeat.autodiscover:
  providers:
    - type: docker
      hints.enabled: false`

filebeat.inputs:
- type: container
  enabled: true
  stream: stdout
  tags: ["json"]
  include_lines: ['.*HTTP/1.1.*', '.*HosCode.*']
  exclude_lines: ['.*monitoring.*']
  paths: 
    - '/var/lib/docker/containers/*/*.log'

output.console:
  enabled: true
  # pretty: true

filebeat --path.config `pwd` --path.data /tmp/beat/data --path.logs /tmp/beat/log -c <filebeat.yml>

1.1.2. Running on Docker

https://www.elastic.co/guide/en/beats/filebeat/7.x/running-on-docker.html

# Pulling the image
docker pull docker.elastic.co/beats/filebeat:7.2.0
# Download this example configuration file
curl -L -O https://raw.githubusercontent.com/elastic/beats/7.2/deploy/docker/filebeat.docker.yml
# Run
docker run -d \
  --name=filebeat \
  --user=root \
  --volume="$(pwd)/filebeat.docker.yml:/usr/share/filebeat/filebeat.yml:ro" \
  --volume="/var/lib/docker/containers:/var/lib/docker/containers:ro" \
  --volume="/var/run/docker.sock:/var/run/docker.sock:ro" \
  docker.elastic.co/beats/filebeat:7.2.0 filebeat -e -strict.perms=false \
  -E output.elasticsearch.hosts=["elasticsearch:9200"]

  filebeat:
    container_name: filebeat
    hostname: filebeat
    #To read the docker socket
    user: root 
    image: docker.elastic.co/beats/filebeat:7.2.0
    environment:
      - ELASTICSEARCH_HOSTS=elasticsearch
    volumes:
      # config of filebeat
      - ./filebeat.docker.yml:/usr/share/filebeat/filebeat.yml:ro
      # needed to persist filebeat tracking data :
      - "filebeat_data:/usr/share/filebeat/data:rw"
      # needed to access all docker logs (read only) :
      - /var/lib/docker/containers:/var/lib/docker/containers:ro
      # needed to access additional informations about containers:
      - /var/run/docker.sock:/var/run/docker.sock:ro
    # command: filebeat -e -strict.perms=false
    command: bash -c "filebeat -e -strict.perms=false >> logs/console.log 2>&1"
    restart: always

1.1.3. Running on Kubernetes

https://www.elastic.co/guide/en/beats/filebeat/7.x/running-on-docker.html

1.2. Configure Filebeat

https://www.elastic.co/guide/en/beats/filebeat/7.x/running-on-kubernetes.html filebeat-kubernetes.yaml

curl -L -O https://raw.githubusercontent.com/elastic/beats/7.x/deploy/kubernetes/filebeat-kubernetes.yaml

1.2.1. Define Inputs

https://www.elastic.co/guide/en/beats/filebeat/master/filebeat-input-container.html Container input <== 读取 docker std out/err

   1 filebeat.inputs:
   2 - type: container
   3   enabled: true
   4   include_lines: ['sometext']
   5   paths: 
   6     - '/var/lib/docker/containers/*/*.log'

1.2.2. Define Outputs

https://www.elastic.co/guide/en/beats/filebeat/7.x/configuring-output.html

https://www.elastic.co/guide/en/beats/filebeat/7.x/console-output.html Configure the Console output

   1 output.elasticsearch:
   2   hosts: ["myEShost:9200"]

1.2.3. Setup to Kibana

https://www.elastic.co/guide/en/beats/filebeat/7.x/load-kibana-dashboards.html

   1 setup.kibana:
   2   host: "mykibanahost:5601"
   3   username: "my_kibana_user"  
   4   password: "YOUR_PASSWORD"

1.3. Load ES index template

https://www.elastic.co/guide/en/beats/filebeat/7.x/filebeat-template.html

Change the index name

   1 output.elasticsearch.index: "customname-%{[agent.version]}-%{+yyyy.MM.dd}"
   2 setup.template.name: "customname"
   3 setup.template.pattern: "customname-*"
   4 
   5 setup.dashboards.index: "customname-*"

1.4. Set up the Kibana dashboards

https://www.elastic.co/guide/en/beats/filebeat/7.x/load-kibana-dashboards.html

1.5. View the sample Kibana dashboards

https://www.elastic.co/guide/en/beats/filebeat/7.x/view-kibana-dashboards.html

   1 output.elasticsearch.index: "customname-%{[agent.version]}-%{+yyyy.MM.dd}"
   2 setup.template.name: "customname"
   3 setup.template.pattern: "customname-*"
   4 
   5 setup.dashboards.index: "customname-*"

2. Modules

https://www.elastic.co/guide/en/beats/filebeat/7.x/filebeat-modules.html

2.1. RabbitMQ module

Read logs of RabbitMQ

3. Use Cases

3.1. filebeat -> elasticsearch -> kibana

https://medium.com/@sanjay.rajak/design-centralize-logging-architecture-using-filebeat-elasticsearch-kibana-6704fe01b7a9 Design centralize logging architecture using Filebeat →ElasticSearch → Kibana

  • filebeat-es-kibana.png

3.2. Docker JSON File Logging Driver with Filebeat as a docker container

https://medium.com/@bcoste/powerful-logging-with-docker-filebeat-and-elasticsearch-8ad021aecd87 Powerful logging with Docker, Filebeat and Elasticsearch

https://docs.docker.com/config/containers/logging/json-file/

  • docker-filebeat.png

4. Reference

MainWiki: FileBeat (last edited 2019-07-22 20:13:40 by twotwo)