ELK Stack Filebeat to Elasticsearch Setup

Prerequisites

Before touching a single config file, let's get the environment sorted. I've seen engineers jump straight into Filebeat installation only to spend two hours debugging connection refused errors that turned out to be firewall rules blocking port 9200. Save yourself the frustration and check the basics first.

This guide assumes you have Elasticsearch 8.x already running and reachable on your network. I'm working with a setup where Elasticsearch lives on

192.168.10.20

and Filebeat is being configured on the host sw-infrarunbook-01 at

192.168.10.25

. Both are running Ubuntu 22.04 LTS.

Here's what you need before starting:

• Elasticsearch 8.x running and healthy — verify with a curl to port 9200 before anything else
• Kibana 8.x installed — technically optional, but you'll regret skipping it when you need to verify ingestion visually
• A service account with credentials — I'm using infrarunbook-admin throughout this guide
• Root or sudo access on the Filebeat host
• The Elasticsearch CA certificate if you're running TLS (and in 8.x, you are — it's on by default)
• Network connectivity with port 9200 open between the Filebeat host and the Elasticsearch node

One thing worth calling out: Filebeat does not require Java. It's written in Go and ships as a single binary, which is one of the reasons it's the right choice for lightweight log shipping. Don't let anyone talk you into using Logstash just for forwarding unless you genuinely need its processing power — Filebeat with ingest pipelines handles the vast majority of use cases with a fraction of the resource overhead.

Verify Elasticsearch is healthy before going any further:

curl -u infrarunbook-admin:changeme \
  --cacert /etc/elasticsearch/certs/http_ca.crt \
  https://192.168.10.20:9200/_cluster/health?pretty

You want to see

"status" : "green"

or at minimum

"yellow"

. If you're getting a connection error or a TLS handshake failure, fix that first. Filebeat will fail silently for a while before you realize it's not shipping anything.

---

Step-by-Step Setup

Step 1: Install Filebeat

Elastic maintains its own APT and RPM repositories, and using them is the cleanest approach — it gives you easy upgrades and ensures version consistency across your fleet. On sw-infrarunbook-01, run the following:

wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | \
  sudo gpg --dearmor -o /usr/share/keyrings/elasticsearch-keyring.gpg

echo "deb [signed-by=/usr/share/keyrings/elasticsearch-keyring.gpg] \
https://artifacts.elastic.co/packages/8.x/apt stable main" | \
  sudo tee /etc/apt/sources.list.d/elastic-8.x.list

sudo apt-get update && sudo apt-get install filebeat

After installation, don't start it yet. The default configuration attempts to connect to localhost on port 9200, which isn't where your Elasticsearch lives. Starting prematurely means you'll get a flood of errors in the journal and potentially put the systemd unit into a fast restart loop before you've even looked at the config.

Step 2: Get the CA Certificate from Elasticsearch

Elasticsearch 8.x enables TLS by default. That's great for security but means Filebeat needs to trust the CA. If you're running a self-managed cluster, the CA certificate lives on the Elasticsearch node at

/etc/elasticsearch/certs/http_ca.crt

. Copy it to the Filebeat host:

sudo mkdir -p /etc/filebeat/certs
sudo chmod 750 /etc/filebeat/certs

sudo scp infrarunbook-admin@192.168.10.20:/etc/elasticsearch/certs/http_ca.crt \
  /etc/filebeat/certs/http_ca.crt

sudo chown root:root /etc/filebeat/certs/http_ca.crt
sudo chmod 640 /etc/filebeat/certs/http_ca.crt

Get the permissions right on this file. The filebeat process runs as the

filebeat

user and needs to read the cert at runtime. This is one of those small details that bites you at 2am when a cert rotation script copies a new cert with root-only permissions and suddenly nothing ships.

Step 3: Configure the Elasticsearch Output

Open

/etc/filebeat/filebeat.yml

. The default file is heavily commented, which is helpful the first time you read it and maddening the tenth. Find the output section and configure it like this:

output.elasticsearch:
  hosts: ["https://192.168.10.20:9200"]
  protocol: "https"
  username: "infrarunbook-admin"
  password: "changeme"
  ssl.certificate_authorities:
    - /etc/filebeat/certs/http_ca.crt

In my experience, the most common mistake here is omitting

protocol: "https"

and then wondering why connections are being rejected. Elasticsearch 8 won't accept plain HTTP on port 9200 by default. If you see

received plaintext http traffic on an https channel

in the Elasticsearch logs, that's exactly what's happening.

Step 4: Configure the Input

The inputs section tells Filebeat which files to tail and how to handle them. Use the

filestream

type — the older

log

type is deprecated and handles log rotation poorly, which we'll cover in the common mistakes section.

filebeat.inputs:
- type: filestream
  id: syslog-input
  enabled: true
  paths:
    - /var/log/syslog
    - /var/log/auth.log
  fields:
    source_host: sw-infrarunbook-01
    environment: production
  fields_under_root: true

The

id

field on each filestream input must be unique. Filebeat uses it to namespace registry entries, and duplicate IDs will cause one input to silently override the other's position tracking. I've seen this cause duplicate events after a restart when someone copied an input block and forgot to update the ID.

Setting

fields_under_root: true

puts your custom fields at the top level of the Elasticsearch document instead of nested under a

fields

key. This makes KQL queries cleaner and prevents broken Kibana dashboards that expect top-level fields.

Step 5: Run Setup Before First Start

Filebeat can push its index template and ILM policy to Elasticsearch automatically. Run setup before starting the service — this ensures the index mappings are correct before any documents arrive. Without the template, fields that should be keywords get analyzed as full text, date fields may not parse correctly, and you'll have a miserable time aggregating anything in Kibana.

sudo filebeat setup -e

The

-e

flag sends logs to stderr so you can watch what's happening in real time. Look for

Successfully loaded template

and

ILM policy successfully loaded

in the output. If you see errors here, fix them before proceeding — don't start the service with a broken template.

Step 6: Start and Enable Filebeat

sudo systemctl enable filebeat
sudo systemctl start filebeat

Give it fifteen seconds, then check the status. Don't immediately jump to the logs — the systemd status output tells you whether the process is running or exited, which is the first thing you need to know.

sudo systemctl status filebeat

---

Full Configuration Example

Here's the complete

/etc/filebeat/filebeat.yml

for sw-infrarunbook-01, stripped of comments and ready to deploy. This configuration ships system logs and application logs from a custom path, with multiline handling for stack traces:

filebeat.inputs:
- type: filestream
  id: syslog-input
  enabled: true
  paths:
    - /var/log/syslog
    - /var/log/auth.log
  fields:
    source_host: sw-infrarunbook-01
    log_type: system
    environment: production
  fields_under_root: true

- type: filestream
  id: app-log-input
  enabled: true
  paths:
    - /var/log/solvethenetwork/*.log
  fields:
    source_host: sw-infrarunbook-01
    log_type: application
    environment: production
  fields_under_root: true
  parsers:
    - multiline:
        type: pattern
        pattern: '^\d{4}-\d{2}-\d{2}'
        negate: true
        match: after

filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false

setup.template.settings:
  index.number_of_shards: 1
  index.number_of_replicas: 1

setup.kibana:
  host: "https://192.168.10.21:5601"
  username: "infrarunbook-admin"
  password: "changeme"
  ssl.certificate_authorities:
    - /etc/filebeat/certs/http_ca.crt

output.elasticsearch:
  hosts: ["https://192.168.10.20:9200"]
  protocol: "https"
  username: "infrarunbook-admin"
  password: "changeme"
  ssl.certificate_authorities:
    - /etc/filebeat/certs/http_ca.crt

setup.ilm.enabled: true

processors:
  - add_host_metadata:
      when.not.contains.tags: forwarded
  - add_cloud_metadata: ~
  - drop_fields:
      fields: ["agent.ephemeral_id", "ecs.version"]
      ignore_missing: true

logging.level: info
logging.to_files: true
logging.files:
  path: /var/log/filebeat
  name: filebeat
  keepfiles: 7
  permissions: 0640

The multiline parser on the application log input groups lines together when they don't start with a date pattern. This is critical for Java or Python applications where a single error event can span twenty lines. Without multiline handling, each line becomes a separate Elasticsearch document and stack traces become useless fragments that are nearly impossible to correlate into a coherent picture.

The

processors

block adds host metadata automatically and strips a couple of fields that are more noise than signal in most environments. The

ecs.version

and

agent.ephemeral_id

fields in particular aren't surfaced in any user-facing Kibana view I've ever actually used. Adjust to match what your dashboards actually reference.

---

Verification Steps

Don't trust

active (running)

as proof that logs are being ingested. I've seen Filebeat stay in a running state while silently failing to ship anything because of a permissions problem on the log files it couldn't read. Verification is a separate step from starting the service, and skipping it will cost you.

Test the output connection before anything else:

sudo filebeat test output

This command is criminally underused. It attempts to connect to Elasticsearch and prints exactly what happens — TLS certificate validation, authentication, TCP connectivity. Run it before starting the service and you'll save yourself significant log-digging time. A successful run looks like this:

elasticsearch: https://192.168.10.20:9200...
  parse url... OK
  connection...
    parse host... OK
    dns lookup... OK
    addresses: 192.168.10.20
    dial up... OK
  TLS...
    security: server's certificate chain verification is enabled
    handshake... OK
  talk to server... OK
  version: 8.13.0

Validate the config file:

sudo filebeat test config

YAML is unforgiving about indentation and Filebeat's config uses indentation to define hierarchy. A single misaligned line will either silently fall back to defaults or prevent startup entirely. Run this any time you make a config change.

Watch the Filebeat logs in real time:

sudo journalctl -u filebeat -f

Look for

Connection to backoff(elasticsearch(https://192.168.10.20:9200)) established

— that confirms the output is live. You should also see

Non-zero metrics in the last 30s

appearing every thirty seconds. This is the stats line and it will show you events being read and published. If you see the connection established message but the metrics show zero events published, the problem is on the input side — Filebeat can connect to Elasticsearch but can't read the source files.

Query Elasticsearch directly to confirm documents are landing:

curl -u infrarunbook-admin:changeme \
  --cacert /etc/filebeat/certs/http_ca.crt \
  "https://192.168.10.20:9200/filebeat-*/_count?pretty"

Run this twice a minute apart and compare the document counts. A rising count confirms active ingestion. If the count is stuck at zero or not growing, work backwards — check Filebeat logs, then check whether the source files are being written to, then check file permissions.

Check that the filebeat user can actually read the log files:

sudo -u filebeat cat /var/log/syslog | head -5

If this returns a permission denied error, add the filebeat user to the

adm

group — that's the group that owns most system logs on Debian-based systems:

sudo usermod -aG adm filebeat
sudo systemctl restart filebeat

---

Common Mistakes

I've helped a lot of people debug Filebeat setups over the years. The same problems come up repeatedly, and almost all of them are preventable with a bit of upfront awareness.

Version mismatch between Filebeat and Elasticsearch. This is the big one. Filebeat 7.x will not communicate cleanly with Elasticsearch 8.x — the index template format changed and there are API incompatibilities that produce confusing errors. Always match major versions. If your cluster is running 8.13, install Filebeat 8.13. Check what you have with

filebeat version

before spending time debugging anything else.

Two outputs enabled simultaneously. Filebeat supports exactly one active output. If you have both

output.logstash

and

output.elasticsearch

uncommented in the config, it won't start. The error message is clear about this, but it regularly gets missed because people are scanning for something more cryptic. Scan for lines starting with

output.

and make sure only one block is uncommented.

Skipping

filebeat setup

before the first run. Without running setup, your indices won't have a template applied. String fields that should be keywords get analyzed as full text, meaning you can't aggregate or filter on them cleanly in Kibana. Numeric fields can end up mapped as strings. Fix this before first run — correcting mappings on an index with data in it means reindexing, which nobody wants to deal with in production. Run

sudo filebeat setup -e

once on initial deployment.

Not restarting after config changes. Filebeat doesn't hot-reload the main

filebeat.yml

. Module configs in

modules.d/

can reload dynamically if you enable it, but the main config file requires a service restart. I've watched engineers make config changes and spend twenty minutes wondering why nothing changed — they were waiting for a reload that was never coming.

Using the deprecated

log

input type. The

log

type tracks files by path. When logrotate renames

syslog

to

syslog.1

and creates a fresh

syslog

, Filebeat sees a new file at the watched path and starts reading from the beginning. You get duplicate events and no way to tell which are real. The

filestream

type tracks files by inode instead, handling rotation correctly. If you're setting up a new instance, use

filestream

from day one.

Registry corruption after an unclean shutdown. Filebeat maintains a registry at

/var/lib/filebeat/registry/

that tracks its read position in every file it monitors. A hard crash or power loss can corrupt it. The symptom is Filebeat appearing to run normally — connected, no errors — but ingesting nothing at all. Deleting the registry and restarting fixes it, but Filebeat will re-read all tracked files from the beginning. Pair the registry delete with an

ignore_older

setting to avoid flooding Elasticsearch with weeks of historical logs:

filebeat.inputs:
- type: filestream
  id: syslog-input
  enabled: true
  paths:
    - /var/log/syslog
  ignore_older: 24h

Get these fundamentals right and Filebeat becomes one of the most dependable components in your logging stack. It genuinely is a well-engineered tool — the problems almost always trace back to configuration, not to Filebeat itself. Once this foundation is solid, layering on Filebeat modules for nginx, PostgreSQL, or system metrics is straightforward, and you'll have pre-built ingest pipelines and Kibana dashboards waiting for you.