~Blog

All about "Elastic Agent" in Elastic SIEM - Part 1

Rohit — Mon, 27 Jan 2025 03:44:11 GMT

Elastic agent is an one stop solution for both collecting of data and also providing endpoint security. Single agent can be configured to collect different forms of data by adding necessary integrations on demand. It is way easier to configure and manage as all the chaos are handled by the stack itself.’

Components needed for a basic SIEM setup in Elastic stack

Elasticsearch
Kibana
Elastic Agent
Fleet server

Elasticsearch is the brain of Elastic stack. Kibana is a very powerful and easy to use UI where we can view and visualize the data that is present in elasticsearch. Elastic Agent and Fleet server are the ones about which we will focus on further.

Types of Elastic Agent (Based on installation):

Fleet managed Elastic Agent

The elastic agent is enrolled to a fleet server through which policy configuration and update can be performed remotely.
Standalone Elastic Agent

The elastic agent runs as standalone agent. For configuring the agent we must manually update the elastic-agent.yml file with updated configs. This is a tiresome work and it is always recommended to go with fleet managed elastic agents.

Building blocks of Elastic Agent

Policy
Outputs
Integrations

Policy

Policy refers to the configuration for elastic agent. Agent policies define the integrations you want to run and the hosts they should run on. You can assign a single Elastic Agent policy to multiple agents, simplifying large-scale configuration management. It contains the details about data inputs and details on where to sent the collected data to. It contains the settings in detail on which data must be collected, from where the data must be fetched, what are the preprocessing that must be performed. If the agent is managed by fleet then details about the fleet will also be available.

Integration

Elastic integrations simplify connecting Elastic with external services and systems, enabling quick insights and actions. They can gather new data sources and typically include ready-to-use assets like dashboards, visualizations, and pipelines for extracting structured fields from logs and events. This allows you to gain insights in just seconds.

When you add an integration, you configure inputs for logs and metrics, such as the path to your Nginx access logs. When you’re done, you save the integration to an Elastic Agent policy. The next time enrolled agents check in, they receive the update.

Outputs

The outputs specifies where to send data. You can specify multiple outputs to pair specific inputs with specific outputs.

Supported outputs are:

Elasticsearch
Kafka
Logstash

Fleet server

Fleet server is technically a elastic agent with a policy that has fleet server integration added to it. Fleet server is like a manager in an office. It is used to manage elastic agents( employees ). It acts as a communication hub between the Elastic Agent and the Elasticsearch cluster. It handles agent enrollment, configuration updates, and data ingestion, ensuring efficient and secure management of multiple agents. By using Fleet Server, administrators can simplify deployment and streamline agent management at scale.

Fleet provides a web-based UI in Kibana for centrally managing Elastic Agents and their policies.

The Agents page provides a view of agent health status, indicating which agents are healthy or unhealthy, along with the last check-in time. It also displays the Elastic Agent binary version and the associated policy.

We can see more about Elastic agent installation and about elastic endpoint in next part! Will update link here soon.

How to Index and Search Documents in Elasticsearch: A Step-by-Step Guide

Rohit — Sun, 05 Jan 2025 18:30:00 GMT

Elasticsearch, a powerful distributed search engine, enables users to efficiently store, search, and analyze large volumes of data in near real-time. Whether you're a developer building a search application or a data analyst working on business insights, understanding how to index and search documents in Elasticsearch is essential. This guide walks you through the basics of indexing and searching documents, complete with practical examples to help you get started.

✅ Prerequisites

Before diving in, ensure you have:

Elasticsearch installed and running: You can install Elasticsearch locally or use a managed service like Elastic Cloud.(There is a 14 day trail available..check it out)
A basic understanding of JSON: Elasticsearch stores and processes data in JSON format.
Tools to interact with Elasticsearch: Use cURL, Kibana, or any HTTP client like Postman to send requests.

📚 Understand the Basics

What is an Index?

An index in Elasticsearch is a collection of documents that share similar characteristics. For example, an 🛒 e-commerce application might have separate indices for products, customers, and orders.

What is a Document?

A document is a single record stored in an index. It is represented in JSON format and contains fields (key-value pairs) that describe the data. For example:

{
  "name": "Wireless Mouse",
  "price": 25.99,
  "in_stock": true
}

Index a Document

Create an Index

To create an index, use the PUT request:

PUT /products

This creates an index named products. By default, Elasticsearch handles the mapping (schema) automatically. You can also define your own mappings for more control.

Index a Single Document

To add a document to the products index, use the POST or PUT request:

POST /products/_doc/1
{
  "name": "Wireless Mouse",
  "price": 25.99,
  "in_stock": true
}

Here:

/products specifies the index.
/_doc/1 indicates the document ID (1 in this case). Elasticsearch assigns an ID automatically if you omit it.

Index Multiple Documents

You can bulk index multiple documents using the _bulk endpoint:

POST /_bulk
{ "index": { "_index": "products", "_id": "1" } }
{ "name": "Wireless Mouse", "price": 25.99, "in_stock": true }
{ "index": { "_index": "products", "_id": "2" } }
{ "name": "Mechanical Keyboard", "price": 89.99, "in_stock": false }

Each pair of lines specifies an action (index) and the document data.

🔍 Search for Documents

Basic Search

To search for all documents in the products index, use the GET request:

GET /products/_search

This returns all documents along with metadata. By default, Elasticsearch retrieves the top 10 results.

Match Query

To search for documents containing specific text, use the match query:

GET /products/_search
{
  "query": {
    "match": {
      "name": "Mouse"
    }
  }
}

This searches for documents where the name field contains the term "Mouse."

Term Query

The term query is used for exact matches:

GET /products/_search
{
  "query": {
    "term": {
      "in_stock": true
    }
  }
}

Filtered Search

To combine queries and filters, use the bool query:

GET /products/_search
{
  "query": {
    "bool": {
      "must": {
        "match": {
          "name": "Keyboard"
        }
      },
      "filter": {
        "term": {
          "in_stock": true
        }
      }
    }
  }
}

This finds documents with name containing "Keyboard" and in_stock set to true.

✏️ Update a Document

To update an existing document, use the _update endpoint:

POST /products/_doc/1/_update
{
  "doc": {
    "price": 24.99
  }
}

This updates the price field of the document with ID 1.

🗑️ Delete a Document or Index

Delete a Document

To delete a document by ID, use the DELETE request:

DELETE /products/_doc/1

Delete an Index

To delete the entire products index, use:

DELETE /products

By following these steps, you’ll be well on your way to mastering Elasticsearch document indexing and search capabilities. Whether you’re building a search engine, analyzing logs, or managing e-commerce data, Elasticsearch provides the tools you need to handle data efficiently. Happy searching! 🎉

Elasticsearch Cluster TLS Encryption

Rohit — Tue, 04 Jun 2024 18:30:00 GMT

Enabling TLS encryption is a great add on protection for you clusters and also most of the compliance certifications need TLS/SSL encrypted communication. It encrypts, both the connection between in between nodes and HTTP API calls. It takes only few steps to enable TLS for your elasticsearch cluster as most of the hard tasks are already handled by the executables provided in elasticsearch zip. If you are not familiar with the concept of TLS then checkout this awesome blog by Andrew Howden in which he explains how TLS works in the simplest way. Now lets start the process of protecting our elasticsearch clusters🫡. We will go with using Self signed certificates for this example blog. In production you can use your companys Authority files to sign the certificates. We will be using elasticsearch-certutil to generate required certificates.

Certificate Authority(CA):

A Certificate Authority (CA) is a trusted entity that issues digital certificates. These certificates verify the ownership/authenticity of a digital asset. The CA's role is to confirm the identity of entities (such as websites) and to bind public keys with those identities through digital certificates. This helps establish trust in a digital communication environment.

X.509 Certificate:

An X.509 Certificate is a digital certificate that uses the X.509 standard to define the format of public key certificates. X.509 certificates are used to establish a secure, encrypted connection between a client (like a web browser) and a server (like a website).

Steps to generate certificates:

‣Generating a certificate authority to sign our X.509 certificates.

./bin/elasticsearch-certutil ca

Executing this will prompt for few questions.

CA file name - you can name it anything you wish to. I will leave it as default.
CA password - Its very important to remember the CA password as we need it for configuration purposes in multiple places. you can also leave it empty. For now i will go with "12345" as password.

//Download elasticsearh zip and extract the same

~/clusters/sample/elasticsearch % ./bin/elasticsearch-certutil ca

This tool assists you in the generation of X.509 certificates and certificate
signing requests for use with SSL/TLS in the Elastic stack.

The 'ca' mode generates a new 'certificate authority'
This will create a new X.509 certificate and private key that can be used
to sign certificate when running in 'cert' mode.

Use the 'ca-dn' option if you wish to configure the 'distinguished name'
of the certificate authority

By default the 'ca' mode produces a single PKCS#12 output file which holds:
    * The CA certificate
    * The CA's private key

If you elect to generate PEM format certificates (the -pem option), then the output will
be a zip file containing individual files for the CA certificate and private key

Please enter the desired output file [elastic-stack-ca.p12]: 
Enter password for elastic-stack-ca.p12 : 

~/clusters/sample/elasticsearch % ls
LICENSE.txt          README.asciidoc      blog                 elastic-stack-ca.p12 lib                  modules              upgrade
NOTICE.txt           bin                  config               jdk                  logs                 plugins

Now you can see that elastic-stack-ca.p12 is created. This certificate authority file contains the following components in it:

Private key
Public certificate which also contains a public key

To view the contents of the .p12 file you can use the below command

openssl pkcs12 -info -in elastic-stact-ca.p12

‣Next we are going to generate Node certificate for enabling encrypting communication in between nodes.

./bin/elasticsearch-certutil cert --ca elastic-stack-ca.p12

Executing this will prompt for few questions.

CA file password - you can name it anything you wish to. I will leave it as default.
Certificate password - We need to remember the Certificate password also. you can also leave it empty. For now i will go with "12345" as password.

Contents of Node Certificate:

Node's Private key
Node certificate ( also contains public key)
CA certificate (only certificate of CA, private key will not be included )

‣ Final step in certificate generation part is to generate a certificate for HTTPS communication. (You can skip this step if you do not want TLS for HTTP communications ).

./bin/elasticsearch-certutil http

Contents of Node Certificate:

HTTP's Private key
HTTP certificate ( also contains public key)
CA certificate (only certificate of CA, private key will not be included)

Both Node and HTTP certificates are X.509 certificates.

## Do you wish to generate a Certificate Signing Request (CSR)?

A CSR is used when you want your certificate to be created by an existing
Certificate Authority (CA) that you do not control (that is, you don't have
access to the keys for that CA). 

If you are in a corporate environment with a central security team, then you
may have an existing Corporate CA that can generate your certificate for you.
Infrastructure within your organisation may already be configured to trust this
CA, so it may be easier for clients to connect to Elasticsearch if you use a
CSR and send that request to the team that controls your CA.

If you choose not to generate a CSR, this tool will generate a new certificate
for you. That certificate will be signed by a CA under your control. This is a
quick and easy way to secure your cluster with TLS, but you will need to
configure all your clients to trust that custom CA.

Generate a CSR? [y/N]N

## Do you have an existing Certificate Authority (CA) key-pair that you wish to use to sign your certificate?

If you have an existing CA certificate and key, then you can use that CA to
sign your new http certificate. This allows you to use the same CA across
multiple Elasticsearch clusters which can make it easier to configure clients,
and may be easier for you to manage.

If you do not have an existing CA, one will be generated for you.

Use an existing CA? [y/N]y

## What is the path to your CA?

Please enter the full pathname to the Certificate Authority that you wish to
use for signing your new http certificate. This can be in PKCS#12 (.p12), JKS
(.jks) or PEM (.crt, .key, .pem) format.
CA Path: /Users/rohit/clusters/sample/elasticsearch/elastic-stack-ca.p12
Reading a PKCS12 keystore requires a password.
It is possible for the keystore's password to be blank,
in which case you can simply press  at the prompt
Password for elastic-stack-ca.p12:

## How long should your certificates be valid?

Every certificate has an expiry date. When the expiry date is reached clients
will stop trusting your certificate and TLS connections will fail.

Best practice suggests that you should either:
(a) set this to a short duration (90 - 120 days) and have automatic processes
to generate a new certificate before the old one expires, or
(b) set it to a longer duration (3 - 5 years) and then perform a manual update
a few months before it expires.

You may enter the validity period in years (e.g. 3Y), months (e.g. 18M), or days (e.g. 90D)

For how long should your certificate be valid? [5y] 10y

## Do you wish to generate one certificate per node?

If you have multiple nodes in your cluster, then you may choose to generate a
separate certificate for each of these nodes. Each certificate will have its
own private key, and will be issued for a specific hostname or IP address.

Alternatively, you may wish to generate a single certificate that is valid
across all the hostnames or addresses in your cluster.

If all of your nodes will be accessed through a single domain
(e.g. node01.es.example.com, node02.es.example.com, etc) then you may find it
simpler to generate one certificate with a wildcard hostname (*.es.example.com)
and use that across all of your nodes.

However, if you do not have a common domain name, and you expect to add
additional nodes to your cluster in the future, then you should generate a
certificate per node so that you can more easily generate new certificates when
you provision new nodes.

Generate a certificate per node? [y/N]N

## Which hostnames will be used to connect to your nodes?

These hostnames will be added as "DNS" names in the "Subject Alternative Name"
(SAN) field in your certificate.

You should list every hostname and variant that people will use to connect to
your cluster over http.
Do not list IP addresses here, you will be asked to enter them later.

If you wish to use a wildcard certificate (for example *.es.example.com) you
can enter that here.

Enter all the hostnames that you need, one per line.
When you are done, press  once more to move on to the next step.


You did not enter any hostnames.
Clients are likely to encounter TLS hostname verification errors if they
connect to your cluster using a DNS name.

Is this correct [Y/n]Y

## Which IP addresses will be used to connect to your nodes?

If your clients will ever connect to your nodes by numeric IP address, then you
can list these as valid IP "Subject Alternative Name" (SAN) fields in your
certificate.

If you do not have fixed IP addresses, or not wish to support direct IP access
to your cluster then you can just press  to skip this step.

Enter all the IP addresses that you need, one per line.
When you are done, press  once more to move on to the next step.


You did not enter any IP addresses.

Is this correct [Y/n]Y

## Other certificate options

The generated certificate will have the following additional configuration
values. These values have been selected based on a combination of the
information you have provided above and secure defaults. You should not need to
change these values unless you have specific requirements.

Key Name: elasticsearch
Subject DN: CN=elasticsearch
Key Size: 2048

Do you wish to change any of these options? [y/N]N

## What password do you want for your private key(s)?

Your private key(s) will be stored in a PKCS#12 keystore file named "http.p12".
This type of keystore is always password protected, but it is possible to use a
blank password.

If you wish to use a blank password, simply press  at the prompt below.
Provide a password for the "http.p12" file:  [ for none]

## Where should we save the generated files?

A number of files will be generated including your private key(s),
public certificate(s), and sample configuration options for Elastic Stack products.

These files will be included in a single zip archive.

What filename should be used for the output zip file? [/Users/rohit/clusters/sample/elasticsearch/elasticsearch-ssl-http.zip] 

Zip file written to /Users/rohit/clusters/sample/elasticsearch/elasticsearch-ssl-http.zip

After generating the necessary certificate copy them to elasticsearch/config/certs folder. That's all for certificate generation part. Now lets configure the same.

Configuring in elasticsearch.yaml:

xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: certs/elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: certs/elastic-certificates.p12

xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.keystore.path: "certs/http.p12"
xpack.security.http.ssl.verification_mode: certificate

Note: If you have set password for your certificate files then you have to add it to elasticsearch's keystore. You can use the below commands to do the same.

./bin/elasticsearch-keystore add xpack.security.transport.ssl.keystore.secure_password
//prompts for node certificates password

./bin/elasticsearch-keystore add xpack.security.transport.ssl.truststore.secure_password
//prompts for node certificates password

./bin/elasticsearch-keystore add xpack.security.http.ssl.keystore.secure_password
//prompts for http certificates password

xpack.security.http.ssl.verification_mode / xpack.security.transport.ssl.verification_mode:

It helps to control the SSL certicate verification level. Based on this configuration we can restrict connection to a finite set of host or IP addresses. For enabling this restriction we need to provide the list of allowed hostnames and IP addresses while generation of certificate. There are three modes available. They are

full:
Validates that the provided certificate: has an issue date that’s within the not_before and not_after dates; chains to a trusted Certificate Authority (CA); has a hostname or IP address that matches the names within the certificate.
certificate:
Validates the provided certificate and verifies that it’s signed by a trusted authority (CA), but doesn’t check the certificate hostname.
none:
It basically disables SSL verification.

In this example we have generated single node and http certificate which we will be using in all nodes and also we have not provided IP address or hostname for certificate generation therefore we will go with certificate mode. Incase you had multiple nodes if you go with full mode cluster will not get created due to Host verification failure.