Ingesting Data into an OpenSearch Cluster with Logstash
Use Logstash to ingest log data into an OpenSearch cluster.
Logstash is a data processing pipeline that can ingest data from a variety of sources, process and transform it, and then send it to a destination such as an OpenSearch cluster.
Prerequisites
Complete the following tasks before proceeding with the steps described in this topic:
- Download and install Java Development Kit (JDK) version 8 or version 11 on your local machine.
-
Create an OpenSearch cluster and make note of the connection information. See Creating an OpenSearch Cluster and Getting an OpenSearch Cluster's Details.
-
Create and connect to a compute instance. The instance must be in the same VCN as the OpenSearch cluster. See Creating a VM Instance.
- Connect to the OpenSearch cluster from your local machine. See Task 3: Test the connection to OCI Search Service – OpenSearch endpoint
Install and Configure LogStash
Download the Logstash OSS with OpenSearch Output Plugin from OpenSearch Ingest Tools (2.8) and install it on your local machine. This is currently only available for Linux and MacOS based systems.
Extract the downloaded Logstash tarball and then navigate to
Logstash
directory.Create a pipeline configuration file,
logstash.conf
, in theconfig
subdirectory of theLogstash
directory.Following is a pipeline configuration file example that reads the log file from your local machine and ingests the log data to an OpenSearch cluster. Replace the applicable text with the details for your cluster. Specify the cluster's API endpoint for
host
, see Getting an OpenSearch Cluster's Details. Foruser
andpassword
, specify a user that has sufficient permissions to ingest data for the OpenSearch cluster. For example, you can use the primary user account for role-based access control that you specified when you created the cluster, see Role-Based Access Control in Search with OpenSearch.input { file{ path => "/path/to/log/file/application.log" start_position => "beginning" } } filter{ #Optional and can be empty } output { opensearch { hosts => "<cluster API endpoint>" user => "<cluster user>" password => "<cluster password>" index => "<index name>" ssl_certificate_verification => true } }
Start Logstash by running the following command:
bin/logstash -f config/logstash.conf
After Logstash starts, it continuously reads the file system for new logs using its file input plugin. Logstash then uses the OpenSearch output plugin to ingest new log data to an OpenSearch cluster.
You can verify that the log data was ingested into the OpenSearch cluster by
connecting to the cluster's OpenSearch Dashboard to check for the index named what
you specified for index
in the pipleline configuration file. See
Task 6: Connect to OpenSearch Dashboards
and Quickstart guide for OpenSearch
Dashboards.