In addition to the core EventProcessor, several tools are provided to facilitate data ingestion, testing, and debugging.

opensearch-fetcher

The opensearch-fetcher is a utility to extract logs from an OpenSearch index and save them as individual JSON files. This is extremely useful for gathering real-world test data for the Playground.

Usage

./opensearch-fetcher -url "http://localhost:9200" -index "v11-log-*" -size 50 -output "./test_data"

Options

FlagDescriptionDefault-urlOpenSearch server URLhttp://192.168.1.18:9200-index(Required) Index or pattern to fetch from-userUsername for authentication-passwordPassword for authentication-outputDirectory to save JSON filesfetched_logs-sizeNumber of logs to fetch100-querySearch query (Query String syntax)*-startStart date (RFC3339)-endEnd date (RFC3339)Current time

log-pusher

The log-pusher is used to send JSON log files to a running EventProcessor instance via gRPC. It simulates an input plugin and is useful for replaying logs or performing load tests.

Usage

./log-pusher -input "./test_data" -socket "/workdir/sockets/engine_server.sock"

Options

FlagDescriptionDefault-input(Required) Directory containing JSON log files-socketPath to the engine_server.sock$WORK_DIR/sockets/engine_server.sock

The log-pusher will recursively walk the input directory, read all .json files, and stream them to the processor. It waits for acknowledgments (ACKs) from the server to ensure data delivery.