Search Logs Using Lucene

A guide to searching logs using Lucene queries in observIQ

In your visualization view or Live Tail, narrow down to the logs that interest you, using simple Lucene queries. Lucene queries have a very user-friendly syntax and offer a wide set of options for search. A Lucene query’s syntax has three parts; a field name, an operator, and a search term which is often enclosed in double quotes. It is important to note that Lucene queries are case-sensitive. “AND”, “OR”, “NOT” are recognized but not “and”, “or” ,”not”.

field: ”term”

Lucene lets you search through your logs based on values of fields or a generic search term within a field. Tabulate below are general Lucene syntaxes that you may use and all these syntaxes are detailed in sections following the tabulation:

Query Type

Search for

Query Syntax

Field search

To search for all logs containing a specific value for a field.

field:“value”

Term search

To search all logs containing a specific numeric value or text.

term

Range search

To search based on a date range, numeric or text range of a field, including the start and end values.

field:[startrange TO endrange]

Range Search

To search based on a date range, numeric or text range in a field, excluding the start and end values.

field:{startrange TO endrange}

Boolean Search

To search for logs that have the value specified in the query for either field1 or field2.

field1:”term1” OR field2:“term2”

Boolean Search

To search for logs which have the values as specified in the query for both field1 and field2

field1:”term1” AND field2:“term2”

Required

To search for logs that must contain the value mentioned after the + sign while the value before the + may or may not exist in the logs.

field1:”term1” +field2:”term2”

Prohibit

To search for logs that do not contain the value specified after the - sign and the values specified before the -sign must be present.

field1:”term1” -field2:”term2

Field Search

You can only search the value/ values of fields that are indexed and listed on the left pane of the Explore→ Discover Page.

Search using Lucene queries all the indexed fields listed on the left pane.Search using Lucene queries all the indexed fields listed on the left pane.

Search using Lucene queries all the indexed fields listed on the left pane.

Syntax field:term May also use field:”term phrase”
Type the name of the field followed by a “:” and the value to search for. If your search term is a single word you can skip using the double quotes to enclose the search value.
Example:
host.name:observiq
But to search for a phrase with more than one word, you must add double quotes to have the entire term considered during the search. When searching for a phrase, the logs retrieved would contain the search phrase in the order it is entered in the query and not the reverse. For instance, a query for a greeting “Good day” would return results with “Good day” and not “day Good”.

Example:
host.name:”observiq simplified”

Any search term

Syntax term
For searches based on any keyword or search term use this syntax. For example, searching for logs related to a browser, simply type the name of the browser and search.

Note:

  • Special characters added to a search term could result in an error or might return a different result. As a best practice use your search terms within double quotes to avoid errors. In the earlier example, searching Mozilla/5.0 would result in an error, while “Mozilla/5.0” as a search term should pull up the logs generated by the browser.
  • Enclose the search term in double quotes if the search term has more than one word or if it is a phrase.

Range Searches

Syntax field:[startrange TO endrange]
Range Queries help you narrow down to logs based on field values between the lower and upper bound specified in the range query. Range Queries can either include or exclude the upper and lower values mentioned in the query.
file.size:[100 TO 1000]
This pulls up all the logs that have files parsed between 100 and 1000MB, including the upper and lower range values specified in the query.
Use this syntax to search for both numeric and non-numeric ranges of field values.
Currency.name:{cn TO jpy}
This search query pulls up all currencies between CN and JPY but not including CN and JPY because we enclosed the ranges here within {} brackets denoting the search ranges be excluded.

Note: Always use square brackets for Inclusive range of queries and curly brackets for excluding the ranges in the results.

Boolean operators

Boolean operators make it easy to combine search terms logically and apply as queries for searching your logs. In Lucene queries, all logical operators are written in the uppercase like AND, OR, NOT. Lucene also allows combining multiple logical operators into a grouped query.

OR

Syntax field1:"term1" OR "term2"
A query with field values separated by the OR operator pulls up logs with either of those values.
When there is no operator separating a boolean Lucene query, the OR operator is executed by default.

Note: It is not mandatory to include an OR between the search terms. However, if the search term(s) is a phrase, it is necessary to include it within double quotes.

AND

Syntax field:"term1" AND field2:"term2"

Required and Prohibit Operators

Lucene makes it easy when querying for fields that you want to include or exclude using the + and - operators

Required +

Syntax field1:”term1” +field2:”term2
Using this query would pull up logs that mandatorily contain the search term2 and may contain the search term1
Example:
agent.name:"mongodb-44-0" +host.ip:"10.33.104.56"

Prohibited -
The - operator is us
Syntax field1:"term1" -field2:"term2"
The - operator is used to pull up logs that do not have the search term/value mentioned after the - operator
Example:
severity:"debug" -log.size:"542"

Fuzzy Searches

Syntax term~
term~value
Use the tilde ~ symbol to perform a fuzzy search. A fuzzy search can be performed only at the end of a single-word search term to find words that have the same or similar spelling to the search term.
Example:
user~

As seen in the image above the search returns logs that have the term close in spelling to the fuzzy search term “user”
Additionally specify a number after ~ to indicate how deviant the results could be from the search term.
Example:
fail~2

The search brings up results for logs with the term "failed" which has a 2 character deviance from the search term “fail”, which is within the limit set. In this case logs with messages containing terms like “failure” will not be included because of the deviance limit set.
This search will find terms like foam and roams. An additional parameter can specify the required similarity.