Using Queries with Nested Lists

What is a Nested List? #

Lucidum includes fields of type “nested list”. These fields behave differently than list fields.

Nested Lists are sometimes called arrays and organize data in a table structure.

For example, Applications is a field of type “nested list”. In the Query tool, the field looks like this:

Note that Applications is the nested list. And Applications::Name, Applications::Source and Applications::Version are sub-fields. You can think of the sub-fields as columns in a table.

It is important to understand how Lucidum performs queries on Nested List fields.

• If you select the field Applications, Lucidum will run the query against each row (element) in the table.

• If you select the a sub-field, for example Applications::Name, Lucidum will first extract all the values from the Name column, create a list, and then run the query against that list.

Example Data for Applications #

For example, suppose we have two assets, Asset-1 and Asset-2.

Asset-1 has the following applications:

• Each row is an element.

• Each column is a sub-field.

So for Asset-1:

• Applications::Name = Python, Docker, Git

• Applications::Version = python (ver 2.7.10), None, git (ver 3.8.10)

• Applications::Source = AWS, Datadog, SentinelOne

Asset-2 has the following applications:

• Each row is an element.

• Each column is a sub-field.

So for Asset-2:

• Applications::Name = Jenkins, Docker, Git

• Applications::Version = None, None, git (ver 3.1.1)

• Applications::Source = AWS, Datadog, SentinelOne

Example 1: Difference Between Nested List Field and Its Sub-fields #

Suppose you wanted to find all assets where the Application name is “Python” and the Version is “3.8.10”. Using the tables above, you would expect to find zero results.

Suppose you wrote a query that searched for Application::Name is “Python” and Application:Version is “3.8.10”.

This query is wrong. It will return Asset-1.

• Lucidum searches the list of names for Asset-1, which is “Python, Docker, Git”.

  • Asset-1 matches Application::Name is “Python”.
• Next, Lucidum searches the list of versions for Asset-1, which is “python (ver 2.7.10), none, and git (ver 3.8.10).
  • Asset-1 also matches Application::Version is “3.8.10”.

• Because Asset-1 matches both parts of the query, the query returns Asset-1.

• Lucidum searches the list of names for Asset-2, which is “Jenkins, Docker, Git”.

  • Asset-2 does not match Application::Name is “Python”.
• Lucidum search the list of versions for Asset-2,. which is “None, None, git (ver 3.1.1).
  • Asset-2 does not match Application::Version is “3.8.10”.

Instead, you want Lucidum to apply the query to all rows (elements) in the table, so you must use the Applications field (not the sub-fields Application::Name, Application::Version, Application::Source).

And a row must match all parts of the query, so you should use the “element matching all” syntax.

The correct query is:

</p;>

This query is correct.

This query returns no results.

• Lucidum would search each row (element) in Applications for Asset-1.
  • There is no row where Application Name is “Python” AND Application Version is “3.8.10”.
• Lucidum would search each row (element) in Applications for Asset-2.
  • There is no row where Application Name is “Python” AND Application Version is “3.8.10”.
• Therefore, the query returns no results.

Example 2: Difference Between Nested List Field and Its Sub-fields #

Suppose you wanted to find all assets that are not running Python. You would expect the query to return Asset-2.

Suppose you wrote a query that used “element matching all” and “Name is not Python”.

This query will examine all rows in the Applications table.

This query is wrong. This query returns Asset-1 and Asset-2. But it should return only Asset-2.

When you use the “Applications” field and “Element matching all”, Lucidum searches all rows (elements) in Applications for Asset-1. And then searches all rows (elements) in Applications for Asset-2.

• For Asset-1, row one has an Application Name of “Python”.
  • Row one does not match the query.
• For Asset-1, row two does not have an Application Name of “Python”.
  • Row two matches the query.
• For Asset-1, row three does not have an Application Name of “Python”
  • Row three matches the query.

• Asset-1 satisfies the query because two rows (elements) match the query. Therefore, Asset-1 matches this query.

• For Asset-2, none of the rows (elements) have an Application Name of “Python”.

  • Asset-2 matches this query.

Instead, you want to use a query that looked at the column Application::Name.

This query is correct.

• First Lucidum would search the Name column for Asset-1, which is “Python, Docker, Git”.
  • Asset-1 does not match, because one of the values of Application::Name is “Python”.
• Next, Lucidum would search the Name column for Asset-2, which is “Jenkins, Docker, Git”.
  • Asset-2 matches the query because none of the values of Application::Name is “Python”.
• The query returns Asset-2.

Example Data for Tags #

Suppose Asset-1 has the following Tags:

• Each row is an element.

• Each column is a sub-field.

So for Asset-1:

• Tag::Key= Environment, Name, Purpose

• Tag::Value = Prod, Lucidum-Public-VM, Pentest

Suppose Asset-2 has the following Tags:

• Each row is an element.

• Each column is a sub-field.

So for Asset-2:

• Tag::Key= Environment, Name, Purpose

• Tag::Value = Dev, Lucidum-Dev-VM, Test

Example 3: Syntax for “element matching all” #

Suppose we query the Tags nested field, which means the query will examine each row.

Suppose we look for assets where the tag Key matches Environment and the tag Value matches Prod.

Suppose we use the “element matching all” syntax. This syntax searches for assets with at least one row that matches both criteria where the tag Key is “Environment” AND the tag Value is “Prod”.

We would expect this query to return Asset-1.

This query returns Asset-1.

• For Asset-1:

  • row one has both a tag Key “Environment” AND a tag Value “Prod”. So that row (element) does match.

  • However, row two does not match and row three does not match.

  • Asset-1 satisfies the query because one row (element) matches the query.

• For Asset-2, none of the rows (elements) have both a tag Key “Environment” AND a tag Value “Prod”.
  • So Asset-2 does not match the query

Example 4: Other Syntax #

• element matching any. This syntax says that at least one row has the tag Key “Environment” string OR the tag Value “Prod”.

  • From our examples, row one for Asset-1 matches both these criteria.

  • Row one for Asset-2 matches the tag Key “Environment”

  • This query returns Asset-1 and Asset-2.

• no element matching all. This syntax says that no rows in the table match both the tag Key “Environment” string AND the tag Value “Prod.

  • For Asset-1, row one has both a tag Key “Environment” AND a tag Value “Prod”. Asset-1 does not match the query.

  • For Asset 2:, no rows have both a tag Key “Environment” AND a tag Value “Prod”. Asset-2 matches the query.

  • The query returns Asset-2

• no element matching any. This syntax says that no rows has the tag Key “Environment” string OR the tag Value “Prod”.

  • For Asset-1, row one has both the tag Key “Environment” string OR the tag Value “Prod”. Asset-1 does not match the query

  • For Asset-2, row one includes the tag Key “Environment”. Asset-2 does not match the query

  • The query returns no assets