Lucidum provides APIs to retrieve data directly from the Lucidum Data Group.
-
The endpoint for /CMDB/v2/data/cmdb queries all fields, both raw and enriched (Lucidum Data Group).
-
The endpoint for /CMDB/v2/data/ldg retrieves only enriched data from the Lucidum Data Group.
This chapter describes the endpoint /CMDB/v2/data/ldg. This endpoint provides a more focused set of results for each asset or user.
What is the Lucidum Data Group? #
The Lucidum Data Group (LDG) is a dataset that includes real-world, verified, accurate, up-to-date data on cybersecurity and IT infrastructure.
-
Lucidum is agentless and ingests read-only API data from all the solutions in your environment, including IT, operations, development, business, network, security, and HR solutions, and structured and unstructured data from data lakes.
-
Lucidum transforms the ingested data using machine learning algorithms, network graph analysis, text mining, and data classification models to build an enriched, normalized dataset for all assets, users, data, and security postures in your environment.
-
For some of the ingested data, Lucidum lightly processes it using rules-based algorithms.
-
Lucidum pushes other ingested data through ML engines and then maps the data to fields in the LDG. These ML engines find relationships between assets, users, and data, determine the applications running on each asset, align all security data about vulnerabilities, risks, and misconfigurations with the appropriate assets, users, and datastores, and apply risk models.
-
Lucidum uses ML engines to deduplicate records for assets and users.
-
Lucidum examines usernames, email addresses, and other fields to deduplicate records for each user.
-
Lucidum uses serial numbers, FQDNs, MAC addresses, network graph analysis, and other fields to deduplicate assets.
-
Lucidum uses several text mining and data classification models to find details about data stores.
-
-
Lucidum normalizes values for asset name, asset type, OS category, OS version, status, username, and vendor, to prevent duplicate records and to enable consistent query results.
-
-
Lucidum stores the enriched, normalized dataset in the Lucidum Data Group (LDG). After processing, the LDG includes deduplicated, normalized, and accurate details about each user, each asset, and each data store.
Headers #
Requests to the endpoints in the Lucidum API v2 must also include a header with:
-
Content-Type of JSON.
Query the LDG About Assets and Users: /CMDB/v2/data/ldg #
The endpoint for querying the Lucidum database, using fields, operators, AND statements, and OR statements, as you would in the Query Builder in the Lucidum UI.
|
Endpoint |
Method |
Parameters in Request Body |
|---|---|---|
|
/CMDB/v2/data/ldg |
POST |
query. See the section below. table. The Lucidum database table to query. Choices are:
|
Query Syntax #
Because requests to the endpoint /CMDB/v2/data/cmdb use the POST method, we can include a request body. The request body with its query parameter looks like this:
{
"query": [
[
{
"searchFieldName": "Owner_Name",
"operator": "exists",
"type": "String",
"value": null
},
]
],
"table": "user",
"paging": {
"page": 0.
"recordsPerPage": 20
},
-
Note that the body uses JSON syntax. For information on JSON syntax, see https://www.w3schools.com/js/js_json_syntax.asp
- Line 2. Define the query parameter.
-
Line 5. The records to retrieve must include the field name “Owner_Name”
-
Line 6. The “Owner_Name” field exists
-
Line 9. The value of “Owner_Name” can be any value
-
Line 8. Asset_Type has a data type of String
-
Line 12. Define the table parameter. We will query the “user” table in the LDG
- The remaining lines define pagination for the results.
Fields #
Fields are one or more properties that you are interested in, such as “open port list”, “ip address”, “country code”, or “risk score”.
Fields are characteristics of the Lucidum objects. For example, a characteristic of a User is the user’s email address. A characteristic of an asset is the asset’s IP address. Usually, a field maps to a column name in a Lucidum database.
In the query parameter, you use the syntax:
"searchFieldName":"<field name>",
where:
-
searchFieldName. In the name: value pair for the field, the name string “searchFieldName” tells the API that the following value is a field name.
-
<field name>. In the name: value pair for searchFieldName, the field name tells the API which field name to search for.
An example of the name:value pair for fields is:
"searchFieldName":"Asset_Type",
In this example, the Lucidum API will examine the field Asset_Type.
Note that the field name must match the case returned by /CMDB/v2/data/metadata/asset or /CMDB/v2/data/metadata/user. For example, “Asset_Type”, not “asset_type” nor “Asset Type”. And “sourcetype”, not “Sourcetype” nor “source type”
- To view a list of all LDG fields for assets, use the query syntax:
{
"query": [
[
{
"searchFieldName": "Asset_Name",
"operator": "exists",
"type": "String",
"value": null
},
]
],
"table": "asset",
"paging": {
"page": 0.
"recordsPerPage": 20
},
}
The results will include all the asset fields in the LDG.
- To view a list of all LDG fields for users, use the query syntax:
{
"query": [
[
{
"searchFieldName": "Owner_Name",
"operator": "exists",
"type": "String",
"value": null
},
]
],
"table": "user",
"paging": {
"page": 0.
"recordsPerPage": 20
},
}
The results will include all the user fields in the LDG.
Operators, Data Types, SmartLabels, Tags, and Values #
For details on other parts of a query:
See the chapter on Endpoints for Assets and Users.
Using And #
For details on using AND in an API query, see the chapter on Endpoints for Assets and Users.
Using OR #
For details on using AND in an API query, see the chapter on Endpoints for Assets and Users.
Pagination #
For details on pagination in an API query, see the chapter on Pagination.
Example cURL Request for Asset Data #
curl --location --request POST 'https://dogfood.lucidum.cloud/CMDB/v2/data/ldg' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer omhsSDpRxOMtOfjXoymU' \
--data '{
"query": [
[
{
"searchFieldName": "Asset_Type",
"operator": "==",
"type": "String",
"value": "VM"
}
]
],
"table": "asset",
"paging": {
"page": 0,
"recordsPerPage": 20
}
}'
This cURL example specifies:
-
Line 1. The cURL call, specifying this request uses a POST method. The URL is host name plus the endpoint for this API.
-
Line 2. The header specifies to return results in JSON.
-
Line 3. The header specifies to use a Bearer Token for authentication and provides the bearer token. The bearer token in an action bearer token.
-
Lines 8-11. The request returns asset records for assets that include the Asset Type field with a value of VM.
-
Line 15. Specifies that this request is for asset data.
-
Lines 16-18. Specify pagination.
Example Python Request for Asset Data #
import requests
import json
import time
url = "/dogfood.lucidum.cloud/CMDB/v2/data/ldg"
authToken = 'Bearer omhsSDpRxOMtOfjXoymU'
payload = {
"query": [
[
{
"searchFieldName": "Asset_Name",
"operator": "exists",
"type": "String",
"value": None
}
]
],
"table": "asset",
"paging": {
"page": 0,
"recordsPerPage": 50
}
}
def create_session():
s = requests.Session()
s.headers = {
'Content-Type': 'application/json',
'Authorization': authToken
}
return s
if __name__ == '__main__':
total_result= []
sess = create_session()
page = 0
while True:
payload['paging']['page'] = page
resp = sess.post(url, data=json.dumps(payload), verify=False)
result = resp.json()
if len(result['data']) == 0:
break
total_result = total_result + result['data']
print(f"Page:{result['page']}")
page = page + 1
time.sleep(1)
print("Total Result Records:", len(total_result))
-
The Python code interacts with the Lucidum APIs and fetches data from the endpoint /CMDB/v2/data/ldg.
-
The code imports the libraries:
-
requests for making HTTP requests
-
json for handling JSON data
-
time for adding delays between requests
-
-
The code constructs three variables: URL (Lucidum API endpoint), authToken (action bearer token), and payload (query to the asset database).
-
The code constructs a JSON payload that includes the query to the asset database in the LDG. The query returns asset records for assets that include the Asset_Name field.
-
The code retrieves data using pagination.
-
The code creates a session with headers, including the variable for the action bearer token.
-
The results are accumulated in a list and printed. The code uses a while loop to continue fetching data until no more records are available.
Example cURL Request for User Data #
This cURL example specifies:
-
Line 1. The cURL call, specifying this request uses a POST method. The URL is host name plus the endpoint for this API.
-
Line 2. The header specifies to return results in JSON.
-
Line 3. The header specifies to use a Bearer Token for authentication and provides the bearer token. The bearer token in an action bearer token.
-
Lines 8-11. The request returns asset records for assets that include the Data_Type field with a value of Customer Support.
-
Line 15. Specifies that this request is for user data.
-
Lines 16-18. Specify pagination.
Example Python Request for User Data #
import requests
import json
import time
url = "/dogfood.lucidum.cloud/CMDB/v2/data/ldg"
authToken = 'Bearer omhsSDpRxOMtOfjXoymU'
payload = {
"query": [
[
{
"searchFieldName": "Owner_Name",
"operator": "exists",
"type": "String",
"value": None
}
]
],
"table": "user",
"paging": {
"page": 0,
"recordsPerPage": 50
}
}
def create_session():
s = requests.Session()
s.headers = {
'Content-Type': 'application/json',
'Authorization': authToken
}
return s
if __name__ == '__main__':
total_result= []
sess = create_session()
page = 0
while True:
payload['paging']['page'] = page
resp = sess.post(url, data=json.dumps(payload), verify=False)
result = resp.json()
if len(result['data']) == 0:
break
total_result = total_result + result['data']
print(f"Page:{result['page']}")
page = page + 1
time.sleep(1)
print("Total Result Records:", len(total_result))
-
The Python code interacts with the Lucidum APIs and fetches data from the endpoint /CMDB/v2/data/ldg.
-
The code imports the libraries:
-
requests for making HTTP requests
-
json for handling JSON data
-
time for adding delays between requests
-
-
The code constructs three variables: URL (Lucidum API endpoint), authToken (action bearer token), and payload (query to the asset database).
-
The code constructs a JSON payload that includes the query to the asset database in the LDG. The query returns asset records for assets that include the Owner_Name field.
-
The code retrieves data using pagination.
-
The code creates a session with headers, including the variable for the action bearer token.
-
The results are accumulated in a list and printed. The code uses a while loop to continue fetching data until no more records are available.
