IRI DarkShield-NoSQL RPC API

by Adam Lewis

IRI DarkShield Version 5 features a Remote Procedure Call (RPC) Application Programming Interface (API) for searching and masking data stored in NoSQL databases. This API supports the execution of DarkShield searching and masking jobs specified in the “New NoSQL Search/Masking Job” wizard in IRI Workbench. You can also embed the same API as middleware in a pipeline outside of IRI Workbench; e.g., from a CI/CD workflow or new program.

Currently supported NoSQL database types ¹ include:

Cassandra
Elasticsearch
MongoDB

The DarkShield NoSQL API is built as a plugin atop the IRI Web Services Platform (codenamed Plankton), which allows you to pick which services you require while utilizing the same hosting, configuration, and logging capabilities provided through the platform.

Before continuing with this article, please familiarize yourself with the operations of the DarkShield-Files API in this article and the base DarkShield API in this article.

The DarkShield-NoSQL API utilizes the DarkShield-Files API to handle binary data, and the base DarkShield API to handle non-binary values extracted from NoSQL sources.

All demos associated with this article can be downloaded from our Git repository here. To run the demos, you will need a working copy of the DarkShield-NoSQL API hosted locally on your computer. Contact your IRI representative for a free trial copy of the software.

NoSQL Search Contexts

The NoSQL API introduces an extension to the File Search Context, called a NoSQL Search Context, for defining search criteria for a NoSQL database. The following snippet of the OpenAPI definition shows the structure of its schema:

The NoSQL Search Context uses a name attribute to uniquely identify a context for performing search operations. The fileSearchContextName attribute indicates a File Search Context that will be associated with the NoSQL Search Context. There is also a configs attribute used to pass parameters related to connecting and interacting with a NoSQL database.

NoSQL Mask Contexts

The NoSQL API introduces an extension to the File Mask Context, a NoSQL Mask Context, for defining masking criteria for a NoSQL database. The following snippet of the OpenAPI definition shows the structure of its schema:

The NoSQL Mask Context name attribute uniquely identifies the context when performing masking operations. The fileMaskContextName attribute indicates the File Mask Context that will be associated with the NoSQL Mask Context. There is also a configs attribute used to pass parameters for connecting and interacting with a NoSQL database.

NoSQL Search Configs

Cassandra configurations:

	Key Name	Description	Optional or Required
Host	hostname	Host(s) for connections. Expected as “hostA” or “hostA,hostB,hostC”.	Optional: if no host is specified, will default to localhost.
Port	port	Port for connection. Expected as integer (i.e. 123458)	Optional: if no port is specified, will default to 9042.
User	username	Username for connection.	Optional: if no user is specified will connect without authenticating.
Password	password	Password for connection.	Optional: if no password is specified will connect without authenticating.
Data Center	dataCenter	Data Center associated with database.	Required
Table	collectionName	Specify table name.	Optional: if no table is specified will search all tables in a database.
DB Type	type	Type is cassandra.	Required

Elasticsearch configurations:

	Key Name	Description	Optional or Required
Host	hostname	Host(s) for connections. Expected as “hostA” or “hostA,hostB,hostC”.	Optional: if no host is specified, will default to localhost.
Port	port	Port for connection. Expected as integer (i.e. 123458)	Optional: if no port is specified, will default to 9200.
User	username	Username for connection.	Optional: if no user is specified will connect without authenticating.
Password	password	Password for connection.	Optional: if no password is specified will connect without authenticating.
Index	collectionName	Specify Index name.	Required
DB Type	type	Type is elasticsearch.	Required

MongoDB configurations:

	Key Name	Description	Optional or Required
URL	url	The url connection string. See MongoDB documentation for supported formats.	Required
Database	databaseName	Specify name of database.	Required
Collection	collectionName	Specify name of collection.	Optional: if no collection is specified, will search all collections in the database.
GridFS Bucket	gridFSBucket	Specify GridFS bucket name.	Optional: if the bucket is specified will ignore collections.
DB Type	type	Type is mongodb.	Required

NoSQL Mask Configs

Cassandra configurations:

	Key Name	Description	Optional or Required
Host	hostname	Host(s) for connections. Expected as “hostA” or “hostA,hostB,hostC”.	Optional: if no host is specified it will default to localhost.
Port	port	Port for connection. Expected as integer (i.e. 12345)	Optional: if no port is specified, will default to 9042.
User	username	Username for connection.	Optional: if no user is specified will connect without authenticating.
Password	password	Password for connection.	Optional: if no password is specified will connect without authenticating.
Data Center	dataCenter	Data Center associated with database.	Required
Table	collectionName	Specify table name.	Optional: if table is specified will write all content from source to target table else will Upsert to tables that share the same name as source.
DB Type	type	Type is cassandra.	Required

Elasticsearch configurations:

	Key Name	Description	Optional or Required
Host	hostname	Host(s) for connections. Expected as “hostA” or “hostA,hostB,hostC”.	Optional: if no host is specified it will default to localhost.
Port	port	Port for connection. Expected as integer (i.e. 12345)	Optional: if no port is specified it will default to 9200.
User	username	Username for connection.	Optional: if no user is specified will connect without authenticating.
Password	password	Password for connection.	Optional: if no password is specified will connect without authenticating.
Index	collectionName	Specify Index name.	Optional: if no index name is provided, will write to an index of the same name as source index.
DB Type	type	Type is elasticsearch.	Required

MongoDB configurations:

	Key Name	Description	Optional or Required
URL	url	The url connection string. See MongoDB documentation for supported formats.	Required
Database	databaseName	Specify name of database.	Required
Collection	collectionName	Specify name of collection.	Optional: if no collection is specified, will write to a collection of the same name as source index.
GridFS Bucket	gridFSBucket	Specify GridFS bucket name.	Optional: if no bucket is specified will upsert to a bucket with the same name as bucket from source.
DB Type	type	Type is mongodb.	Required

DarkShield-NoSQL API Endpoints

The NoSQL API is an extension of the Files API, which is an extension of the DarkShield base API. As such the NoSQL API requires not only NoSQL Search/Mask Contexts, but File Search/Mask Contexts and base API Search/Mask Contexts to be created via their respective endpoints.

A breakdown of API endpoint calls is as follows:

Search Job
- /api/darkshield/searchContext.create
- /api/darkshield/files/fileSearchContext.create
- /api/darkshield/nosql/nosqlSearchContext.create
- /api/darkshield/nosql/nosqlSearchContext.search
- /api/darkshield/nosql/nosqlSearchContext.destroy
- /api/darkshield/files/fileSearchContext.destroy
- /api/darkshield/searchContext.destroy
Mask Job
- /api/darkshield/maskContext.create
- /api/darkshield/files/fileMaskContext.create
- /api/darkshield/nosql/nosqlMaskContext.create
- /api/darkshield/nosql/nosqlMaskContext.mask
- /api/darkshield/nosql/nosqlMaskContext.destroy
- /api/darkshield/files/fileMaskContext.destroy
- /api/darkshield/maskContext.destroy
Search and Mask Job
- /api/darkshield/searchContext.create
- /api/darkshield/maskContext.create
- /api/darkshield/files/fileSearchContext.create
- /api/darkshield/files/fileMaskContext.create
- /api/darkshield/nosql/nosqlSearchContext.create
- /api/darkshield/nosql/nosqlMaskContext.create
- /api/darkshield/nosql/nosqlSearchContext.mask
- /api/darkshield/nosql/nosqlSearchContext.destroy
- /api/darkshield/nosql/nosqlMaskContext.destroy
- /api/darkshield/files/fileSearchContext.destroy
- /api/darkshield/files/fileMaskContext.destroy
- /api/darkshield/searchContext.destroy
- /api/darkshield/maskContext.destroy

Search/Mask Contexts and File Search/Mask Contexts Example

Below are examples of requests made to generate Search/Mask Contexts and File Search/Mask Contexts, using Postman to display the body of the POST request.

Search Context with email matcher

File Search Context

Mask Context with hash rule

File Mask Context

Cassandra NoSQL Search/Mask Context Example

Below are examples of requests made to generate NoSQL Search/Mask Contexts for Cassandra, using Postman to display the body of the POST request.

Cassandra NoSQL Search Context

Cassandra NoSQL Mask Context

Elasticsearch NoSQL Search/Mask Context Example

Below are examples of requests made to generate NoSQL Search/Mask Contexts for Elasticsearch, using Postman to display the body of the POST request.

Elasticsearch NoSQL Search Context

Elasticsearch NoSQL Mask Context

MongoDB NoSQL Search/Mask Context Example

Below are examples of requests made to generate NoSQL Search/Mask Contexts for MongoDB, using Postman to display the body of the POST request.

MongoDB NoSQL Search Context

MongoDB NoSQL Mask Context

NoSQL Search and Mask Operation

Performing search and mask operations using Search Contexts and Mask Contexts previously created

Original Mongo document

Emails were masked in the Mongo document

If you would like help searching or masking data in your NoSQL database – or in any other data source(s) – please contact your IRI representative or email darkshield@iri.com.