menu

Introduction

Welcome to MinusOneDB. Here you will find documentation, solutions and samples showing how to use the MinusOneDB platform. People interested in event capture should look at our event capture solution. Alternatively, use our ops api, environment api, and m1 client documentation to to build out your own solutions. You can immediately create and play with your own free trial here.

Most of the examples follow linux/unix conventions though we hope they will be fairly straightforward to follow in other environments. If, for this or any other reason, you run into any issues, need more help, or have a problem not covered by any of our existing solutions or samples, please let us know. We are always here to help!

m1 client

The m1 client is the command line interface to minusonedb. In addition to allowing you to execute all of the endpoints available in ops or one of your environments, the m1 client simplifies authentication and operations that require coordination between https://ops.minusonedb.com and your environments. For brevity's sake only m1 authentication and orchestration operations are described below; see the service api references for additional m1 sample usages.

Getting Started

Step 1: Download the latest m1 client zip file.

Step 2: Unzip the m1.zip file to a location on your computer.

Step 3: Add the unzipped m1 directory to your path based on your operating system's instructions.

Step 4: The m1 client requires java. Install java if it is not already present on your computer.

Step 5: Try running m1 in a terminal. You should see something like:

Usage: m1 <cmd> <args...>

If the m1 command is not found, make sure the m1 directory containing the m1.jar zip was successfully added to your path. If java is not found, doublecheck your java installation steps.

Signing Up

Only an email address is required to obtain a MinusOneDB ops login. Your ops login will let you create a free trial or manage any accounts and environments that other MinusOneDB users have shared with you. We will not send outbound marketing emails to you; if you receive an email from us it is only in support of the service we provide; our terms of use define how we want to do business with you.

To sign up, run:

m1 signup

You will be prompted for a username (which must be a valid email address) and a password. If successful, those credentials will be saved to a local directory and used subsequently to get tokens to access ops services.

From here you can keep on reading about how to use the m1 client or go start your free trial.

Authenticating

Connect to https://ops.minusonedb.com

m1 auth

Connect to your environment at https://envName-accountName.minusonedb.com

m1 auth envName-accountName

Connect to https://ops.minusonedb.com by running the following:

m1 auth

You will be prompted for your MinusOneDB username (email) and password; these credentials will be saved to a local directory and used subsequently to get tokens to access ops services.

Connect to one of your environments by running the following command, replacing envName and accountName appropriately:

m1 auth envName-accountName

As with ops, you will be prompted for your credentials to access your environment.

You will not need to rerun m1 auth unless your credentials change.

Creating an Environment

A sample m1 environment configuration file

{
    "environment": {
        "create" : {
            "servers": 1
        },
        "init": {
            "username": "admin",
            "password": "change this password"
        },
        "stores": [{
            "store": "index",
            "type": "standard",
            "shards": 1,
            "replicas": 1
        }],
        "sessionStores": [{
            "store": "stash",
            "shards": 1,
            "replicas": 1
        }],
        "configure": [{
            "system": {
                "geo": "true",
                "publish-permissive": "true"
            }
        },{
            "user/update": {
                "username": "admin",
                "rights": ["admin","get","publish","schema","sessionUpdate","sessionQuery"]
            }
        },{
            "user/add": {
                "username": "flush",
                "password": "change this password",
                "rights": ["publish","sessionUpdate","sessionQuery"]
            },
            "user/add": {
                "username": "instrument",
                "password": "this password does not matter",
                "rights": ["publish", "sessionUpdate"]
            }
        },{
            "store/autocommit": {
                "store": "index",
                "seconds": 2
            }
        },{
            "schema/add": {
                "properties": "@../schema/instrument.json"
            }
        },{
            "schema/add": {
                "properties": "@../schema/geo.json"
            }
        },{
            "schema/add": {
                "properties": "@../schema/sample.json"
            }
        }]
    }
}

Create a new environment by running the following:

m1 orchestrate deploy envName environment-configuration.json

envName is the name of the new environment to create. The configuration file specifies desired environment parameters -- the number of data processing servers, data store and session store topology, users and rights, as well system property configuration.

Sample environment template files are available with our downloadable samples. To use an environment template file:

Step 1: Make a copy of the environment template file.

Step 2: Set your admin username and password in the init block.

Step 3: Create any additional users who should have access this environment. The example configuration to the right creates users flush and instrument.

Step 4: You can make any other desired edits to the template file here.

Step 5: Run m1 orchestrate deploy envName your-copied-config-from-step-1.json.

Adding Stores

Add a data store named index and a session store named stash

m1 orchestrate addStores test-m1 addStores.json

An example addStores.json that adds a store and a session store:

{
    "environment": {
        "stores": [{
            "store": "index",
            "type": "standard"
        }],
        "sessionStores": [{
            "store": "stash"
        }]
    }
}

Create and add any number of data stores or session stores to an existing environment by running the following:

m1 orchestrate addStores envName-accountName addStores.json

As a convenience, you can use an environment config file as an argument to m1 orchestrate addStores; elements other than the stores and sessionStores entries will be ignored.

orchestrate addStores adds data store and session store configurations to an environment and then provisions corresponding data store and session store server resources via https://ops.minusonedb.com.

Dropping a Store

Drop a session store named index

m1 orchestrate dropStore store test-m1 index

Drop a session store named stash

m1 orchestrate dropStore session test-m1 stash

Drop and destroy a data store by running the following:

m1 orchestrate dropStore store envName-accountName dataStoreToDrop

This operation drops the data store configuration in the environment and then destroys the data store server resources via https://ops.minusonedb.com.

Drop and destroy a session store by running the following.

m1 orchestrate dropStore session envName-accountName sessionStoreToDrop

This operation drops the session store configuration in the environment and then destroys the session store server resources via https://ops.minusonedb.com.

Configuration File Format

eventsSessionStore.json template

{
    "environment": {
        "create" : {
            "servers": 1
        },
        "init": {
            "username": "admin",
            "password": "change this password"
        },
        "stores": [{
            "store": "index",
            "type": "standard",
            "shards": 1,
            "replicas": 1
        }],
        "sessionStores": [{
            "store": "stash",
            "shards": 1,
            "replicas": 1
        }],
        "configure": [{
            "system": {
                "geo": "true",
                "publish-permissive": "true"
            }
        },{
            "user/update": {
                "username": "admin",
                "rights": ["admin","get","publish","schema","sessionUpdate","sessionQuery"]
            }
        },{
            "user/add": {
                "username": "flush",
                "password": "change this password",
                "rights": ["publish","sessionUpdate","sessionQuery"]
            },
            "user/add": {
                "username": "instrument",
                "password": "this password does not matter",
                "rights": ["publish", "sessionUpdate"]
            }
        },{
            "store/autocommit": {
                "store": "index",
                "seconds": 2
            }
        },{
            "schema/add": {
                "properties": "@../schema/instrument.json"
            }
        },{
            "schema/add": {
                "properties": "@../schema/geo.json"
            }
        },{
            "schema/add": {
                "properties": "@../schema/sample.json"
            }
        }]
    }
}

The environment configuration file is a json map describing an environment. A fairly complete example configuration is shown to the right; a description of each key element of the configuration file is described below: - environment - create: Parameters for env/create (like the number of data processing servers). - init: Your initial admin user/password. Note how this element corresponds to init. - stores: A list of data stores to add. - sessionStores: a list of session stores to add. - configure: a list of maps representing environment service operations to configure a store. Note how a user/update, /system, or schema/add elements in the example configuration relate to the corresponding api method. This pattern can be extended to other environment service endpoints.

Start your free trial!

The best way to learn how to use new technology is to play with it. To that end, we offer a 1 week free trial that allows you to use minusonedb and judge its value for yourself. Please contact us if there is anything that we can do to make your trial more effective. We are always here to help.

To set up a free trial environment:

Step 1: Set up the m1 client and sign up if you have not already done so.

Step 2: Create a trial environment by running the following, changing the parameters as appropriate:

m1 orchestrate deploy yourtrialname-trial events.json

Replace yourtrialname with whatever you would prefer (the string can only contain alphanumeric characters and cannot be in use by another trial user). Replace events.json with an m1 environment config file that suits your needs. The various event capture sample walkthroughs contain example templates or you can make your own.

The orchestrate command will take a 5-10 minutes to complete. Once your trial environment is up, you can interact with it in the same way you would a paid minusonedb environment: see your trial environment configuration, register buckets to load your own data from S3, or use any of the environment services to interact with your environment now hosted at https://yourtrialname-trial.minusonedb.com.

Note the following limitations with trial environments:

If at any point you wish to pause or end your trial, simply destroy your active trial environment (you can use any time you still have left on a future free trial). If you need to extend your trial or you want to become a minusonedb customer, reach out to us.

Event capture

Our event capture solution should have everything you need to understand user behavior in any context. The samples below illustrate common patterns for identifying, storing, and querying events. You can create events when a customer adds an item to a shopping cart, swipes right on a mobile app, leaves your site or anything else of interest to you. Or you can apply the same patterns to capture data from devices for monitoring or telemetry purposes.

To get started by instrumenting a website or other JavaScript app continue to the next section. If you would prefer to explore using simulated traffic jump to the Event Simulator setup instructions. This can be useful for testing before your website goes live or for testing large volumes of events.

Instrument Your Website

See how users interact with your web application.

Setup Time: 20 minutes

Run Time: Immediate

This tutorial shows how to instrument your web application using MinusOneDB. This will allow you to understand how users are interacting with your website in great detail. Once you have completed the steps below, user interaction events will be sent from your website to your minuseondb environment. These events can be queried in near real-time to understand how users are interacting with your site. This tutorial will also guide you through some common customization paths to further enrich the data you can capture.

Prerequisites

Step 1: Download and unzip the instrumentation sample.

Step 2: Set up the m1 client if you have not already done so.

Step 3: You will need a MinusOneDB environment:

Instructions

Step 1: Copy the instrumentation.js, M1Events.js, and lib/jquery-3.6.0.min.js files from the instrument/src sample directory into the same folder as your web application's other JavaScript files. There is no need to copy the jQuery file if your site already uses it.

To make a new user with appropriately restrictive rights:

m1 envName-accountName user/add -username instrument -password ChangeMeNow! -rights '["publish"]'

curl https://envName-accountName.minusonedb.com/user/add \
-d 'username=instrument&password=ChangeMeNow!&rights=["publish"]' \
-H "m1-auth-token: $adminToken"

To restrict the rights of an existing user to only publish:

m1 envName-accountName user/update -username instrument -rights '["publish"]'

curl https://envName-accountName.minusonedb.com/user/update \
-d 'username=instrument&rights=["publish"]' \
-H "m1-auth-token: $adminToken"

Step 2: Set the environment name and user account in the "REQUIRED PARAMETERS" section of the copied instrumentation.js file. Typically this account should be a system account not used for any other purpose. This account should only have the publish right.

Step 3: Source the JavaScript files copied above by adding the following lines to the head of your html file (or other location where you import JavaScript files); this is often in an index.html file.

<script src="js/jquery-3.6.0.min.js" type="text/javascript" charset="utf-8"></script>

<script src="js/instrumentation.js" type="text/javascript" charset="utf-8"></script>

<script src="js/M1Events.js" type="text/javascript" charset="utf-8l"></script>

Add localhost.somethingunique to the cors domain whitelist:

m1 envName-accountName system/cors/add -domains localhost.somethingunique

curl https://envName-accountName.minusonedb.com/system/cors/add \
-d "domains=localhost.somethingunique" \
-H "m1-auth-token: $myToken"

Step 4: The sample instrumentation.js file contains reasonable defaults for instrumenting a single page web app; sessionStart events are recorded when a user first comes to a site and view events are created as users navigate via hyperlinks. instrumentation.js contains some other sample event capture patterns that you may want to adopt or customize for your own needs.

Step 5: You will need to add your website's domain to your environment's CORS domain whitelist. You can add as many domains as needed. If you are testing on localhost, add a unique entry to your hosts file that resolves to localhost, i.e., 127.0.0.1 localhost.somethingunique and then add that as whitelist entry using one of the commands to the right. We encourage you to add the entry to hosts rather than directly adding localhost as a whitelisted domain to prevent any chance of running into a CORS related vulnerability.

See the 5 most recent events:

m1 envName-accountName query -store index -q '*' -sort "_m1.receivedUTC desc" -rows 5

curl https://envName-accountName.minusonedb.com/query \
-d "store=index&q=*&sort=_m1.receivedUTC desc&rows=5" \
-H "m1-auth-token: $myToken"

See events by state:

m1 envName-accountName query -store index -q '*' -json '{
  "facet": {
    "by_state": {
      "field": "_m1.geo.subdivision_1_iso_code",
      "type": "terms"
    }
  }
}'

curl https://envName-accountName.minusonedb.com/query -d 'store=index&q=*&json={
  "facet": {
    "by_state": {
      "field": "_m1.geo.subdivision_1_iso_code",
      "type": "terms"
    }
  }
}' -H "m1-auth-token: $myToken"

Step 6: Open up your site in a browser; if your site was hosted at localhost, be sure to use the hosts entry you created above in place of localhost (i.e. http://localhost.somethingunique:8080/index.html rather than http://localhost:8080/index.html). As you navigate around your site, events will flow into your MinusOneDB environment; you can query your events using our reporting application or by using the sample queries to the right.

Troubleshooting Tip: If the queries above are not returning results, open your browser's console and check if there are any errors. Doublecheck each of the steps above, making sure your site has loaded the custom JavaScript, your user credentials are correct and there are no other obvious errors. If you get stuck, reach out to us.

Congratulations! You have now successfully instrumented your website. You are capturing web events as they occur and you can query this data in near real-time. Continue reading for some guidance on taking this to production.

Next Steps

Before deploying these changes to your production web site:

Add minusonedb.com to the cors domain whitelist:

m1 envName-accountName system/cors/add -domains minusonedb.com

curl https://envName-accountName.minusonedb.com/system/cors/add \
-d "domains=minusonedb.com" \
-H "m1-auth-token: $myToken"

Step 1: In instrumentation.js, set the async parameter to true. This allows the MinusOneDB servers to optimize the storage of events in your production environment.

Step 2: Add your production website domain to your MinusOneDB environment's cors whitelist. Replace minusonedb.com with your domain as in the examples to the right.

Step 3: There are a number of sections in instrumentation.js that may be of further interest. In particular, consider adding heartbeat events to gain a much richer time series based perspective of your users' behavior.

Heartbeats

See how users interact with your web application over time.

Setup Time: 5 or 25 minutes (depends on whether you have worked through the instrumentation sample).

Run Time: Immediate

Heartbeats are periodic events that record that a user is still present. They allow you to understand how long a user stays on your site and when they leave it. Heartbeats directly allow you to calculate the number of concurrent users on your site that can then be broken down by geography or individual page. In addition to the notion of presence, any attributes available on heartbeat events allow for a time series perspective that is not possible otherwise. What pages on your site do users spend the longest time on? What is the most frequently viewed viewed page right before a user leaves your site? Heartbeat events can help answer these and many other questions.

The volume of heartbeats events will typically dwarf that of user interaction events. While we have found a great deal of benefit in heartbeat events at a 5 second frequency, some customers will want to optimize for cost. This can done by lowering the heartbeat frequency or by not persisting heartbeats at all and instead using them only to calculate user departure. See configuring heartbeat events with a session store or continue reading to persist all heartbeat events. We strongly encourage persisting heartbeats to start. The persisted heartbeats will be more cost-effective until you have collected 1-2 million hours of recorded user interactions. By then you will have a better understanding of the value of these and other events to your business.

Prerequisites

Step 1: Work through the steps of the Instrument Your Website sample above.

Instructions

See the 3 most recent heartbeat events:

m1 envName-accountName query -store index -q "event.type:heartbeat" -sort "_m1.receivedUTC desc" -rows 3

curl https://envName-accountName.minusonedb.com/query \
-d "store=index&q=event.type:heartbeat&sort=_m1.receivedUTC desc&rows=3" \
-H "m1-auth-token: $myToken"

Step 1: In the instrumentation.js file you edited above, configure an appropriate HEARTBEAT_INTERVAL_MS value and uncomment the line that saves all heartbeats.

Step 2: Open up your site in a web browser. Heartbeat events will periodically be sent to your minusonedb environment. You can query your events using our reporting application or by using the sample queries to the right.

Heartbeats on a Budget

Capture heartbeats in ephemeral storage to understand concurrent user metrics while reducing costs for large sites.

Setup Time: 10 or 30 minutes (depends on whether you have worked through the instrumentation sample).

Run Time: Immediate

While heartbeat events offer many benefits, their number will typically exceed the number of non-heartbeat user interaction events. You may not wish to persist a high volume of potentially similar heartbeat events, while still desiring analytics around user presence. Rather than storing heartbeats in a datalake and indexed store, this sample shows how a client can send heartbeats to a session store and have a separate session flusher process determine when sessions have timed out. In other words, this sample allows you to record only the last (and most analytically interesting) heartbeat for a given session and foregoes persisting the intervening ones. Note that the added cost of the session store is roughly the equivalent in price of storing an additional 1-2 million hours of user interactions; at smaller volumes persisting heartbeats will be more cost-effective.

Prerequisites

Step 1: Work through the steps of the Instrument Your Website sample. You can save a step if you use the eventsSessionStore.json environment template file rather than the events.json described in that sample.

*Step 2: Download and unzip the session flusher sample.

An example addSessionStore.json to add a new session store with name stash:

{
    "environment": {
        "stores": [
        ],
        "sessionStores": [{
            "store": "stash",
            "shards": 1,
            "replicas": 1
        }]
    }
}

If you have already created an environment from events.json you can modify it to look like the eventsSessionStore environment by performing the following operations. These commands will add a session store, create a flush user with appropriate permissions, and allow the instrument user to write to the session store.

m1 orchestrate addStores envName-accountName envKey addSessionStore.json

m1 envName-accountName user/add -username flush -password "change this password" -rights '["publish", "sessionUpdate", "sessionQuery"]'

m1 envName-accountName user/update -username instrument -rights '["publish", "sessionUpdate"]'

Step 3: Install and configure python if you have not already done so.

Instructions

Configure the Session Flusher

Sample config.json.sample for reference:

{
    "server"   : "server name (ex: https://live-m1.minusonedb.com)",
    "session"  : "session name (ex: stash)",
    "username" : "username",
    "password" : "password",

    "timeoutSeconds" : "time (in seconds) before the flusher considers a session to have timed out;
                      you'll generally want this to be 3-5x your heartbeat period."
}

Step 1: In the flusher/src directory, copy config.json.sample to config.json. Edit your environment, environment credentials, session store, and data store in config.json

Step 2: Run python3 -u flusher.py config.json

Step 3: You should see some output saying that there is "nothing to flush". flusher.py checks every few seconds to see if there are timed out sessions.

Step 4: Leave flusher.py running; you will come back to it after configuring instrumentation.js

Configure instrumentation.js

Step 1: Open the instrumentation.js file you have already edited and find the "Using a session store with heartbeats" section.

See heartbeats in the session store:

m1 envName-accountName session/query -store stash -q "*"

curl https://envName-accountName.minusonedb.com/session/query \
-d "store=stash&q=*" \
-H "m1-auth-token: $myToken"

Step 2: Configure the session store name, configure the heartbeat frequency, and uncomment blocks of code as directed in instrumentation.js

Step 3: Open up your site in a web browser. Heartbeat events will be periodically sent to your configured session store.

Step 4: Query for heartbeats in the session store. You should see a document representing your active session.

See persisted sessionTimeout events:

m1 envName-accountName query -store index -q "type:sessionTimeout" -rows 5

curl https://envName-accountName.minusonedb.com/query \
-d "store=index&q=type:sessionTimeout" \
-H "m1-auth-token: $myToken"

Step 5: Now close the browser that was visiting your site -- heartbeats will no longer be sent to the session store. Navigate back to where you ran flusher.py. After the appropriate amount of time has elapsed (configurable via the timeoutSeconds parameter in the flusher.py config file), you will see a "Preparing to flush" message in the flusher.py output signifying that flusher.py found your session that has timed out.

Step 6: Additionally, you will see a sessionTimeout persisted event recording the time your session timing out. You can use the sample query to the right to retrieve these events.

Congratulations! You now have system that allows you to determine the number of users that are actively using your site at any point in time at a minimal cost.

Event Simulator

Generate synthetic web events to familiarize yourself with the MinusOneDB event capture solution rather than instrumenting your own website.

Setup Time: 20 minutes

Run Time: Immediate

This python sample simulates user interactions on a web page. Use it to quickly create some test data to gain familiarity with MinusOneDB concepts. Run a number of simulator processes concurrently to simulate high traffic. Use in concert with the session flusher to test your own session flushing criteria.

Prerequisites

Step 1: Download and unzip the event simulator sample.

Step 2: Set up the m1 client if you have not already done so.

Step 3: You will need a MinusOneDB environment:

Step 4: Install and configure python if you have not already done so.

Instructions

config.json.sample for reference:

{
    "server"   : "server name (ex: https://live-m1.minusonedb.com)",
    "session"  : "session name (ex: stash).  Set this param to null to simulate persisting heartbeats",
    "username" : "username",
    "password" : "password"
}

In the simulator samples directory, copy config.json.sample to config.json. Edit your environment, environment credentials, session store, and data store in config.json.

Simulate Web Events

Run this command to execute the python simulator:

python3 simulateClient.py config.json

The script simulates actions users perform on a webpage: arriving at your site, navigating to pages, and leaving the site. Heartbeat events are also simulated; they will be sent to a session store if you have configured one. Otherwise heartbeats will be archived in your environment like all other events.

Simulate Many Web Events

See heartbeats in the session store:

m1 envName-accountName session/query -store stash -q "*"

curl https://envName-accountName.minusonedb.com/session/query \
-d "store=stash&q=*" \
-H "m1-auth-token: $myToken"

See the 5 most recent events:

m1 envName-accountName query -store index -q '*' -sort "_m1.receivedUTC desc" -rows 5

curl https://envName-accountName.minusonedb.com/query \
-d "store=index&q=*&sort=_m1.receivedUTC desc&rows=5" \
-H "m1-auth-token: $myToken"

See events by state:

m1 envName-accountName query -store index -q '*' -json '{
  "facet": {
    "by_state": {
      "field": "_m1.geo.subdivision_1_iso_code",
      "type": "terms"
    }
  }
}'

curl https://envName-accountName.minusonedb.com/query -d 'store=index&q=*&json={
  "facet": {
    "by_state": {
      "field": "_m1.geo.subdivision_1_iso_code",
      "type": "terms"
    }
  }
}' -H "m1-auth-token: $myToken"

To simulate N concurrent users interacting with a webste, run the following:

Step 1: ./runSim config.json N

Step 2: Running tail -f *.log will give you a view into what is happening with your simulated clients.

Step 3: When you are done simulating, you can stop the running simulateClient processes by running ./stopSim

Step 4: (If using a Session Store) Query for heartbeats in the session store. You should see a document representing your active session.

Step 5: You can query persisted events using our reporting application or by using the sample queries to the right.

General purpose samples

Data analytics samples show a number of common data loading and management patterns when using MinusOneDB. Learn how to load large amounts of data as quickly as possible or how to resize stores as your usage grows.

Load Sample Data

Load 50 million Reddit comments in an hour.

Setup Time: 15 minutes

Data Load Time: 40 minutes

This python sample shows how to load 50 million Reddit comments into a MinusOneDB environment as quickly as possible. It illustrates common MinusOneDB patterns to maximize loading performance that you can use when you load your own data. Loading larger datasets works in the same way; with a corresponding increase in provisioned hardware, any dataset can be predictably loaded in the same amount of time. Once loaded, explore the sample Reddit dataset using the m1 client or our reporting application.

Prerequisites

Step 1: Download and unzip the reddit loading sample.

Step 2: Set up the m1 client if you have not already done so.

Step 3: You will need a MinusOneDB environment:

Note: Because this sample uses more than 1 data processing server, it will not work with our trial environments. Contact [email protected] if you want to try this before becoming a MinusOneDB customer.

Step 4: Install and configure python if you have not already done so.

Instructions

For reference, here is a copy of ops.json.sample

{
    "server"   : "https://ops.minusonedb.com",
    "username" : "<username>",
    "password" : "<password>"
}

For reference, here is a copy of env.json.sample

{
    "server"   : "https://envName-accountName.minusonedb.com",
    "username" : "<username>",
    "password" : "<password>"
}

Your environment key can be found using the following: m1 ops env/list -account accountId

Step 1: Navigate to the bulkload-reddit/src sample directory.

Step 2: Copy ops.json.sample to ops.json and update ops.json with your ops.minusonedb.com username and password.

Step 3: Copy env.json.sample to env.json and update your environment name, username, and password.

Step 4: Run python3 bulkload-sample.py ops.json env.json

The script will print out progress as it performs the following actions:

See the number of Reddit comments:

m1 <envname> query -store index -q "*"

curl https://<envname>.minusonedb.com/query \
-d "store=index&q=*" \
-H "m1-auth-token: $myToken"

See the top 10 most popular subreddits:

m1 <envname> query -store index -q "*" -json '{
    "facet": {
        "by_score": {
            "field": "subreddit",
            "type": "terms", 
            "sort" : "count desc"
            "limit" : 10,
        }
    }
}'

curl https://<envname>.minusonedb.com/query -d 'store=index&q=*&json={
    "facet": {
        "by_score": {
            "field": "subreddit",
            "type": "terms",
            "sort" : "count desc",
            "limit" : 10
        }
    }
}' -H "m1-auth-token: $myToken"

You can now explore the Reddit data with the sample queries to the right or with our reporting application.

Ops api reference

The ops services documented below are hosted at https://ops.minusonedb.com/ and allow you to manage your account(s) and environments. You can interact with all of the service endpoints described below via conventional HTTP clients (curl, postman) or programmatically in your desired language or framework. We offer the m1 client which simplifies credential management and environment configuration.

Services use typical HTTP response codes.

2xx: The request operation worked as as expected.
401: The token used for the request was invalid.
403: The user associated with the token used for the request does not have sufficient rights to perform the requested operation.
500: The request could not be completed.

Authentication and Access

/signup

m1 signup

curl https://ops.minusonedb.com/signup \
-d "[email protected]&password=passphrase"

Creates a user for you on ops. If successful, you can immediately authenticate using the username and password you supplied.

/auth

m1 auth

curl https://ops.minusonedb.com/auth \
-d "[email protected]&password=passphrase"

Sample Response:

eFRxichv7cTeqks99XD+iNBqR3LlgILIZ3nhKieS8c8qajV9+JcNqSqZfngvS+gJM3IBKnvI2Crd66WOIswtlr7ps5DnOtCp5KjJu/lUKN2bCiUUGKJKHc8wYgUKienHOoL00LIbsLvCMQjcjoW9U+u88ZEqqvyE/qhSmTJSONdnEicbHYzy9pjJ0+RcrFyP

Log in and retrieve token for service access.

/user/password

m1 ops user/password -password newPassphrase

curl https://ops.minusonedb.com/user/password \
-d "password=newPassphrase" \
-H "m1-auth-token: $myToken"

No response when successful.

Change user password

A correct auth token must be included in the request.

Accounts

/account/list

m1 ops account/list

curl https://ops.minusonedb.com/account/list \
-H "m1-auth-token: $myToken"

Sample Response:

[
    {
        "account" : "31e1192e-5415-4d85-a45e-b88897c96c1b",
        "name" : "m1",
        "rights" : [
            "view",
            "manage"
        ],
        "username" : "[email protected]"
    }
]

List all accounts the current user has access to

/account/user/list

m1 ops account/user/list -account 31e1192e-5415-4d85-a45e-b88897c96c1bd

curl https://ops.minusonedb.com/account/user/list?account=31e1192e-5415-4d85-a45e-b88897c96c1bd \
-H "m1-auth-token: $myToken"

Sample Response:

[ {
  "account" : "31e1192e-5415-4d85-a45e-b88897c96c1bd",
  "rights" : [ "manage", "view" ],
  "username" : "[email protected]"
}, {
  "account" : "31e1192e-5415-4d85-a45e-b88897c96c1bd",
  "rights" : [ "admin", "owner" ],
  "username" : "[email protected]"
} ]

List all users associated with this account

/account/user/add

m1 ops account/user/add -account 31e1192e-5415-4d85-a45e-b88897c96c1b -username [email protected] -rights '["view"]'

curl https://ops.minusonedb.com/account/user/add \
-d 'account=31e1192e-5415-4d85-a45e-b88897c96c1b&[email protected]&rights=["view"]' \
-H "m1-auth-token: $myToken"

No response when successful.

Add access to an account for the specified user

/account/user/update

m1 ops account/user/update -account 31e1192e-5415-4d85-a45e-b88897c96c1b -username [email protected] \
-rights '["manage", "view"]'

curl https://ops.minusonedb.com/account/user/update \
-d 'account=31e1192e-5415-4d85-a45e-b88897c96c1b&[email protected]&rights=["manage","view"]' \
-H "m1-auth-token: $myToken"

No response when successful.

Update accounts rights for specified user

/account/user/remove

m1 ops account/user/remove -account 31e1192e-5415-4d85-a45e-b88897c96c1b -username [email protected]

curl https://ops.minusonedb.com/account/user/remove \
-d "account=31e1192e-5415-4d85-a45e-b88897c96c1b&[email protected]" \
-H "m1-auth-token: $myToken"

No response when successful.

Remove all access to account for specified user

Environments

/env/get

m1 ops env/get -env 24e27632d234452ba37e60936c0e88eb

curl https://ops.minusonedb.com/env/get?env=24e27632d234452ba37e60936c0e88eb \
-H "m1-auth-token: $myToken"

Sample Response:

{
    "bucket" : "m1-24e27632d234452ba37e60936c0e88eb",
    "servers" : 1,
    "accountName" : "m1",
    "dns" : "test-m1.minusonedb.com",
    "name" : "test-m1",
    "account" : "31e1192e-5415-4d85-a45e-b88897c96c1bd",
    "key" : "24e27632d234452ba37e60936c0e88eb",
    "sourceBuckets" : [
        "m1-resources"
    ]
}

Retrieve metadata associated with the specified environment

/env/list

m1 ops env/list -account 31e1192e-5415-4d85-a45e-b88897c96c1b

curl https://ops.minusonedb.com/env/list?account=31e1192e-5415-4d85-a45e-b88897c96c1b \
-H "m1-auth-token: $myToken"

Sample Response:

{
    "bucket" : "m1-24e27632d234452ba37e60936c0e88eb",
    "servers" : 1,
    "accountName" : "m1",
    "dns" : "test-m1.minusonedb.com",
    "name" : "test-m1",
    "account" : "31e1192e-5415-4d85-a45e-b88897c96c1b",
    "key" : "24e27632d234452ba37e60936c0e88eb",
    "sourceBuckets" : [
        "m1-resources"
    ]
}

List all environments associated with the specified account

/env/create

m1 ops env/create -env newenvironment-m1 -servers 2

curl https://ops.minusonedb.com/env/create -d "env=newenvironment-m1&servers=2" \
-H "m1-auth-token: $myToken"

Sample Response:

{
    "bucket" : "m1-24e27632d234452ba37e60936c0e88eb",
    "servers" : 1,
    "accountName" : "m1",
    "dns" : "newenvironment-m1.minusonedb.com",
    "name" : "newenvironment-m1",
    "account" : "31e1192e-5415-4d85-a45e-b88897c96c1b",
    "key" : "24e27632d234452ba37e60936c0e88eb",
    "sourceBuckets" : [
        "m1-resources"
    ]
}

Create a new environment.

/env/rename

m1 ops env/rename -env myenv-m1 -name mynewenv-m1

curl https://ops.minusonedb.com/env/rename -d "env=myenv-m1&name=mynewenv-m1" \
-H "m1-auth-token: $myToken"

No response when successful.

Renames an existing environment. Can be used to move an environment from one account to another but the user must have manage permissions for both the source and destination accounts.

/env/destroy

m1 ops env/destroy -env 24e27632d234452ba37e60936c0e88eb

curl https://ops.minusonedb.com/env/destroy \
-d "env=24e27632d234452ba37e60936c0e88eb" \
-H "m1-auth-token: $myToken"

No response when successful.

Destroy an environment and all associated resources

/env/rescale

m1 ops env/rescale -env 24e27632d234452ba37e60936c0e88eb -servers 2 -threshold 2

curl https://ops.minusonedb.com/env/rescale \
-d "env=24e27632d234452ba37e60936c0e88eb&servers=2&threshold=2" \
-H "m1-auth-token: $myToken"

No response when successful.

Change the number of data processing servers for the specified environment

/env/reboot

m1 ops env/reboot -env 24e27632d234452ba37e60936c0e88eb -group index 

curl https://ops.minusonedb.com/env/reboot \
-d "env=24e27632d234452ba37e60936c0e88eb&group=index" \
-H "m1-auth-token: $myToken"

No response when successful.

Reboots all servers in a server group; a server group encompasses the servers associated for a particular data store or session store.

/env/store/create

m1 ops env/store/create -env 24e27632d234452ba37e60936c0e88eb -store index -type standard -shards 1 -replicas 2

curl https://ops.minusonedb.com/env/store/create \
-d "env=24e27632d234452ba37e60936c0e88eb&store=index&type=standard&shards=1&replicas=2" \
-H "m1-auth-token: $myToken"

No response when successful.

Create a new data store for the specified environment.

/env/store/destroy

m1 ops env/store/destroy -env 24e27632d234452ba37e60936c0e88eb -store index

curl https://ops.minusonedb.com/env/store/destroy \
-d "env=24e27632d234452ba37e60936c0e88eb&store=index" \
-H "m1-auth-token: $myToken"

No response when successful.

Destroy a data store from the specified environment.

/env/session/create

m1 ops env/session/create -env 24e27632d234452ba37e60936c0e88eb -store stash -shards 1 -replicas 2

curl https://ops.minusonedb.com/env/session/create \
-d "env=24e27632d234452ba37e60936c0e88eb&store=stash&shards=1&replicas=2" \
-H "m1-auth-token: $myToken"

No response when successful.

Create a new session store for the specified environment.

/env/session/destroy

m1 ops env/session/destroy -env 24e27632d234452ba37e60936c0e88eb -store stash

curl https://ops.minusonedb.com/env/session/destroy \
-d "env=24e27632d234452ba37e60936c0e88eb&store=stash" \
-H "m1-auth-token: $myToken"

No response when successful.

Destroy a session store from the specified environment.

/env/bucket/register

m1 ops env/bucket/register -env 24e27632d234452ba37e60936c0e88eb -bucket myBucket

curl https://ops.minusonedb.com/env/bucket/register \
-d "env=24e27632d234452ba37e60936c0e88eb&bucket=myBucket" \
-H "m1-auth-token: $myToken"

No response when successful.

Register a new bucket for the specified environment

/env/bucket/deregister

m1 ops env/bucket/deregister -env 24e27632d234452ba37e60936c0e88eb -bucket myBucket

curl https://ops.minusonedb.com/env/bucket/deregister \
-d "env=24e27632d234452ba37e60936c0e88eb&bucket=myBucket" \
-H "m1-auth-token: $myToken"

No response when successful.

Deregister an existing bucket from the specified environment

Rights

Rights are case-sensitive strings associated with a specific account; a user may have access to many (or no) accounts with varying levels of access. The collection of rights is extensible for application-specific uses. The list below details all ops.minusonedb.com rights.

owner: user can delete the account or change payment information.
admin: user can manage users and assign rights.
view: user can view all account assets including environment names and metadata.
manage: user can manage environment assets including deploying or undeploying infrastructure as well as creating and destroying environments.

Environment api reference

The services documented below let you interact with one of your MinusOneDB environments. All of the service endpoints described can be accessed via conventional HTTP clients (curl, postman) or programmatically in your desired language or framework. We offer the m1 client which simplifies credential management and environment interaction.

Services use typical HTTP response codes.

2xx: The request operation worked as as expected.
401: The token used for the request was invalid.
403: The user associated with the token used for the request does not have sufficient rights to perform the requested operation.
500: The request could not be completed

Authentication and Access

/auth

m1 auth test-m1

curl https://test-m1.minusonedb.com/auth \
-d "username=admin&password=passphrase"

Sample Response:

eFRxichv7cTeqks99XD+iNBqR3LlgILIZ3nhKieS8c8qajV9+JcNqSqZfngvS+gJM3IBKnvI2Crd66WOIswtlr7ps5DnOtCp5KjJu/lUKN2bCiUUGKJKHc8wYgUKienHOoL00LIbsLvCMQjcjoW9U+u88ZEqqvyE/qhSmTJSONdnEicbHYzy9pjJ0+RcrFyP

Log in and retrieve token for service access. By default, tokens must be used from the machine they were generated and will expire after a user defined time.

/user/password

m1 test-m1 user/password -password "newpassphrase"

curl https://test-m1.minusonedb.com/user/password \
-d "password=newpassphrase" \
-H "m1-auth-token: $myToken"

No response when successful.

Change user password

Users and Tokens

/user/list

m1 test-m1 user/list

curl https://test-m1.minusonedb.com/user/list \
-H "m1-auth-token: $myToken"

Sample Response:

[ {
  "username" : "user",
  "rights" : [ "schema", "get", "publish" ]
}, {
  "username" : "admin",
  "rights" : [ "schema", "sessionQuery", "get", "publish", "admin", "sessionUpdate" ]
} ]

List all users and associated metadata.

/user/get

m1 test-m1 user/get -username user

curl https://test-m1.minusonedb.com/user/get?username=user \
-H "m1-auth-token: $myToken"

Sample Response:

[ {
  "username" : "user",
  "rights" : [ "schema", "get", "publish" ]
} ]

Retrieve metadata for specified user.

/user/add

m1 test-m1 user/add -username newUser \
-password passphrase -rights '["schema","get"]'

curl https://test-m1.minusonedb.com/user/add \
-d 'username=newUser&password=passphrase&rights=["schema","get"]' \
-H "m1-auth-token: $myToken"

No response when successful.

Add new user and specify rights.

/user/update

m1 test-m1 user/update -username user -rights '["get", "publish"]'

curl https://test-m1.minusonedb.com/user/update \ 
-d 'username=user&rights=["get", publish"]' \
-H "m1-auth-token: $myToken"

No response when successful.

Update user metadata.

/user/remove

m1 test-m1 user/remove -username user

curl https://test-m1.minusonedb.com/user/remove \
-d "username=user" \
-H "m1-auth-token: $myToken"

No response when successful.

Remove user; the user will no longer be able to access the associated environment.

/token/list

m1 ops token/list

curl https://ops.minusonedb.com/token/list \
-H "m1-auth-token: $myToken"

Sample Response:

[ {
  "expires" : 1637190931886,
  "id" : "f4946f10-c6d6-4345-9a8f-de6aec884186",
  "ip" : "120.1.88.123",
  "username" : "[email protected]" 
} ]

List all tokens.

/token/data

m1 test-m1 token/data -token eFRxichv7cTeqks99XD+iNBqR3LlgILIZ3nhKieS8c8qajV9+JcNqSqZfngvS+gJM3IBKnvI2Crd66WOIswtlr7ps5DnOtCp5KjJu/lUKN2bCiUUGKJKHc8wYgUKienHOoL00LIbsLvCMQjcjoW9U+u88ZEqqvyE/qhSmTJSONdnEicbHYzy9pjJ0+RcrFyP

curl https://test-m1.minusonedb.com/token/data \
-d "token=eFRxichv7cTeqks99XD+iNBqR3LlgILIZ3nhKieS8c8qajV9+JcNqSqZfngvS+gJM3IBKnvI2Crd66WOIswtlr7ps5DnOtCp5KjJu/lUKN2bCiUUGKJKHc8wYgUKienHOoL00LIbsLvCMQjcjoW9U+u88ZEqqvyE/qhSmTJSONdnEicbHYzy9pjJ0+RcrFyP"
-H "m1-auth-token: $myToken"

Sample Response:

{
    "expires" : 1637456697412,
    "ip" : "96.253.115.241",
    "id" : "687569bd-dfde-463f-b85d-dd8b1a7ea50c",
    "user" : "delete"
}

Decodes the elements that make up a token.

/token/delete

m1 test-m1 token/delete

curl -X POST https://test-m1.minusonedb.com/token/delete -H "m1-auth-token: $myToken"

No response when successful.

Deletes all active tokens in an environment. This will require that all users reauthenticate.

Schema

/schema

m1 test-m1 schema

curl https://test-m1.minusonedb.com/schema - "m1-auth-token: $myToken"

Sample Response:

{
  "_m1rand" : {
    "large" : false,
    "name" : "_m1rand",
    "description" : "Random number assigned at publish time used for sampling use cases.",
    "type" : "double",
    "required" : true,
    "multi" : false
  },
  "myTags" : {
    "large" : false,
    "name" : "myTags",
    "description" : "Represents important tags.",
    "type" : "string",
    "required" : false,
    "multi" : true
  },
  "_m1key" : {
    "large" : false,
    "name" : "_m1key",
    "description" : "Unique identifier across an environment.",
    "type" : "integer",
    "required" : true,
    "multi" : false
  },
  "myBoolean" : {
    "large" : false,
    "name" : "myBoolean",
    "type" : "boolean",
    "required" : false
  }
}

Retrieves current schema.

/schema/add

m1 test-m1 schema/add -properties '[
    {
        "name": "myTags",
        "type": "string",
        "multi": true,
        "description": "Represents important tags."
    },
    {
        "name": "myBoolean",
        "type": "boolean"
    }
]'

m1 test-m1 schema/add -properties @fileWithSchema.json


curl https://test-m1.minusonedb.com/schema/add 'properties=[
    {
        "name": "myTags",
        "type": "string",
        "multi": true,
        "description": "Represents important tags."
    },
    {
        "name": "myBoolean",
        "type": "boolean"
    }
]' -H "m1-auth-token: $myToken"


curl https://test-m1.minusonedb.com/schema/add "properties=$(<$fileWithSchema.json)" -H "m1-auth-token: $myToken"

No response when successful.

Add properties to the environment schema. Can be called as many times as desired to add new property definitions.

Each property is a map with the following keys
- name (string, required)
- type (string, required, one of string, text, integer, double, date)
- multi (boolean, default false, if true the property can contain multiple values)
- description (string, default null, comment describing property, purely for informational purposes)

/schema/wipe

m1 test-m1 schema/wipe

curl -X POST https://test-m1.minusonedb.com/schema/wipe -H "m1-auth-token: $myToken"

No response when successful.

Deletes all user defined properties in the environment schema. Note that this does not modify data that has already been written to the data lake.

Archive

/write

m1 test-m1 write \
-items '[{"property1":{"ip":"127.0.0.1"},"session":null,"event":{"id":"key98765","time":946702800,"type":"sessionStart"}}]' -publish true

curl https://test-m1.minusonedb.com/write \
-d 'items=[{"property1" : {"ip" : "127.0.0.1"}, "session" : null, "event": {"id": "key98765", "time": 946702800, "type": "sessionStart"}}]&publish=true' \
-H "m1-auth-token: $myToken"

No response when successful.

Save data to the archive.

In a production setting, if you are publishing data immediately after archiving, we recommend you set publish-permissive to true.

Each archived item will have the origin ip address and reception time appended to it. See property-ip and property-received system parameter documentation for more details.

When geo is enabled, geographic metadata will be appended to each item based on the available ip address.

Data Lake

/publish

m1 test-m1 publish -s3file "s3file=s3://m1-public/reddit/us.jsonl.gz"

curl https://test-m1.minusonedb.com/publish \
-d "s3file=s3://m1-public/reddit/us.jsonl.gz" \
-H "m1-auth-token: $myToken"

Sample Response

Saved 30000-46570 (16570 total)
Saved 50000-66376 (32946 total)
Saved 70000-86492 (49438 total)
Saved 90000-92104 (51542 total)

Load files from S3 into the data lake.

/next

m1 test-m1 next

curl https://test-m1.minusonedb.com/next \
-H "m1-auth-token: $myToken"

Sample Response:

10000

Retrieve the next _m1key that will be assigned via /publish.

/get


```m1
m1 test-m1 get -ids "[10000,20000,30000]" -properties '["_m1key","session.id"]' 

curl https://test-m1.minusonedb.com/get \
-d 'ids=[10000,20000,30000]&properties=["_m1key","session.id"]' \
-H "m1-auth-token: $myToken"

Sample Response:

{
  "_m1key" : 10000,
  "session.id" : "67397f5e-b6b1-4cb9-9b88-b6396d3e1ae4"
}, {
  "_m1key" : 20000,
  "session.id" : "76a9cad0-88f8-46e3-bbae-40256f2c9267"
}, {
  "_m1key" : 30000,
  "session.id" : "acd6540d-b8cd-470e-b6e8-703cd1a48870"
} ]

Retrieve any number of rows from the data lake via _m1key property.

/range

m1 test-m1 range -start 10000 -end 30000

curl https://test-m1.minusonedb.com/range \
-d 'start=10000&end=30000&properties=["_m1key","session.id"]' \
-H "m1-auth-token: $myToken"

Sample Response:

{
  "_m1key" : 10000,
  "session.id" : "67397f5e-b6b1-4cb9-9b88-b6396d3e1ae4"
}, {
  "_m1key" : 20000,
  "session.id" : "76a9cad0-88f8-46e3-bbae-40256f2c9267"
}, {
  "_m1key" : 30000,
  "session.id" : "acd6540d-b8cd-470e-b6e8-703cd1a48870"
} ]

Retrieve all rows from datalake with _m1key values between start (inclusive) and end (exclusive)

Data Stores

/query

Queries the specified data store.

The easiest way to learn the query is by studying the following simple examples.

Return all documents:

m1 test-m1 query -store "index" -q '*' -rows 10

curl https://test-m1.minusonedb.com/query \
-d "store=index&q=*&rows=10" \
-H "m1-auth-token: $myToken"

The simplest query is * which will match all documents. Add a rows parameter to get data other than a count.

All events with from a particular IP address:

m1 test-m1 query -store "index" -q "_m1.ip:128.218.229.26"

curl https://test-m1.minusonedb.com/query \
-d "store=index&q=_m1.ip:128.218.229.26" \
-H "m1-auth-token: $myToken"

To find all events from a particular ip address use a query like _m1.ip:128.218.229.26. In general, the form is property:value.

All events from 2 IP addresses:

m1 test-m1 query -store "index" -q '_m1.ip:("128.218.229.26" OR "72.200.1.74")'

curl https://test-m1.minusonedb.com/query \
-d "store=index&q=_m1.ip:("128.218.229.26" OR "72.200.1.74")" \
-H "m1-auth-token: $myToken"

To find all events from more than ip address, use OR to separate multiple values: _m1.ip:("128.218.229.26" OR "72.200.1.74").

More than 1 constraint:

m1 test-m1 query -store "index" -q "_m1.ip:("128.218.229.26" OR "72.200.1.74") AND event.type:sessionStart"

curl https://test-m1.minusonedb.com/query \
-d "store=index&q=_m1.ip:("128.218.229.26" OR "72.200.1.74") AND event.type:sessionStart" \
-H "m1-auth-token: $myToken"

To use multiple constraints, use an AND between individual property terms. The query _m1.ip:(128.218.229.26 OR 72.200.1.74) AND event.type:sessionStart will return events of type sessionStart from 2 the specified ip addresses.

Match against a range of values for a given property:

m1 test-m1 query -store "index" -q "_m1.receivedUTC:[2021-12-25T00:00:00Z TO 2021-12-26T00:00:00} -sort '_m1.receivedUTC desc'"

curl https://test-m1.minusonedb.com/query \
-d "store=index&q=_m1.receivedUTC:[2021-12-25T00:00:00Z TO 2021-12-26T00:00:00}&sort=_m1.receivedUTC desc" \
-H "m1-auth-token: $myToken"

Ranges let you search against a numeric or date interval: q=_m1.receivedUTC:[2021-12-25T00:00:00Z TO 2021-12-26T00:00:00Z} will return all events received on 12/25/2021. Use [] for inclusive ranges and {} for exclusive ranges.

A more complicated query:

m1 test-m1 query -store "index" -q '_m1.receivedUTC:[2021-12-25T00:00:00Z TO 2021-12-26T00:00:00Z} AND (_m1.ip:"128.218.229.26") AND NOT (event.type:sessionStart)'

curl https://test-m1.minusonedb.com/query \
-d 'store=index&q=_m1.receivedUTC:[2021-12-25T00:00:00Z TO 2021-12-26T00:00:00Z} AND (_m1.ip:"128.218.229.26") AND NOT (event.type:sessionStart)' \
-H "m1-auth-token: $myToken"

These simple concepts can be combined and composed to make more complicated queries. Our reports interface can help you build queries and display results.

/index

m1 test-m1 index -store index -start 10000 -end 30000

curl https://test-m1.minusonedb.com/index \
-d "store=index&start=10000&end=30000" \
-H "m1-auth-token: $myToken"

No response when successful.

Writes all rows in the datalake with _m1key values between start (inclusive) and end) (exclusive) to the associated store. Often used when resizing a datastore.

/store/list

m1 test-m1 store/list

curl https://test-m1.minusonedb.com/store/list \
-H "m1-auth-token: $myToken"

Sample Response:

[ {
  "shards" : 1,
  "autocommit" : 2,
  "replicas" : 1,
  "name" : "index",
  "active" : true,
  "type" : "standard"
} ]

Return a list of all configured data stores.

/store/autocommit

m1 test-m1 store/autocommit -store index -seconds 2

curl https://test-m1.minusonedb.com/store/autocommit \
-d "store=index&seconds=2" \
-H "m1-auth-token: $myToken"

No response when successful.

Set autocommit frequency of a given store. The autocommit frequency governs how often a store makes recently written data available for searching. For most near real time uses cases we recommend a value of 5; a 0 or negative value will turn off autocommit which is recommended for bulk loading use cases.

/store/add

m1 test-m1 store/add -store index -shards 1 -replicas 2

curl https://test-m1.minusonedb.com/store/add \
-d "store=index&shards=1&replicas=2" \
-H "m1-auth-token: $myToken"

No response when successful.

Add specification for a data store. You will need a corresonding /env/store/create ops service call to provision your data store servers.

The m1 client can help streamline operations that require coordination between https://ops.minusonedb.com and an environment.

/store/drop

m1 test-m1 store/drop -store index

curl https://test-m1.minusonedb.com/store/drop \
-d "store=index" \
-H "m1-auth-token: $myToken"

No response when successful.

Removes the specification for a data store. You will need to make a corresponding /env/store/destroy ops service call so as not to incur charges for provisioned servers.

The m1 client can help streamline operations that require coordination between https://ops.minusonedb.com and an environment.

/store/backup/create

m1 test-m1 store/backup/create -store index -name myBackup

curl https://test-m1.minusonedb.com/store/backup/create \
-d "store=index&name=myBackup" \
-H "m1-auth-token: $myToken"

Sample Response:

{
  "shards" : 1,
  "s3path" : "m1-a8e5404b42688d1bab855d9ed2b7f03e/backup/5ed16f88-1282-41b6-a990-67635ce5036a/",
  "name" : "myBackup",
  "id" : "5ed16f63-1282-41b6-a880-67635ce5036a",
  "status" : "working"
}

Starts a backup of the store; use /store/backup/list to monitor status of the backup.

/store/backup/restore

m1 test-m1 -store index -backup 5ed16f63-1282-41b6-a880-67635ce5036a

curl https://test-m1.minusonedb.com/store/backup/restore \
-d "store=index&backup=5ed16f63-1282-41b6-a880-67635ce5036a" \
-H "m1-auth-token: $myToken"

No response when successful.

Begins the restore process of a store from a saved backup. Any data that was previously in the store will be lost once the restore process begins. Use /health to monitor status of the restore; the store server group will require a /reboot to complete the restore process.

/store/backup/list

m1 test-m1 store/backup/list

curl https://test-m1.minusonedb.com/store/backup/list \
-H "m1-auth-token: $myToken"

Sample Response:

[ {
  "shards" : 1,
  "s3path" : "m1-a8e5404b42688d1bab855d9ed2b7f03e/backup/5ed16f88-1282-41b6-a990-67635ce5036a/",
  "created" : 1636565879150,
  "name" : "myBackup",
  "id" : "5ed16f63-1282-41b6-a880-67635ce5036a",
  "status" : "completed"
} ]

Return a list of all backups associated with this environment.

Session Stores

/session/update

m1 test-m1 session/update -store stash -ops '[{"upsert": [{"ip": "65.29.29.190","session": {"id": "0295aed9-5872-4586-8086-57d3d8c40fb8","start": 1612497996184}}]},{"delete": ["78ba3dac-3e45-4e17-940e-e66584882ce4"]}]'

curl https://test-m1.minusonedb.com/session/update -d 'store=stash&ops=[{"upsert":[
  {
    "ip": "65.29.29.190",
    "session": {
      "id": "0295aed9-5872-4586-8086-57d3d8c40fb8",
      "start": 1612497996184
    },
    "user": {
      "id": "f7661f25-7c49-4f3b-9185-ec6566892878",
    },
    "event": {
      "id": "6dd089ce-5a96-4fac-a405-43c1ab3eb6b3",
      "time": 1612497996000,
      "type": "heartbeat",
      "data": {
        "a": "timers",
        "b": "string",
        "foo": "epe",
        "bar": "string"
      }
    }
}]},
{"delete" : ["78ba3dac-3e45-4e17-940e-e66584882ce4"]}]' \
-H "m1-auth-token: $myToken"

No response when successful.

Upsert and/or delete data stored in a specified session store.

/session/query

m1 test-m1 session/query -store stash -q "_m1.ip:64.150.5.135"

curl https://test-m1.minusonedb.com/session/query \
-d 'store=stash&q=_m1.ip:64.150.5.135' \
-H "m1-auth-token: $myToken"

Sample Response:

{
    "count" : 1,
    "items" : [
        {
            "settings" : {
                "doNotTrack" : "boolean",
                "language" : "string",
                "cookies" : false
            },
            "_m1" : {
                "receivedUTC" : 1635539079099,
                "ip" : "64.150.5.135"
            },
            "session" : {
                "start" : 1635538913819,
                "id" : "0006184d-2961-497a-9203-38e39ed65976"
            },
            "screen" : {
                "orientation" : "landscape",
                "availableWidth" : 3530,
                "availableHeight" : 196,
                "width" : 668,
                "height" : 1775
            },
            "id" : "0e168aaf-0af2-46f1-af91-65ca8a257752",
            "time" : 1635539086000,
            "page" : {
                "domain" : "www.site.com",
                "timeOnPage" : 100,
                "url" : "string"
            },
            "window" : {
                "width" : 2000,
                "height" : 3800
            },
            "event" : {
                "type" : "heartbeat"
            },
            "user" : {
                "id" : "b3c9f536-f3a8-4fd7-8c70-cd904de65443"
            }
        }
    ]
}

Retrieve matching records from a session store.

The syntax and semantics of /session/query are the same as /query where possible. Because session stores accept items that have a deeply nested structure, dotted notation is used to signify nesting. In the example to the right, the archived item contained an m1 key that was a map containing an ip key. To query against that propery in /session/query, use m1.ip as the property name.

/session/list

m1 test-m1 session/list

curl https://test-m1.minusonedb.com/session/list \
-H "m1-auth-token: $myToken"

Sample Response:

[ {
  "shards" : 1,
  "replicas" : 1,
  "name" : "stash",
  "key" : "session.id"
} ]

Return a list of all configured session stores with metadata.

/session/key

m1 test-m1 session/key -store stash -key "session.id"

curl https://test-m1.minusonedb.com/session/key \
-d "store=stash&key=session.id" \
-H "m1-auth-token: $myToken"

No response when successful.

Set the key to be used for identity tests.

/session/add

m1 test-m1 session/add -store stash -key "session.id" -shards 1 -replicas 1

curl https://test-m1.minusonedb.com/session/add \
-d "store=stash&key=session.id&shards=1&replicas=1" \
-H "m1-auth-token: $myToken"

No response when successful.

Add configuration for a newly deployed session store. Called after a new store is deployed via MinusOneDB ops services. Use the m1 client to provision hardware and update your environment with one command.

/session/drop

m1 test-m1 session/drop -store stash

curl https://test-m1.minusonedb.com/session/drop \
-d "store=stash" \
-H "m1-auth-token: $myToken"

No response when successful.

Removes configuration for a session store. Typically called after the store is destroyed via MinusOneDB ops services. Use the m1 client to decommission server resources and update your environment with one command.

System Configuration

Each system parameter is described in detail below; a GET of /system/parameter will let you see the current value of parameter while a POST to the same path will let you change parameter's value. System parameters are extensible for application-specific usecases.

/system

m1 test-m1 system

m1 test-m1 system -publish-permissive true -geo true -cors '["minusonedb.com"] -token-expire-ms 86400000

curl https://test-m1.minusonedb.com/system/ \
-H "m1-auth-token: $myToken"

curl https://test-m1.minusonedb.com/system/ \
-d 'publish-permissive=true&geo=true&cors=["minusonedb.com"]&token-expire-ms=86400000' \
-H "m1-auth-token: $myToken"

Sample Response:

{
  "init" : "true",
  "cors" : "[\"minusonedb.com\"]",
  "token-expire-ms" : "720000000",
  "publish-permissive" : "false",
  "property-partial" : "_m1.partial",
  "password-min-length" : "8",
  "property-session-id" : "session.id",
  "geo" : "false",
  "property-received" : "_m1.receivedUTC",
  "bucket" : "m1-dbd016368bb03434b4210674d451950a",
  "property-ip" : "_m1.ip",
  "region" : "us-east-1"
}

Return all environment system parameters or set multiple system parameters at once.

/system/init

m1 test-m1 system/init

curl https://test-m1.minusonedb.com/system/init \
-H "m1-auth-token: $myToken"

Return true if the environment is initialized and false otherwise. Setting this parameter outside of /init is not supported at this time.

/system/bucket

m1 test-m1 system/bucket

curl https://test-m1.minusonedb.com/system/bucket \
-H "m1-auth-token: $myToken"

Return the S3 bucket used to store environment data. This is set at /init time and should be the the bucket associated with the environment at creation time. Setting this parameter outside of /init is not supported at this time.

/system/region

m1 test-m1 system/region

curl https://test-m1.minusonedb.com/system/region \
-H "m1-auth-token: $myToken"

Return the AWS region used for this environment's infrastructure. No value other than us-east-1 is currently supported. Setting this parameter is not supported at this time.

/system/password-min-length

m1 test-m1 system/password-min-length

m1 test-m1 system/password-min-length -value 8

curl https://test-m1.minusonedb.com/system/password-min-length \
-H "m1-auth-token: $myToken"

curl https://test-m1.minusonedb.com/system/password-min-length \
-d "value=8" -H "m1-auth-token: $myToken"

Return or set the minimum password length for users of the environment. The default is 8.

/system/token-expire-ms

m1 test-m1 system/token-expire-ms

m1 test-m1 system/token-expire-ms -value 86400000

curl https://test-m1.minusonedb.com/system/token-expire-ms \
-H "m1-auth-token: $myToken"

curl https://test-m1.minusonedb.com/system/token-expire-ms \
-d "value=86400000" -H "m1-auth-token: $myToken"

Return or set the default token expiration time (ttl). The default is 720000000 (200 hours).

/system/publish-permissive

m1 test-m1 system/publish-permissive

m1 test-m1 system/publish-permissive -value true

curl https://test-m1.minusonedb.com/system/publish-permissive \
-H "m1-auth-token: $myToken"

curl https://test-m1.minusonedb.com/system/publish-permissive \
-d "value=true" -H "m1-auth-token: $myToken"

Return or set whether rows that fail schema validation will cause entire blocks of published data to be rejected atomically. If true, rows that fail schema validation will be persisted to the extent possible; all properties that pass validation will be persisted. Rows that are partially persisted will have have their property-partial property set to true. If false, attempts to publish rows that fail schema validation will fail with an error. The default value is false.

/system/property-partial

m1 test-m1 system/property-partial

m1 test-m1 system/property-partial -value "_m1.partial"

curl https://test-m1.minusonedb.com/system/property-partial \
-H "m1-auth-token: $myToken"

curl https://test-m1.minusonedb.com/system/property-partial \
-d "value=_m1.partial" -H "m1-auth-token: $myToken"

Return or set the property used to signal when rows are only partially persisted. This is only effectively used when publish-permissive is set to true. The specified property must be available in the environment schema. The default value is _m1.partial.

/system/property-ip

m1 test-m1 system/property-ip

m1 test-m1 system/property-ip -value "_m1.ip"

curl https://test-m1.minusonedb.com/system/property-ip \
-H "m1-auth-token: $myToken"

curl https://test-m1.minusonedb.com/system/property-ip \
-d "value=_m1.ip" -H "m1-auth-token: $myToken"

Return or set the property name where the ip address of the request origin machine will be added to items sent to the archive layer or session store. Items that already contain a non-null value in this property will be left unchanged. Additionally, data in this property will be used for ip address based geocoding if geo is suitably configured. The specified property must be available in the environment schema. The default value of this property is _m1.ip.

/system/property-received

m1 test-m1 system/property-received

m1 test-m1 system/property-received -value "_m1.receivedUTC"

curl https://test-m1.minusonedb.com/system/property-received \
-H "m1-auth-token: $myToken"

curl https://test-m1.minusonedb.com/system/property-received \
-d "value=_m1.receivedUTC" -H "m1-auth-token: $myToken"

Return or set the property name where server receipt time will be added to items sent to the archive layer or session store. The specified property must be available in the environment schema. The default value of this property is _m1.receivedUTC.

/system/geo

m1 test-m1 system/geo

m1 test-m1 system/geo -value true

curl https://test-m1.minusonedb.com/system/geo \
-H "m1-auth-token: $myToken"

curl https://test-m1.minusonedb.com/system/geo \
-d "value=true" -H "m1-auth-token: $myToken"

Return or specify whether geo data will be appended to items sent to the archive layer or session store. Geographic data is based on ip address. The property-ip system parameter specifies where to look for the ip address in a to be archived document. The schema of an environment with geo turned on must include the set of geo properties in the environment schema (which can be mapped to alternate property names as desired).

/system/cors

m1 test-m1 system/cors

m1 test-m1 system/cors -value '["minusonedb.com","yourdomain.com"]'

curl https://test-m1.minusonedb.com/system/cors \
-H "m1-auth-token: $myToken"

curl https://test-m1.minusonedb.com/system/cors \
-d 'value=["minusonedb.com","yourdomain.com"]' -H "m1-auth-token: $myToken"

Return or set the list of domains that will be accepted for CORS requests; MinusOneDB environment will accept requests from all specifies domains and their subdomains. Use /system/cors/add and /system/cors/remove to add or remove individual domains.

/system/cors/add

m1 test-m1 system/cors/add -domains '["domain1.com","domain2.com"]'

curl https://test-m1.minusonedb.com/system/cors/add \
-d 'domains=["domain1.com","domain2.com"]' -H "m1-auth-token: $myToken"

Add CORS domains.

/system/cors/remove

m1 test-m1 system/cors/remove -domains '["domain1.com","domain2.com"]'

curl https://test-m1.minusonedb.com/system/cors/remove \
-d 'domains=["domain1.com","domain2.com"]' -H "m1-auth-token: $myToken"

Remove CORS domains.

Operations

/ping

m1 test-m1 ping

curl https://test-m1.minusonedb.com/ping

Sample Response:

true

Simple boolean, always returns true in non-error states. Indicates the minimal functioning of the system as a whole. Use /health for a deeper diagnostic.

/init

m1 test-m1 init

m1 test-m1 init -username admin -password passphrase -bucket m1-24e27632d234452ba37e60936c0e88eb

curl https://test-m1.minusonedb.com/init

curl https://test-m1.minusonedb.com/init \
-d "username=admin&password=passphrase&bucket=m1-24e27632d234452ba37e60936c0e88eb"

A POST will initialize a new environment; this operation can only be performed once. GET calls to /init will return a boolean indicating whether the system has been initialized.

/health

m1 test-m1 health

curl https://test-m1.minusonedb.com/health \
-H "m1-auth-token: $myToken"

Sample Response:

{
    "geo" : {
        "status" : "PASSED",
        "message" : "Data loaded."
    },
    "sessions" : {
        "status" : "PASSED",
        "message" : "",
        "details" : {
            "Node stash shard 0 replica 0" : {
                "status" : "PASSED",
                "message" : "Status 200"
            }
        }
    },
    "system" : {
        "status" : "PASSED"
    },
    "stores" : {
        "status" : "PASSED",
        "message" : "",
        "details" : {
            "Node index shard 0 replica 0" : {
                "status" : "PASSED",
                "message" : "Status 200"
            }
        }
    },
    "overall" : {
        "status" : "PASSED"
    },
    "id" : {
        "status" : "PASSED"
    },
    "lake" : {
        "status" : "PASSED"
    },
    "db" : {
        "status" : "PASSED"
    },
    "users" : {
        "status" : "PASSED"
    }
}

Returns the status of all environment componentss.

Rights

An example response when a user does not have sufficient rights:

{
  "message" : "User test does not have requested right in set [\"admin\"]",
  "status" : 403
}

Users must have one of the rights listed in the service description in order to use the service. Rights are case-sensitive strings. The list below details all MinusOneDB managed environment rights.

admin: user can manage users, assign rights, and perform typical operational tasks (backup, restore, health checks, etc).
schema: user can edit and view the environment schema.
get: user can query data the data lake or data stores.
publish: user can publish data to the data lake.
sessionQuery: user can query all session stores.
sessionUpdate: user can add, update, or delete data from all session stores.

You can use /user/update to update the rights of a user. The collection of rights is extensible for application-specific uses.

Troubleshooting

Common API Errors

401 Errors

Sample 401 Response:

{
  "message" : "Invalid Token",
  "status" : 401
}

Most ops and environment services require an authentication token. If a particular request fails with a 401 error, doublecheck the following:

/token/data can provide some additional diagnostic information that may be helpful in understanding why a token is not working.

Consider using the m1 client to manage the token life cycle for you.

Java Setup

Successful java -version output: you already have a JRE installed

openjdk version "16.0.1" 2021-04-20
OpenJDK Runtime Environment (build 16.0.1+9-Ubuntu-120.04)
OpenJDK 64-Bit Server VM (build 16.0.1+9-Ubuntu-120.04, mixed mode, sharing)

Unsuccessful java -version output: you need to install a JRE.

java: command not found

The m1 client requires a Java Runtime Environment (JRE) version 1.8 or higher. You may already have a java runtime installed on your computer. To check, run the following:

java -version

The exact output may vary (some samples are to the right) but if it looks like you do not have a JRE installed, download and install the latest for your platform from https://adoptium.net/.

Python Setup

To run any MinusOneDB samples that require python, you will need to install python3 and some 3rd party python packages.

Step 1: Install python3

Step 2: Install 3rd party libraries used in MinusOneDB samples.

Ubuntu

sudo apt-get install python3-boto3 python3-pebble

Windows & Mac

pip install boto3 pebble requests

If you are using a python package manager other than pip, refer to its documentation to install the boto3 and pebble packages on your system.