Typed data: data points

Sending data involves data points and data streams from the Remote Manager point of view, and also a new concept called data point collection from the Cloud Connector API:

A data point represents a value at a specific moment in time and possibly at a geographical localization. To complete the information, a data point must belong to a data stream, which contains metadata common to all the data points on it.
A data stream defines common information for all data points that belong to it, such as its stream ID (the name given in Remote Manager), type (float, integer, string, etc.), and units (C, km/h, seconds, etc.) among other information applicable to all data points of such stream.
A data point collection is a group of data points (from one or more data streams) pending to be uploaded to their Remote Manager stream(s). When sending a data point collection all the data points in it are pushed to their data stream(s) in the same transaction, minimizing network usage.

To send data points to Remote Manager, follow these steps:

1. Create a data point collection

2. Add a data stream to the collection

3. Add a data point

4. Send data point collection to Remote Manager

5. Destroy data point collection

For a list of possible errors receiving device request from Remote Manager, see Description of errors.

Digi Embedded Yocto includes a sample application demonstrating how to receive device requests from Remote Manager.

1. Create a data point collection

The first step before working with data points is to create an empty collection. You can use the ccapi_dp_create_collection() function.

ccapi_dp_error_t ccapi_dp_create_collection(ccapi_dp_collection_handle_t * const dp_collection)

Parameter	Description
dp_collection	Address of the handler to the struct that will hold the data point collection. May be NULL when returning if the creation process fails.

Creating a data point collection

ccapi_dp_collection_handle_t dp_collection;
ccapi_dp_error_t error;
 
[...]
 
/* 1. Create the data point collection */
error = ccapi_dp_create_collection(&dp_collection);
if (dp_collection == NULL) {
    log_error("Cannot create data point collection, error %d", error);
}
 
[...]

Function ccapi_dp_create_collection() returns a ccapi_dp_error_t error code indicating whether a failure occurs during the process. The following subset of ccapi_dp_error_t errors can be returned by ccapi_dp_create_collection():

CCAPI_DP_ERROR_NONE
CCAPI_DP_ERROR_INVALID_ARGUMENT
CCAPI_DP_ERROR_LOCK_FAILED

See Description of errors for more information.

2. Add a data stream to the collection

Before adding a data point to a collection, you must add at least one data stream to the collection. This step explains how many parameters the data stream will hold and in which order they are expected.

A data stream defines:

Common information for the data points belonging to a data stream, such as unit, type, or stream ID.
How a data point is added to a collection, that is, how many parameters and in which order they are expected.

There are two functions. The first one obviates the last two parameters:

ccapi_dp_error_t ccapi_dp_add_data_stream_to_collection(ccapi_dp_collection_handle_t const dp_collection, 
                                                        char const * const stream_id, char const * const format_string)
ccapi_dp_error_t ccapi_dp_add_data_stream_to_collection_extra(ccapi_dp_collection_handle_t const dp_collection, 
                                                              char const * const stream_id, char const * const format_string, 
                                                              char const * const units, char const * const forward_to)

Parameter	Description
dp_collection	Data point collection to add the data stream to. It must have been initialized with ccapi_dp_create_collection().
stream_id	Name of the data stream destination in Remote Manager. It must be unique. For example, name "temperature" creates a stream in Remote Manager with the path <DeviceID>/temperature, where <DeviceID> is the ID of your device.
format_string	Null terminated string to define the data type and how it is expected. See explanation below.
units*	String to define the unit of the data in the stream, such as, seconds, C, etc. NULL for no units.
forward_to*	Name of the data stream to replicate data points to Can be NULL** if not used.

^{* only for ccapi_dp_add_data_stream_to_collection_extra() function}

^{** as soon as a data point is stored in the data stream, it is also copied or replicated in the data stream whose name is in the forward_to parameter.}

Adding a data stream to the collection

ccapi_dp_collection_handle_t dp_collection;
ccapi_dp_error_t error;
 
[...]
 
/* 2. Add the data stream "temperature". Its data points attributes will be:
    * Float value
    * Location required
    * Timestamp will be added by Remote Manager
    * Quality not present
*/
error = ccapi_dp_add_data_stream_to_collection(dp_collection, "temperature", "float loc");
if (error != CCAPI_DP_ERROR_NONE) {
    log_error("Cannot add temperature data stream to collection, error %d", error);
}
 
[...]

The format_string is a combination of some keywords separated by a white space. Each keyword defines an attribute of the data points that are going to be added to the data stream:

Data type. Required attribute to indicate the type of the data points value to be added to this data stream.
Timestamp. Optional attribute to indicate the format of the timestamp of the data points.
Location. Optional attribute to indicate if the data points location is going to be captured.
Quality. Optional attribute to indicate if the sample quality is going to be captured.

The following options are available for each attribute:

Information type	Keyword	Description
Data type	int32	32-bit signed integer value
	int64	64-bit signed integer value
	float	Floating point value
	double	Double precision floating point value
	string	Null-terminated string
	json	Null-terminated string that represents a JSON object
	geojson	Null-terminated string that represents a GeoJSON object
Timestamp	[Not present]	Timestamp will be added by Remote Manager according to the time of the upload
	ts_epoch	Structure holding the seconds elapsed since 00:00:00 UTC on 1 January 1970 and the milliseconds portion
	ts_epoch_ms	64-bit unsigned integer with the number of milliseconds since 00:00:00 UTC on 1 January 1970
	ts_iso	Null-terminated string with the timestamp in ISO 8601 format
Location	loc	Location structure, with latitude, longitude and altitude (optional) as floating point values
Quality	qual	Data Point quality value, as an integer

Note The order of the keywords in the format_string is very important. It defines the expected order when adding a data point (like in printf/scanf functions).

For example:

The format_string "int32 loc" means that all data points added to the data stream must pass a 32-bit integer as the first argument and the location structure as the second one. The timestamp will be added by Remote Manager.
The format_string "string ts_epoch_ms loc" means that all data points added to the data stream must pass a null-terminated string as the first argument, a 64-bit unsigned integer for the timestamp as the second one, and the location structure as the third one.

Functions ccapi_dp_add_data_stream_to_collection() and ccapi_dp_add_data_stream_to_collection_extra() return a ccapi_dp_error_t error code indicating whether any failure occurs during the process. The following subset of ccapi_dp_error_t errors can be returned:

CCAPI_DP_ERROR_NONE
CCAPI_DP_ERROR_INVALID_ARGUMENT
CCAPI_DP_ERROR_INVALID_STREAM_ID
CCAPI_DP_ERROR_INVALID_FORMAT
CCAPI_DP_ERROR_INVALID_UNITS
CCAPI_DP_ERROR_INVALID_FORWARD_TO
CCAPI_DP_ERROR_INSUFFICIENT_MEMORY
CCAPI_DP_ERROR_LOCK_FAILED

See Description of errors for more information.

You can also manage the streams and data points in a collection with the functions:

Function	Description
ccapi_dp_error_t ccapi_dp_remove_data_stream_from_collection(ccapi_dp_collection_handle_t const dp_collection, char const * const stream_id)	Removes the given stream (stream_id) from the dp_collection data points collection
ccapi_dp_error_t ccapi_dp_get_collection_points_count(ccapi_dp_collection_handle_t const dp_collection, uint32_t * const count)	Returns in count the number of data points stored in dp_collection
ccapi_dp_error_t ccapi_dp_clear_collection(ccapi_dp_collection_handle_t const dp_collection)	Removes all data streams information and the data points stored in dp_collection

These functions return a ccapi_dp_error_t error code indicating whether any failure occurs during the process. The following subset of ccapi_dp_error_t errors can be returned:

CCAPI_DP_ERROR_NONE
CCAPI_DP_ERROR_INVALID_ARGUMENT
CCAPI_DP_ERROR_INVALID_STREAM_ID⁽¹⁾⁽²⁾
CCAPI_DP_ERROR_LOCK_FAILED⁽¹⁾

^{(1) not for ccapi_dp_get_collection_points_count()}

^{(2) not for ccapi_dp_clear_collection()}

3. Add a data point

A data point represents a single value that is stored at a specific time in a data stream in Remote Manager. A data point is defined by its attributes:

Value. Required attribute, that represents the sample value to store. Its type must match with the type selected in the format_string when the data stream was added to the collection.
Timestamp. Optional value to specify when the value was captured. Its type must match with the type selected in the format_string.
Location. Optional value (ccapi_location_t) to establish the location of the device when the sample was taken. Only required if loc keyword was included in the format_string.
Quality. Optional value (int32_t) to define the quality of the sample. Only required if qual keyword was included in the format_string.

The data point value is required to add a data point to a collection. The rest of the attributes are optional, only those ones in the format_string of ccapi_dp_add_data_stream_to_collection() need to be provided. See 2. Add a data stream to the collection.

The function ccapi_dp_add() allows you to add data points to a data point collection providing the name of the stream which they belongs and the required data point attributes.

ccapi_dp_error_t ccapi_dp_add(ccapi_dp_collection_handle_t const dp_collection, char const * const stream_id, ...)

Parameter

Description

dp_collection

Data point collection to add the data stream to. It must have been initialized with ccapi_dp_create_collection().

stream_id

Name of the data stream destination in Remote Manager. It must be unique.

...

Attribute values of the data point to be added (value, timestamp, location, quality).

They must match (in number and order) the format_string specified when the stream_id data stream was added with ccapi_dp_add_data_stream_to_collection().

See 2. Add a data stream to the collection.

Function ccapi_dp_add() returns a ccapi_dp_error_t error code indicating whether any failure occurs during the process. The following subset of ccapi_dp_error_t errors can be returned:

CCAPI_DP_ERROR_NONE
CCAPI_DP_ERROR_INVALID_ARGUMENT
CCAPI_DP_ERROR_INVALID_STREAM_ID
CCAPI_DP_ERROR_INSUFFICIENT_MEMORY
CCAPI_DP_ERROR_LOCK_FAILED

See Description of errors for more information.

Adding a data point to the collection

ccapi_dp_collection_handle_t dp_collection;
ccapi_dp_error_t error;
int const FIVE_MINUTES = 5 * 60;
int i;
 
[...]
 
error = ccapi_dp_add_data_stream_to_collection(dp_collection, "temperature", "float loc");
 
[...]
  
for (i = 0; i < 10; i++) {
    float temperature;
    ccapi_location_t location;
 
    temperature = get_sensor_temperature();
    get_gps_location(&location.latitude, &location.longitude);
    location.elevation = 90;
  
    /* 3. Add data points.
          Pass temperature as a float and the location as it is specified in the format_string */
    error = ccapi_dp_add(dp_collection, "temperature", temperature, &location);
    if (error != CCAPI_DP_ERROR_NONE) {
        log_error("Cannot add temperature data point, error: %d", error);
    }
    sleep(FIVE_MINUTES);
}
 
[...]

You can get the number of data points stored in a collection using ccapi_dp_get_collection_points_count() function.

ccapi_dp_error_t ccapi_dp_get_collection_points_count(ccapi_dp_collection_handle_t const dp_collection,
                                                      uint32_t * const count)

4. Send data point collection to Remote Manager

Once all data points are added to the proper data stream in your data point collection, send them to Remote Manager using the following function:

ccapi_dp_error_t ccapi_dp_send_collection(ccapi_transport_t transport, ccapi_dp_collection_handle_t const dp_collection)

Parameter	Description
transport	Must always be CCAPI_TRANSPORT_TCP
dp_collection	Data point collection to send to Remote Manager

All data points in a collection are sent to their respective data stream when the collection is uploaded to Remote Manager, regardless the order of addition.

Function ccapi_dp_send_collection() returns a ccapi_dp_error_t error code indicating whether any failure occurs during the process. The following subset of ccapi_dp_error_t errors can be returned:

CCAPI_DP_ERROR_NONE
CCAPI_DP_ERROR_INVALID_ARGUMENT
CCAPI_DP_ERROR_INSUFFICIENT_MEMORY
CCAPI_DP_ERROR_LOCK_FAILED
CCAPI_DP_ERROR_CCAPI_NOT_RUNNING
CCAPI_DP_ERROR_TRANSPORT_NOT_STARTED
CCAPI_DP_ERROR_INITIATE_ACTION_FAILED
CCAPI_DP_ERROR_RESPONSE_BAD_REQUEST
CCAPI_DP_ERROR_RESPONSE_UNAVAILABLE
CCAPI_DP_ERROR_RESPONSE_CLOUD_ERROR
CCAPI_DP_ERROR_STATUS_CANCEL
CCAPI_DP_ERROR_STATUS_INVALID_DATA
CCAPI_DP_ERROR_STATUS_SESSION_ERROR

See Description of errors for more information.

Sending data points to Remote Manager

ccapi_dp_collection_handle_t dp_collection;
ccapi_dp_error_t error;
int const FIVE_MINUTES = 5 * 60;
 
[...]
 
/* 1. Create the data point collection */
error = ccapi_dp_create_collection(&dp_collection);
if (dp_collection == NULL) {
    log_error("Cannot create data point collection, error %d", error);
}
 
/* 2. Add the data stream "temperature". Its data points attributes will be:
    * Float value
    * Location required
    * Timestamp will be added by Remote Manager
    * Quality not present
*/
error = ccapi_dp_add_data_stream_to_collection(dp_collection, "temperature", "float loc");
if (error != CCAPI_DP_ERROR_NONE) {
    log_error("Cannot add temperature data stream to collection, error %d", error);
    goto done;
}
  
for (;;) {
    int i;
  
    for (i = 0; i < 10; i++) {
        float temperature;
        ccapi_location_t location;
 
        temperature = get_sensor_temperature();
        get_gps_location(&location.latitude, &location.longitude);
        location.elevation = 90;
  
        /* 3. Add data points.
              Pass temperature as a float and the location as it is specified in the format_string */
        error = ccapi_dp_add(dp_collection, "temperature", temperature, &location);
        if (error != CCAPI_DP_ERROR_NONE) {
            log_error("Cannot add temperature data point, error: %d", error);
        }
        sleep(FIVE_MINUTES);
    }
  
    /* 4. Send the data points in the collection to Remote Manager */
    error = ccapi_dp_send_collection(CCAPI_TRANSPORT_TCP, dp_collection);
    if (error != CCAPI_DP_ERROR_NONE) {
        log_error("Cannot send data points, error %d", error);
            break;
    }
}
 
done:
    /* 5. Destroy the collection */
    ccapi_dp_destroy_collection(dp_collection);
 
[...]

5. Destroy data point collection

At any time, you can free a collection’s allocated resources by calling ccapi_dp_destroy_collection(). Also, a collection can be cleared with ccapi_dp_clear_collection() which means to remove all the data streams information and the data points stored.

ccapi_dp_error_t ccapi_dp_destroy_collection(ccapi_dp_collection_handle_t const dp_collection)

Parameter	Description
dp_collection	Data point collection to destroy

Function ccapi_dp_destroy_collection() returns a ccapi_dp_error_t error code indicating whether any failure occurs during the process. The following subset of ccapi_dp_error_t errors can be returned:

CCAPI_DP_ERROR_NONE
CCAPI_DP_ERROR_LOCK_FAILED

See Description of errors for more information.

Description of errors

Error	Description
CCAPI_DP_ERROR_NONE	Operation finished successfully.
CCAPI_DP_ERROR_INVALID_ARGUMENT	In most cases this refers to empty strings or NULL pointers in structures.
CCAPI_DP_ERROR_INVALID_STREAM_ID	When adding a data stream to a collection this error might mean that the provided stream name contains invalid characters or it already exists in the collection. When adding a data point, it means that such stream name does not exist in the collection.
CCAPI_DP_ERROR_INVALID_FORMAT	When creating a data stream, the provided format string contains invalid keywords or characters or is ambiguous (repeated keywords).
CCAPI_DP_ERROR_INVALID_UNITS	The units can be NULL, but not an empty string.
CCAPI_DP_ERROR_INVALID_FORWARD_TO	When creating a data stream, the forward to must contain only valid characters.
CCAPI_DP_ERROR_INSUFFICIENT_MEMORY	Cloud Connector encountered problems allocating memory to perform the requested operation. Your system may have run out of resources.
CCAPI_DP_ERROR_LOCK_FAILED	Cloud Connector encountered problems using synchronization mechanisms. Your system may have run out of resources.
CCAPI_DP_ERROR_CCAPI_NOT_RUNNING	You missed calling initialize_cloud_connection() function or it failed.
CCAPI_DP_ERROR_TRANSPORT_NOT_STARTED	You missed calling start_cloud_connection() function or it failed.
CCAPI_DP_ERROR_INITIATE_ACTION_FAILED	An unexpected failure on lower communication layers occurred. You can check the specific error on the logging output.
CCAPI_DP_ERROR_RESPONSE_BAD_REQUEST	The collection was uploaded but Remote Manager response indicated that it has invalid data.
CCAPI_DP_ERROR_RESPONSE_UNAVAILABLE	Remote Manager response indicates that service is not available at this moment and can be retried later. The account also may not have sufficient permissions for data points service.
CCAPI_DP_ERROR_RESPONSE_CLOUD_ERROR	Remote Manager encountered an error while handling the request.
CCAPI_DP_ERROR_STATUS_CANCEL	The operation did not finish properly because it was canceled by the user (probably by stopping Cloud Connector).
CCAPI_DP_ERROR_STATUS_INVALID_DATA	Invalid data passed to lower communication layers; this error should never be returned.
CCAPI_DP_ERROR_STATUS_SESSION_ERROR	Indicates an error on lower communication layers.

Data points example

The Upload data points example demonstrates the usage of the Cloud Connector send data API. In this example, a integer value is incremented every five seconds and added to the "incremental" data stream in a data point collection. The collection is sent to Remote Manager when it contains 10 samples.

This example is included in Digi Embedded Yocto. Go to GitHub to look at the application source code.