Data services is an optional feature that is used to send data to and from Device Cloud. The Data Streams feature within Device Cloud allows for time-series data storage and data visualization.
Virtually any type of data can be stored and you can create real-time charts to monitor your data streams. Data streams are fully searchable and the data within a stream can be rolled up into time interval summaries. Data is stored and replicated in multiple secure, commercial-grade storage systems to ensure complete data protection and is based on GB usage per day, and is billed monthly.
Time series data involves two concepts:
To use the Data Streams feature and send data points to Device Cloud, you need to:
Data service enablement is defined by the enable_data_service property located in the Services Settings section. Set it to True in the Cloud Connector configuration file, connector_config.txt, to enable the Data Services feature.
enable_data_service=True
You also need to configure all the settings described in the Connecting to Device Cloud topic to establish a proper connection with Device Cloud.
|
|
The configuration of Cloud Connector comprehends more settings that are not going to be covered in this section. Previous and next topics also cover some of the configuration that might be needed. For more information about all the settings, read the Cloud Connector Configuration section. |
A DataStream object (com.etherios.connector.core.DataStream) represents the destination of data points in Device Cloud. The name of the stream is needed to instantiate a DataStream object, the units of the stream and a list of other data streams names to replicate data points to can also be provided in the constructor. These attributes can be configured using the corresponding set method.
A DataStream object with name "temperature" will create a stream in Device Cloud with the path <DeviceID>/temperature, where <DeviceID> is the Device ID of your device.
DataStream instantiation |
import com.etherios.connector.core.DataStream;
[...]
// New temperature data stream
DataStream temperatureStream = new DataStream("temperature", "C", new String[0]);
[...]
|
A data point is defined by the class DataPoint (com.etherios.connector.core.DataPoint) and it represents a single value that it is stored at a specific time in a data stream in Device Cloud.
The data point value and the stream destination (a DataStream object) are required to create a DataPoint. Other attributes can be specified for it (description, time stamp, quality, location) directly in the constructor or using the right set method.
If the time stamp is not provided, the system or Device Cloud time will be used as a time stamp for the data points. The static method useSystemTime allows you to establish this default value to one or the other.
DataPoint instantiation |
import com.etherios.connector.core.DataPoint; import com.etherios.connector.core.system.Location; [...] DataStream temperatureStream = ...; [...] // If no timestamp is provided, use the system time as data point timestamp DataPoint.useSystemTime(true); // Current temperature data point with description, quality location and temperature data stream as destination DataPoint temperatureDataPoint = new DataPoint(getCurrentTemperatureInCelsius(), temperatureStream, "Current temperature", null, 0, getCurrentLocation()); [...] |
Once all data points are created, send them to Device Cloud using the sendDataPoint method of Cloud Connector.
| Send a single data point to Device Cloud |
[...] CloudConnector connector = ...; DataPoint temperatureDataPoint = ...; // Send to Device Cloud the temperature data point connector.sendDataPoint(temperatureDataPoint); [...] |
Create a List of DataPoints to send all their data points stored inside at the same time.
| Send to multiple data points |
import java.util.ArrayList; [...] CloudConnector connector = ...; DataPoint temperatureDataPoint = ...; ArrayList<DataPoint> dataPointsList = new ArrayList<>(); dataPointsList.add(temperatureDataPoint); // Send to Device Cloud the list of data points connector.sendDataPoint(dataPointsList); [...] |
In order to maximize the speed and throughput of Device Cloud, device uploads have certain limits:
If you need to minimize the data traffic, use the binary concise alternative format to send data points to Device Cloud. It satisfies the requirement of being concise and less verbose than the sendDataPoint method: you can specify a simple binary value, when you push this value to Device Cloud it will be stored as is, in the exact binary format you provided.
To use this format, binary data points have to be defined. They are represented by the class BinaryDataPoint (com.etherios.connector.core.BinaryDataPoint) and contain a single value that it is stored at a specific time in a data stream in Device Cloud, similar to the DataPoint objects with the following differences:
units and forwardTo attributes of the associated DataStream object will not be used.BinaryDataPoint instantiation |
import com.etherios.connector.core.BinaryDataPoint; [...] DataStream binaryStream = ...; [...] // Binary data point with 'binaryStream' as destination BinaryDataPoint binaryDataPoint = new BinaryDataPoint(getBinarySample(), binaryStream); [...] |
Once the binary data point is created, use the method sendBinaryDataPoint of the CloudConnector object to send it to Device Cloud.
BinaryDataPoint instantiation |
[...] CloudConnector connector = ...; BinaryDataPoint binaryDataPoint = ...; // Send to Device Cloud the binary data point connector.sendBinaryDataPoint(binaryDataPoint); [...] |
The binary concise mechanism has the following limitations:
BinaryDataPoint object must be smaller than 64KB.For binary data and if you do not need the additional information provided with the standard data points (geolocation, local timestamp, ...), then we recommend you use binary concise format. However, you can't create multiple data points in a single request using binary concise format. To create multiple binary data points in a single request, you can use DataPoint objects (instead of BinaryDataPoint objects) with your byte array as data, it will be sent as a base64 encoded string to Device Cloud.
Your application might need to be notified about the data points events, mainly about the status of the data points transaction to Device Cloud.
The listener, MyEventsListener, that we already registered in Listening to Connection Events, notifies this circumstance by calling the method notifyDataPointEvent with a DataPointEvent. This last object represents the event thrown when the status of the data point transaction is received from the server.
notifyDataPointEvent implementation example |
[...]
public class MyEventsListener implements EventsListener {
[...]
public void notifyDataPointEvent(DataPointEvent event) {
switch (event.getStatus()) {
case DataPointEvent.STATUS_SUCCESS:
System.out.println("Data points successfully sent");
[...]
break;
case DataPointEvent.STATUS_BAD_REQUEST:
System.out.println("Error sending data points "
+ ": " + event.getStatusMessage());
[...]
break;
case DataPointEvent.STATUS_UNAVAILABLE:
System.out.println("Error sending data points "
+ ": " + event.getStatusMessage());
[...]
break;
case DataPointEvent.STATUS_SERVER_ERROR:
System.out.println("Error sending data points "
+ ": " + event.getStatusMessage());
[...]
break;
default:
System.out.println("Unknown status sending data points "
+ ": " + event.getStatusMessage());
[...]
break;
}
}
[...]
}
|
There is not any specific API permission to send data to Device Cloud, this will depend on what data you need to upload to the Cloud and how you access to it.
Data streams can be remotely accessed within Device Cloud by navigating to Data Services > Data Streams.
Here, your time-series data are stored for an indefinite period. Virtually any type of data can be stored and you can create real-time charts to monitor the data streams in data visualization. Within this page you can view a list of all your data streams as well as create, edit and delete data streams. You can then select a data stream from your list and view a chart of that data stream’s data points based on several options and using several pre-defined time periods, or simply view the raw data associated with a data stream.
For a DataStream object created with name "temperature", the entry in the Data Streams list of Device Cloud will be called <DeviceID>/temperature, where <DeviceID> is the Device ID of your device.
The same list will display the last value received, its location, units, data type and description.
Below a graphic representation of the selected data stream will be displayed in the Charts tab, and a list of its data points in the Raw Data tab.
It is also possible to use Web Services to access this data by navigating to Documentation > API Explorer and then exploring the various samples available within Examples > Data Streams.
For example, to get all data from a stream select Examples > Data Streams > Get all data from stream, then change the name of your data stream in the Path box. For our example above where our data stream is called <DeviceID>/temperature (where <DeviceID> is the Device ID of your device) the Path box should contain the string /ws/DataPoint/<DeviceID>/temperature. Once it is properly configured, click the Send button, the answer will contain the data points in this stream:
<?xml version="1.0" encoding="ISO-8859-1"?> <result> <resultSize>13</resultSize> <requestedSize>1000</requestedSize> <pageCursor>e377f199-b407-11e3-99c3-bc764e1023af</pageCursor> <requestedStartTime>-1</requestedStartTime> <requestedEndTime>-1</requestedEndTime> <DataPoint> <id>0ce29e9d-b407-11e3-99c3-bc764e1023af</id> <cstId>5582</cstId> <streamId>00080003-00000000-0300018A-BD58F613/temperature</streamId> <timestamp>1395742812519</timestamp> <serverTimestamp>1395742818223</serverTimestamp> <data>25.46</data> <description>Current temperature</description> <quality>99</quality> </DataPoint> [...] <DataPoint> <id>e377f199-b407-11e3-99c3-bc764e1023af</id> <cstId>5582</cstId> <streamId>00080003-00000000-0300018A-BD58F613/temperature</streamId> <timestamp>1395743172530</timestamp> <serverTimestamp>1395743177894</serverTimestamp> <data>24.740000000000002</data> <description>Current temperature</description> <quality>99</quality> </DataPoint> </result> |
The Data Points Sample and Binary Data Points Sample demonstrate the Data Streams feature by showing how new data points can be uploaded to Device Cloud.
The samples include their own connector_config.txt file to properly configure Cloud Connector to establish a connection and send data points to Device Cloud.
The Data Points Sample code sends multiple data points to Device Cloud by:
DataStream objects.DataPoint objects.sendDataPoint method.The Binary Data Points Sample code sends single binary data points to Device Cloud by:
DataStream objects.BinaryDataPoint objects.sendBinaryDataPoint method.