How to build Sensorthings API Queries

The SensorThings API was an experimental interface providing access to location metadata and provisional measurements from the last 10 days. After an evaluation period, we’ve decided that we won’t be promoting our SensorThings implementation to production status. The API will be formally sunset once replacement APIs are made available.

REST Information

GETS

The OGC SensorThings API is a RESTful API that is based on the Odata protocol. For the public facing service, only GETs are supported. Major functions and endpoint of the API:

GET	Description
labs.waterdata.usgs.gov/sta/v1.1/	Get index - provides an overview of the SensorThings response.
labs.waterdata.usgs.gov/sta/v1.1/Type	Get all of type - includes Locations, Things, Data Streams, Observed Properties, or Observations.
labs.waterdata.usgs.gov/sta/v1.1/Type(id)	Get one of a type - requests information related to a specific device.
labs.waterdata.usgs.gov/sta/v1.1/Type(id)/Entity	Get linked entity- requests a list of information associated with a specific Thing.
labs.waterdata.usgs.gov/sta/v1.1/Type(id)/EntitySet	Get all linked - provides data associations between object types.

Most interactions with sensorthings will use GETs against one of the type endpoints. In general, searching from more specific to less specific will greatly increase performance

End Points

The entity types referenced throughout the API documentation refer to 8 core classes of information.

Core Class	Description
Locations	The physical location of a Thing. It is possible for a Thing to move to a different location.
Things	A monitoring location or room where a sensor is mounted.
Datastreams	An entry point for a time series from a Thing with a Sensor that is measuring an ObservedProperty.
Observations	A single measurement value
ObservedProperties	A description of what is being measured by a Sensor at a specific Location.
Sensor	Future Development - A description of the device, value, and measurement method.
FeatureOfInterest	Future Development - The object on which the measurement was performed.
HistoricalLocation	Future Development - The historical location of a Thing if it was moved.

Most queries with the SensorThings API will begin with one of the entity types.

Create a Custom Response

Comparison Operators

Operator	Description	Example
eq	Equal	/ObservedProperties?$filter=name eq ‘CO2’
ne	Not equal	/ObservedProperties?$filter=name ne ‘CO2’
gt	Greater than	/Observations?$filter=result gt 5
ge	Greater than or equal	/Observations?$filter=result ge 5
lt	Less than	/Observations?$filter=result lt 5
le	Less than or equal	/Observations?$filter=result le 5

Response Parameters

There are many ways a user may wish to custom tailor the response from the SensorThings API for various usage types. For this purpose, SensorThings API provides the following request parameters. When creating a query, separate parameter with & character and each request parameter can appear only once, unless used inside the $expand parameter.

Parameter	Description
$top	Specify the number of returned Things. The default setting is 100. The max is 1000.
$skip	Use for paging. Skip over the first n records and provide records from the n + 1 on.
$count	Return the total number of objects in the response. The usual default setting for $count is false. Always be careful with count, because count is one of the most expensive kinds of queries to perform.
$orderBy	Specify returned objects are ordered by a specific attribute in ascending or descending order.
$select	Specify the exact attributes provided in the response.
$filter	Specify filters to control returned entities (Things?).
$expand	Create a response returning multiple object types nested within each other.

Most interactions with sensorthings will use GETs against one of the type endpoints. In general, searching from more specific to less specific will greatly increase performance

Result Formats

The SensorThing API provides responses in a standard JSON format. This parameter can be used to request the server to return a different format than the standard JSON format.

$resultFormat Parameter	Description
$resultFormat=DataArray	Returns `Observations` in a compact way by stripping out the JSON keys and returning those keys at the top of the response. It only works on collections of `Observations`.
$resultFormat=GeoJSON	Returns a result that can be displayed on a web map.
$resultFormat=CSV	Returns a flat, CSV format. Details about CSV results can be found at FROST-Server Documentation Site .

Example Queries

To improve readability, these examples are not url encoded. Include the url to use these examples.

All discharge observations for the last two days. This data will require paging over ten responses

https://labs.waterdata.usgs.gov/sta/v1.1/ObservedProperties('00060')/Datastreams?
$expand=Observations(
	$select=phenomenonTime,result,parameters;
	$filter=phenomenonTime gt now() sub duration'P2D';
  $orderby=phenomenonTime asc;$top=1000)
&$top=1000

All Things, with their Location and Datastreams. For each Datastream, the Sensor, ObservedProperty, and two most recent Observations

https://labs.waterdata.usgs.gov/sta/v1.1/Things?
	$select=name,properties,description,@iot.id
	&$expand=
	Locations
		($select=name,description,properties,location,@iot.id)
			,Datastreams
			($select=name,description,@iot.id
			;$expand=
				Sensor
					($select=name,description,@iot.id)
				,ObservedProperty
					($select=name,description,@iot.id)
				,Observations
					($select=result,phenomenonTime,parameters,resultQuality,@iot.id
					;$orderby=phenomenonTime desc
					;$top=2
					)
				)

All Things, with their Location and Datastreams. For each Datastream, the Sensor, ObservedProperty, and two most recent Observations, filtered by locations that are tied to the Colorado River. To determine the correct Geographic Names Information System (GNIS) id to use, you can use a distinct query for Colorado, and then follow mainstem URLs to find the right Colorado River for your query.

https://labs.waterdata.usgs.gov/sta/v1.1/Things?
	$select=name,properties,description,@iot.id
	&$filter=Locations/properties/mainstemNameAtOutletGNIS eq 45730
  &$expand=
	Locations
		($select=name,description,properties,location,@iot.id)
			,Datastreams
			($select=name,description,@iot.id
			;$expand=
				Sensor
					($select=name,description,@iot.id)
				,ObservedProperty
					($select=name,description,@iot.id)
				,Observations
					($select=result,phenomenonTime,parameters,resultQuality,@iot.id
					;$orderby=phenomenonTime desc
					;$top=2
					)
				)

The phenomenonTime and result of the first 100 observations of a datastream, ordered by phenomenonTime that overlap with the time frame from 2023-01-02 07:00:00 UTC to 2023-01-04 07:00:00 UTC (2 days):

https://labs.waterdata.usgs.gov/sta/v1.1/Datastreams('2463374192ca43468cad14834898f975')/Observations
	?$orderby=phenomenonTime asc
	&$top=100
	&$select=phenomenonTime, result
	&$filter=overlaps(phenomenonTime, 2023-01-02T07:00:00Z/2023-01-04T07:00:00Z)

The observations of a Datastream, for the last day:

https://labs.waterdata.usgs.gov/sta/v1.1/Datastreams('2463374192ca43468cad14834898f975')/Observations
	&$orderby=phenomenonTime asc
	&$filter=phenomenonTime gt now() sub duration'P1D'

Datastreams that have data for the ObservedProperty with id '00010' (Temperature, water, degrees Celsius):

https://labs.waterdata.usgs.gov/sta/v1.1/Datastreams?
	$filter=ObservedProperty/@iot.id eq '00010'

ObservedProperties that are measured at the same station as the ObservedProperty with name Temperature:

https://labs.waterdata.usgs.gov/sta/v1.1/ObservedProperties?
	$filter=Datastream/Thing/Datastreams/ObservedProperty/name eq 'Temperature, water, degrees Celsius'

Functions work for Ordering:

Datastreams?$orderby=length(name) desc

Query Design Overview

The SensorThings API provides 8 entity types (12 with all extensions). All data can be reached from each entity type. To get Observations one could start:

directly from /Observations
from /Datastreams and expand Observations
from /FeaturesOfInterest and expand Observations
from /Things and expand Datastreams & Observations
from /ObservedProperties and expand Datastreams & Observations
from /Sensors and expand Datastreams & Observations
from /Locations and expand Things, Datastreams & Observations

The following options can provide the same data but with different structures. All these options can provide same data but with a different structure.

data queries graphic

The data is least structured when starting directly from /Observations. In this case each item in the resulting list is one Observation, but Observations for all Datastreams (Things/ObservedProperties) or FeaturesOfInterest are mixed together.

Individual timeseries for Datastreams are lost and it is up to the client to sort the data again. When expanding Datastreams and from there ObservedProperties or Things, the data for the Datastreams is repeated for each Observation. This result is very much like a CSV file, and quite inefficient.

The resulting data structure looks like this:

Observation + Datastream + ObservedProperty + Thing
Observation + Datastream + ObservedProperty + Thing
Observation + Datastream + ObservedProperty + Thing

It is not possible to get “the latest Observation for each …” since there is only one list, thus only one “latest Observation.”

On the other side of the spectrum is starting from /Things. Each item in the resulting list will be a Thing, and for each Thing the Datastreams can be expanded, and for each of those the Observations.

Thing 1
- Datastream 1 + ObservedProperty 1
  - Observation 1
  - Observation 2
- Datastream 2 + ObservedProperty 2
  - Observation 3
  - Observation 4
Thing 2
- Datastream 3 + ObservedProperty 1
  - Observation 5
  - Observation 6
- Datastream 4 + ObservedProperty 2
  - Observation 7
  - Observation 8

There is some duplication in the ObservedProperties when fetching data in this way, but the structure is easy to put on a map. Because of the way that queries are structured, starting with a narrow data format will often lead to better performance.

References

What is OData ?

The Open Data Protocol (OData) is a data access protocol built on core protocols like HTTP and commonly accepted methodologies like REST for the web. There are various kinds of libraries and tools can be used to consume OData services.

Learn more about this protocol in this tutorial: https://www.odata.org/getting-started/basic-tutorial/ .

Additional Resources

Internet of Water
- SensorThings API Introduction
- STA Resources
Frost Server
British Geological Survey Sensor Data
- Example Python Notebook
- How-to Video

How to build Sensorthings API Queries

REST Information

GETS

End Points

Create a Custom Response

Comparison Operators

Response Parameters

Result Formats

Example Queries

Query Design Overview

Starting From /Observations

Starting From /Things

References

What is OData ?

Additional Resources

How to build Sensorthings API Queries

REST Information

GETS

End Points

Create a Custom Response

Comparison Operators

Response Parameters

Result Formats

Example Queries

Recent discharge observations

Give Me Everything!

Give Me Everything on the Colorado River!

Overlapping Time Frames

The Previous Day

Filters on Related Entities

Ordering by Function

Query Design Overview

Starting From /Observations

Starting From /Things

References

What is OData ?

Additional Resources