ones and zeros flying around at a high speed

Metadata streams

Axis devices have three different metadata streams; PTZ, event, and analytics stream. You can read more about each stream on this page.

PTZ metadata stream

The PTZ stream delivers the data corresponding to the actual PTZ moves. It can be received by appending "ptz=all" in the RTSP URL. Below is a sample flow of PTZ move action. It only delivers data if there is movement. Otherwise it's silent.

PTZ move action

Sample flow of PTZ move action:

Metadata sample flow

Sample packages

Here are some sample packages showing idle, moving, and position:

Event stream

The event stream shows all the events coming from a device. These could be I/O events, temperature events based on the operating conditions of the device, status of the PTZ presets, status of network or storage connection, ACAP events, etc. It can be received by appending "event=on" in the RTSP URL. It's like a global stream that is shared among the device channels. In other words, if you have a multisensor unit or different view areas, event stream would be identical from all channels, since I/O or other properties are shared. It only delivers data if there is movement. Otherwise it's silent, just like the PTZ stream. It's not meant to be used for sharing data, but more on the trigger information. It's also good for live alerts.

Output

Sample:

Metadata sample showing event output

I/O

Sample package:

Metadata showing sample package of event I/O

VMD4

Sample package:

Metadata sample package showing VMD4 output

Analytics stream

The analytics stream is different from metadata streams. It's a real ongoing stream, meaning that it constantly delivers packages even when there's no activity. It's designed for sharing data and it's good for post actions.

To support use cases where analytics metadata is a requirement, for example forensic search, Axis has made available analytics metadata as part of later firmware releases. Analytics metadata was first introduced in Axis OS version 9.50 (9.80 LTS Axis OS release has the same version as 9.50), and with Axis OS 10.6 object classification metadata was added.

Output

Sample:

Metadata sample showing analytics output

Axis devices produces analytics metadata based on the ONVIF profile M format

You can see explanations of what it contains in the image below.

  • Like the other streams, it has its own branch called "VideoAnalytics" and the timestampt. 
  • In addition compared to other streams, it has a source field. 
  • It has information for bounding boxes and polygons. Currently bounding boxes and polygons are the same if you use AXIS Object Analytics as a source.
  • It also contains object classification data like Human or Vehicle and the probability score. If you have a deep learning camera, it also presents a sub category, such as car in this example.
Sample showing different metadata

Motion Object Tracking Engine as source

Below is a sample package that has "Motion Object Tracking Engine" as source. It has real polygons consisting of many points. Since Axis OS 10.9 it also has a source value.

Sample package showing metadata with motion object tracking as source

No activity in scene

Here is a sample when there's no activity in the scene:

Sample package showing metadata when no activity

The type of metadata also depends on the device

For example, to receive object classification metadata, specific camera models are required. Over time, as Axis introduces new devices with more processing power and deep learning capabilities, most of the portfolio will produce object classification metadata.

What's new

AXIS OS 10.11

For new products and after defaulting the devices, AXIS Object Analytics will become the default metadata producer. Old configured systems will work as it is once you upgrade to AXIS OS 10.11.

  • AXIS Object Analytics will also deliver objects that cannot be classified. See sample package.
  • Added support for HumanFace, LicensePlate, Bike, Bus, Car and Truck as new object detection classes to machine learning cameras. See affected products in the AXIS OS portal.
  • Added information about supported features for metadata producers in web GUI(Axis Device Assistant). See how it looks.
  • Corrected an issue where changing rotation required restart of the device for the metadata in Axis Object Analytics. See affected products in the AXIS OS portal.
AXIS OS 10.10

Restructure of AXIS Object Analytics (AOA) and Motion Object Tracking Engine (MOTE) metadata producers. AOA ACAP is no longer the producer for object classification metadata.

The metadata from the Axis object analytics provider will add two new classifications and associated bounding boxes for: 

  • HumanFace
  • LicensePlate (location of the license plate on the image)

Note: The additional classifications will only be available on DL cameras.

For releases after AXIS OS 10.10, it is good to be aware that:

  • There might be new metadata producers added
  • The factory default metadata producer might change
  • The contents of the metadata from the producers that already exist may evolve with new classifications and other additional data.
     

For more information, see changes in Metadata Analytics stream AXIS OS 10.10.

AXIS OS 10.9 Motion Object Tracking Engine (MOTE) data got a source name. The main difference is Source=VideoMotionTracker, the rest is the same. See example package.
AXIS OS 10.6

With the release of Axis OS 10.6, you are able to retrieve object classification data, e.g. humans and vehicles.

To be able to retrieve object classification data, you need a device that has support for Axis Object Analytics (AOA). AOA requires ARTPEC-7 or higher and, a certain amount of memory.

See list of supported models

What about cameras that don't support Axis Object Analytics?

Such devices will still produce metadata but without object classification. This means that you will get Object ID, bounding boxes, center of gravity, and polygons. Axis refers to this data as MOTE (Motion Object Tracking Engine) data.

In Axis OS 10.6, the device handles the rotation of the metadata. Before 10.6, the metadata stream needs to be rotated according to the camera rotation. For more accurate conversion, you can use the “transform matrix” in SDP.

What are the benefits of introducing analytics metadata?

We made these changes to get ONVIF Profile M conformance and to help support use cases related to forensic search.

The primary use case for the current analytics metadata is forensic search. Depending on the filters you apply, you may use this data for other purposes too. Metadata rotation is another improvement that simplifies integration. Also, this data is ONVIF profile M conformant. It is supported by not only the high-end models, but also many other camera models.

What should developers take into consideration?

The metadata analytics stream is delivered via the RTSP stream with a special parameter. You need to append “analytics=polygon” in your RTSP request.

For example:

  • Neither video nor audio, just metadata enabled analytics stream

rtsp:///axis-media/media.amp?video=0&audio=0&analytics=polygon

  • The available video&audio in camera=1 channel and metadata analytics stream

rtsp:///axis-media/media.amp?camera=1&analytics=polygon

  • Video, audio and all metadata streams

rtsp:///axis-media/media.amp?video=1&audio=1&analytics=polygon&event=on&ptz=all

Time synchronization is vital in this implementation, so the same NTP server should be used in the camera, clients, and the VMS server.

While implementing forensic search, you should pay attention to “Source” and “Object ID” in the XML document. Object ID may disappear and appear again depending on the scene. In 10.6, MOTE data did not have a "Source" field. In 10.9 and later, VideoMotionTracker is added as a source. AOA data has “=objectanalytics”. NB! When AOA is activated, not every frame contains object classification. It depends on the scene. So, while merging the query results, object classification is not reliable, the Object ID is more reliable. However, Object ID may change as well but you can follow it via Object tree specification.

Another important point to keep in mind is that the analytics metadata stream provides "raw" data that you need to filter while querying. It constantly delivers XML files even if there is nothing going on in the scene. It is not good enough to use directly like normal event data in the event stream, where event stream only sent packages once something occurs.