Radar-video fusion camera integration

Introduction and integration tips for radar-video fusion cameras.

Next-level detection and visualization

Get wide-area intrusion protection and reliable 24/7 detection with a fusion of two powerful technologies: video and radar. This unique device provides state-of-the-art deep learning-powered object classification for next-level detection and visualization.

Introduction

Radar-video fusion and AXIS Q1656-DLE Radar-Video Fusion Camera.

How does radar-video fusion cameras work?

Fusion is performed on two levels in the radar-video fusion camera:

1. Visual fusion: The radar detections and classifications are fused in the video, resulting in a visualization of the radar inside the video.

2. Analytics fusion: The radar detections are fused with the detections and classifications from the video analytics, resulting in a combined analytics output where the respective strengths of both technologies are merged:

  • The distance, position, speed and direction from the radar
  • The position in the video plan and class from the video


Example:

1. If an object appears at 50 meters distance from the device, it may be too small for the video analytics to detect, but the radar would pick it up.

2. The radar detections are fused into the image plane and can be used to raise events inside Axis Object Analytics, or a search based on the analytics metadata stream.

3. The visual fusion guides the operator to where an event occurred since it is mapped to the visual plane.

4. As the object approaches, it is detected by video analytics.

5. The radar detections are fused with video analytics and the combined output is of higher quality and with more information than the technologies can provide individually. 

The device fuses two powerful technologies (radio frequency and high-resolution video) to deliver reliable detection 24/7 regardless of weather and lighting conditions.

Why radar-video fusion?

  • Your video analytics work well in close proximity, but may fail to detect objects at a far distance or in dark conditions.
  • Your radar work well overall, but from time to time it may miss object classifications. The same applies to the camera where low light or IR reflection from rain could impair the video.
     

AXIS Q1656-DLE analyzes two sources to provide full situational awareness and availability:

  • The device fuses the tracks from both technologies, leading to better performance by combining video analytics with the strenghts of radar technology such as speed, distance, and direction of movement.
  • The combination provides the perfect platform for analytics ensuring greater accuracy where each technology complements the other.
  • By adding extra dimensions such as speed and distance to your camera, the camera becomes an even more powerful tool to accurately detect and classify objects of interest.

What's the difference between AXIS Q1656-DLE and AXIS Q1656-LE?

AXIS Q1656-DLE Radar-Video Fusion Camera and AXIS Q1656-LE Box Camera have almost the same naming, but there is an enormous difference in their capabilities.  Using AXIS Q1656-DLE provides you with:

  • Two devices in one: a radar and a camera for effective cost on maintenance and installation
  • Unique visual and analytics fusion between radar and video
  • Absolute speed and distance from the radar inside analytics metadata and Axis Object Analytics
  • The fusion gives more detections on distance, in challenging weather and light, and the possibility to choose preferred sensitivity on the detections
note

Fusion relies on the factory calibration of both technologies. Do not change the lens or tamper with the radar unit as fusion may break.

How will external white light illumination improve detection performance? 

It is recommended to have 50 lux in the detection area (see Axis Object Analytics user manual) to have reliable detection. Leaving the light on permanently during the night can incur high electricity costs, especially when larger areas need to be covered. 

As an alternative to using permanent illumination, you can use radar motion detection as a trigger for the external illuminators. You can individually control illuminators mounted at different positions by connecting different radar zones to the different illuminators.  

Why is the bounding box not covering the object precisely?

If the bounding box is not located exactly in the right place, it’s because there is no video analytics detection there. You are seeing the projection of the radar detection in the image, and that is not as accurate as a video analytics box. If the box is too high or low, make sure that the installation height is set correctly. It could also be due to elevation differences in the scene, such as a sloping road, a hill, or a depression.

What are the application areas?

  • AXIS Q1656-DLE is designed for outdoor installation in open area coverage for cases such as high accuracy detection for critical infrastructure.
  • AXIS Q1656-DLE can also be used for parking lot monitoring, as the user can enjoy advanced video analytics along with additional radar parameters like distance to the moving object and speed. The maximum speed supported is 55 km/h (35 mph), making it perfect for traffic monitoring in such cases.

Are there any limitations?

  • By the launch of AXIS Q1656-DLE, the capabilities of the analytics metadata stream will be available under the feature flag starting from AXIS OS 11.2 firmware.
  • AXIS Q1656-DLE is not designed to be used for people counting, especially not in crowded areas.

Integration guideline

AXIS Q1656-DLE Radar-Video Fusion Camera

Before you start, we recommend that you read the radar integration guidelines and metadata integration guidelines as you will get data from both technologies with AXIS Q1656-DLE.

In terms of integration, AXIS Q1656-DLE have two video channels and two analytics metadata streams corresponding to the video streams:

  • One video stream is the same as AXIS Q1656-LE, but its metadata analytics stream will get input from both the radar and cameras detections and classifications (regardless of whether the information is available from both of them or not). 
  • The second video stream is from the radar sensor only, which visualizes radar analytics in a typical video stream. The second streams metadata analytics is a radar-based analytics metadata stream, which is similar to the conventional radar metadata stream structure. See the radar integration guidelines for more information.
  • AXIS Q1656-DLE has only got one real video channel (the other channel being the radar video channel), compared to the eight video channels in AXIS Q1656 (due to the possibility to create view areas). It is not possible to create multi-view areas on the AXIS Q1656-DLE. See the consideration section at the end of this guideline for more information.
  • You can identify the radar stream via param.cgi call. "root.ImageSource.IX.Type=Radar" is valid for the radar channel. In AXIS Q1656-DLE, it is "root.ImageSource.I1.Type=Radar". The normal video channels do not have a type parameter.
     

RTSP URL for camera channel with video and audio:

/axis-media/media.amp?camera=1&video=1&audio=1

RTSP URL for radar channel with video and audio:

/axis-media/media.amp?camera=2&video=1&audio=1

RTSP URL for analytics radar-video fusion metadata stream for the camera channel:

/axis-media/media.amp?camera=1&video=0&audio=0&analytics=polygon

Subscribe to the radar-video fusion metadata stream

  • The user can subscribe to the radar-video fusion metadata stream, starting from AXIS OS 11.2.
  • A feature flag should be enabled on the camera to start receiving the fused combination of the metadata stream from both technologies.
  • Starting from AXIS OS 11.5, the feature will be enabled by default and there will not be a need to enable the flag. See AXIS OS release notes for future updates.
note

The feature flag is available from AXIS OS 11.2. Enabling the feature flag will not be needed starting from AXIS OS 11.5 (Scheduled for May 2023).

You can enable the flag by entering the command:

'http://192.168.0.90/axis-cgi/featureflag.cgi' \
"Content-Type: application/json" --data \
'{"apiVersion": "1.0", "method": "set", "params":{"flagValues":{"radar_video_fusion_metadata":true}}, "context": " "}'

Which yields to:

{
  "apiVersion": "1.0",
  "context": " ",
  "method": "listAll",
  "data": {
    "flags": [
      {
        "name": "radar_video_fusion_metadata",
        "value": true,
        "description": "Include Radar Video Fusion in AnalyticsSceneDescription metadata.",
        "defaultValue": false
      },

To verify that it is enabled, the command below list the enabled feature flags:

'http://192.168.0.90/axis-cgi/featureflag.cgi' \
"Content-Type: application/json" --data \
'{"apiVersion": "1.0", "method": "listAll", "context": " "}'

Which yields to:

{
  "apiVersion": "1.0",
  "context": " ",
  "method": "listAll",
  "data": {
    "flags": [
      {
        "name": "radar_video_fusion_metadata",
        "value": false,
        "description": "Include Radar Video Fusion in AnalyticsSceneDescription metadata.",
        "defaultValue": false
      },

Restart the device

The new fields in the Radar-Video Fusion metadata stream should be present. Switching the radar transmission off also changes the metadata stream (in real time).

RTSP URL for analytics metadata stream for the radar channel:

/axis-media/media.amp?camera=2&video=0&audio=0&analytics=polygon

RTSP URL for the event stream is the same as all other Axis devices and it is valid for channels:

/axis-media/media.amp?video=0&audio=0&event=on

Metadata fields

AXIS Q1656-DLE is placed on the active track and will get the same capabilities of AXIS OS.

  • Since it is fed by the combination of radar and camera, the main analytics working on this device is Axis Object Analytics
  • If your customer require Video Motion Detection, they should use AXIS Q1656-LE instead

A sample frame with the new fields can look like this (new fields are marked as bold)

<?xml version="1.0" ?>
<tt:SampleFrame xmlns:tt="http://www.onvif.org/ver10/schema" Source="AnalyticsSceneDescription">
   <tt:Object ObjectId="101">
      <tt:Appearance>
         <tt:Shape>
            <tt:BoundingBox left="-0.6" top="0.6" right="-0.2" bottom="0.2"/>
            <tt:CenterOfGravity x="-0.4" y="0.4"/>
            <tt:Polygon>
               <tt:Point x="-0.6" y="0.6"/>
               <tt:Point x="-0.6" y="0.2"/>
               <tt:Point x="-0.2" y="0.2"/>
               <tt:Point x="-0.2" y="0.6"/>
            </tt:Polygon>
         </tt:Shape>
         <tt:Color>
            <tt:ColorCluster>
               <tt:Color X="255" Y="255" Z="255" Likelihood="0.8" Colorspace="RGB"/>
            </tt:ColorCluster>
         </tt:Color>
         <tt:Class>
            <tt:ClassCandidate>
               <tt:Type>Vehical</tt:Type>
               <tt:Likelihood>0.75</tt:Likelihood>
            </tt:ClassCandidate>
            <tt:Type Likelihood="0.75">Vehicle</tt:Type>
         </tt:Class>
         <tt:VehicleInfo>
            <tt:Type Likelihood="0.75">Bus</tt:Type>
         </tt:VehicleInfo>
         <tt:GeoLocation lon="-0.000254295" lat="0.000255369" elevation="0"/>
         <tt:SphericalCoordinate Distance="40" ElevationAngle="45" AzimuthAngle="88"/>
      </tt:Appearance>
      <tt:Behaviour>
         <tt:Speed>20</tt:Speed>
         <tt:Direction yaw="20" pitch="88"/>
      </tt:Behaviour>
   </tt:Object>
   <tt:ObjectTree>
      <tt:Delete ObjectId="1"/>
   </tt:ObjectTree>
</tt:SampleFrame>

For video-only targets the metadata would still look like this:

<?xml version="1.0" ?>
<tt:SampleFrame xmlns:tt="http://www.onvif.org/ver10/schema" Source="AnalyticsSceneDescription">
   <tt:Object ObjectId="101">
      <tt:Appearance>
         <tt:Shape>
            <tt:BoundingBox left="-0.6" top="0.6" right="-0.2" bottom="0.2"/>
            <tt:CenterOfGravity x="-0.4" y="0.4"/>
            <tt:Polygon>
               <tt:Point x="-0.6" y="0.6"/>
               <tt:Point x="-0.6" y="0.2"/>
               <tt:Point x="-0.2" y="0.2"/>
               <tt:Point x="-0.2" y="0.6"/>
            </tt:Polygon>
         </tt:Shape>
         <tt:Color>
            <tt:ColorCluster>
               <tt:Color X="255" Y="255" Z="255" Likelihood="0.8" Colorspace="RGB"/>
            </tt:ColorCluster>
         </tt:Color>
         <tt:Class>
            <tt:ClassCandidate>
               <tt:Type>Vehical</tt:Type>
               <tt:Likelihood>0.75</tt:Likelihood>
            </tt:ClassCandidate>
            <tt:Type Likelihood="0.75">Vehicle</tt:Type>
         </tt:Class>
         <tt:VehicleInfo>
            <tt:Type Likelihood="0.75">Bus</tt:Type>
         </tt:VehicleInfo>
      </tt:Appearance>
   </tt:Object>
   <tt:ObjectTree>
      <tt:Delete ObjectId="1"/>
   </tt:ObjectTree>
</tt:SampleFrame>

For radar-only targets with no video history, the metadata would still look like this:

<?xml version="1.0" ?>
<tt:SampleFrame xmlns:tt="http://www.onvif.org/ver10/schema" Source="AnalyticsSceneDescription">
   <tt:Object ObjectId="101">
      <tt:Appearance>
         <tt:Shape>
            <tt:BoundingBox left="-0.6" top="0.6" right="-0.2" bottom="0.2"/>
            <tt:CenterOfGravity x="-0.4" y="0.4"/>
            <tt:Polygon>
               <tt:Point x="-0.6" y="0.6"/>
               <tt:Point x="-0.6" y="0.2"/>
               <tt:Point x="-0.2" y="0.2"/>
               <tt:Point x="-0.2" y="0.6"/>
            </tt:Polygon>
         </tt:Shape>
                 <tt:Class>
            <tt:ClassCandidate>
               <tt:Type>Vehical</tt:Type>
               <tt:Likelihood>0.75</tt:Likelihood>
            </tt:ClassCandidate>
            <tt:Type Likelihood="0.75">Vehicle</tt:Type>
         </tt:Class>
         <tt:VehicleInfo>
            <tt:Type Likelihood="0.75">Vehicle</tt:Type>
         </tt:VehicleInfo>
         <tt:GeoLocation lon="-0.000254295" lat="0.000255369" elevation="0"/>
         <tt:SphericalCoordinate Distance="40" ElevationAngle="45" AzimuthAngle="88"/>
      </tt:Appearance>
      <tt:Behaviour>
         <tt:Speed>20</tt:Speed>
         <tt:Direction yaw="20" pitch="88"/>
      </tt:Behaviour>
   </tt:Object>
   <tt:ObjectTree>
      <tt:Delete ObjectId="1"/>
   </tt:ObjectTree>
</tt:SampleFrame>

The new metadata fields:

GeoLocation

  • Provides the longitude and latitude with respect to the camera.
  • You can enter the camera's actual coordinate and you have the metadata stream the actual coordinate of the moving object.
     

The GeoLocation is presented like this:

<tt:GeoLocation lon="-0.000254295" lat="0.000255369" elevation="0"/>

Spherical Coordinate:

  • The spherical coordinates system is commonly used in mathematics. It provides the distance, which is the actual distance from the camera to the moving object, Azimuth θ, and Elevation φ measured by the radar.
     

The Spherical Coordinate is presented like this:

<tt:SphericalCoordinate Distance="40" ElevationAngle="45" AzimuthAngle="135"/>

Spherical coordinates system

Speed:

  • The absolute speed of the detected object is measured by the radar.
  • All the speed information provided in the metadata is measured by radar technology.
  • The unit is meters per second.
     

The Speed is presented like this:

<tt:Speed>20</tt:Speed>

Direction of movement: 

  • The direction element describes the direction of movement of an object in the Geolocation orientation with angles Yaw ψ and Pitch θ , provided by the radar.
  • The range of yaw ψ,  is between -180 and +180 degrees, where 0 is rightward and 90 is away from the device.
  • The range of Pitch θ is between -90 and 90.
     

The direction of movement is presented like this:

<tt:Direction yaw="20" pitch="88"/>

Spherical coordinates system

Considerations Axis Q1656-DLE Radar-Video Fusion Camera

  • The video stream rotation feature is not available.
  • Corridor format functionality is not available.
  • Multi-view areas will not be possible on this camera.
  • No digital PTZ on the camera.
To top