Source: Horus View and Explore B.V.

Introduction

Designing real-world applications using rapidly changing technologies can be challenging for companies in terms of bringing their product to market. Building a complete AI system is hard. One needs experience in the fields such as cameras, embedded systems, networking, data acquisition, and storage. Therefore cooperation between companies, operating in different fields, can be a huge benefit to all parties involved. It’s a challenge that different fields have different and often non-overlapping sets of knowledge, thus forming liabilities for the project reaching its completion.

To address the latter, this blog will cover the following topics:

1. How a graphical representation, such as the Horus Linking Lab, can help design complex systems Horus Framework, including Linking Lab, in a nutshell
2. How to build a GDPR compliant camera system, using the Horus Framework and NavInfo Europe anonymization model
3. Sand Motor, an example of a real-world application using the Horus Framework

Graphical design of complex systems

Most fields employ graphical tools that assist the designer in their tasks. Well-known tools include Quartus (FPGA), Matlab & Simulink, and Altium Designer (PCB). One such tool that allows designers to visualize and design applications that span different domains is the Horus Framework with Linking Lab as its graphical user interface.

The image above shows Linking Lab with an application that encompasses many different parts a typical AI/software/hardware project may require. These include cameras (physical), video-transcoding, AI inference, networking, and custom client code.

For the designer, there is no difference between adding a physical camera or a Docker container to the pipeline. This allows one to discuss concepts such as networking (“I would like my data to be sent over there”) with disregard of the underlying technology (TCP/UDP/unicast/multicast).

Linking Lab provides a “discovered” tab (figure 2) that lists all system-specific components. Components range from system services to physical devices. The framework keeps scanning for new components. So when an extra sensor is connected it will appear automatically.

The Horison Framework in a nutshell

The Horison Framework contains all the necessary tools and programs that are needed to deliver an end product. This includes the creation of Linux images, toolchains, application server (System V2), design tool (Linking Lab), and user interfaces (Web Builder). This blog will cover the application server and design tool briefly.

System V2

The main application is called System V2 (V2), this is an application server that will run the designs made with the Linking Lab software. In many cases, V2 is running on a remote machine, for example, an embedded board like an NVIDIA Jetson Xavier AGX, while the designer is working in Linking Lab on his local desktop. The workflow is straight-forward: design, save, upload, and run.

Linking Lab

An application engineer uses Linking Lab to design a system. All V2 systems make themselves known on the local network (figure 3), and you simply select which device you would like to configure. After this, Linking Lab will reflect resources that are available for the specific device such as cameras, GPU’s and services.

Building GDPR compliant live camera systems

NavInfo Europe is an experienced AI technology provider based in the Netherlands, develops its robust computer vision and AI solutions, and has proven expertise by successfully deploying our robust solutions to automotive, road authorities, and smart city providers.

Protecting privacy is becoming mandatory in many territories across the world and there has been a noticeable increase in privacy-related regulations globally, including the GDPR in Europe, CCPA in California, CSL in China, and APPI in Japan. To support organizations being compliant with these regulations, we developed a state-of-the-art AI model to anonymize personal information like faces and license plates. For more on how personal data can be protected, discover NavInfo Europe’s anonymization solution. 

This model incorporated in a Deepstream 6 container in the combination with Horus Framework, enables real-time video anonymization.  Below we will show how to use the Horus Framework to build a system grabbing video data, pass it through the container and use the output in the system again. All this runs real-time on a Jetson Xavier AGX in the local network.

Steps

1. Download image and flash the Jetson

To prepare the Jetson with the complete Horus Framework there are images available for download here. The Jetson images are built using Yocto and based on the NVIDIA Jetpacks (images for other devices are available too). To flash the device, unpack the archive on a Linux machine, put it into recovery mode, connect the USB port and run the doflash.sh script.

The images provided are built to run in a production environment. The core rootfs is read-only and there are overlays in place to store configuration data. By using Mender, the system is also able to be updated over- the-air or by transferring an update artifact.

2. Linking Lab and activation

After flashing the Jetson, it’s ready to be configured. To do this, get Horus Linking Lab for either Linux or Windows.

Start Linking Lab. This brings you to the devices overview screen. By default Linking Lab also starts a local V2 application server. This is shown by the This system entry in the overview. If the Jetson is in the same local network as the local machine, it will be discovered automatically. Simply click on the Jetson to connect to it and start configuring.

To get access to the complete component library, a license should be activated. Go to the License Manager to enter the license or start a free trial. This license is then bound to the connected device.

3. Docker configuration

To use a Deepstream 6 Docker image on the Jetson with the Horus Framework a couple of things have to be configured. First, an extra storage device is needed to store the images. For example, by installing an NVME SSD or USB drive.

After installing the storage device, mount it to a preferred location and make it persistent in /etc/fstab. Then configure Docker to use this mount as the data root. In /etc/docker/daemon.json add:

{
“data-root”: “/mnt/docker”
}

Now restart Docker ( systemctl restart docker ) and pull the container from your registry to use it in V2.

4. Application design

In this blog, we design an application with a Deepstream 6 container provided to us by NavInfo (the end result is shown in figure 1) this Deepstream 6 container has the following interfaces which System V2 can use to stream data:

• V4L input
• V4L output
• TCP JSON output

To start designing the application, open the Graph Builder window in Linking Lab. Here, all the available components are shown which can be used on the device. By simply dragging the components onto the graph and connecting them the application can be constructed. After “setting” the pipeline, the application will be uploaded to the System V2 application server on the Jetson. For example, to grab and view a video stream from a discovered GenICam camera, go to the Discovered tab, drag it onto the graph and configure it as follows:

The Discovered tab (figure 2) shows all the detected components on the Jetson, this includes the previously pulled Docker container. To use a Deepstream 6 container in V2 drag it onto the graph like any other component. Double click the Docker component to show its properties. The settings property is a JSON string containing Docker properties directly passed to the Docker daemon. Make sure it contains at least the following:

{“HostConfig”:{“Runtime”:“nvidia”,”Privileged”:true,”NetworkMode”:“host”}}

This allows the container to access the NVIDIA dependencies and use the Jetsons hardware. The other properties give an easy way to configure mounts, ports (although not needed in NetworkMode: Host), and environment variables. Using these properties it’s possible to set up a working Deepstream 6 container.

To use all the features V2 offers, a bit more configuration is required. For example, V2 supports a wide range of cameras (V4L, GenICam, ONVIF, etc.). To use these as an input for the Deepstream 6 container, there is support for the V4L loopback driver. However, not all cameras output the correct video format, so a couple of components may be needed to transform it.

First, the Media Converter allows for rescaling and transforming of pixel formats. This can be used to create the required format for the Deepstream 6 container. But as not all video format ID’s are compatible with the V4L FourCC codes, the FourCC Setter component was built to apply the correct code onto the video buffers. Thus making sure that the video stream is correctly interpreted by V4L.

When the video stream is in the correct size and format and has the proper FourCC code, it can be streamed to the V4L output component. This component will create a new capture device in Linux which can be mounted in the Deepstream 6 container. To do this add the following mount to the Docker component:

/dev/video<X>:/dev/video<Y>

Where  X is the V4L identifier of the Xavier and  Y is the V4L identifier used in the container. Now the container will use the V4L device created by V2. Because of the broad range of features in V2, this allows many more things to be done with the video stream. V2 supports projections, stitching, and much more. All this without altering the Deepstream 6 container or AI model.

To use the results produced by the container (video stream, detection data, etc.) in V2 there are multiple options. For video, the most efficient manner would be to let the Docker container write to another instance of the same V4L loopback driver. V2 will discover the capture device like any other V4L capture device. To do this, first, add another mount to the Deepstream 6 container with a second V4L loopback device. Then rediscover devices while the container is running to detect the newly created capture device and drag it onto the graph.

The information of the capture device is stored in the pipeline (figure 4) so that the next time it will be loaded automatically. However, because this device can be created sometime after the Horus pipeline is started it is possible to connect a clock component to the init pipe of the V4L Capture component. This will make sure that the component will retry opening the stream on a set interval. Another, less efficient, option would be to stream through a network protocol (RTSP, TCP, UDP).

To receive detection data separate from the video stream, a TCP or UDP stream can be used. Using the Network Reader a raw socket can be opened to receive ASCII strings. Depending on the string representations this string can be converted to V2’s it internal message protocol. This makes it possible to use the detection data to trigger all kinds of things (video scaling, IO control, etc.). However, because not all protocols are open there is the Code Block component. This component allows third parties to create their own component implementation to convert their proprietary protocol to V2’s messaging protocol.

5. Transforming the NavInfo Europe protocol with Code Block

To show the ease of the Code Block component, the JSON output of the NavInfo container is transformed to the Horus protocol. The complete implementation can be found here. In this case, the container outputs a JSON string in the following format:

{
“height”: 480,
“width”: 640,
“objects”: [

{

“bbox”: {
“xmin”: 280,
“ymin”: 187,
“xmax”: 314,
“ymax”: 229
},

“category”: “1”,
“score”: 47
},

{

“bbox”: {

“xmin”: 385,
“ymin”: 148,
“xmax”: 421,
“ymax”: 193

},
“category”: “1”,
score”: 57
}

]

}

Each object represents a detected object, for this model a face or license plate. The Code Block iterates all the detected objects and creates a new message for each detection. This message contains three sensor values: a vector of doubles representing the detection bounding box, a string value with the detected category, and a double value with the detection’s score. When outputted using the Console Writer component (a useful component for debugging purposes), it looks as follows:

============================================

Source ID : AI Json parser
Bytes : 151

:: Message ::

Stamp : 1633955257823108
Format : UtcEpoch
Source : Grabber

:: Sensor ::

Name : bbox
Structure : VECTOR
Units : Unknown
Dimension 0 : 4
Values : 280 187 314 229
Name : category
Structure : SCALAR
Units : Unknown
Value : 1
Name : score
Structure : SCALAR
Units : Unknown
Value : 47

These messages can then be used to trigger other components or systems to act when something is detected. Using a Sensor Filter component a single scalar value can be checked for a certain value. Using a Matrix Filter component the bbox vector can be scanned for a certain area within the image. Using the Network Reader/Writer components the data can be sent to a completely different device and trigger something elsewhere.

Sand Engine, a real-world application

The Sand Engine is a large-scale peninsula where Delft University of Technology researches multiple subjects regarding sustainable coastal protection. To register the changes over time, Horus was asked to record the beach.

There are limited resources present, limited network bandwidth (4G) and power constraints. Because of the power limitations but high video processing demands, the Jetson Xavier AGX was chosen as heart of the system.

Two high resolution cameras were chosen with wide angle lenses. The Jetson debayers and then encodes the images to cope with the limited 4G bandwidth. To further help save bandwidth and network costs, V2 allows the user to rescale the images and change the framerate on the fly.

The next step in bringing the system further would be to utilize the power of Deepstream 6 to control dynamic video scaling. An AI that looks for changes in the environment which in turn controls the framerate and resolution of the video stream. For example, sending higher resolution images more often when the weather is more volatile. To help with the limited bandwidth of 4G T-Mobile has offered to give access to their 5G network that is available in the area.