Apollo Cyber Record Viewer

Introduction

This website can be used to understand or validate the data saved in the Baidu Apollo's recording files (produced by cyber_recorder command).

Cyber Recording File

The recording file is a binary file of a fixed format. It consists of 1 Header section, 1 Index section and 1+ Channel sections and 1+ Chunk HEADER + Chunk Body sections, saved in the following order

[Header Section] [empty padding] [Channel Section 1] [Channel Section 2] ... [Chunk Header 1] [Chunk Body 1] [Chunk Header 2] [Chunk Body 2] ... [Index Section]

Section Binary format

Each section has a fixed length section head (16 bytes) followed by flexible length section body. Sections have the following structure:

positiondataDescription
0 - 3XX XX XX XX
The first four bytes records signed 32 bit integer (little endian) whose value is the section type. The section types are defined in the file: cyber/proto/record.proto
  • "SECTION_HEADER": 0
  • "SECTION_CHUNK_HEADER": 1
  • "SECTION_CHUNK_BODY": 2
  • "SECTION_INDEX": 3
  • "SECTION_CHANNEL": 4
4 - 700 00 00 00These four bytes are empty for padding purpose.
8 - 15XX XX XX XX XX XX XX XXThe next 8 bytes records signed 64 bit integer (little endian) whose value is the number of bytes of the section body (which follows the section head), a.k.a. section body size.
16 - (16 + section body size - 1)XX XX XX XX ...
The section body is parsed according to the section type defined in the section head. Section body are the encoded messages using protobuf libraries and thus can be decoded using corresponding libraries generated by protoc.
For example, the first section is usually the Header Section, and the section body can be decoded by: Header.deserializeBinary(buf).toObject(), where Header is imported from protobuf libraries generated by protoc.

Section Semantics

Section TypeDescription
SECTION_HEADER
Header section records the following information:
  • chunk_interval: limits the duration of each chunk
  • segment_interval: limits the duration of each recording segment file
  • index_position: the starting position of the last index section
  • chunk_number: number of chunks saved in this file
  • channel_number: number of channels in total in this file
  • begin_time
  • end_time
  • message_number: number of messages in total
  • size: total file size
  • ...
SECTION_CHANNELChannel Section records the channel name and message type. e.g. name: "/apollo/perception/obstacles" and type: "apollo.perception.PerceptionObstacles"
SECTION_CHUNK_HEADERThe chunk header section indicates the start of a message chunk. It records the chunk's start timestamp and end timestamp as well as total number of messages saved in the chunk body section.
SECTION_CHUNK_BODY
The chunk body section follows its chunk header section. It contains a message list which consists of all the cyber messages recorded during the period of the current chunk, ordered by message time. Each message in the list contains the following fields:
  • channelName: e.g. /apollo/sensor/gnss/corrected_imu
  • time: message timestamp
  • content: a string representing the protobuf encoded binary data of the message. The content of the message can be decoded by MessageClass.deserializeBinary(string).toObject(), where MessageClass is imported from protobuf libraries generated by protoc
SECTION_INDEXThe index section is located at the end of each recording file. Its starting position is saved in the Header section. It saves the index list which records the starting positions and other information (e.g. number of messages, begin time, end time etc.) of all the Channel sections, Chunk Header sections and Chunk Body sections.

Usage

  1. Generate protobuf libraries using protoc.
    1. Enter Apollo terminal
    2. Create a temporary folder, e.g. /apollo/protobuf_out
    3. Use following command to generate protobuf libraries and save to the protobuf_out folder
      find modules/ cyber/ -name "*.proto" | grep -v node_modules | xargs protoc --js_out=import_style=library,binary:protobuf_out
    4. If you encounter errors related to conflicts in yolo.proto and yolo4.proto, you may comment out unused proto files and try again.
  2. Import the generated JavaScript libraries
    1. Navigate to the top of this website, and click the "Choose files" button in the middle.
    2. Navigate to the protobuf_out folder and select all the .js files
    3. Wait for some time and let the browser load those libraries. When ready, the "choose file" button on the right will be enabled.
  3. Import the cyber recording file. Currently only supports one file at a time.
    1. Navigate to the top of the website, and click the "Choose file" button at the right end.
    2. Select any recording file
    3. Wait for some time to parse the file. When the parsing is done, you can scroll down and check out the results.

Author

Apollo's 社区布道师 Tang Yun (ntutangyun [at] gmail [dot] com)

My current research interest is software testing of Autonomous vehicles. Feel free to drop me an email for research collaborations. :)

The repository address is https://github.com/ntutangyun/cyber-recorder-viewer. Feel free to create forks and pull requests.

Kindly acknowledge if you use any code from this repository :)