auv_nav

Usage

auv_nav has 3 commands:

parse, which reads raw data from multiple sensors and writes it in chornological order to a text file in an intermediate oplab data format.
process, which outputs previously parsed data in a format that the 3D mapping pipeline can read
convert, which converts previously parsed data to ACFR format at this moment.

Parse (`auv_nav parse`)

This command reads raw data from multiple sensors and writes it in chornological order to a text file in an intermediate oplab data format.

Input files read by `auv_nav parse`

All input files are read from the dive subfolder of the “raw” folder

vehicle.yaml
mission.yaml
All files referenced in mission.yaml (e.g. DVL, INS, USBL, depth sensor logs, image timestamp files, usually saved in subfolders of the dive folder)

Output files written by `auv_nav parse`

All ouput files are written to the dive subfolder of the “processed” folder

mission.yaml (This is a copy of the mission.yaml file from the input folder, with metadata about program version, the date and time, host computer name and username appended.)
vehicle.yaml (Same as above.)

And if output format “oplab” is selected (this is the default):

A subfoder “nav” in the dive folder containing
- nav_standard.json
- json_data_info.html
- timestamp_history.html

Usage:

auv_nav parse [-h] [-F] [--merge] path [path ...]

positional arguments:
  path         Folderpath where the (raw) input data is. Needs to be a subfolder of 'raw' and contain the mission.yaml
               configuration file.

optional arguments:
  -h, --help   show this help message and exit
  -F, --Force  Force file overwite
  --merge      Merge multiple dives into a single JSON file. Requires more than one dive PATH.

Process (`auv_nav process`)

This command expects a parsed dive with an intermediate file nav_standard.json present and runs all localisation filters specified in the configuration file auv_nav.yaml, or uses the defaults if this file is not present.

Input files read by `auv_nav process`

The following files from the dive subfolder in the “processed” folder:

nav/nav_standard.json (generated by auv_nav parse when run in “oplab” mode).
vehicle.yaml (copied there by auv_nav parse)
mission.yaml (copied there by auv_nav parse)

The following file from the dive subfolder in the “configuration” folder:

auv_nav.yaml (if it does not exist, a default auv_nav.yaml file is generated, which the user needs to edit as required)

Output files written by `auv_nav process`

A subfolder starting with “json_renav_” followed by start and end date and time (e.g. json_renav_20180802_154058_20180803_210450) containing csv and html files is written to the dive subfolder in the “processed” folder.

Input files read by `auv_nav process`

The following files from the dive subfolder in the “processed” folder:

nav/nav_standard.json (generated by auv_nav parse when run in “oplab” mode).
vehicle.yaml (copied there by auv_nav parse)
mission.yaml (copied there by auv_nav parse)

The following file from the dive subfolder in the “configuration” folder:

auv_nav.yaml (if it does not exist, a default auv_nav.yaml file is generated, which the user needs to edit as required)

Output files written by `auv_nav process`

A subfolder starting with “json_renav_” followed by start and end date and time (e.g. json_renav_20180802_154058_20180803_210450) containing csv and html files is written to the dive subfolder in the “processed” folder.

Usage:

auv_nav process [-h] [-F] [-s START_DATETIME] [-e END_DATETIME] [-v] path

positional arguments:
  path                  Path to folder where the data to process is. The folder has to be generated using auv_nav
                        parse.

optional arguments:
  -h, --help            show this help message and exit
  -F, --Force           Force file overwite
  -s START_DATETIME, --start START_DATETIME
                        Start date & time in YYYYMMDDhhmmss from which data will be processed. If not set, start at
                        beginning of dataset.
  -e END_DATETIME, --end END_DATETIME
                        End date & time in YYYYMMDDhhmmss up to which data will be processed. If not set process to
                        end of dataset.
  -v, --verbose         Enable verbose output to terminal

The algorithm replicated the same folder structure as the input data, but instead of using a subfolder of ‘raw’, a folder ‘processed’ is created (if doesn’t already exist), with the same succession of subfolders as there are in ‘raw’. The mission.yaml and the vehicle.yaml files are copied to this folder. The interlaced data is written in oplab format to a file called ‘nav_standard.json’ in a subfolder called ‘nav’.

Convert (`auv_nav convert`)

The algorithm will read in the nav_standard.json file obtained after the parsing and will write the required formats and outputs.

At v0.2.2 the following conversions are available:

oplab_to_acfr: Converts a processed dive to ACFR navigation formats by saving a dRAWLOGS_cv folder with its navigation solutions called combined.RAW.auv, mission.cfg and stereo_pose.est.
acfr_to_oplab
hybis_to_oplab
koyo20rov_to_oplab

At v0.1.6 the following output formats are available:

acfr: The AFCR format uses a ‘dRAWLOGS_cv’ folder name and outputs its navigation solution to a file called ‘combined.RAW.auv’ as well as to a ‘mission.cfg’ to the processing folder root.

If output format “acfr” is selected:

mission.cfg
A sublfolder “dRAWLOGS_cv” with a file named combined.RAW.auv

auv_nav convert usage:

auv_nav convert [-h] {oplab_to_acfr,acfr_to_oplab,hybis_to_oplab,koyo20rov_to_oplab} ...

positional arguments:
  {oplab_to_acfr,acfr_to_oplab,hybis_to_oplab,koyo20rov_to_oplab}
    oplab_to_acfr       Converts an already processed dive to ACFR format
    acfr_to_oplab       Converts a VehiclePosEst.data and/or a StereoPosEst.data to OPLAB csv format
    hybis_to_oplab      Converts a hybis navigation file to oplab CSV format
    koyo20rov_to_oplab       Converts TCM and ship_logs files to oplab CSV format

optional arguments:
  -h, --help            show this help message and exit

The algorithm will read in the nav_standard.json file obtained after the parsing and will write the required formats and outputs. At v0.1.6 the following output formats are available:

acfr: The AFCR format uses a ‘dRAWLOGS_cv’ folder name and outputs its navigation solution to a file called ‘combined.RAW.auv’ as well as to a ‘mission.cfg’ to the processing folder root.
hybis: Given a Hybis navigation file and their image folder, it generates the required CSV files for correct_images or other software. It does not generate any intermediate json file.
koyo20rov Given a koyo20rov dive folder, it reads in the vehicle.yaml; mission.yaml; FileTime.csv for three cameras; TCM.csv; and two ship_log folder .csv files. It generates dead reckoning .csv files for two colour cameras and LM165_at_dvl.

Inputs and outputs of `auv_nav`

When running auv_nav, the user indicates the path of the dive folder. Typically auv_nav is run from the dive folder, in which case the (relative) path is “.”. This folder must be a subfolder (can be nested in multiple levels of subfolders) of a folder called eihter “raw”, “processed” or “configuration”. auv_nav automatically reads and writes files in the corresponding subfolders of these 3 folders, i.e. reads raw inputs from the subfolder of “raw”, configurations from the corresponding subfolder in the “configuration” folder and writes the outputs to the corresponding subfolder in “processed”. When writing files, missing subfolders are automatically created. If one or several of the output files already exist, auv_nav aborts unless overwriting is enabled with the command line key -F or --Force.

Examples

An example data set can be downloaded from here:

`https://console.cloud.google.com/storage/browser/university-southampton-   squidle/raw/year/cruise/platform/YYYYMMDD_hhmmss_platform_sensor/`

Make sure you copy the folder format from ‘raw/year/cruise/platform/YYYYMMDD_hhmmss_platform_sensor/’

For OPLAB output format:

Parse raw data into json file format ‘nav_standard.json’ and visualise data output by parsing the target directory where mission.yaml is stored.

auv_nav parse -F '<container directory>/raw/year/cruise/platform/YYYYMMDD_hhmmss_platform_sensor/'

Example of output:

'oplab' - nav_standard.json
[{"epoch_timestamp": 1501974125.926, "epoch_timestamp_dvl": 1501974125.875, "class": "measurement", "sensor": "phins", "frame": "body", "category": "velocity", "data": [{"x_velocity": -0.075, "x_velocity_std": 0.200075}, {"y_velocity": 0.024, "y_velocity_std": 0.200024}, {"z_velocity": -0.316, "z_velocity_std": 0.20031600000000002}]},
{"epoch_timestamp": 1501974002.1, "class": "measurement", "sensor": "phins", "frame": "inertial", "category": "orientation", "data": [{"heading": 243.777, "heading_std": 2.0}, {"roll": 4.595, "roll_std": 0.1}, {"pitch": 0.165, "pitch_std": 0.1}]},
{"epoch_timestamp": 1501974125.926, "epoch_timestamp_dvl": 1501974125.875, "class": "measurement", "sensor": "phins", "frame": "body", "category": "altitude", "data": [{"altitude": 31.53, "altitude_std": 0.3153}, {"sound_velocity": 1546.0, "sound_velocity_correction": 0.0}]},
{"epoch_timestamp": 1501974002.7, "epoch_timestamp_depth": 1501974002.674, "class": "measurement", "sensor": "phins", "frame": "inertial", "category": "depth", "data": [{"depth": -0.958, "depth_std": -9.58e-05}]},
{"epoch_timestamp": 1502840568.204, "class": "measurement", "sensor": "gaps", "frame": "inertial", "category": "usbl", "data_ship": [{"latitude": 26.66935735000014, "longitude": 127.86623359499968}, {"northings": -526.0556603025898, "eastings": -181.08730736724087}, {"heading": 174.0588800058365}], "data_target": [{"latitude": 26.669344833333334, "latitude_std": -1.7801748803947248e-06}, {"longitude": 127.86607166666667, "longitude_std": -1.992112444781924e-06}, {"northings": -527.4487693247576, "northings_std": 0.19816816183128352}, {"eastings": -197.19537408743128, "eastings_std": 0.19816816183128352}, {"depth": 28.8}]},{"epoch_timestamp": 1501983409.56, "class": "measurement", "sensor": "unagi", "frame": "body", "category": "image", "camera1": [{"epoch_timestamp": 1501983409.56, "filename": "PR_20170816_023649_560_LC16.tif"}], "camera2": [{"epoch_timestamp": 1501983409.56, "filename": "PR_20170816_023649_560_RC16.tif"}]}
]

Extract information from nav_standard.json output (start and finish time can be selected based on output in step 2)

auv_nav process -f oplab -s 20170817032000 -e 20170817071000 '<container directory>/processed/year/cruise/platform/YYYYMMDD_hhmmss_platform_sensor/'

The setting used to run this can be updated by modifying the following file, which takes repository default parameters on the first run. ‘/configuration/year/cruise/platform/YYYYMMDD_hhmmss_platform_sensor/auv_nav.yaml’`

For ACFR output format:

Parse raw data into combined.RAW.auv and mission.cfg

auv_nav convert '<container directory>/processed/year/cruise/platform/YYYYMMDD_hhmmss_platform_sensor/'

Example of output:

'acfr' - combined.RAW.auv
    PHINS_COMPASS: 1444452882.644 r: -2.29 p: 17.21 h: 1.75 std_r: 0 std_p: 0 std_h: 0
    RDI: 1444452882.644 alt:200 r1:0 r2:0 r3:0 r4:0 h:1.75 p:17.21 r:-2.29 vx:0.403 vy:0 vz:0 nx:0 ny:0 nz:0 COG:0 SOG:0 bt_status:32768 h_true:0 p_gimbal:0 sv: 1500
    PAROSCI: 1444452882.644 298.289
    VIS: 1444452882.655 [1444452882.655] sx_073311_image0003805_AC.tif exp: 0
    VIS: 1444452882.655 [1444452882.655] sx_073311_image0003805_FC.tif exp: 0
    SSBL_FIX: 1444452883 ship_x: 402.988947 ship_y: 140.275056 target_x: 275.337171 target_y: 304.388346 target_z: 299.2 target_hr: 0 target_sr: 364.347071 target_bearing: 127.876747

Folder Structure

The output files are stored in a mirrored file location where the input raw data is stored as follows with the paths to raw data as defined in mission.yaml

e.g.
    raw /<YEAR> /<CRUISE> /<DIVE>   /mission.yaml
                                    /vehicle.yaml
                                    /nav/gaps/
                                    /nav/phins/
                                    /image/r20170816_023028_UG069_sesoko/i20170816_023028/

For this example, the outputs would be stored in the follow location, where folders will be automatically generated

e.g.
    processed /<YEAR> /<CRUISE> /<DIVE> /nav/nav_standard.json
                                        /dRAWLOGS_cv/combined.RAW.auv
                                        /mission.cfg

Example of Required YAML Configuration Files

These files need to be in the root raw folder. Further examples can be found in default_yaml folder.

mission.yaml: This file describes the mission’s details and parameters of each sensor (e.g. where is the filepath of the data, its timezone format, etc).

#YAML 1.0
version: 2

origin:
  latitude: 22.745000
  longitude: 153.266667
  coordinate_reference_system: wgs84

velocity:
  format: ae2000  # phins or ae2000
  filepath: nav/ae_log/  # where is the filepath located
  filename: pos171123064542.csv  # what is the filename
  timezone: jst  # what time zone is the timestamps in
  timeoffset: 0.0  # what is the time offset
  std_factor: 0.01 # what is the std factor
  std_offset: 0    # what is the std offset

orientation:
  format: ae2000  # phins or ae2000
  filepath: nav/ae_log/
  filename: pos171123064542.csv
  timezone: jst
  timeoffset: 0.0
  std_factor: 0.01
  std_offset: 0

depth:
  format: ae2000  # phins or ae2000
  filepath: nav/ae_log/
  filename: pos171123064542.csv
  timezone: jst
  timeoffset: 0.0
  std_factor: 0.01
  std_offset: 0

altitude:
  format: ae2000  # phins or ae2000
  filepath: nav/ae_log/
  filename: pos171123064542.csv
  timezone: jst
  timeoffset: 0.0
  std_factor: 0.01
  std_offset: 0

usbl:
  format: usbl_dump  # gaps or usbl_dump
  filepath: nav/ssbl/
  filename: 20171123_AE2000f_LOG.csv  # filename of data, only for usbl_dump format
  timezone: 9
  timeoffset: 0.0
  label: T1  # selected usbl label
  std_factor: 0.01
  std_offset: 0

image:
  format: seaxerocks_3  # acfr_standard or seaxerocks_3
  cameras:
    - name: Cam51707923 # fore
      path: image/SeaXeroxData20171123_095119/Xviii/Cam51707923
    - name: Cam51707925 # aft
      path: image/SeaXeroxData20171123_095119/Xviii/Cam51707925
    - name: LM165
      path: image/SeaXeroxData20171123_095119/LM165
      records_laser: true
  timezone: jst
  timeoffset: 0.0

vehicle.yaml This file describes the location of the sensors relative to the defined position (origin) of the vehicle.

#YAML 1.0
origin: #centre of robot
  surge_m: 0
  sway_m: 0
  heave_m: 0
  roll_deg: 0
  pitch_deg: 0
  yaw_deg: 0

# distance with reference to origin/centre of robot
usbl:
  surge_m: 0 
  sway_m: 0 
  heave_m: -0.289 
  roll_deg: 0
  pitch_deg: 0
  yaw_deg: 0

ins:
  surge_m: -0.09 
  sway_m: 0 
  heave_m: 0 
  roll_deg: 0
  pitch_deg: 0
  yaw_deg: 0

dvl:
  surge_m: -0.780625
  sway_m: 0
  heave_m: 0.204
  roll_deg: 0
  pitch_deg: 0
  yaw_deg: 0

depth:
  surge_m: 0 
  sway_m: 0 
  heave_m: 0 
  roll_deg: 0
  pitch_deg: 0
  yaw_deg: 0

fore: # Front camera
  surge_m: 0.262875
  sway_m: 0
  heave_m: 0.5 
  roll_deg: 0
  pitch_deg: 0
  yaw_deg: 0

aft: #Back Camera
  surge_m: 0.012875
  sway_m: 0
  heave_m: 0.5 
  roll_deg: 0
  pitch_deg: 0
  yaw_deg: 0

laser: # Laser
  surge_m: 0.147875
  sway_m: 0
  heave_m: 0.5 
  roll_deg: 0
  pitch_deg: 0
  yaw_deg: 0

Example Dataset for testing

An example dataset can be downloaded from the following link with the expected folder structure: https://drive.google.com/drive/folders/0BzYMMCBxpT8BUF9feFpEclBzV0k?usp=sharing

Download, extract and specify the folder location and run as

OPLAB format:
auv_nav parse -f oplab ~/raw/2017/cruise/dive
auv_nav process -f oplab -s 20170816032345 -e 20170816034030  ~/processed/2017/cruise/dive
Convert to ACFR
auv_nav convert -f acfr ~/raw/2017/cruise/dive

The coordinate frames used are those defined in Thor Fossen Guidance, Navigation and Control of Ocean Vehicles

i.e. Body frame: x-direction: +ve aft to fore y-direction: +ve port to starboard z-direction: +ve top to bottom i.e. Intertial frame: north-direction: +ve north east-direction: +ve east down-direction: +ve depth downwards

Parameter naming conventions long and descriptive names should be used with all lower case letters.

auv_nav

Usage

Parse (auv_nav parse)

Input files read by auv_nav parse

Output files written by auv_nav parse

Usage:

Process (auv_nav process)

Input files read by auv_nav process

Output files written by auv_nav process

Input files read by auv_nav process

Output files written by auv_nav process

Usage:

Convert (auv_nav convert)

Inputs and outputs of auv_nav

Examples

Folder Structure

Example of Required YAML Configuration Files

Example Dataset for testing

Parse (`auv_nav parse`)

Input files read by `auv_nav parse`

Output files written by `auv_nav parse`

Process (`auv_nav process`)

Input files read by `auv_nav process`

Output files written by `auv_nav process`

Input files read by `auv_nav process`

Output files written by `auv_nav process`

Convert (`auv_nav convert`)

Inputs and outputs of `auv_nav`