Module: camera

Executive Summary

The goal of the camera module is to simulate a camera in the Basilisk codebase. Although images are provided by the visualization, they are renders of the Unity engine and are not necessarily representative of a camera. The module reads in an image from a file, or in the simulation as a pointer to image data, then corrupts it according to input parameters.

Module Assumptions and Limitations

The practical limitation of this module is that it decodes and re-encodes the images that are corrupted. Outside of this design choice, the limitations are limited to the corruption methods used to replicate real camera physics. A Gaussian Dark Current might not always be a good model to represent such a phenomenon.

Message Connection Descriptions

The following table lists all the module input and output messages. The module msg connection is set by the user from python. The msg type contains a link to the message structure definition, while the description provides information on what this message is used for.

Module I/O Messages

Msg Variable Name

Msg Type

Description

imageInMsg

CameraImageMsgPayload

camera input message

cameraConfigOutMsg

CameraConfigMsgPayload

camera parameters output message

imageOutMsg

CameraImageMsgPayload

camera output message

Detailed Module Description

Overview of Corruption Sequence

This modules pulls heavily from the OpenCV library to perform a series of image corruptions. The following methods are implemented and executed in the sequence shown:

  • Gaussian noise on the image

  • Blurring

  • Dark Current

  • HSV adjustments

  • BGR adjustments

  • Dead and stuck pixels

  • Random Cosmic rays

Doxygen documentation for OpenCV can be found here.

Overview of Corruptions

  1. Gaussian noise is designed to simulate camera sensor noise. This is done by using the addWeighted OpenCV method and scaling the noise according to the input parameter. The noise is zero-mean with standard deviation equal to twice the gaussian noise parameter. The image is then thresholded at the 6-sigma value of the noise parameter in order to keep the background dark.

  2. Blurring is implemented using the standard OpenCV blur function with size specified by the blur parameter. This type of blur is referred to as a box blur. The only requirement is that the blur parameter be an odd number.

  3. Dark current is due to thermal properties of the CCD or CMOS sensor in use: as electrons are created independently of incoming light, they are captured in the pixel potential wells and appear to be a signal. Dark current noise is the statistical variation of this phenomenon. In Basilisk, dark current noise is added with a Gaussian noise model with zero standard deviation and mean of 15 × D, where D is another input to the module.

  4. The image color can be changed in the HSV color space. The hue value is rotated by a specified amount of radians, while the saturation and value component can be adjusted through a positive or negative percentage values. The hue value rotates such that 360 degrees becomes 0 degrees. The saturation and value components are limited to [0,255].

  5. The image color can be changed in the BGR color space. All color channel changes are specified through an integer percentage value that can be either positive or negative. The resulting integer color value is limited to lie within [0,255].

  6. Dead and stuck pixels are also implemented as a static perturbation. A dead pixel is a pixel on the sensor that always reads black. A stuck pixel is a pixel on the sensor that always reads white. At the initialization of the run, a random number generates arbitrary pixel coordinates and either saturates them dark or light. The corrupted pixels stay the same throughout the simulation and provide an example of a static artifact.

  7. Camera sensor with relatively high energies, they can saturate a line of pixels and corrupt the image. Cosmic rays are modeled in Basilisk by randomly choosing a point on the sensor as well as second point within a maximal distance of the first one. The abundance of cosmic rays on an image depend on the shutter speed amongst other parameters, and the module allows to toggle the frequency and quantity of such events.

To read more about the corruptions and for example pictures see section 5.2 of Dr. Thibaud Teil’s thesis.

Because each successive filter is applied on top of the previous, the order in which they are applied is very important. Currently Basilisk does not support a custom order with out directly modifying the source code. The order is as shown in the following table describing the filter and the parameters to control the filter. This order was determined in part by trying to match a simulated image to a real image of mars and also based on what makes sense. In the following parameters a value of 0 turns this corruption off and is the default value. Any filter that should be supplied must be provided a non-zero filter parameter value.

Order of Corruptions

Corruption

Parameters

Notes

Gaussian Noise

gaussian

[double] Adds noise with a mean of 0 and standard deviation of 2 * scaling parameter

Blur

blurParam

[double] Determines the size of the box blur. Blur size parameter must be odd

Dark Current

darkCurrent

[double] Adds noise with mean of 15 * scaling factor and standard deviation of 0

HSV Adjust

hsv

[3D vector of doubles] First parameter is given in radians and determines the hue shift. Second two parameters are scaling factors for saturation and value

BGR Adjust

bgrPercent

[3D vector of ints] Parameters correspond to scaling factors for blue, green, and red

Salt/Pepper

saltPepper

[double] Probability of both stuck and dead pixels is calculated as 0.00002 * scaling parameter

Cosmic Rays

cosmicRays

[double] Adds the specified number of cosmic rays

User Guide

The test and these few lines show an example setup for the module.

 1moduleConfig.imageInMsg.subscribeTo(inputCamMsg)
 2
 3moduleConfig.filename = ""
 4moduleConfig.saveImages = 0
 5# If images are to be saved, add the directory to which they
 6should be saved
 7#moduleConfig.saveDir = '/'.join(imagePath.split('/')[:-1]) + '/'
 8
 9#Camera config values
10moduleConfig.cameraIsOn = 1
11moduleConfig.sigma_CB = [0,0,1]
12
13#Noise Values
14moduleConfig.gaussian = 2
15moduleConfig.darkCurrent = 1
16moduleConfig.saltPepper = 2
17moduleConfig.cosmicRays = 1
18moduleConfig.blurParam = 3
19moduleConfig.hsv = [30*macros.D2R, 0, 0]

These scalar double values are written such that 0 provides no corruption of that type and 10 provides very high levels of errors (not bounding though)

class Camera : public SysModel
#include <camera.h>

visual camera class

Public Functions

Camera()

The constructor for the Camera module. It also sets some default values at its creation.

~Camera()

This is the destructor

void UpdateState(uint64_t currentSimNanos) override

This module reads an OpNav image and extracts circle information from its content using OpenCV’s HoughCircle Transform. It performs a greyscale, a bur, and a threshold on the image to facilitate circle-finding.

Parameters

currentSimNanos – The clock time at which the function was called (nanoseconds)

Returns

void

void Reset(uint64_t currentSimNanos) override

This method performs a complete reset of the module. Local module variables that retain time varying states between function calls are reset to their default values.

Parameters

currentSimNanos – current time (ns)

Returns

void

void hsvAdjust(const cv::Mat&, cv::Mat &mDst)

Adjusts the HSV values of each pixel. Can be used to shift the hue, saturation, and brightness of an image.

Parameters

  • mSrc – source image

  • mDst – destination of modified image

Returns

void

void bgrAdjustPercent(const cv::Mat&, cv::Mat &mDst)

Adjusts the BGR values of each pixel by a percent value. Can be used to simulate a sensor with different sensitivities to B, G, and R.

Parameters

  • mSrc – source image

  • mDst – destination of modified image

Returns

void

void addGaussianNoise(const cv::Mat&, cv::Mat &mDst, double, double)

Adds gaussian noise to an image. Can be used to add color noise and dark current.

Parameters

  • mSrc – source image

  • mDst – destination of modified image

  • Mean – mean pixel value

  • StdDev – standard deviation of pixel value

Returns

void

void addSaltPepper(const cv::Mat&, cv::Mat &mDst, float, float)

Adds dead and hot pixels to an image.

Parameters

  • mSrc – source image

  • mDst – destination of modified image

  • pa – probability of dead pixels

  • pb – probability of hot pixels

Returns

void

void addCosmicRay(const cv::Mat&, cv::Mat &mDst, float, double, int)

Adds a cosmic ray to an image. The ray is modelled as a single pixel wide white line.

Parameters

  • mSrc – source image

  • mDst – destination of modified image

  • probThreshhold – probability of getting a ray each frame

  • randOffset – if adding multiple rays pass in the number of each to guarantee a random ray

  • maxSize – max length of cosmic ray

Returns

void

void addCosmicRayBurst(const cv::Mat&, cv::Mat &mDst, double)

Adds a user specified number of cosmic rays to an image. Rays are modelled as a single pixel wide white line. The maximum length is hard-coded as 50 pixels.

Parameters

  • mSrc – source image

  • mDst – destination of modified image

  • num – number of cosmic rays to be added

Returns

void

void applyFilters(cv::Mat &mSource, cv::Mat &mDst)

Applies all of the various perturbations to an image with user specified levels. Each parameter is a double scaling actor. A parameter of 0 will result in the respective perturbation not being applied.

Parameters

  • mSource – source image

  • mDst – destination of modified image

Returns

void

Public Members

std::string filename = {}

Filename for module to read an image directly.

ReadFunctor<CameraImageMsgPayload> imageInMsg

camera image input message

Message<CameraImageMsgPayload> imageOutMsg

camera image output message

Message<CameraConfigMsgPayload> cameraConfigOutMsg

The name of the CameraConfigMsg output message.

std::string saveDir = {}

The name of the directory to save images.

uint64_t sensorTimeTag = {}

[ns] Current time tag for sensor out

int32_t saveImages = {}

[-] 1 to save images to file for debugging

char parentName[MAX_STRING_LENGTH] = {}

[-] Name of the parent body to which the camera should be attached

Camera parameters

int cameraIsOn = {}

[-] Is the camera currently taking images

int cameraID = {1}

[-] Is the camera currently taking images

int resolution[2] = {512, 512}

[-] Camera resolution, width/height in pixels (pixelWidth/pixelHeight in Unity) in pixels

uint64_t renderRate = {}

[ns] Frame time interval at which to capture images in units of nanosecond

double fieldOfView = {0.7}

[r] camera y-axis field of view edge-to-edge

double cameraPos_B[3] = {}

[m] Camera position in body frame

double sigma_CB[3] = {}

[-] MRP defining the orientation of the camera frame relative to the body frame

char skyBox[MAX_STRING_LENGTH] = {"black"}

[-] name of skyboz in use

int postProcessingOn = {}

Enable post-processing of camera image. Value of 0 (protobuffer default) to use viz default which is off, -1 for false, 1 for true.

double ppFocusDistance = {}

Distance to the point of focus, minimum value of 0.1, Value of 0 to turn off this parameter entirely.

double ppAperture = {}

Ratio of the aperture (known as f-stop or f-number). The smaller the value is, the shallower the depth of field is. Valid Setting Range: 0.05 to 32. Value of 0 to turn off this parameter entirely.

double ppFocalLength = {}

[m] Valid setting range: 0.001m to 0.3m. Value of 0 to turn off this parameter entirely.

int ppMaxBlurSize = {}

Convolution kernel size of the bokeh filter, which determines the maximum radius of bokeh. It also affects the performance (the larger the kernel is, the longer the GPU time is required). Depth textures Value of 1 for Small, 2 for Medium, 3 for Large, 4 for Extra Large. Value of 0 to turn off this parameter entirely.

double gaussian = {}

Gaussian noise level.

Noise paramters

double darkCurrent = {}

Dark current intensity.

double saltPepper = {}

Stuck and Dark pixels probability.

double cosmicRays = {}

Random cosmic rays (number)

double blurParam = {}

Blur over image in pixels.

Eigen::Vector3d hsv = {Eigen::Vector3d::Zero()}

(double) HSV color correction, H (-pi/pi) hue shift, S and V are percent multipliers

Eigen::Vector3d bgrPercent = {Eigen::Vector3d::Zero()}

(int) BGR color correction values as percent

BSKLogger bskLogger

&#8212; BSK Logging

Private Members

uint64_t localCurrentSimNanos = {}
void *pointImageOut = {nullptr}

void pointer for image memory passing