Structure

The SPIF file uses the NetCDF4 format. NetCDF4 is a structured binary file format capable of containing large datasets and has automatic compression utilities. NetCDF4 is widely supported on a variety of platforms and environments.

In a similar fashion to the CF (Climate and Forecast) Conventions, the SPIF conventions define a minimum structure, in terms of groups, variables, and attributes, for compliance. Any additional data contained within the file is optional but should not conflict with the standards. In this documentation any suggested but optional data for inclusion are given in italics. Due to the focussed type of data, SPIF conventions are more demanding of variable and attribute names than the CF conventions are. SPIF follows the CF scoping guidelines in that dimensions are visible to all child groups.

The SPIF group structure is below with optional groups shown in italics

root
│
├── instrument 1
│ ├── “core”
│ ├── “raw”
│ ├── “aux”
│ ├── level-0
│ │ └── level-1
│ └── level-2
│
├── instrument 2
:

Contents of a SPIF file are;

root

The root of the SPIF file contains global attributes associated with the file.

instrument group

Data from each instrument channel data is included within a separate group. All metadata relevant to an instrument channel plus universal variables are included in the instrument root. Many instruments will only have a single channel but in all cases the instrument group exists to maintain a consistant structure.

core group

The spif-core group contains the raw image data. Coordinate variables are image_num and pixel.

raw group

The spif-raw group contains the raw image data read directly from the binary file. These data include buffer header information that may have been stripped out of the spif-core data variables and may be useful for deciphering data corruption issues and the like.

aux group

The spif-aux group contains additional data which is generated by a given instrument but is not required for image processing. This data is included to maintain integrity of the original dataset, making SPIF a suitable format for long term archiving.

level-0 group

Particle information and derived parameters in units of pixels/slices. Coordinate variable is particle.

level-1 group

Particle information and derived parameters in physical units. Coordinate variable is particle.

level-2 group

Higher-level, possibly aggregated, data products derived from level-1. Coordinate variable is (probably) time.

Mandatory and Optional Content

The SPIF definition is somewhat constrained to ensure that standard-compliant files contain all the information/data required for future processing. SPIF files must contain, as a minimum, a set of mandatory parameters. That is groups, attributes, and (coordinate) variables.

In addition to these mandatory netCDF parameters, a SPIF file can be extended with additional groups, attributes, and (coordinate) variables. It may assist users if these added parameters are familiar and so some suggested optional parameters are given;

Further information on optional groups and their contents are described in SPIF Extensions and Optional Extras.

Instrument Channels

Data is contained within instrument channel groups inside the SPIF file. Only data from a single probe is contained in a single SPIF file however some instruments may have more than one channel, for example the SPEC 2DS has vertical and horizontal channels with each detecting particles independently, essentially behaving as independent instruments. For a probe that is made up of multiple instruments (e.g., 2DS, 3V-CPI, etc) where the particle detection will not be coincident, data is split into different instrument groups as this allows keeping these data together in a single file. The instrument channel group names are not pre-defined but should be self-explainatory with further information included as attributes of these groups.

Image Array Structure

Flattened image data is stored in the instrument core group as a 1-dimensional array. The length of this array is the product of the number of images, given by the dimension image_num, and the width and height of each image. Depending on the type of instrument, the width and/or height may be fixed or variable for each image.

Note

It is (in)conceivable that a future probe may natively produce 3-dimensional images. In this case the 1-dimensional image array would still work perfectly well, just another variable of length in this new dimension would need to be added. Something for a new version of the convention once/if that ever happens!

In order to reconstruct the 2-dimensional images from the flattened array two variables are provided, width and height, these are of length image_num.

Note

It is possible that width is given as a scalar variable instead of an array of length image_num?

More details about this is included in the section describing the Instrument Core Group.