Skip to content

Random Segmented Data Files

Jump to bottom
mfvalin edited this page Mar 17, 2020 · 4 revisions

Table of Contents

file layout (tentative)

file 

  it is valid for a file to contain records having a different endianness
 
  <-------------------- segment 0 -------------------->   <-------------------- segment N -------------------->
  +--------+------+--------+------+-----------+-------+   +--------+------+--------+------+-----------+-------+
  | S O S  | data | ...... | data | directory | E O S |...| S O S  | data | ...... | data | directory | E O S | ....
  +--------+------+--------+------+-----------+-------+   +--------+------+--------+------+-----------+-------+
   ^                                              |        ^                                              |       
   +-----------------------------------------<----+        +-----------------------------------------<----+
   ^                                                                                                      |
   +-----------------------------------------<----------------------- fused case --------------------<----+
 
   S O S : Start Of Segment
   E O S : End Of Segment (contains "pointer" to beginning of segment or beginning of 1st segment in file
   segments may get fused in which case the directory of the last segment would cover the whole file
   concatenated files are a VALID segmented file

records

Start Of Segment 

   LNG   length      : 3 64 bit elements
   ZR    zero byte   : used as endianness detector
   RT    record type : 3 (Start Of Segment)
   SIGN  file type signature (APPL is FsTd for "RPN standard" files)
   Note: RT == 0 is ILLEGAL

End Of Segment 

   LNG   length      : 3 64 bit elements
   ZR    zero byte   : used as endianness detector
   RT    record type : 4 (End Of Segment)
   SEGL  length of segment in BYTES, including S O S and E O S ( SEGL(lo) | ( SEGL(hi) << 32 ) )
   Note: RT == 0 is ILLEGAL

data record 

   LNG   length      : in 64 bit elements (0 -> 2 PBytes) = LNG(lo) | ( LNG(hi)<<32 )
   ZR    zero byte   : used as endianness detector
   RT    record type : 1 (data)
   Note: RT == 0 is ILLEGAL
   data payload is a sequence of 32 bit items
data payload structure

directory record 

   LNG   length      : in 64 bit elements 
   ZR    zero byte   : used as endianness detector
   RT    record type : 2 (directory block)
   Note: RT == 0 is ILLEGAL
   directory is a sequence of 32 bit items
directory structure

metadata

compatible metadata 

 (to preserve old API usage when possible)
     search metadata  (FIXED, mandatory)
      datev  (old style)
      ip1    (lower 4 bits masked)
      ip2    (lower 4 bits masked)
      ip3    (lower 4 bits masked)
      typvar (first 2 only, rest masked)
      nomvar (first 4 only, rest masked)
      etiket (first 12 only, rest masked)
     informative metadata (FIXED, mandatory)
      deet (lower 24 bits only)  (in seconds)
      npas (lower 26 bits only)
      ni   (lower 24 bits only)
      nj   (lower 24 bits only)
      nk   (lower 20 bits only)
      dtyp
      nbits
      grtyp (first 1 only)
      ig1  (lower 24 bits only)
      ig2  (lower 24 bits only)
      ig3  (lower 24 bits only)
      ig4  (lower 24 bits only)

new metadata 

     search metadata (variable with mandatory items ?)
      dateo   full 64 bits in micro/nano seconds
      ip1/2/3 extended to full 32 bits (4 extra bits for mantissa)
      typvar  extended to 4/5 characters ? case sensitivity ?
      nomvar  extended to 5 characters ? case sensitivity ?
      etiket  extended to 20 characters ? case sensitivity ?
      type+value (the new IPn) how many pairs (TPn+FPn|UPn|SPn) 
                 (Type+Float/Unsigned/Signed) (32 bits, coded)
     informative metadata (variable with mandatory items ?)
      deet    extended to full 32 bits (with LNG style encoding) (0bb 10b 110)
      npas    extended to full 32 bits
      dtyp
      nbits
      grtyp (extended to ? characters)
      ig1/2/3/4 (extended to 32 bits)
      extra geoloc  TBD

Clone this wiki locally