Unsafe and reading from files

Einar wrote (long ago),

In the article you wrote on unsafe image processing, you announced a follow-up that would "cover some more unsafe scenarios, including dealing with existing structures on disk". But then I guess something came up, and that article never surfaced (to my knowledge and loss, respectively). How about a blog post instead?

Einar's first mistake was believing anything that I write about future columns. My track record at getting back to topics is pretty poor - so bad, in fact, that I've tried to stop making promises about what I'll write about next.

Let's look at some code from a GPS application I've been playing around with for quite some time. This code opens a file and reads a mess of binary structures out of it, and recreates them as real structures.

  public unsafe static GpsDataset Load(string filename)
{
GpsDataset dataset = new GpsDataset();

   FileInfo fileInfo = new FileInfo(filename);

int count = (int) (fileInfo.Length / sizeof(GpsPosition));

   using (FileStream fileStream = new FileStream(filename, FileMode.Open))
{
BinaryReader binaryReader = new BinaryReader(fileStream);

    for (int i = 0; i < count; i++)
{
byte[] buffer = binaryReader.ReadBytes(sizeof(GpsPosition));
dataset.positions.Add(new GpsPosition(buffer));
}
}
return dataset;
}

It starts by getting the size of the file, and then computing how many items are in the file (I could have just read till end-of-file, but that seemed less elegant. Know the size ahead of time allows me to pre-allocate my positions ArrayList (not that I did that, but it *allows me to* do that...)).

Then it's off to reading the chunks out, into a byte[] buffer. It would be considerably more efficient if I stuck with a stream and read the bytes into the same buffer over and over rather than re-allocated a whole buffer, but a) I haven't gotten around to optimizing this yet b) I wrote it to be flexible, when I didn't know what I'd be storing c) The UI operations are far slower than what I'm doing here and d) did I say it wasn't yet optimized.

Anyway, this buffer gets passed off to the constructor for GpsPosition, which looks like this:

 [StructLayout(LayoutKind.Sequential, Pack=1)]
public unsafe struct GpsPosition
{
  public float AltitudeAboveWGS84;
public float EstimatedPositionError;

  public float EstimatedHorizontalError;
public float EstimatedVeriticalError;
public PositionFix PositionFix;
public double TimeOfWeek;

  public double Latitude;
public double Longitude;
public float VelocityEast;
public float VelocityNorth;
public float VelocityUp;
public float AltitudeAboveMSL;
public short LeapSeconds;
public int WeekNumberDays;

  public GpsPosition(byte[] data)
{
fixed (byte* pData = data)
{
this = *(GpsPosition*) pData;
}
}
}

The constructor simply fixes the data buffer so it's okay to get a pointer to it, then uses a cast and pointer dereference to copy the contents of the buffer over the struct.

Note the StructLayout attribute at the beginning of the routine. In it, I set the packing to 1. By default, the runtime will give you nicely aligned structures that are more efficient to access by padding. In this case, I need the structure to match the definition from the Garmin API spec *exactly*, so I have to set packing to 1.

That's about all there is to it. Define the structure, get a byte[] of the right size, and then copy the data over in a fixed block.

Comments

• Anonymous
  April 13, 2004
  What would be my second mistake, I wonder (not that I doubt having made one...)?
  
  Interestingly, I first read your article on unsafe code looking for ways to optimize code reading GPS data(!) Unfortunately, the GPS file was in ascii format, so I couldn't apply the technique you're showing here. Someone should take the time to produce an optimized yet flexible .NET library for reading GPS data from the serial port, a binary file, an ascii file etc. Perhaps someone already has?
• Anonymous
  April 13, 2004
  Oh, I forgot to say thanks for pulling my request out of bit oblivion and making a blog post of it! Thanks :)
• Anonymous
  April 13, 2004
  Oh, and I take it there's no equally simple solution for transforming an existing big-endian structure into a little-endian struct?
• Anonymous
  April 19, 2004
  I like the pointer manipulation you are using to cast the struct. What are your thoughts on code like the following?
  
  public Boxcar(byte [] data)
  {
  GCHandle hBoxCar = GCHandle.Alloc(data,
  GCHandleType.Pinned);
  
  IntPtr pBoxCar = hBoxCar.AddrOfPinnedObject();
  
  Marshal.PtrToStructure(pBoxCar, this);
  
  hBoxCar.Free();
  }
  
  This uses the same struct layout attribute as in your example and it’s definitely not as simple as yours.
  
  Thanks
  -Mike