GSoC 2: Electric Boogaloo

Sometimes, you just need to know when to quit.

And I don’t mean from a project perspective. Well, I do, but not entirely in that regard.

See, there are a handful of “required tests” that are listed in the IOOS Glider Quality Control Manual. Some of them aren’t implemented. The syntax test was one of them, as there are so many different ways that such a test could be implemented. This entry talks a little bit about how I started exploring what seemed like a simple test, and slowly learned a lot more computer science than was appropriate for everyday glider quality control.

Disclaimer: I’m not an expert on this, and much of this entry is paraphrased from my notes across multiple days. If you’re curious to learn more about this journey - see my GSoC notes here.

Let’s run through the definition of the test on page 10 of our 2016-version of the manual:

Test 2) Syntax Test (required)

Check to ensure that the message is structured properly

I’m already worried about this one. The use of the word “message” rather than “data” or some other message suggests that this is handling something abnormal. Like a message packet. Most users of the IOOS Toolbox are likely to be people that just want to quality control their data and they may not have access to the “message”-level data that preceeds all the science units.

Received data message (full message) contains the proper structure without any indicators of flawed transmission such as parity errors. Possible tests are: a) the expected number of characters (NCHAR) for fixed length messages equals the number of characters received (REC_CHAR), or b) passes a standard parity bit check, cyclic redundancy check (CRC), etc. Many such syntax tests exist, and the user should select the best criteria for one or more syntax tests.

This is concerning, as someone that wants the QARTOD toolbox to be as complete as possible. This paragraph goes a step further, describing the message as a “full message” and clarifies that this test is indeed, intended for transmission tests.

Fortunately, it outlines what the requirements for such a test would be. It defines NCHAR and REC_CHAR as lengths of characters that should be received. Perhaps REC_CHAR is assigned programmatically, perhaps it’s done manually. Whatever the case, the manual is outlining this test. Well, the first of two. Next, it introduces running binary-level checks: A standard parity bit check or a CRC (cyclic redundancy check). As it turns out, the parity bit check is a subset of the CRC.

We’ll get back to the CRC check later. There is still more to unpack here. The manual then says “Many such syntax tests exist…” and that the user “…should select the best criteria [themselves] for one or more syntax tests”. This assumes a lot of the user of the QARTOD toolbox. First, that the user will know what to do with binary data (which is often to pass off to pyglider or dbdreader and have it do the scary stuff). That includes knowing how the data is formatted and what all the individual bytes mean. That’s easier said than done. There are many manufacturers that skip the raw step and provide their users with scientifically-valid, human-readable ASCII. I was able to find a description of the raw data for Slocum gliders, which are used by the majority of operators in the United States, but understanding the doc itself has a certain entry level that is required.

Next, the user needs to know what all the syntax tests are that can be done on binary data and which fits their use case best. I don’t quite know what those are. I’m pretty sure I wouldn’t know or care about those tests as a student, considering how disconnected this would feel from any undergraduate (or even graduate) coursework. I’m familiar with MD5 hashes - a way of quickly checking if data has been modified. A blog that says to use it at your own risk is one of the first results in my flavor of Google algorithm when I look up the different types of hash tests. Even then, it isn’t entirely clear where this fits into my daily routine as a data analyst.

See, it’s completely possible that these syntax tests don’t belong in the toolbox. Nowadays, it seems like both the Slocum and SeaExplorer gliders transmit data from the glider to the satellite, and then that data is transmitted back to a manufacturer’s data center prior to delivery to the glider’s owner. Something like a CRC or MD5 hash test would apply during one of those data transmission steps, not so much during data processing by a student/tech/analyst. In my brief exchanges with the greater oceanographic glider community, there seem to be few teams that work with the binary files. Most of the parsing is done by Teledyne or Alseamar.

(A borrowed example from Baumgartner et al., highlighting some of what a glider pipeline looks like)

Furthermore, there’s nothing to test. I don’t have examples of glider binaries that fail the CRC test. I know there’s a CRC value assigned the the header of raw Slocum files, but there isn’t a way to test what the file should have been before transmission. It isn’t really emulatable without having the data direct from a glider, and the data direct from a satellite.

Overall, the syntax tests were an interesting rabbit hole.

• Why was there no code in the toolbox for such a test?
  • Most users don’t know what it means, or how to use it, considering they don’t have examples or work with that form of the data.
• Why didn’t the glider community agree on what it was?
  • Different manufacturers have different instructions. At the IOOS GDAC, there are dozens of different sensors which lead to different configs and no two data collections will be exactly the same. As such, it’s hard to automate or define in general.
• Why was it so hard to get examples of glider binary files?
  • The types of tests described in the glider manual take place earlier in the data pipeline than most users would have access to. They’re done by the manufacturers in automated routines coded up by experienced engineers. Perhaps this step used to be more appropriate, considering how telecommunications and AUV technology has advanced since the manual was written. The dearth of “raw” data files speaks for how few users have access to them.

It all came down to a lot of sleuthing which mandated some networking with other glider experts. It was ultimately a handy endeavour, however, to better understand the data flowpath and see where the QARTOD toolbox fits in said flowpath. Perhaps this can be of help for the IOOS Glider committee, should the manual be updated for its 10-year anniversary.