Senior Thesis: Part Two

This article discusses the second part of my senior thesis, namely the work on simulation testing and writing some opening sections. It is part of a series of articles on my senior thesis; use the “thesis” tag to see all of them. This is a living document; once the thesis moves into the tracer phase, I will write another part.

Simulator Verification

After the last couple of months of hard work, the simulator now works! This is a huge step forwards, since now we have reliable tracking that provably works against the IABP dataset. However, with over 2,700 buoys, not all simulations will perfectly match even the cleaned real-world dataset, so further work is currently locked behind understanding the simulator’s failure modes.

So far, production runs of the simulator have the following outcomes:

• Works perfectly (the simplest and fortunately most common case)
• Data input issues:
  • IABP data outputs nonsensical GPS coordinates (need to improve cleaning process, since there’s often a valid path within the noise)
  • Polar Pathfinders has no data for a specific tile (in which case we cannot test against that buoy since we lack comparisons)
  • Buoy in real life moves too close to shore for Polar Pathfinders (we cannot ever get closer than 25km offshore due to data limits)
• Simulator code issues:
  • Run is far too short (usually solved by switching from weekly to daily vector data)
  • Plotting fails to render (solved by increasing the error handling in the R plotter)
  • Incorrect out-of-season behavior (in development)

I am still in the middle of manually reviewing all buoy model runs to see what the failure modes are for each, which is letting me catch some of the most difficult-to-find bugs within the simulator. Reviewing each run is needed to find ones that are beneficial for the thesis as benchmarks, even if there were no bugs; I want runs that span the whole melt-freeze season and play nice with the restrictions of the Polar Pathfinders dataset. Further work is also needed to write a cleaning method more sophisticated than the ones currently available in the literature, since review indicates that there are clearly useful buoys in there that are only non-viable for this benchmarking merely because of some extraneous GPS coordinates that haven’t been properly removed (but there are just too many to delete them manually).

Continuous Integration

Once we’re done verifying all the runs, how do we make sure the simulator continues to produce reasonable and physically accurate results? The best way to go about it is to steal from the software engineering world and write some continuous integration code. What this means is that on every upload of new code to GitHub, the remote server will run the code and check to see if the behavior is what we expect the code to reliably do, and if it doesn’t match, warns about this discrepancy.

Given that this simulator must match the Polar Pathfinders dataset as closely as possible to enable doing useful science with it, and has a lot of complex behavior that could emerge by combining the various simulator options in different ways, I think writing tests is paramount to ensuring that no future code subtly breaks anything going forward. Currently this is happening with the hypothesis and coverage.py Python libraries to help ensure coverage both over many possible bad inputs and code in the project; I would do this for the R code as well, but that is not complex enough to justify the hassle at the moment.

The other metric I am looking to add is an “error score”, which is essentially the distance between where the simulated buoy is and the actual buoy is (matched by closest timestamp). This score could then be calculated for each run; if it ever changes between code commits, there’s some level of nondeterminism. This is both more complex to write and more complex to test (since it requires running the whole sim all over again), but once it’s done, it should then be substantially harder to introduce mistakes by accident.

Tracing Data

In the coming months, once the Introduction and Methods sections are written, I intend to start extending the model to trace data along the path, so that we can see climate variables from a buoy’s-eye-view, so to speak. This will enable actually addressing the question this thesis is all about, since running the tracers and starting from many locations and times all across the Arctic should yield more than enough data to find such a relationship, if it exists. At present, this is still quite a long ways away.