test failed due to non-unique log file

[davepacheco]

While running part of the Omicron test suite in a loop, I ran into this:

failures:

---- db::queries::external_ip::tests::test_next_external_ip_is_restricted_to_pools stdout ----
log file: "/dangerzone/omicron_tmp/omicron_nexus-598f187aac60c458-test_next_external_ip_is_restricted_to_pools.3174.55.log"
thread 'db::queries::external_ip::tests::test_next_external_ip_is_restricted_to_pools' panicked at 'called `Result::unwrap()` on an `Err` value: Os { code: 17, kind: AlreadyExists, message: "File exists" }', /home/dap/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ed0b9c8/dropshot/src/test_util.rs:476:51
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace


failures:
    db::queries::external_ip::tests::test_next_external_ip_is_restricted_to_pools

test result: FAILED. 84 passed; 1 failed; 0 ignored; 0 measured; 50 filtered out; finished in 25.10s

error: test failed, to rerun pass `-p omicron-nexus --lib`

The log file has the test name, some Cargo identifier, pid, and a per-process counter. But it still looks like we conflicted with an old log file. I say that because this failure happened in the last 24 hours (so, after 2022-11-10T23:00Z), but that path exists from before that:

 $ ls -lrt /dangerzone/omicron_tmp/omicron_nexus-598f187aac60c458-test_next_external_ip_is_restricted_to_pools.3174.55.log
-rw-r--r--   1 dap      staff        965 Nov  3 19:47 /dangerzone/omicron_tmp/omicron_nexus-598f187aac60c458-test_next_external_ip_is_restricted_to_pools.3174.55.log

This seems probably unlikely in CI, but possible in dev environments where failed test run files may stick around for a while.

[davepacheco]

The problem here is in dropshot and I've filed oxidecomputer/dropshot#479 for it.

[davepacheco]

It's a little surprising to hit this because we generally remove log files from successful tests. I think this probably tends to happen when you set TMPDIR to the same directory all the time and you run the test suite a lot. Over time, failures leave around log files and eventually you hit a collision. Given that, I think you can work around this by either clearing out $TMPDIR or setting a fresh TMPDIR for each run.