Add test sharding, proactive clean, and retry logic for self-hosted CI

[sbryngelson]

Summary

Hardens self-hosted CI with test sharding, retry logic, and script deduplication.

Test sharding & retry

• Add --shard i/n flag to ./mfc.sh test — splits tests via modular arithmetic for even distribution
• Frontier GPU matrix now runs 2 shards per interface (acc/omp), halving wall-clock time
• Zero-test guard on both --only and --shard — empty results raise an error instead of silent green CI
• GitHub runner tests retry up to 5 sporadic failures using tests/failed_uuids.txt
• Abort path cleans failed_uuids.txt to prevent stale retries

--only filter improvements

• UUIDs use OR logic (match any), labels use AND logic (match all)
• --only matching zero tests now raises an error instead of silently passing

CI script consolidation

• Merge submit-bench.sh into submit.sh for all 3 clusters (frontier, frontier_amd, phoenix) — submit.sh auto-detects bench vs test mode from the submitted script's basename
• Unify frontier/ and frontier_amd/ scripts via directory-name detection — build.sh, bench.sh, submit.sh, and test.sh are now byte-identical across both directories
• Net deletion of 3 files and ~120 lines of duplicated shell code

Other

• Frontier test jobs use --qos=normal on batch partition (1h59m, CFD154 account)
• --requeue on Phoenix SLURM jobs for preemption recovery
• Build retry wrapper (3 attempts with clean between)
• Pin nick-fields/retry to commit SHA for security on self-hosted runners
• Lint-gate must pass before self-hosted tests run
• Skip benchmark workflow for bot review events

Depends on: #1170

Test plan

• Frontier GPU tests run in 2 shards per interface and complete within 2h
• Phoenix tests pass with --requeue and preemption recovery
• Lint-gate blocks self-hosted tests on lint failure
• GitHub runner retry logic fires on ≤5 test failures
• Benchmark jobs submit correctly via merged submit.sh (bench mode auto-detected)
• frontier/ and frontier_amd/ scripts are identical and detect cluster correctly
• --shard with zero resulting tests raises an error (not silent pass)

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 44.04%. Comparing base (1412eb2) to head (73fd804).
⚠️ Report is 5 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1171      +/-   ##
==========================================
- Coverage   44.05%   44.04%   -0.02%     
==========================================
  Files          70       70              
  Lines       20496    20499       +3     
  Branches     1991     1993       +2     
==========================================
- Hits         9029     9028       -1     
- Misses      10328    10330       +2     
- Partials     1139     1141       +2     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.