rancher/dashboard#19828 by codyrancher, draft, "[Experiment]", no issue. Head 2cb7332 on master 55438bb. Reviewed 2026-11-12 in two passes: Cody's process from codyrancher/ai-skills at 8b419f0 (A) and ours as a fresh agent (B). Every finding was rechecked at the head commit before it got here. Nothing is posted.
Preview: none. The PR changes only Cypress support code and the workflow, so the evidence is the head CI run, its artifacts, and the PR's own server.ts run locally on CI recordings. Cypress was not run locally.
@rancher/cypress consumers and the Jenkins config.RANCHER_VERSION (1), and a logged-in hook is answered with a pre-login 401 (2).| Mode | Attempt 1 | Attempt 2 | Failures |
|---|---|---|---|
full (e2e-test) | 17 / 18 | 17 / 18 | kubewarden before all, also on master |
| generate | 16 / 18 | 18 / 18 | |
| integration | 12 / 18 | 13 / 18 | edit-fake-cluster (admin manager): finding 1 · user-retention (std usersAndAuths): finding 2 · compliance, logging (admin charts): shell window timing, 0 misses · not diagnosed: kontainer-drivers, name-annotation (admin manager), helmop (admin fleet), banners, performance (admin generic) |
The PR body's numbers (integration 10/18) come from run 43194123695 at 17482b6, three commits back. Generate adds 252 and integration 140 job minutes to full's 230.
RANCHER_VERSION; the new integration job does not. CURRENT_RANCHER_VERSION comes from it, so edit-fake-cluster.spec.ts expects https://ranchermanager.docs.rancher.com//how-to-guides/... while the replayed app links /v2.16/. The job is the third copy of e2e-test and has drifted twice: generate also lost the "Detect Rancher license" step, so its tags never get --@noPrime (no effect on master today, CI runs community rancher/rancher:head).docs.rancher.com//. Rechecked in the attempt-2 log.Integration doesn't get `RANCHER_VERSION` like full and generate do, so the log says "Rancher version: not provided" and `edit-fake-cluster.spec.ts` expects a `docs.rancher.com//` link, which fails it in admin manager on both attempts. Generate also dropped the Prime check that full has, so the two would run different tests on a Prime image.
lookup borrows the next test's recordings and pickEarly takes that test's first state, whatever the hook already did. user-retention's test clears its session first, so its first GET /v3/principals is a 401 (write 0) and the later one a 200 (write 3). Replay is faster than Rancher, the app's request lands in the before all hook after its login, gets the 401, and the hook fails with "Unauthorized 401: must authenticate".before all GET /rancherversion -> 200
before all GET /rancherversion -> 200
before all GET /v1/uiplugins -> 200
before all GET /version -> 200
before all GET /v1/management.cattle.io.settings?pagesize=100000&exclude=metadata.managedFields -> 200
before all GET /v1-public/authproviders?pagesize=100000&exclude=metadata.managedFields -> 200
before all POST /v1-public/login -> 200
before all POST /v1/ext.cattle.io.selfuser -> 201
before all GET /v1/management.cattle.io.users?pagesize=1 -> 200
before all GET /v1/management.cattle.io.user/u-fzw97 -> 200
before all GET /v1/uiplugins -> 200
before all GET /v3/principals (made inside the hook in CI) -> 401 {"type":"error","status":"401","message":"Unauthoriz
When a `before all` hook asks for something it didn't record, this borrows the next test's first recording, whatever the hook already did. user-retention's test clears its session first, so the hook's logged-in `GET /v3/principals` gets that test's pre-login 401, and the standard_user usersAndAuths job fails on both attempts. I got the same 401 replaying that job's recording through this server locally.
misses, the afterEach log or the spec summary, so the PR's own check ("logs no no recording: lines") passes while different data is served.replicas: 0 got the recorded replicas: 3 200; a team-b filter got team-a's rows; a request only another spec recorded as standard_user was served to admin; the only miss was a path recorded nowhere. Test (Pass B, CI recording): a metadata.name=nobody filter got the full user list. CI: admin charts logged 45 "replaying a similar one" lines and 0 misses while 2 tests failed.GET recorded query GET /v1/things?filter=metadata.namespace=team-a
-> 200 {"data":[{"id":"team-a/one"}]}
GET a different filter GET /v1/things?filter=metadata.namespace=team-b
-> 200 {"data":[{"id":"team-a/one"}]}
PUT a different body PUT /v1/things/team-a/one {"spec":{"replicas":0}}
-> 200 {"id":"team-a/one","spec":{"replicas":3}}
GET only another spec/user recorded GET /v1/secrets
-> 200 {"data":[{"id":"recorded-by-another-spec-and-user"}]}
GET nothing recorded GET /v1/nothing
-> 404 {"type":"error","status":404,"code":"NotFound","message":"e2e integration mode has no recording of GET /v1/nothing"}
test-end misses: {"misses":["GET /v1/nothing"]}
server console:
[e2e integration] listening on https://127.0.0.1.sslip.io:18443
[e2e integration] recordings: /var/folders/6v/k6yp126x6dd36kv4g1xtv1vr0000gn/T/rec-WLyQf6/admin
[e2e integration] indexed 3 distinct requests across 2 recording sets
[e2e integration] no exact recording, replaying a similar one: GET /v1/things?filter=metadata.namespace=team-b (demo > saves a thing)
[e2e integration] no recording: GET /v1/nothing (demo > saves a thing)Only a request with no recording anywhere counts as a miss. A save with a different body still gets the recorded success, a GET with a different filter gets the other filter's rows, and another spec's or user's recording can answer too, so integration can pass a change to what the UI sends, and a failing spec still logs "0 without a recording".
test.yaml runs on push to master and release-*, and generate and integration are gated only on the ci/skip-e2e label. So master would run all three modes, while the description says it keeps running against a real Rancher, and any replay failure turns the run red. Auto-retry already re-ran every run of this PR.These jobs have no PR condition and can't fail softly, so every push to master would also run generate and integration (the description says master keeps running only full), and with 13 of 18 integration jobs green every run goes red and auto-retry runs it again. They also add about 390 job minutes per run.
POST /v1-public/login is stored with its body, and its response keeps Set-Cookie: R_SESS=token-.... Without TEST_RECORDINGS_DIR they go to cypress/recordings/, which .gitignore doesn't cover. CI only has the throwaway password on a k3d Rancher that dies with the job, so nothing real leaks here. The PR's test steps run generate against API=https://<rancher>, and the code ships in @rancher/cypress.e2e-recordings-standard_user@userMenu@usersAndAuths: three HARs with the login body and R_SESS. Test: git check-ignore --no-index cypress/recordings/... exits 1, while cypress/videos matches .gitignore:95 (rechecked).Recordings keep the login body with the password and the `R_SESS` cookie, and `cypress/recordings` isn't in `.gitignore`. In CI that's only the throwaway `password`, but following the PR's test steps against a real Rancher leaves a real password and a live token in untracked files. Could those be left out and the folder ignored?
pump moves past a Steve frame whose resourceType isn't watched yet instead of holding it. Frames are timed from the recording (capped at 1.5s), so a watch sent a little later than against Rancher means the resource.change never arrives and the UI waits for a state that never comes.resource.start; at 50ms it gets both. Rerun after checking Pass B's harness copies server.ts unchanged. Not tied to a CI failure.A frame for a resource type the UI hasn't watched yet is skipped for good, so if the browser sends its watch a little later than it did against Rancher, that change never arrives. On a small made-up recording, watching at 400ms instead of 50ms lost it.
What's the plan for PRs? Integration replays this run's generate artifact, so it always waits for a real Rancher run. If PRs replay master's recordings later, how does a PR that changes requests get new ones?
HAR_FILE claim in the descriptionHAR_FILE dev replay already reads". har-file.js only keeps responses with content.text, and these keep bodies in _file, so it kept 0 of 32 responses from user-retention.har. Pass B ran it; I rechecked the code.The description says the `HAR_FILE` dev replay already reads this format, but `har-file.js` only keeps responses with `content.text`, so it dropped all 32 from `user-retention.har`. Did you mean it's HAR in general?
This says it returns the Cypress tasks it needs, but it returns the server.
| Pass A: Cody's process | Pass B: ours | |
|---|---|---|
| Kept findings | 3 comments | 3 blockers, 4 suggestions, 4 nits, 4 questions |
| Before-all hook gets a 401 (2) | ✓ reproduced on the CI recording | ✓ reproduced, plus a tested one-line guard |
| Silent fallbacks (3) | ✓ writes, filters, other spec and user | ✓ filters, other scopes; other users as a question |
| Workflow on master, red, cost (4) | ✓ master trigger | ✓ plus soft-fail, auto-retry, +390 job minutes |
| Missing RANCHER_VERSION (1) | ✗ missed; saw only the Prime drift and dropped it | ✓ from the admin manager log |
| Credentials in recordings (5) | seen, dropped as CI-only | ✓ kept for local runs |
| Socket frames lost (6) | – | ✓ synthetic |
| HAR_FILE claim (Q2) | – | ✓ ran the loader |
| Nits | – (his rule: never post one) | 4, one kept |
| Acceptance table | ✓ 10 criteria from the description | – |
| Full mode and consumers | ✓ extendConfig, the Jenkins config in qa-infra-automation | ✓ virtual-clusters-ui, rancher-ai-ui |
| Output | acceptance files, 3 condensed comments, 2 harnesses | review, 7-dimension grid, checked list, draft, 6 harnesses |
| Cost | about 45 min in this session, about 45 tool calls | 20 min, 99 tool calls, 258k tokens |
Lesson for both: when a PR adds a job that copies an existing one, diff the two jobs' env and steps key by key. Pass A noticed one dropped step and stopped there.
| Criterion | Verdict | How | |
|---|---|---|---|
| AC1 | full unchanged | Met | Server returns early, hook gated on e2eMode; full 17/18, the failure is on master too |
| AC2 | generate records each passing test | Met | Artifact: one HAR page per test and hook; only passing tests committed |
| AC3 | integration without Rancher | Partly met | No Rancher step; 13/18 legs pass (findings 1 and 2) |
| AC4 | a write replays the same body | Partly met | Preferred, not required (finding 3) |
| AC5 | a read recorded after that write | Met (reading) | priorWrite / lastWrites |
| AC6 | a socket frame once its write is made | Met (reading) | Gives up after 5s and sends anyway |
| AC7 | CI runs all three and charts timings | Met | e2e-timings job; the PR body table |
| AC8 | master keeps running against Rancher | Not met | Master would also run generate and integration (finding 4) |
| AC9 | @rancher/cypress consumers | Met | ws already a dependency; extendConfig doesn't call setupNodeEvents |
| AC10 | the manual test steps | Could not check | Cypress isn't run locally; "no no recording: lines" can't show exact replay |
setupNodeEvents with virtual-clusters-ui and rancher-ai-ui; e2e-timings.js has read-only permissions and doesn't write to the PR body (it ran and parsed all 54 jobs); scripts/e2e-recordings runs no shell; on the retry integration downloaded the newer recordings.| Kubewarden failure in full admin navigation (Pass B question) | Same before all failure on master, run 42312559879; master's last 6 test.yaml runs are red. Not this PR. |
| No unit tests for the matcher (Pass B suggestion) | True, and Jest ignores cypress/. Left out while it's an experiment whose matching changes every commit; worth raising once the design settles. |
stop() never called, any for parsed JSON, new env vars not documented (Pass B nits) | Look odd, mislead no one, and the last is premature for an experiment. |
scripts/e2e-recordings copies serialize (Pass A) | Five lines, commented as a copy. Taste. |
Nice experiment, and full mode looks untouched. I dug into the red integration jobs on the latest run, and a few of them come from the replay or the workflow rather than flaky tests.
.github/workflows/test.yaml:252RANCHER_VERSION like full and generate do, so the log says "Rancher version: not provided" and edit-fake-cluster.spec.ts expects a docs.rancher.com// link, which fails it in admin manager on both attempts. Generate also dropped the Prime check that full has, so the two would run different tests on a Prime image.cypress/support/e2e-modes/server.ts:963before all hook asks for something it didn't record, this borrows the next test's first recording, whatever the hook already did. user-retention's test clears its session first, so the hook's logged-in GET /v3/principals gets that test's pre-login 401, and the standard_user usersAndAuths job fails on both attempts. I got the same 401 replaying that job's recording through this server locally.cypress/support/e2e-modes/server.ts:903.github/workflows/test.yaml:203cypress/support/e2e-modes/server.ts:48R_SESS cookie, and cypress/recordings isn't in .gitignore. In CI that's only the throwaway password, but following the PR's test steps against a real Rancher leaves a real password and a live token in untracked files. Could those be left out and the folder ignored?cypress/support/e2e-modes/server.ts:371.github/workflows/test.yaml:205cypress/support/e2e-modes/recordings.ts:11HAR_FILE dev replay already reads this format, but har-file.js only keeps responses with content.text, so it dropped all 32 from user-retention.har. Did you mean it's HAR in general?cypress/support/e2e-modes/config.ts:60 (optional)