Building a Regression Pack the Team Will Actually Defend

Delivery Already Ran in Jira

Every squad worked in its own Jira project. Stories carried the acceptance criteria, Fix Versions carried the release, and defects were raised against the work item that caused them. The delivery half of the process had a single source of truth and had done for years.

The Test Half Lived Somewhere Else

Test cases sat in a shared spreadsheet, organised by feature area. Each release, a tester copied the relevant tabs into a new sheet, marked results in a colour column, and pasted a summary into the release ticket.

The spreadsheet was not the problem on its own. The problem was that nothing connected a row in it to the story that justified the row’s existence. When a story changed, the case did not know.

What That Cost at Release Time

• Scope was argued about rather than derived, usually in the release call.
• Cases written against superseded requirements kept passing, which is worse than failing.
• Results were a colour in a cell, so the release decision rested on one person’s reading of the sheet.
• A hotfix had no regression scope at all — there was no time to build one.

Scope Was Chosen From Memory

The regression pack for a release was assembled by asking the squads what they had touched. The answer was accurate for the work people remembered and silent about everything else — a shared library bumped a version, a settlement rule was edited in passing, a feature flag flipped.

That produced two failure modes in turn. A pack wide enough to be safe could not be finished in the window. A pack narrow enough to finish was not one anybody would sign.

Results Did Not Point Back at the Release

Because the sheet did not know about Fix Versions, there was no way to ask the only question that matters at a release gate: has everything in this release been covered, and what is the state of it?

Requirement Changes Were Invisible to the Cases

A story edited after its cases were written left no trace in the test material. The team found the mismatch during execution, when a tester read a step that no longer matched the product — or did not find it at all.

A Second Tool Was Not an Option

The team had looked at standalone test management platforms before. Each one asked them to maintain a second copy of the project structure, keep it synchronised, and administer a separate set of users. The spreadsheet had already shown what happens to a system that sits outside the place people work.

Halden Pay chose Sherlock. because it is a Jira app, not a platform with a Jira connector. The requirement list was short and all of it was about location:

• Test cases stored as Jira work items, in the project that owns the feature.
• A case linked to the story it came from, so a change to one is visible from the other.
• Release scope derived from the Fix Version the team already maintains.
• Atlassian user management and billing — no separate vendor relationship, no extra accounts.
• Nothing to synchronise, so no question about which system is right.

The AI generation mattered to them second, not first. The team’s position was that a tool which writes cases into the wrong place is worse than one that writes none.

Setup and Import

Installation took an afternoon. The existing spreadsheet was imported feature area by feature area, starting with settlement — the area with the most expensive failure mode — rather than all at once.

No integration work, no data-mapping project and no dedicated administrator were involved. The two testers did the configuration themselves over the first week.

A Baseline per Release

Each release line now has an approved baseline: the set of cases that represent the product as it was signed off. The next release is compared against it, so the pack for a release is the difference between the two, not a judgement call.

Cases Linked to Requirements

Every imported case was attached to the story or epic it verifies. Where no such story existed, the gap was itself the finding — the team located a set of settlement checks that no longer matched any current requirement.

Review Before a Run

When a requirement changes, Sherlock. marks the cases that drew on it and proposes an updated version. A tester approves, edits or rejects each one. Nothing is rewritten without a person agreeing to it — a condition the team set before the trial began.

One Report at the Gate

The release report is generated from the run, not assembled by hand: scope, coverage against the Fix Version, pass and fail state, and the defects raised. It is the same artefact the reviewer and the squads read.

A Scope That Can Be Defended

The regression pack is now derived from the release rather than recalled. When someone asks why a case is in or out, the answer is a link, not a recollection.

Changes Surface Before Execution, Not During

Cases affected by an edited requirement are flagged while there is still time to act on them. The team reviews the proposed updates as part of release preparation instead of discovering mismatches mid-run.

Hotfixes Get a Pack Too

Because scope is derived rather than built, a hotfix now carries a regression pack proportional to what it changed — something the old process could not produce inside the time available.

Testing Is Visible to the Squads

Developers see the cases attached to their stories without opening another tool, and raise defects against the run from the same screen. Test state stopped being information that only the two testers held.

One Place, One Record

Requirements, cases, runs, defects and the release report are work items in the same Jira projects. Nothing is exported, nothing is reconciled, and there is no second system to keep honest.

• Import the riskiest area first. Starting with settlement meant the first week of use was spent on the cases that actually carry consequence.
• Treat an unlinkable case as a finding. Cases with no current requirement behind them are not an import problem — they are coverage that stopped meaning anything.
• Keep approval human. Proposed updates are reviewed before they land. The team was explicit that this is what made generated cases usable at all.
• Put it where the work already is. The spreadsheet did not fail because it was a spreadsheet. It failed because it sat outside Jira, and so did every answer it held.

Halden Pay already had a disciplined delivery process in Jira. What it lacked was a test process with the same properties: a single location, a traceable record, and a scope anyone could reconstruct.

With Sherlock. the regression pack is derived from the release, cases stay attached to the requirements that justify them, and the release decision rests on a report generated from the run. The testing half of the process now lives where the delivery half always did.