Anatomy of a test case

What a test case contains: identifier, purpose, the requirement it proves, preconditions with PICS and PIXIT, the test body, expected observations, pass criteria, verdicts and post-conditions, and how conformance, interoperability and lab test cases differ. With one example written out.

Why the structure matters

A test case is a small contract. Anyone who runs it must reach the same verdict on the same product, and anyone who reads the verdict must know which requirement it proves. The conformance methodology of ISO/IEC 9646 gets there in steps: requirements of the standard become test purposes, test purposes become test cases, and the test cases are grouped into a test suite. The ISO 15118 conformance parts follow this path; ISO 15118-4 writes its test cases in TTCN-3, and ISO 15118-21 and -23 describe theirs in tables.

The work item for ISO 15118-22 names the parts of a test case plainly: pre-conditions, test behaviour, expected results to decide pass or fail, and post-conditions that return the system under test to a safe state. The fields below follow that order.

The fields

FieldWhat it holdsTypical mistake
IdentifierA stable id: suite, side, topic, number (for example DC-07). Never reused.Renumbering, so old reports point to the wrong case
PurposeOne sentence: what behaviour the case provesTwo behaviours in one case, so a fail does not say which
RequirementThe clause or requirement id of the standard it traces toNo trace, so nobody can tell whether the case is still valid after a new edition
System under test and testerWhich side is tested (EVCC or SECC) and what plays the other sideTesting both real products and calling it conformance
PreconditionsThe claimed options (PICS), the parameters (PIXIT), the initial state, the equipmentAn initial state that depends on the previous case
Test bodyThe stimulus steps, in order, with the values the tester sendsSteps a person reads differently, such as "wait a while"
Expected observationsMessages, fields, timings and electrical values that must appearOnly the final state, not the path to it
Pass criteriaThe rule that turns observations into a verdict, with its limitsLimits in the text instead of in the parameter set
VerdictPass, fail or inconclusive, with the reasonFail for a case that could not run
Post-conditionsHow the product returns to a safe, known stateLeaving a DC output energised for the next case

PICS, PIXIT and verdicts

The PICS (protocol implementation conformance statement) lists the options a product claims: AC or DC, EIM or Plug and Charge, TLS, which protocols. It selects which test cases apply. The PIXIT (protocol implementation extra information for testing) gives the values the tester needs: addresses, timeouts the product uses, maximum voltage and current, certificates. It parameterises the selected cases. A case whose option the PICS does not claim is not run, so it cannot fail. More in Conformance testing: ISO 15118-4, -5 and Testivals.

VerdictMeaning
PassThe observations match the test purpose: the product did what the requirement asks
FailAn observation contradicts the requirement: a wrong message, a wrong value, a limit exceeded
InconclusiveNothing contradicts the requirement, but the case could not reach the point where it decides, for example because the product stopped for an unrelated reason

TTCN-3 also knows the values none (no verdict set yet) and error (a fault in the test system itself). The TTCN-3 core language (ETSI ES 201 873-1, ITU-T Z.140) orders the values none, pass, inconc, fail, error, and a verdict can only get worse during a run, never better.

Conformance, interoperability and a lab plan

Conformance test caseInteroperability test caseLab's own test plan
Other sideA test system that plays the peer and can misbehave on purposeA real productEither, often a simulator
SourceThe conformance part of the standardEvent or program plans (CharIN, ChargeX)Customer needs, field faults, regulations
FaultsInjected by the tester: late answers, wrong ids, bad certificatesMostly real use: plug, stop, unplug, pull the networkBoth, plus environment: heat, cold, grid
VerdictPass, fail, inconclusive against one requirementWorks or does not, per pairingAs the plan defines
BlameAlways the system under testOpen: either side, or the standardAs investigated
RepeatableYes, by designOnly as far as the two products areDepends on the plan

The ChargeX interoperability plan shows the second kind: each case has a purpose, a setup, a procedure and pass criteria, but the scenarios are those a driver meets, such as a partly inserted connector, a broken latch, or the internet dropping during power transfer.

One test case written out

FieldContent
IdentifierDC-07 (from DC test cases, worked)
PurposeThe charger raises its output to the voltage the car asks for in pre-charge, within the tolerance and before the timeout
RequirementIEC 61851-23 pre-charge; the PreCharge messages and timing of ISO 15118-2 (or DC_PreCharge of ISO 15118-20)
System under testSECC with its power stage (the charger)
TesterEV simulator with an EVCC stack, a DC load or battery emulator and a voltage probe at the inlet
PICSDC, ISO 15118-2, EIM; TLS optional
PIXITMaximum and minimum output voltage of the charger; pre-charge tolerance and timeout to judge against
PreconditionsSession set up to the end of CableCheck with isolation valid; battery emulator at 400 V behind open contactors
Test body1. Send PreChargeReq with EVTargetVoltage 400 V and EVTargetCurrent 2 A. 2. Repeat PreChargeReq until EVSEPresentVoltage is within tolerance of 400 V, or the timeout expires. 3. Close the EV contactors. 4. Send PowerDeliveryReq Start.
Expected observationsEvery PreChargeRes answers with ResponseCode OK within the message timeout; EVSEPresentVoltage rises steadily; the probe at the inlet agrees with EVSEPresentVoltage
Pass criteriaPresent voltage within tolerance of the target before the pre-charge timeout; no response later than the message timeout
FailTimeout reached with the voltage outside tolerance, a response code other than OK, or a late response
InconclusiveThe session ended before PreCharge for a reason outside the case, for example a CableCheck failure
Post-conditionsPowerDeliveryReq Stop, WeldingDetection, SessionStopReq; output below a safe voltage before the connector unlocks
Observed in this tool byDC-01 (pre-charge reaches the target), TIM-01 (response times), the Curve tab (voltage over time)

The tolerance and the timeout are parameters, not constants. The default limit profile uses 20 V and 7 s, both flagged verify: the first is common practice for IEC 61851-23 pre-charge, the second V2G_EVCC_PreCharge_Timeout from secondary sources.

Writing good test cases

  • One behaviour per case. A case named "session works" fails without saying why.
  • Every limit comes from the parameter set, so the same case runs against another edition or another product class.
  • Write the negative case next to the positive one: pre-charge that reaches the target, and pre-charge whose output stalls.
  • Record evidence: a capture of the powerline traffic, a Control Pilot log, and meter readings. A verdict without evidence cannot be reviewed.
  • End every case in a safe state, and check it: no voltage on the inlet, contactors open, connector unlocked.

In this tool

Each check of the analyzer works like the pass criteria of a test case: it names its reference, reads its limits from the limit profile, and ends in pass, fail or not applicable. Not applicable plays the part of a case that the PICS excludes. The demo precharge-fail shows the negative variant of the example above, and iso20 a passing DC session. See Limit profiles: the numbers behind every check for the parameters and Session end verdicts for how the checks add up.

Sources