Running test cases

Judge a capture against a test case: pass, fail or inconclusive, per file of a batch or from the command line, with JUnit for a test bench.

What a test case checks

The worked test cases on the Learn shelf on testing say what each run has to show. Those a capture can decide are built into the analyzer as expectations on its own analysis: the verdict (outcome, category, likely party), checks that have to pass, checks that must not fail, checks that have to fail because the test injects that fault, and the state each stage has to end in.

ResultWhen
passEvery expectation holds
failAt least one does not: a check failed, a check the test expects to fail passed, the verdict or a stage is not the one expected
inconclusiveNothing failed, but the capture cannot decide: a check that has to pass did not apply (no Control Pilot log, no TLS), or a stage is not in the capture
  • A test records one run. When a file holds several sessions, each has to meet the expectations: one failing session fails the test, so a retry that worked does not hide the attempt that did not. Sessions the test cannot judge count only when none can be judged.
  • A negative test, such as DC-06 with an insulation fault, passes when the capture shows the fault and the reaction the standard asks for. The same capture fails the positive test DC-05.
  • A pass says what the capture shows against the expectations. It is not a certification result: the test body, the setup and the equipment are yours to document.

In batch mode

  • Name each capture after its test case, the way a test bench numbers its records: DC-06_insulation_fault.pcapng, AC-03 run 2.pcapng. The id has to stand on its own, so DC-061 is not DC-06.
  • On the Batch page, set Test cases to From the file names, or pick one test case for every file, then drop the folder.
  • The table gets a column with the result of each file; open a result to read why. Export JUnit writes one test per file and test case, a failure for fail and skipped for inconclusive. Export test results writes the same as CSV.

In the analyzer

  • Open the capture and the Test cases tab beside the findings. It takes the test case from the file name, or pick one; the result covers every session of the file.
  • The report you export then carries a Test cases section with each result and why, in the full report and the one page summary.
  • Write a test plan opens an editor: checks by id, the outcome, the category, the likely party and the stages, checked as you type. Use this plan keeps it for the tab and for batch mode; Download plan saves it as JSON for the command line.

From the command line

node dist-cli/v2g-analyze.mjs DC-06_fault.pcapng --testcase auto --junit dc-06.xml
node dist-cli/v2g-analyze.mjs run.pcapng cp.csv --testcase AC-03,BE-02
node dist-cli/v2g-analyze.mjs run.pcapng --testplan lab-plan.json
  • --testcase takes ids separated by commas, or auto for the ids in the file name.
  • --testplan reads test cases of your own; without --testcase it runs all of them.
  • With test cases, they decide the exit code: 0 when every one passes, 1 when one fails or is inconclusive, 2 when an id is unknown or the plan cannot be read. The JUnit file gets a Test cases suite in front of the per check suites, --json a testCases list and --html a Test cases section.

A test plan of your own

A test plan is a JSON file with a list of test cases, or an object with a testCases list. A test case of the plan with the id of a built-in one replaces it. Checks are named by id, including the custom checks of the active limit profile; the plan is refused when it names a check the analyzer does not have, repeats an id, or expects nothing.

{
  "testCases": [
    {
      "id": "LAB-12",
      "title": "Cable check with 90 kOhm insulation fault",
      "reference": "IEC 61851-23",
      "expect": {
        "outcome": ["Never charged"],
        "fail": ["SQ-05"],
        "noFail": ["TIM-01"],
        "stages": [
          { "name": "Cable check", "status": ["failed"] },
          { "name": "Energy transfer", "status": ["not-reached"] }
        ]
      }
    }
  ]
}
Field of expectMeaning
outcomeThe verdict outcome is one of these: Completed, Stopped early, Never charged, Incomplete capture
categoryThe verdict category is one of these, as the verdict names it
likelyPartyThe likely party is one of these: EV, EVSE, Either, Unknown
passThese checks run and pass; one that does not apply makes the test inconclusive
noFailThese checks do not fail; one that does not apply is fine
failThese checks run and fail
stagesEach stage, by the name the stage strip shows, ends in one of the states: passed, failed, not-reached, not-in-capture

Built-in test cases

Ids match the worked test cases on the Learn shelf; a letter marks a variant that injects a fault. The test cases that need equipment beyond a capture (a firmware update, a payment terminal) stay on the Learn pages; the OCPP boot (BE-01) needs an OCPP log that starts with the BootNotification.

IdTest caseExpectsPasses on the demo
AC-01Basic charging with PWM onlypass CP-01, CP-02, CP-03, PW-01, PW-02; no failure of CP-10, PF-01, PF-02, PF-03ac-ok
AC-025 percent start for HLCpass CP-03, CP-04, SL-01; no failure of SEC-01, SEC-02, SEC-04ac-ok
AC-03ISO 15118-2 AC session with EIMoutcome Completed; pass SQ-01, SQ-02, SQ-03, SQ-04, SQ-05, SQ-06, SQ-07, TIM-01, TIM-04, VT-01, VT-02; no failure of CP-06, CP-07, TIM-03, TIM-06; Energy transfer passedac-ok
AC-04Limit change by PWMpass PW-01; no failure of CP-03
AC-05Limit change by schedulepass SQ-08, SQ-09; no failure of PW-01renegotiation
AC-06Cable rating by PPpass PP-01, PW-01; no failure of PF-04ac-ok
AC-07Three phase and single phasepass PW-01; no failure of IS20-10
AC-08Stop by the carpass PW-03, CP-08; Stop passedac-ok
AC-09Stop by the chargerpass PW-03; no failure of SQ-09
AC-10Pause and resumepass SQ-11; no failure of PW-03pause-resume
AC-11Control Pilot short and charger faultpass CP-02, CP-09; no failure of PW-03
AC-12No SLAC after 5 percentpass CP-05; no failure of SL-07
AC-13Loss of communication while chargingoutcome Stopped early; pass PW-03; no failure of TIM-03
AC-14Ventilation request (state D)pass CP-02
AC-15Reactive power setpointpass IS20-11; no failure of IS20-10
DC-01SLAC under attenuationpass SL-01, SL-02, SL-03, SL-04, SL-05, SL-06, SL-07, SL-08; no failure of SL-09iso20
DC-02SECC discoverypass SD-01, SD-02, SD-03, SD-04, SD-05, SD-06, TC-01; no failure of TC-03iso20
DC-03TLS handshakepass TL-01, TL-02, TL-03, TL-04, PK-01; no failure of TL-05, TL-06, TL-07, TL-08, PK-02, PK-04, SD-04, SEC-03tls-ok
DC-03bTLS with an untrusted charger certificate is refusedoutcome Never charged; category TLS failure; fail TL-01; TCP / TLS failedtls-fail
DC-04Protocol handshakepass HS-01, HS-02; no failure of HS-03ac-ok
DC-05Cable check, normalpass TIM-02, SQ-05, SQ-01; Cable check passediso20
DC-06Cable check with insulation faultoutcome Never charged; fail SQ-05; Cable check failed; Energy transfer not-reachedcablecheck-fail
DC-07Pre-charge within tolerancepass DC-01, TIM-01; Pre-charge passediso20
DC-08Pre-charge timeoutoutcome Never charged; fail DC-01; Pre-charge failed; Energy transfer not-reachedprecharge-fail
DC-09Current demand followingpass TIM-01, TIM-05, PW-05; no failure of PW-01, PW-07, IS20-02, IS20-04; Energy transfer passediso20
DC-10Charger power limitationno failure of PW-01, IS20-03iso20
DC-11Renegotiationpass IS20-09, SQ-01; no failure of SQ-09
DC-12Bidirectional discharge limitspass IS20-01, IS20-02, IS20-03, IS20-04, IS20-05, IS20-07, IS20-08
DC-13Stop by the caroutcome Completed; pass SQ-01, TC-02; no failure of PW-03; Stop passediso20
DC-14Emergency stop or stop at the chargercategory EVSE stopped; no failure of SQ-09, PW-03emergency-stop
DC-15Welding detectionpass DC-02, SQ-01; Stop passedemergency-stop
DC-16Pause and resumeno failure of SQ-11, IS20-06pause-resume
DC-17CHAdeMO basic sessionpass CH-01, CH-02, CH-03, CH-04, CH-05, CH-06, CAN-01, CAN-02chademo
DC-17bCHAdeMO charger faultoutcome Stopped early; likely party EVSE; fail CH-04chademo-stop
DC-19Limits of both sides heldpass LM-01, LM-02, LM-03, LM-04, LM-05din-ok
DC-18GB/T 27930 basic sessionpass GB-01, GB-02, GB-03, GB-04, GB-05, GB-06, GB-07, GB-08, CAN-01gbt
DC-18bGB/T 27930 stop by a BMS faultoutcome Stopped early; category Vehicle fault; likely party EVgbt-bst
BE-01Bootpass OC-06, OC-01; no failure of OC-02, OC-04
BE-02Authorization acceptedpass TIM-02, SQ-05; Setup passedac-ok
BE-03Authorization rejectedoutcome Never charged; category Authorization not granted; pass XL-04ocpp-rejected
BE-05Transaction eventsoutcome Completed; pass PW-06, OC-01, OC-02, XL-01, XL-02, XL-05, CDR-01, CDR-02, CDR-03; no failure of OC-03, OC-05, OC-07, OC-08, OC-09, CDR-04ac-ok
BE-06Meter valuespass PW-04, PW-06, OC-01; no failure of SQ-10, XL-03, SM-01, SM-02, SM-03ac-ok
BE-10Charging profileno failure of PW-01, SQ-08ac-ok
BE-11Contract certificate installationpass TIM-01, SQ-05
BE-12Plug and Charge authorizationpass SQ-05, TIM-02, TL-01, PK-03, PK-05, XL-06; Setup passed
BE-13Expired certificatesoutcome Never charged; fail PK-01pnc-expired