[PR #4189] [MERGED] feat(cli): add support for JUnit reporter #4692

New issue

Closed

opened 2026-03-17 02:12:24 +03:00 by kerem · 0 comments

kerem commented

2026-03-17 02:12:24 +03:00

Owner

📋 Pull Request Information

Original PR: https://github.com/hoppscotch/hoppscotch/pull/4189
Author: @jamesgeorge007
Created: 7/16/2024
Status: ✅ Merged
Merged: 7/26/2024
Merged by: @jamesgeorge007

Base: next ← Head: feat/cli-junit-reporter

📝 Commits (5)

11d1ab9 feat: add support for JUnit reporter in the CLI
7e2cbff test: increase coverage
241b859 chore: bump CLI version
5c4db95 chore: update report export format
c75cca5 chore: cleanup

📊 Changes

15 files changed (+1197 additions, -64 deletions)

View changed files

📝 packages/hoppscotch-cli/package.json (+6 -3)
➕ packages/hoppscotch-cli/src/__tests__/e2e/commands/__snapshots__/test.spec.ts.snap (+529 -0)
📝 packages/hoppscotch-cli/src/__tests__/e2e/commands/test.spec.ts (+163 -2)
➕ packages/hoppscotch-cli/src/__tests__/e2e/fixtures/collections/test-junit-report-export-coll.json (+150 -0)
📝 packages/hoppscotch-cli/src/commands/test.ts (+2 -2)
📝 packages/hoppscotch-cli/src/handlers/error.ts (+4 -0)
📝 packages/hoppscotch-cli/src/index.ts (+16 -1)
📝 packages/hoppscotch-cli/src/types/commands.ts (+1 -0)
📝 packages/hoppscotch-cli/src/types/errors.ts (+1 -0)
📝 packages/hoppscotch-cli/src/utils/collections.ts (+76 -37)
📝 packages/hoppscotch-cli/src/utils/pre-request.ts (+4 -1)
➕ packages/hoppscotch-cli/src/utils/reporters/junit.ts (+178 -0)
📝 packages/hoppscotch-cli/src/utils/request.ts (+7 -6)
📝 packages/hoppscotch-cli/tsconfig.json (+2 -1)
📝 pnpm-lock.yaml (+58 -11)

📄 Description

Closes HFE-536, #4085.

Description

This PR adds the ability to create JUnit report export for collection runs in the CLI via a newly added --reporter-junit [path] flag. The report gets generated under the name hopp-junit-report.xml in the current path if the path is not specified.

hopp test <file_path_or_id> --env <file_path_or_id> --reporter-junit [path]

Report format

Consider a collection with two requests:

`req1`

Sends a post request to echo.hoppscotch.io.
Content type set to application/json.

Raw body specified as

 {
  "key": "raw-body-value"
 }

// Test script
pw.test("Status code is 200", () => {
  const { status } = pw.response;

  pw.expect(status).toBe(200);
  pw.expect(status).toBeLevel2xx();
});

pw.test("Check JSON response properties", () => {
  const { method, args, data } = pw.response.body

  pw.expect(method).toBe("POST");
  pw.expect(args.key).toBe("query-param-value");

  // Failure: requires `JSON.parse` to be applied over `data`
  pw.expect(data.key).toBe("raw-body-value");
});

pw.test("Check headers", () => {
  const { headers } = pw.response.body;

  pw.expect(headers["custom-header"]).toBe("custom-header-value");

  // Error: `toHaveLength()` reports an error for values other than string or array
  pw.expect(headers["non-existent-header"]).toHaveLength(10);
});

`req2`

Invalid endpoint.

Reference error within the test script.

// Test script
pw.test("Status code is 200", ()=> {
    // `TEST_SCRIPT_ERROR` Script execution failed: Should be `pw.response.body`
    pw.expect(pw.body.status).toBe(200);
});

The JUnit XML report contents generated are:

<?xml version="1.0" encoding="UTF-8"?>
<testsuites tests="7" failures="1" errors="1" time="0.003">
  <testsuite name="coll/req1" time="0.003" timestamp="2024-07-22T14:24:03.826Z" tests="7" failures="1" errors="1">
    <testcase name="Status code is 200 - Expected '200' to be '200'" classname="coll/req1"/>
    <testcase name="Status code is 200 - Expected '200' to be 200-level status" classname="coll/req1"/>
    <testcase name="Check JSON response properties - Expected 'POST' to be 'POST'" classname="coll/req1"/>
    <testcase name="Check JSON response properties - Expected 'query-param-value' to be 'query-param-value'" classname="coll/req1"/>
    <testcase name="Check JSON response properties - Expected 'undefined' to be 'raw-body-value'" classname="coll/req1">
      <failure type="AssertionFailure" message="Expected 'undefined' to be 'raw-body-value'"/>
    </testcase>
    <testcase name="Check headers - Expected 'custom-header-value' to be 'custom-header-value'" classname="coll/req1"/>
    <testcase name="Check headers - Expected toHaveLength to be called for an array or string" classname="coll/req1">
      <error message="Expected toHaveLength to be called for an array or string"/>
    </testcase>
  </testsuite>
  <testsuite name="coll/req2" time="0" timestamp="2024-07-22T14:24:03.829Z" tests="0" failures="0" errors="0">
    <system-err><![CDATA[
      REQUEST_ERROR - Error: getaddrinfo ENOTFOUND invalid-url
      TEST_SCRIPT_ERROR - Script execution failed: TypeError: Cannot read properties of undefined (reading 'status')]]></system-err>
  </testsuite>
</testsuites>

Test suites are kept at each request level with the naming convention <parent-collection-name</<child-collection-name>/<request-name>. The name combines all the collection names till the request.
Each request can have multiple test suites corresponding to the pw.test() scripting API method with nested test cases. In JUnit reports, nested test suites are flattened, so the test cases corresponding to pw.expect() invocations appear as direct children to the request-level test suite, with names prefixed with the respective test suite description. In the generated report, req1 has three test suites, each containing different test cases. These test cases are all represented through the testcase elements, with the test suite descriptions serving as prefixes.
Assertion failures are indicated by the failure element nested under the testcase element, while errors during assertions via the error element.
The classname attribute at each testcase level is set to the same value as the name attribute at the testsuite level.
For errors reported at the request level - invalid URL, reference error in the test script, etc are represented by the system-err element with the specifics under CDATA. It is to be noted that the report follows a convention where errors reported at the request level are compiled in a single CDATA element separated by newlines.
The time taken to execute the test cases (exclusive of the request execution time) is set at each request-level test suite element via the time attribute and the total time is recorded at the top-level test suite.
There is a timestamp attribute set at all request-level test suite elements indicating the date in ISO string format.
The no. of test cases, as well as failed and errored cases, are kept track via the tests, failures and errors attributes respectively at all the request-level test suite elements and the effective count at the root test suite. When errors are reported at the request level, all the above attributes will be set to 0 since the test suite halts and no further test cases get executed.

Note to reviewers

We evaluated both junit-report-builder and xmlbuilder2 libraries and decided to go ahead with the latter, since it offers more fine-grained control, being a generic XML document builder catering to our requirements.

To verify the behaviour:

Run pnpm i at the project root (CLI now includes a postinstall script that runs the build script) and navigate to the CLI package path.
Ensure there is a valid Hoppscotch collection export available in path. Also, try with the collection linked above and those under e2e test fixtures /src/__tests__/e2e/fixtures/collections/.....
Generate a JUnit report with the following command:
```
bin/hopp.js <file_path> --reporter-junit
```

The report gets generated under the name hopp-junit-report.xml in the current path. Now, specify a different path as an argument to the --reporter-junit flag.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/hoppscotch/hoppscotch/pull/4189 **Author:** [@jamesgeorge007](https://github.com/jamesgeorge007) **Created:** 7/16/2024 **Status:** ✅ Merged **Merged:** 7/26/2024 **Merged by:** [@jamesgeorge007](https://github.com/jamesgeorge007) **Base:** `next` ← **Head:** `feat/cli-junit-reporter` --- ### 📝 Commits (5) - [`11d1ab9`](https://github.com/hoppscotch/hoppscotch/commit/11d1ab929ae076c197e6cd0dae70b2bc0ee3a656) feat: add support for JUnit reporter in the CLI - [`7e2cbff`](https://github.com/hoppscotch/hoppscotch/commit/7e2cbff721e09314e738e030937be6025b3e66b6) test: increase coverage - [`241b859`](https://github.com/hoppscotch/hoppscotch/commit/241b8590d71ef185045adad6827adb6c83759475) chore: bump CLI version - [`5c4db95`](https://github.com/hoppscotch/hoppscotch/commit/5c4db959b5ac8157f7bef5bafa640cf583d51f05) chore: update report export format - [`c75cca5`](https://github.com/hoppscotch/hoppscotch/commit/c75cca5ce23f263e98588b0fe869cc35a518a2e3) chore: cleanup ### 📊 Changes **15 files changed** (+1197 additions, -64 deletions) <details> <summary>View changed files</summary> 📝 `packages/hoppscotch-cli/package.json` (+6 -3) ➕ `packages/hoppscotch-cli/src/__tests__/e2e/commands/__snapshots__/test.spec.ts.snap` (+529 -0) 📝 `packages/hoppscotch-cli/src/__tests__/e2e/commands/test.spec.ts` (+163 -2) ➕ `packages/hoppscotch-cli/src/__tests__/e2e/fixtures/collections/test-junit-report-export-coll.json` (+150 -0) 📝 `packages/hoppscotch-cli/src/commands/test.ts` (+2 -2) 📝 `packages/hoppscotch-cli/src/handlers/error.ts` (+4 -0) 📝 `packages/hoppscotch-cli/src/index.ts` (+16 -1) 📝 `packages/hoppscotch-cli/src/types/commands.ts` (+1 -0) 📝 `packages/hoppscotch-cli/src/types/errors.ts` (+1 -0) 📝 `packages/hoppscotch-cli/src/utils/collections.ts` (+76 -37) 📝 `packages/hoppscotch-cli/src/utils/pre-request.ts` (+4 -1) ➕ `packages/hoppscotch-cli/src/utils/reporters/junit.ts` (+178 -0) 📝 `packages/hoppscotch-cli/src/utils/request.ts` (+7 -6) 📝 `packages/hoppscotch-cli/tsconfig.json` (+2 -1) 📝 `pnpm-lock.yaml` (+58 -11) </details> ### 📄 Description Closes HFE-536, #4085. ### Description This PR adds the ability to create JUnit report export for collection runs in the CLI via a newly added `--reporter-junit [path]` flag. The report gets generated under the name `hopp-junit-report.xml` in the current path if the path is not specified. ```sh hopp test <file_path_or_id> --env <file_path_or_id> --reporter-junit [path] ``` ### Report format Consider [a collection](https://gist.github.com/jamesgeorge007/90c90ff4b8a49a756e68cc32638722e7) with two requests: #### `req1` - Sends a `post` request to `echo.hoppscotch.io`. - Content type set to `application/json`. - Raw body specified as ```json { "key": "raw-body-value" } ``` ```js // Test script pw.test("Status code is 200", () => { const { status } = pw.response; pw.expect(status).toBe(200); pw.expect(status).toBeLevel2xx(); }); pw.test("Check JSON response properties", () => { const { method, args, data } = pw.response.body pw.expect(method).toBe("POST"); pw.expect(args.key).toBe("query-param-value"); // Failure: requires `JSON.parse` to be applied over `data` pw.expect(data.key).toBe("raw-body-value"); }); pw.test("Check headers", () => { const { headers } = pw.response.body; pw.expect(headers["custom-header"]).toBe("custom-header-value"); // Error: `toHaveLength()` reports an error for values other than string or array pw.expect(headers["non-existent-header"]).toHaveLength(10); }); ``` #### `req2` - Invalid endpoint. - Reference error within the test script. ```js // Test script pw.test("Status code is 200", ()=> { // `TEST_SCRIPT_ERROR` Script execution failed: Should be `pw.response.body` pw.expect(pw.body.status).toBe(200); }); ``` The JUnit XML report contents generated are: ```xml <?xml version="1.0" encoding="UTF-8"?> <testsuites tests="7" failures="1" errors="1" time="0.003"> <testsuite name="coll/req1" time="0.003" timestamp="2024-07-22T14:24:03.826Z" tests="7" failures="1" errors="1"> <testcase name="Status code is 200 - Expected '200' to be '200'" classname="coll/req1"/> <testcase name="Status code is 200 - Expected '200' to be 200-level status" classname="coll/req1"/> <testcase name="Check JSON response properties - Expected 'POST' to be 'POST'" classname="coll/req1"/> <testcase name="Check JSON response properties - Expected 'query-param-value' to be 'query-param-value'" classname="coll/req1"/> <testcase name="Check JSON response properties - Expected 'undefined' to be 'raw-body-value'" classname="coll/req1"> <failure type="AssertionFailure" message="Expected 'undefined' to be 'raw-body-value'"/> </testcase> <testcase name="Check headers - Expected 'custom-header-value' to be 'custom-header-value'" classname="coll/req1"/> <testcase name="Check headers - Expected toHaveLength to be called for an array or string" classname="coll/req1"> <error message="Expected toHaveLength to be called for an array or string"/> </testcase> </testsuite> <testsuite name="coll/req2" time="0" timestamp="2024-07-22T14:24:03.829Z" tests="0" failures="0" errors="0"> <system-err><![CDATA[ REQUEST_ERROR - Error: getaddrinfo ENOTFOUND invalid-url TEST_SCRIPT_ERROR - Script execution failed: TypeError: Cannot read properties of undefined (reading 'status')]]></system-err> </testsuite> </testsuites> ``` - Test suites are kept at each request level with the naming convention `<parent-collection-name</<child-collection-name>/<request-name>`. The name combines all the collection names till the request. - Each request can have multiple test suites corresponding to the `pw.test()` scripting API method with nested test cases. In JUnit reports, nested test suites are flattened, so the test cases corresponding to `pw.expect()` invocations appear as direct children to the request-level test suite, with names prefixed with the respective test suite description. In the generated report, `req1` has three test suites, each containing different test cases. These test cases are all represented through the `testcase` elements, with the test suite descriptions serving as prefixes. - Assertion failures are indicated by the `failure` element nested under the `testcase` element, while errors during assertions via the `error` element. - The `classname` attribute at each `testcase` level is set to the same value as the `name` attribute at the `testsuite` level. - For errors reported at the request level - invalid URL, reference error in the test script, etc are represented by the `system-err` element with the specifics under `CDATA`. It is to be noted that the report follows a convention where errors reported at the request level are compiled in a single `CDATA` element separated by newlines. - The time taken to execute the test cases (exclusive of the request execution time) is set at each request-level test suite element via the `time` attribute and the total time is recorded at the top-level test suite. - There is a `timestamp` attribute set at all request-level test suite elements indicating the date in ISO string format. - The no. of test cases, as well as failed and errored cases, are kept track via the `tests`, `failures` and `errors` attributes respectively at all the request-level test suite elements and the effective count at the root test suite. When errors are reported at the request level, all the above attributes will be set to `0` since the test suite halts and no further test cases get executed. ### Note to reviewers We evaluated both [junit-report-builder](https://github.com/davidparsson/junit-report-builder) and [xmlbuilder2](https://github.com/oozcitak/xmlbuilder2) libraries and decided to go ahead with the latter, since it offers more fine-grained control, being a generic XML document builder catering to our requirements. To verify the behaviour: - Run `pnpm i` at the project root (CLI now includes a `postinstall` script that runs the `build` script) and navigate to the CLI package path. - Ensure there is a valid Hoppscotch collection export available in path. Also, try with the collection linked above and those under e2e test fixtures `/src/__tests__/e2e/fixtures/collections/....`. - Generate a JUnit report with the following command: ```sh bin/hopp.js <file_path> --reporter-junit ``` > The report gets generated under the name `hopp-junit-report.xml` in the current path. Now, specify a different path as an argument to the `--reporter-junit` flag. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>