Issue/Question

While doing the API Conformance Scan, we are getting invalid response because it looks like the Scan is taking random input data and sending the request to provided endpoint URL, which will not work.

Where can we provide the test input request details with valid input test data to have Scan send the requests to provided endpoint?

Solution/Answer

The goal of the Conformance Scan is to test the resilience of your API and mimic the typical requests hackers would do (they would not follow the documentation to attack your API!)

Conformance Scan aims to find occurrences in which a call made outside of the contract is handled properly, as per the defined contract.

Conformance scans are done in two phases:

• Happy Path requests: A happy path request is a valid request generated directly from the OpenAPI definition of your API, designed and expected to always succeed. Conformance Scan generates and sends this request to each operation defined in your API, and validates the responses it received.
• Conformance Scan tests: Those are tests generated automatically from the OpenAPI contents and are using random data, wrong data types and data outside of established constraints.

Happy Path requests are a benchmark to determine if the API is working properly when invoked with proper data. For that phase, the engine will in priority consume valid data which is taken from the examples present in the OAS file. Otherwise it will generate data from the regular expressions, enums or just the type of the data.

See: https://docs.42crunch.com/latest/content/concepts/api_contract_conformance_scan.htm#scrollNav-3

For a happy path request to be a success, the response must be either successful or expected: the received HTTP status code must be 200—399, or 404 (because the likelihood that the scan manages to generate a value that matches an existing ID is vanishingly small). Otherwise, the happy path request fails. If a happy path request fails, the operation in question is skipped in the scan, because any results for it would be inconclusive without a successful benchmark.

For Conformance Scan tests, the requests and parameter values that Conformance Scan generates are random, as is the order in which the requests are sent to the API. That means that you cannot specify test input request details that scan will use against an API. The scan actually uses improper types as well (like string when you expect an integer).

For example, if your API returns a 200 (which means everything worked well) when invoked with wrong data, that is a scan error. It should really return a 400 to indicate to consumers that the data is incorrect. Additionally, the scan will detect that your response does not match (most likely) the schema for the response for code 200.

You can read more about it in the following link:
https://docs.42crunch.com/latest/content/concepts/api_contract_conformance_scan.htm