OpenAPI Specification Test Suite

This repository contains a set of JSON objects that implementers of OpenAPI Specification validation libraries can use to test their validators. The test suite repository exists to verify specified behavior defined by the OpenAPI Specification and should not be confused with a style guide. It is not intended to demonstrate how specifications ought to be written. Tests may appear unusual or unintuitive, but they exist solely to exercise behavior prescribed by the specification.

It is meant to be language agnostic and should require only a JSON parser. The conversion of the JSON objects into tests within a specific language and test framework of choice is left to be done by the validator implementer.

The recommended workflow of this test suite is to clone the main branch of this repository as a git submodule or git subtree. The main branch is always stable.

Coverage

All OpenAPI Specification releases should be well covered by this suite, including versions 3.0, 3.1, and 3.2.

Additional coverage is always welcome, particularly for bugs encountered in real-world implementations. If you see anything missing or incorrect, please feel free to file an issue or submit a PR.

Introduction to the Test Suite Structure

The tests in this suite are contained in the tests directory at the root of this repository. Inside that directory is a subdirectory for each released version of the specification.

In addition to the version-specific subdirectories, one additional entry is present:

1. latest/: a symbolic link which points to the directory for the most recent release (which may be useful for implementations providing specific entry points for validating against the latest version of the specification)

Inside each version directory there are a number of .json files each containing a collection of related tests. The grouping is generally by the specification object under test (e.g., info.json contains tests for the Info Object).

Each .json file consists of a single JSON array of test cases.

Terminology

For clarity, we first define this document's usage of some testing terminology:

term	definition
test suite	the entirety of the contents of this repository, containing tests for multiple different releases of the OpenAPI Specification
test case	a grouping of related tests sharing a common description, each containing an OpenAPI document and an expected validation result
test	within a test case, a single example containing a description, an OpenAPI document, and a boolean indicating whether the document is valid under the specification
test runner	a program, external to this repository and authored by a user of this suite, which executes each of the tests in the suite

An example illustrating this structure is immediately below, and a JSON Schema containing a formal definition of the contents of test case files can be found alongside this README.

Sample Test Case

Here is a single test case, containing one or more tests:

{
    "description": "The test case description",
    "tests": [
        {
            "description": "a valid OpenAPI document",
            "document": {
                "openapi": "3.1.0",
                "info": { "title": "My API", "version": "1.0.0" },
                "paths": {}
            },
            "valid": true
        },
        {
            "description": "an OpenAPI document missing a required field",
            "document": {
                "openapi": "3.1.0",
                "paths": {}
            },
            "valid": false
        }
    ]
}

Subdirectories Within Each Version Directory

A specification version test directory may contain one or more subdirectories:

1. optional/: Contains tests that are considered optional. These may cover behaviors that are not strictly required by the specification but are potentially useful to an implementer, or behaviors that only apply to certain implementation environments.

Using the Suite to Test a Validator Implementation

The test suite structure was described above.

If you are authoring a new validator implementation, or adding support for an additional version of the specification, this section describes how to implement a test runner which passes tests to your validator.

How to Implement a Test Runner

Presented here is a possible implementation of a test runner. The precise steps described do not need to be followed exactly, but the results of your own procedure should produce the same effects.

To test a specific version:

• Configure your implementation to validate against the OpenAPI Specification version matching the test directory name (e.g., 3.1 for the tests/3.1/ directory)

• Walk the filesystem tree for that version's subdirectory and for each .json file found:

  • for each test case present in the file:

    • load (or log) the test case description from the "description" property for debugging or output

    • for each test in the "tests" property:

      • load the OpenAPI document from the "document" property
      • load (or log) the individual test description from the "description" property for debugging or output
      • use your validator to determine whether the document is valid according to the specification
      • if the result from your implementation matches the value found in the "valid" property, your implementation correctly implements the specific behavior
      • if the result does not match, or your implementation errors or crashes, your implementation does not correctly implement the specific behavior

If your implementation supports multiple versions, run the above procedure for each version supported.

Invariants

The test suite makes the following claims about its tests:

1. Each "document" value is a complete OpenAPI document (not a fragment), unless explicitly noted otherwise in the test's "comment" field.
2. The "valid" value is authoritative — it reflects the behavior prescribed by the specification, not what any particular implementation does.
3. Each .json file validates against the test-schema.json meta-schema.

Contributing

Contributions are welcome! When adding new tests, please ensure:

1. Each test targets a specific, well-defined behavior from the specification.
2. The "description" fields are clear and concise.
3. Include a "comment" field when the test exercises non-obvious behavior.
4. Tests files validate against test-schema.json.
5. Valid and invalid examples are both represented for each behavior being tested.