Repository navigation
Use OpenAPI to define MDS API request shapes and query parameters #281
Description
The existing documentation of query parameters and headers is in narrative format in the Provider and Agency README files. This is useful to give an overview of what's allowed, but because it is not programmatically enforced and left up to interpretation, it's common to see divergences from the spec. [1]
I'd like to propose adding API endpoint parameter definitions to Provider (and maybe Agency?) endpoints using a format designed to support automatic validation.
Implementers of MDS endpoints and software that interacts with them would be able to use these definitions to validate the shape of query parameters and headers for incoming requests. Additionally, a rigorous definition of the request shape would let MDS evolve to support more advanced request formats in a safe and transparent way.
A framework frequently used for this kind of schema is OpenAPI v3, an evolution of Swagger API definition formats. It has a lot of OSS and third party tools for API documentation and validation that may come in useful, and is reasonably well documented. Based on prior experience, I'd recommend using this format. [2]
[1] A similar thing to this already exists for validating the shape of responses via JSON Schema, and has been extremely useful for deciding if an API response conforms to the specification. However, JSON Schema is a toolkit for asserting the shape of JSON objects and does not really support defining the shape of request parameters and headers.
[2] I also considered suggesting JSON Hyper Schema (narrative intro), an extension to JSON Schema that builds a vocabulary for defining URI shapes. However, it is very young and does not have a really mature ecosystem of tools around it yet.
Activity
[-]Use OpenAPI to define Provider API request shapes and query parameters[/-][+][0.4.x] Use OpenAPI to define Provider API request shapes and query parameters[/+][-][0.4.x] Use OpenAPI to define Provider API request shapes and query parameters[/-][+]Use OpenAPI to define Provider API request shapes and query parameters[/+]dyakovlev commentedon May 23, 2019
notes from 5/23 call
some good thoughts about tool applications in the readme here - https://github.com/wework/json-schema-to-openapi-schema
also, having OpenAPI defs allows use of all of these tools which is pretty neat
craastad commentedon May 23, 2019
I'm all for this, especially the choice of using the OpenAPI or JsonSchema tooling. I found the PHP tooling with JsonSchema to have issues with performance and completeness (i.e. adding massive overhead to requests to build responses and not not rendering nulls in paging params). I find the tooling to be better with OpenAPI. Please reach out if you'd like help with implementing or validating this change.
hunterowens commentedon Jul 10, 2019
Any progress on this @dyakovlev @craastad ?
dyakovlev commentedon Jul 11, 2019
not from my side, sorry
marie-x commentedon Aug 8, 2019
Agree that OpenAPI is the way to go, for all of MDS (Provider, Agency, Policy, Audit ...), not just Provider, and agree that we should target this for 0.4.0.
craastad commentedon Aug 22, 2019
I'm happy to look into this over the weekend.
hunterowens commentedon Oct 16, 2019
sadly, it looks like the work integrating OpenAPI and JSONSchema is still a little incomplete (/ being led by a team at WeWork, so YMMV). I'm gonna assign this to the future milestone.
10 remaining items
schnuerle commentedon Apr 4, 2022
There has been a call for something similar in CDS.
openmobilityfoundation/curb-data-specification#87
With MDS 2.0 the schema is going to need a major update. Typically we've relied on volunteers to complete this work each release, or even paid contractors to speed things up and double check work. I wonder if Swagger/OpenAPI could replace our json schema validation? And/or if we may need to fundraise the development of schema updates or OpenAPI creation in order to keep this for MDS 2.0 if there is not a volunteer or group willing to take on the work. I will bring it up with both WGSCs.
[-]Use OpenAPI to define Provider API request shapes and query parameters[/-][+]Use OpenAPI to define MDS API request shapes and query parameters[/+]marie-x commentedon Apr 4, 2022
I am in favor!
schnuerle commentedon Apr 13, 2022
If time allows, we will bring this up for discussion at the public Working Group meeting tomorrow, so please comment here or attend and bring your thoughts.
schnuerle commentedon Aug 11, 2022
We are getting more traction for OpenAPI in CDS.
JSON Schemas (as we've used already in MDS) are used for describing and validating data structures, which has been useful.
OpenAPI uses JSON Schemas internally as part of its data structure. Additionally it specifically documents the description (and sometimes validation) of HTTP requests parameters and responses for endpoints.
I think this addition of the endpoint documentation, definitions, and validation will be valuable for MDS consumers and producers and general users. I also think we can leverage most of the existing MDS JSON Schema work and incorporate it into the OpenAPI definitions. There are some tools to 'automate' this conversion for some syntax, notably this one made by Stoplight.
We will continue to work on this with CDS, and the idea there is to have a separate repo for the OpenAPI definitions. These will still be tied to release versions, but can be updated outside of the more formal spec release cycles. We will see how this plays out.
For MDS, the existing JSON Schema definitions would be removed from the spec itself and pulled out.
I also wanted to note that other specs like GBFS are still using JSON Schema, and I don't see talk of them moving to OpenAPI. Notably they put this schema in a separate repo, like I am proposing here.
schnuerle commentedon Dec 19, 2022
We should be working soon on creating OpenAPI for MDS 2.0 in its own repo referenced by MDS.
marie-x commentedon Dec 22, 2022
I strongly support OpenAPI for MDS 2.0, although I don't have the bandwidth to do it myself.
schnuerle commentedon Feb 7, 2023
First draft of modifying MDS to support OpenAPI is happening with PR #832. This work to migrate to OpenAPI in a new repo should take about 2 months. More details coming soon.
schnuerle commentedon May 5, 2023
The OpenAPI work is done for now with great work by @thekaveman!
See the end result here: https://github.com/openmobilityfoundation/mds-openapi
And visualized on Stoplight here: https://openmobilityfnd.stoplight.io/docs/mds-openapi/