Show HN: OpenAPI to Terraform Provider Code Generation(speakeasyapi.dev)
speakeasyapi.dev
Show HN: OpenAPI to Terraform Provider Code Generation
https://www.speakeasyapi.dev/
2 comments
> This sounds pretty cool, but I'd love to know a bit more about how you handle the impedance mismatch between OpenAPI and an IaaC provider. So far I've only dabbled in making small changes to existing providers but to my somewhat uninformed eyes it seems like a "draw the rest of the owl" situation.
So the impedance mismatch is tackled in a few different ways:
CRUD:
The "operation" that interacts with the entity looks in an OpenAPI spec like "POST /entity", "GET /entity/{id}", "DELETE /entity". CRUD semantics aren't visible at this layer without making a bunch of heuristic guesses about the ways that people use (and mis-use) REST semantics.
Rather than guess, we add OpenAPI extensions to operations that guide these semantics. I.e. If `POST /entity` is used to create the entity, it is annotated with `x-speakeasy-entity-operation: MyEntity#create`. Similarly `#update`, `#read`, `#delete` for all other operations that interact with resources in terraform.
Entity Attributes:
The "entity" doesn't usually look the same across all CRUD request/responses. I.e. more attributes (e.g. `id`) are often returned in a response, that aren't in the request.
To tackle this, we annotate every JSON Schema in request/response with `x-speakeasy-entity: MyEntity` that applies to an interaction with an entity in terraform. Some versions of this might be bigger/smaller depending on API semantics. To build the terraform schema entry, we merge all of these together, applying inference logic to work out how each attribute is interacted with in the CRUD requests to determine the terraform properties.
E.g. if an attribute is returned in a CREATE API response, but isn't in a CREATE API Request, it's marked as `Computed`. If one attribute is in the UPDATE API Request, but not another, one is marked as `ForceNew` via a terraform plan extension (i.e. only modifiable with a full delete/create cycle) whereas the other is left alone.
The `type` / `format` that are visible in JSON Schemas make their way to runtime validations, e.g. ensuring that `format: date` are runtime validated in `YYYY-MM-DD` format.
In total there's around 35 different inference rules so far to cover how different bits of the JSON Schema / OpenAPI specification map into terraform state.
Hoisting: The "entity" that is been managed isn't always at the root level of a request/response body, but might be hidden at some deeper level.
E.g. 1: if a response body looked like `{data: {the-entity}}`, we annotate `the-entity` JSON Schema level and "hoist" that to the top level. Anything at a "higher" level is inlined into the resource.
E.g. 2: A parameter in a CREATE request will automatically be inlined into `the-entity`, even if not defined in the JSON Schema of a request body. E.g. `POST /{workspace_id}/entity` marked as `x-speakeasy-entity-operation: MyEntity#create` will not only have whatever request body is defined in the terraform type, but it will also gain an attribute `workspace_id` marked as `ForceNew` in the root of the state.
> Writing some terraform providers is very likely in our not-to-distant future, so I created an account and imported our spec. It failed validation and wouldn't let me proceed, and the errors provided weren't helpful. When I have a bit more time I'll throw it at another OpenAPI validator to try to work out where the problems are, but we're using this spec for code generation for several clients already and haven't had any issues. It's dynamically generated from our backend.
Thanks for letting us know. There's definitely gaps in our validation library: we operate on a relatively high level of strictness to minimize the complexity of code generation, but we're constantly trying to loosen it.
> One more thing, I noticed you have logos for stripe, twilio and plaid on your site, above a "Learn how SDKs help" button, which 404's. If those are your customers that's pretty cool, but otherwise using other companies' logos is a bit shady.
We'll remove them. It's meant to be an illustrator in how improving developer experience directly impacts revenue for a company, directly linking out to some sources. However definitely don't want us to appear shady. Some of our more recognizable customers are listed in the "Trusted By" section in the header.
So the impedance mismatch is tackled in a few different ways:
CRUD:
The "operation" that interacts with the entity looks in an OpenAPI spec like "POST /entity", "GET /entity/{id}", "DELETE /entity". CRUD semantics aren't visible at this layer without making a bunch of heuristic guesses about the ways that people use (and mis-use) REST semantics.
Rather than guess, we add OpenAPI extensions to operations that guide these semantics. I.e. If `POST /entity` is used to create the entity, it is annotated with `x-speakeasy-entity-operation: MyEntity#create`. Similarly `#update`, `#read`, `#delete` for all other operations that interact with resources in terraform.
Entity Attributes:
The "entity" doesn't usually look the same across all CRUD request/responses. I.e. more attributes (e.g. `id`) are often returned in a response, that aren't in the request.
To tackle this, we annotate every JSON Schema in request/response with `x-speakeasy-entity: MyEntity` that applies to an interaction with an entity in terraform. Some versions of this might be bigger/smaller depending on API semantics. To build the terraform schema entry, we merge all of these together, applying inference logic to work out how each attribute is interacted with in the CRUD requests to determine the terraform properties.
E.g. if an attribute is returned in a CREATE API response, but isn't in a CREATE API Request, it's marked as `Computed`. If one attribute is in the UPDATE API Request, but not another, one is marked as `ForceNew` via a terraform plan extension (i.e. only modifiable with a full delete/create cycle) whereas the other is left alone.
The `type` / `format` that are visible in JSON Schemas make their way to runtime validations, e.g. ensuring that `format: date` are runtime validated in `YYYY-MM-DD` format.
In total there's around 35 different inference rules so far to cover how different bits of the JSON Schema / OpenAPI specification map into terraform state.
Hoisting: The "entity" that is been managed isn't always at the root level of a request/response body, but might be hidden at some deeper level.
E.g. 1: if a response body looked like `{data: {the-entity}}`, we annotate `the-entity` JSON Schema level and "hoist" that to the top level. Anything at a "higher" level is inlined into the resource.
E.g. 2: A parameter in a CREATE request will automatically be inlined into `the-entity`, even if not defined in the JSON Schema of a request body. E.g. `POST /{workspace_id}/entity` marked as `x-speakeasy-entity-operation: MyEntity#create` will not only have whatever request body is defined in the terraform type, but it will also gain an attribute `workspace_id` marked as `ForceNew` in the root of the state.
> Writing some terraform providers is very likely in our not-to-distant future, so I created an account and imported our spec. It failed validation and wouldn't let me proceed, and the errors provided weren't helpful. When I have a bit more time I'll throw it at another OpenAPI validator to try to work out where the problems are, but we're using this spec for code generation for several clients already and haven't had any issues. It's dynamically generated from our backend.
Thanks for letting us know. There's definitely gaps in our validation library: we operate on a relatively high level of strictness to minimize the complexity of code generation, but we're constantly trying to loosen it.
> One more thing, I noticed you have logos for stripe, twilio and plaid on your site, above a "Learn how SDKs help" button, which 404's. If those are your customers that's pretty cool, but otherwise using other companies' logos is a bit shady.
We'll remove them. It's meant to be an illustrator in how improving developer experience directly impacts revenue for a company, directly linking out to some sources. However definitely don't want us to appear shady. Some of our more recognizable customers are listed in the "Trusted By" section in the header.
Thanks for the explanation. It'll be interesting to see how this strategy performs with our current API - it has a few rough edges that stem from being designed for consumption by internal clients only at this stage.
Hey HN, after a fair few months of iteration, we're excited to share our latest offering in Speakeasy: auto-generation of Terraform providers using OpenAPI specifications.
The Problem: Building a Terraform Provider to expose an API via Infrastructure As Code (e.g. HCL, Pulumi, CDK) is expensive, error-prone, and highly repetitive.
However, if you don't have a mature terraform provider many products will not even be considered by mature organisations with a mandate to automate-all-the-things.
Our Solution:
1. Deep Integration with OpenAPI: Just point Speakeasy to your OpenAPI spec. Every change, every tweak, every evolution of your API is monitored, and we adapt in real-time.
2. Automated Terraform Provider Generation: Instead of manually coding a Terraform provider, Speakeasy synthesizes one for you, ensuring it remains in sync with your API's latest version.
3. Smart Schema Semantics: Based on CRUD operations, Speakeasy can smartly deduce and apply Terraform schema attributes like Computed, Optional, and Force Replace.
4. Continuous GitHub PRs: With each OpenAPI spec alteration, PRs are raised automatically against your Terraform provider repository, ensuring seamless and continuous integration.
Why Speakeasy for Terraform?
While several tools play around the fringes, none offer 100% automation via code synthesis. We've built and extensively tested our Terraform Provider Generation engine from OpenAPI, and have been in production with real customers for the last 6 months.
* It is possible to generate an OpenAPI specification for almost any server side framework (even things like ProtoBuf via REST Gateways) entirely automatically.
* Once your OpenAPI specification is automatically generated from your codebase, Speakeasy enables subsequent integration artifacts, like SDKs and Terraform Providers, to be automatically maintained with close-to-zero engineering effort.
* Once an API is exposed via a Terraform Provider, it becomes usable by the entire IaC ecosystem like Terraform, CDK, and Pulumi through the use of bridging tools.
* Speakeasy will also generate documentation, usage examples, support/guide you through the launch, and upgrade the provider automatically as the ecosystem matures.
Dive Deeper:
- Explore our product: https://www.speakeasyapi.dev
- Explore our CLI: https://github.com/speakeasy-api/speakeasy
- Explore our largest yet terraform provider: https://github.com/airbytehq/terraform-provider-airbyte
- Explore exposing a terraform provider via Pulumi: https://www.speakeasyapi.dev/post/pulumi-terraform-provider
- Explore a toy example: https://github.com/speakeasy-sdks/terraform-provider-hashicu...
A massive thanks to our early adopters and the vibrant Terraform community for guiding our journey.
HN, we’re eager for your insights. Whether it’s rigorous feedback, burning queries, or just wanting to geek out over Terraform and APIs, hit us up!
The Problem: Building a Terraform Provider to expose an API via Infrastructure As Code (e.g. HCL, Pulumi, CDK) is expensive, error-prone, and highly repetitive.
However, if you don't have a mature terraform provider many products will not even be considered by mature organisations with a mandate to automate-all-the-things.
Our Solution:
1. Deep Integration with OpenAPI: Just point Speakeasy to your OpenAPI spec. Every change, every tweak, every evolution of your API is monitored, and we adapt in real-time.
2. Automated Terraform Provider Generation: Instead of manually coding a Terraform provider, Speakeasy synthesizes one for you, ensuring it remains in sync with your API's latest version.
3. Smart Schema Semantics: Based on CRUD operations, Speakeasy can smartly deduce and apply Terraform schema attributes like Computed, Optional, and Force Replace.
4. Continuous GitHub PRs: With each OpenAPI spec alteration, PRs are raised automatically against your Terraform provider repository, ensuring seamless and continuous integration.
Why Speakeasy for Terraform?
While several tools play around the fringes, none offer 100% automation via code synthesis. We've built and extensively tested our Terraform Provider Generation engine from OpenAPI, and have been in production with real customers for the last 6 months.
* It is possible to generate an OpenAPI specification for almost any server side framework (even things like ProtoBuf via REST Gateways) entirely automatically.
* Once your OpenAPI specification is automatically generated from your codebase, Speakeasy enables subsequent integration artifacts, like SDKs and Terraform Providers, to be automatically maintained with close-to-zero engineering effort.
* Once an API is exposed via a Terraform Provider, it becomes usable by the entire IaC ecosystem like Terraform, CDK, and Pulumi through the use of bridging tools.
* Speakeasy will also generate documentation, usage examples, support/guide you through the launch, and upgrade the provider automatically as the ecosystem matures.
Dive Deeper:
- Explore our product: https://www.speakeasyapi.dev
- Explore our CLI: https://github.com/speakeasy-api/speakeasy
- Explore our largest yet terraform provider: https://github.com/airbytehq/terraform-provider-airbyte
- Explore exposing a terraform provider via Pulumi: https://www.speakeasyapi.dev/post/pulumi-terraform-provider
- Explore a toy example: https://github.com/speakeasy-sdks/terraform-provider-hashicu...
A massive thanks to our early adopters and the vibrant Terraform community for guiding our journey.
HN, we’re eager for your insights. Whether it’s rigorous feedback, burning queries, or just wanting to geek out over Terraform and APIs, hit us up!
Writing some terraform providers is very likely in our not-to-distant future, so I created an account and imported our spec. It failed validation and wouldn't let me proceed, and the errors provided weren't helpful. When I have a bit more time I'll throw it at another OpenAPI validator to try to work out where the problems are, but we're using this spec for code generation for several clients already and haven't had any issues. It's dynamically generated from our backend.
One more thing, I noticed you have logos for stripe, twilio and plaid on your site, above a "Learn how SDKs help" button, which 404's. If those are your customers that's pretty cool, but otherwise using other companies' logos is a bit shady.