Show HN: Pythagora – auto generated integration tests for Node.js API servers
1 comments
This is a brilliant idea. One of those "why hasn't that happened yet??" ones.
Is there any way for me to obscure PII data easily? or anonymise any data recognised as file contents or base64? That'd help.
Have you considered hooking up alerts. For example to run `npx pythagora --script ./path/to/your/server.js --mode test` but with an option to pass a key/function to send alerts(perhaps just picking up test fail and dealing with it in the CI is better though) if a test fails that previously passed (wherever that command is run, so in the CI or if manually ran).
We do use socket.io for a massive amount of our requests so I don't think it'd cover that (happy if wrong?) but I will look into if we can add a pull request for socket.io support, if you are interested in that scope creep... As that would be very powerful.
Is there any way for me to obscure PII data easily? or anonymise any data recognised as file contents or base64? That'd help.
Have you considered hooking up alerts. For example to run `npx pythagora --script ./path/to/your/server.js --mode test` but with an option to pass a key/function to send alerts(perhaps just picking up test fail and dealing with it in the CI is better though) if a test fails that previously passed (wherever that command is run, so in the CI or if manually ran).
We do use socket.io for a massive amount of our requests so I don't think it'd cover that (happy if wrong?) but I will look into if we can add a pull request for socket.io support, if you are interested in that scope creep... As that would be very powerful.
Thanks James!
Yea, agree, I'm also curious why this doesn't exist yet. I thought there might be a tech roadblock but there really isn't any - especially since Node released Async Context. We did a POC for covering Redis and requests to 3rd party APIs and everything looks very doable, just need some time to implement it.
Re PII, interesting question. We were thinking Pythagora will be used in a local/qa environment and not in production so that there won't be a need to anonymise data. However, it's good to have it in mind.
Re alerts, yes, definitely. I think that's a great idea for using Pythagora in a CICD pipeline. Especially since, once the test suite grows, it will be super handy to get alert for failed tests after branches merge.
Re sockets, yes, they are definitely on the roadmap - my previous startup used mostly sockets so hoping we cover them soon and we would LOVE if you want to contribute. If you have any questions or need anything, just let me know, I'd be happy to help you with the contribution.
Yea, agree, I'm also curious why this doesn't exist yet. I thought there might be a tech roadblock but there really isn't any - especially since Node released Async Context. We did a POC for covering Redis and requests to 3rd party APIs and everything looks very doable, just need some time to implement it.
Re PII, interesting question. We were thinking Pythagora will be used in a local/qa environment and not in production so that there won't be a need to anonymise data. However, it's good to have it in mind.
Re alerts, yes, definitely. I think that's a great idea for using Pythagora in a CICD pipeline. Especially since, once the test suite grows, it will be super handy to get alert for failed tests after branches merge.
Re sockets, yes, they are definitely on the roadmap - my previous startup used mostly sockets so hoping we cover them soon and we would LOVE if you want to contribute. If you have any questions or need anything, just let me know, I'd be happy to help you with the contribution.
I just released my first npm package so wanted to share it here.
Pythagora is a Node.js module that creates automated integration tests for MERN/MEAN apps by recording server activity.
To integrate Pythagora, you need to paste one line of code to your repository and run the Pythagora capture CLI command. Then, just play around with your app and from all API requests and database queries Pythagora will generate integration tests.
When running tests, it doesn’t matter what database is your Node.js connected to or what is the state of that database. Actually, that database is never touched or used —> instead, Pythagora creates a special, ephemeral `pythagoraDb` database, which it uses to restore the data before each test is executed, that was present at the time when the test was recorded. Because of this, tests can be run on any machine or environment.
Also, *if the request updates the database, after the API returns the response, Pythagora checks the database to see if it was updated correctly*.
Finally, Pythagora tracks (using istanbul/nyc) lines of code that were triggered during tests, so you know how much of your code is covered.
Our goal is to help teams that don’t have time to properly manage their test suite to cover their entire codebase with as little as 1 hour of their time. We tested Pythagora on open source clones of apps like Reddit, IG, etc. and we were able to get to 50% of code coverage within 10 minutes and to 90% within 1 hour of playing around these apps.
Feel free to try out the Pythagora npm package - https://www.npmjs.com/package/pythagora
And the source code on Github - https://github.com/Pythagora-io/pythagora
We’re excited to hear what you think.
How do you write integration tests for an API server? Would you consider using Pythagora instead/along with your system?
If not, we would love to hear what are your concerns and why this wouldn’t work for you?