Sidekiq Pro version was inspiration for me to create a pro version for my Ruby gem called knapsack that automatically splits tests in parallel on CI nodes/jobs.
With the knapsack_pro version I focused on common needs of users and there was simple validation method to verify if particular feature was needed. People just pay if it brings value.
In case of free open source tool, people tend to want something but you don't know if they actually will you use it and sometimes people demand a thing while you have to put the hard work.
When you do paid version of your tool then somehow people are polite. :)
Nowadays I have also support for running parallel tests in JS.
If you have no tests at all I would start with simple E2E tests to ensure your critical parts of app are working like sign up page and billing. You can add later other important sections of your app to ensure happy paths for your features are working fine.
When I started my own product for Ruby & JS developers https://knapsackpro.com I was doing mostly unit testing and later relayed on E2E tests for testing user dashboard to ensure happy paths are covered.
It's very easy to introduce weird bugs and it's much faster to run CI build to test your app than manually verify if it works correctly in your browser when you make some changes.
I started with writing open source Ruby gem https://github.com/ArturT/knapsack for running tests in parallel on CI and later I gave a talk on local meetup event for Ruby developers.
With time I wrote more blog posts https://docs.knapsackpro.com and shared on Reddit, Twitter, LinkedIn profile or Ruby groups. From time to time posts landed in RubyWeekly newsletter.
Also asking users for recommendation might help.
I connected with each person on LinkedIn I talked to or who signed up and I followed them on Twitter. People who like your product can easily see you publish new content and can like or share it.
End-to-end tests tend to take a lot of time when project grows. Some tricks like recording API calls with Puffing Billy can help save some time but still majority time is spend in browser.
This problem triggered me to work on knapsack ruby gem to solve problem how to split tests across parallel CI servers and run my whole test suite faster.
E2E tests can vary in time execution so I decided to split them in a dynamic way between CI nodes to auto balance tests speed between nodes.
For annoying flaky features tests, I use rspec-retry gem to repeat the test a few times before marking it as failed. It helped for integration tests with external sandbox API.
I noticed discourse had a lot of flaky tests while using their repo to test my knapsack_pro ruby gem to run test suite with CI parallelisation. A few articles with CI examples of parallelisation can be found here https://docs.knapsackpro.com
I need to try the latest version of discourse code, maybe now it will be more stable to run tests in parallel.
Back in 2014 I work on a project where we had ~15min test suite running on CI and this was painful so I started working on open source solution knapsack for CI parallelisation for Ruby tests.
It's easy to lose yourself as a developer in problem-solving mode instead of looking at the big picture of a project. It's great to have people around you that reminds you the code is just a part of the more complex story - how to help your client solve a business problem.
I turned my open source ruby gem knapsack into indie SaaS product. I was inspired by sidekiq gem. Here is the short story:
In 2014 I had a problem with large test suite in my customer Ruby on Rails project. I developed the open source gem knapsack to split tests among CI nodes in parallel.
In 2015 I had an idea to turn it into small SaaS product and by the end of the year, I had a first early adopter Buildkite team.
This year I released a new solution to dynamically split RSpec tests among CI nodes and I tested it with a few companies like Pivotal, VTS, Namely.
On one hand would be great to live in a world with small and fast test suites but on the other hand, we know how it is in the reality. There are large projects with dozen minutes or several hours test suite time execution.
I'm curious if there are similar problems with getting optimal time execution for large test suite across multiple CI nodes in projects done in other languages than Ruby. Have you experienced that problem? I'm thinking about building my tool in a different language and would be nice to test solution on the real project.
I'd love to hear your story about working with big test suites and what languages & tools you use. :)
I created ruby gem knapsack to speed up testing and I built a Pro version which is SaaS https://knapsackpro.com
I definitely learned a lot during last year while emailing with people interested in the product. Thanks to that I improved my tool iteratively while the early adopters discovered new areas or edge cases about testing I didn't even think about.
It's pretty popular problem. I recently got this book "The Positioning Manual for Technical Firm" by Philip Morgan from my friend and it's basically about how to move from generalist to in-demand specialist. The book has exercises how to determine for who and what service you could provide to position yourself as specialist. Maybe you will find this useful.
Do you have large test suites in your company projects? How big are they? How do you deal with long running tests?
Probably keeping a few small projects with fast test suites instead of monolith would be a nice way but we all know it's not always that easy.
I'd like to build a client library for auto-balancing tests across CI nodes in other languages than Ruby and I'd love to learn what test runners tools you use in your favorite programming languages.
When I released my first ruby gem for rspec test suite parallelisation I decided to write a short blog post about it. If you don't have your own blog you can write the post on your company blog. Thanks to that you will get some promotion because your company or colleagues can tweet or share it on Facebook.
Another thing I did was to post my article on Reddit in the ruby group. This seems to be a validated way of getting some traction. Make sure your friend read the blog post before publishing so you can iterate after feedback to make it interesting for people who will first time hear about your tool. :)
What do you think about http://sidekiq.org gem approach where you have free open source tool and paid pro version.
I did similar thing with my open source gem knapsack for test suite parallelisation in ruby. I run pro version at https://knapsackpro.com
I spin off https://knapsackpro.com from the knapsack gem and we are helping our customers run fast CI builds.