Love it! In the meantime, you can check out one of our customers who was able decrease their delivery speeds by nearly 90% and improve predictability to >95%!
The webinar link is below - pretty cool to see how teams use metrics to really enhance the way they work. Even cooler to hear how happy the engineering team was after removing all those annoying friction points in the development process :)
TBH I love the skepticism here. It's important to think about metrics critically in this way to make sure we're building truly useful applications of data.
We've found that evaluating individuals, stack ranking teams, and measuring an individual's 'productivity' to be an ultimately futile effort.
It's more important to focus on team delivery as described in many research studies. My favorite being the book Accelerate by nicole forsgren. If you haven't, I do encourage you to check it out!
As the founder of Haystack I can be seen as biased so its probably more helpful to checkout their research across thousands of teams on what makes an 'elite engineering team' and how elite engineering teams ultimately help drive successful organizations across profitability, market share, and customer satisfaction :)
Thanks for the heads up, looking into this now! :)
In the meantime, we've opened up our team Slack (link below) if you'd like more info on Haystack (or just want to jam about metrics with other eng leaders from companies like Microsoft and The Economist).
Loving all the Gitlab requests! If you sign up and select "Gitlab" as your tool of choice, it will put you on our waitlist and help us prioritize that work :)
1. While we do use pull requests as a unit of measure, we don't actually consider # of commits and # of pull requests as great proxies for productivity. We mainly track Cycle Time, Deployment Frequency, Change Failure Rate, and Throughput which help encapsulate speed vs. quality of an engineering teams. It's important for engineering metrics to correlate to engineering work. Specifically, that the engineering team has full control over those metrics. When looking from an Epic, Story, or Task perspective - this is often controlled by other parties, making it more difficult to know if engineering itself is improving or not. In addition to this, Epics often don't reflect actual engineering work in the same way that pull requests in Github do making them problematic to focus on.
2. Because we focus on engineering work itself from Github. Haystack can be used with any methodology so long as the team is using Git!
3. From an engineering perspective, the main goal is '# of successful iterations'. Product's main goal is that those successful iterations are pointed in the right direction. Looking at # of successful iterations (aka speed, deployment frequency, and quality) allow us to see how often we deploy value to customers and what is the quality of those deployments. Focusing on these metrics allow engineering teams to improve 'how they deliver'.
4. We actively don't look at individual engineers and we especially don't compare them using any metric since we've seen that lead to some bad takeaways like the one you mentioned. We help measure at the team-level to simply highlight improvements and bottlenecks rather than cross-compare any team or individual which is like comparing apples to oranges
I think it's really important to focus on team improvement and enablement. These tools if used correctly can be incredibly powerful. One in particular, Haystack, actually just focuses on team trends and alerts for things like burnout, concurrent work, idle prs, etc. giving teams more touch points for communication so they can resolve issues.
It's really not meant to 'watch' anyone, but geared towards being more helpful than simply measuring for the sake of measuring.
Anyways, to wrap it all up - I think it's about the intent of the tool and what's it's used for. It should be used to enable things like communication, advocation and improvement. Certainly not to judge, compare or rank (which I've seen in GitPrime and WayDev).
The webinar link is below - pretty cool to see how teams use metrics to really enhance the way they work. Even cooler to hear how happy the engineering team was after removing all those annoying friction points in the development process :)
https://www.youtube.com/watch?v=KrSc9EOrE_4