Hi HN! I'm Antoine, co-founder and CEO at Okay (YC W20). We are building analytics for engineering teams and we have released our new application on Github Marketplace.
Getting good metrics on engineering work is hard. On the one hand, you can build your own pipeline by ingesting API data from Github, JIRA, your CI tool, etc. Then you have to ETL that data, link it to teams and services, and finally select a dashboarding tool to display metrics. That gives you ultimate flexibility at a high cost of maintenance. On the other hand, you can go with out-of-the-box products but a lot of them focus on very specific metrics, such as DORA Metrics, PR throughput, etc. - not necessarily what you'll be interested in tracking, and you might even disagree with the metrics definitions or ethos of the product.
We've gone through several iterations of Okay and we think we now have the right balance between flexibility and ease-of-use. We solve ingestion + ETL and we give dashboarding features, but we leave users free to define metrics using our no-code query builder (with plain SQL coming soon). We specifically avoid forming opinions on what users should actually be building in terms of dashboards - in that sense, we are closer to an observability tool than to "development analytics" products. We are seeing a range of use-cases, such as incident tracking (with JIRA and PagerDuty data), sprint reviews (JIRA / Github), velocity & DORA (mainly Github + deployment data). We are building on BigQuery, DBT for the ETL and our custom ingestion code for API syncs. Our frontend is using Vue with a mix of D3.js and common charting libraries.
We'd love to get the community feedback on this current product approach! This Github app makes it easy to start on our free plan.
The job market for top engineers is supply-constrained, as demonstrated by the rapid increase in SWE wages in the past 15 years. If you agree that top engineers are the primary input to these companies' success, then it makes sense for them to adopt a "hoarding" behavior.
In other words, these companies have over-hired as a way to prevent competitors from hiring these same engineers. Although this has created a situation where the company has hired past the theoretical "productive" point, it was still a rational behavior.
Now that the tide is turning, the productivity goal becomes relatively more important than the competitive goal. In the long run though, I don't think the job market will fundamentally change - there is still a shortage of top engineering talent in the US.
80% of the value of OKRs is in the discussions they create, not in the actual list of OKRs that is produced in the process. The more an organization focuses on the technical aspects of producing a perfect OKR list or on "grading" the OKRs, the less value they will have.
In particular, the "KR" part - quantifiable outcomes for the goals - usually helps in clarifying vague projects or ideas that may otherwise harm the team's focus.
It's similar to the famous Warren Buffett's advice on identifying your priorities: pick 20 ideas you have and identify the top 5 goals that you absolutely want to get done; then, throw away the other 15 goals and make sure that you never get tempted to work on them.
It's great to see deeper pieces about developer productivity - in this case starting from first principles and the actual experience of developers. The notion of feedback loops has also been explored in e.g. https://martinfowler.com/articles/developer-effectiveness.ht...
The bad reputation of developer productivity metrics comes from the misguided assumption that developers should be measured. The better approach is to treat developers as customers of the management team / engineering enablement team /etc. In that sense, developer productivity is actually a measurement of management effectiveness / organizational health.
Once you build your developer productivity approach on this basis, what needs to be measured becomes much clearer - for example: interruptions caused by meetings, performance of local tooling like local builds, latency of CI, number of on-call pages triggering outside business hours, etc.
The right set of metrics depends on your team and can be sourced from surveys and just talking to people about what's painful on a daily basis. As a quick and dirty solution, I'd even recommend piping Github webhooks and other events to a product analytics tool like Amplitude or Mixpanel. You'd be surprised how fast you can understand things like build or CI latency by using a classic funnel visualization.
A lot of great engineering teams are migrating to this approach, especially when they have a dedicated platform / enablement / productivity engineering team.
Yes, engagement has a direct impact on the performance of individuals, and managers are responsible for creating an engaging environment - which shows why individual, team and manager performance are hard to disentangle.
Measuring development performance at the individual level is futile given the collaborative nature of the work and its complexity. Team and manager performance are more tractable.
Team performance is highly influenced by external disruptions, which is why the culture of engineering management since the 1980s has been centered around "removing blockers". Focusing on building a smooth-running operation (good hardware and software, fast code reviews, fast deployments, minimum amount of meetings, etc.) is probably what this article means by DecisionOps.
Re: managers, one formula I've used in the past (inspired by Andy Grove I believe) for managers' performance is a combination of the following: how is their team delivering ? How are they helping neighboring teams deliver as well? Do people want to work for this person (internal transfers + hiring) ? Or do people want to leave this team ?
Currently we only support these ecosystems because most of our early customers are on these platforms. We do have a tracing-like API that you can use to send custom events, but that would require more integration work of course.
For time spent on local builds, we expose a tracing-like API where you can tag the build id, start and end events, as well as connect it back to the right team. This custom event gets joined with everything else. I'll reach out to you on your email!
Sure! We've seen teams starting from situations where 60-70% of their week is spent in meetings, so these users benefit from calendar analysis. We encourage users to increase Maker Time (2 hours or more un-interrupted), which is inspired by http://www.paulgraham.com/makersschedule.htm. This notion of having enough time to code or focus on complex tasks has been shown to correlate positively with self-reported productivity and engagement.
Another example is a manager who noticed that one engineer on their teams was carrying 70% of the PR review load for the team. This was creating a situation where this person was burning out under the review load, but they also assumed that everyone was doing this same amount. We actually see this problem a lot, where people get used to bad situations just out of habit. In this case, the manager re-organized the code review process to make it more fair.
"Becoming better" points to the question of what is a good tech leader and how you can evaluate yourself as you make progress on this path. In my experience and according to Google's research in this area (https://rework.withgoogle.com/guides/managers-identify-what-...), there are specific traits to cultivate:
1. Is a good coach / Supports career discussions and discusses performance
"Leader As Coach: Strategies for Coaching & Developing Others" by Mary Dee Hicks and David Peterson. An in-depth, very practical manual on how to coach your reports according to their specific needs and personalities.
"Crucial Conversations: Tools for Talking When Stakes Are High" by Kerry Patterson et al. Helps you prepare and approach difficult conversations with empathy. Especially useful for performance conversations.
2. Creates an inclusive team environment, showing concern for success and well-being
"The Five Dysfunctions of a Team: A Leadership Fable" by Patrick Lencioni. If there is one book to read about team dynamics, this is the one.
"First, Break All the Rules" by Marcus Buckingham. Slightly more general than the 5 dysfunctions, but with a lot of good content about psychological safety and what makes team work.
3. Has a clear vision & strategy / is a good communicator
"High output management" by Andy Grove. Probably the most famous book on this list. It gives you a good introduction to setting goals, communicating about your team's work inwards and outwards. It also features more "system thinking" than the other books - how everything fits together as organizations scale, which is useful for developing vision and strategy.
I'd also recommend reading biographies and memoirs of famous leaders (e.g. Churchill, Marcus Aurelius). Strategic thinking can be easier to learn through examples and pattern recognition than through abstract / self-help books.
4. Is productive and results-oriented / has key technical skills to advise the team
A lot of new managers over-index on the people management side of the job and forget about the key responsibility of a technical leader: choosing a direction, leading others in this direction and delivering results.
"Accelerate: The Science of Lean Software and DevOps: Building and Scaling High Performing Technology Organizations" by Nicole Forsgren et al. One of the most recent, well-researched books into how to measure the productivity and quality of software teams.
"The Art of Scalability: Scalable Web Architecture, Processes, and Organizations for the Modern Enterprise" by Martin Abbott et al. This book uniquely blends the technical, organizational and people management aspects of technical leadership.