Team building is pretty constant across all the “flavors” of agile. Building good communication patterns and trust isn’t coupled to the “how we execute” details
This is a standard agile exercise called “Rules of Engagement”. Should be run with any forming team. I’m always surprised that this knowledge isn’t more commonplace.
It depends on the day but not such a high percentage usually. Our design phases don’t last as long (software only).
Virtual meeting fatigue is real because there aren’t as many natural breaks. It is important to make sure that people know they can excuse themselves for water/food/bio breaks.
I’ve also found that virtual whiteboarding is slower for me as I’m used to picking up a dry erase marker. It takes time to adjust to the different work stream but a huge benefit is the quality of the end product (no sloppy hand writing, easy to modify).
I don’t have any direct advice geared towards your situation as our white boarding and planning phases are quite different in my industry.
In addition to the software, make sure your team has good hardware. A screen large enough with high enough resolution is critical to see the whiteboard. What’s also important is that video from meetings takes up screen real estate and with multiple participants, having enough size and resolution to see them all helps you stay connected. External webcams that can be adjusted are helpful also.
Thanks for putting this together. I’d recommend adding some indicator that the results load async. I thought you had zero restaurants in Austin and left the page after a second because there was no loading indicator. In fact, until I got to Pittsburgh, I thought it was a site with no listings asking for my email (because Pittsburgh loads way faster).
(Adding an index to your database for city or however the queries are formed would probably also help).
Sometimes it’s nice to have a list view. Google maps only provides geolocated results in a small list so you’d have to move around the city quite a bit to find the whole list.
It can be useful to be remembered of places you like but haven’t been in a while as well.
This was essentially the story of how we built a successful software company at cPanel. Our customers would describe things like “I want a button that does this” or “a script that does that” and we had to make many assumptions about what they really wanted. Finally after determine we were spending massive amounts of time on unused features, we developed everything as a workable “beta” feature. The ones that we got a ton of praise or complaints on kept getting developed. The ones with silence stayed the same.
This was before heat map software and click tracking which can do a great job of augmenting customer feedback.
The key part that we found out is that if you take this approach, you have to couple it with high quality software support. We hired and trained folks who knew the entirety of our business and tech (Linux, DNS, HTTP, hosting, etc). Our tech support reps were paid much more highly than others in the industry but it gave us more benefits than just being able to support customers when beta software was ship to prod, it also gave us a training ground for future software engineers, QA folks, SysAdmins, and Product Managers.
What I think a lot of people forget is that software is one piece of the puzzle in solving a problem, it’s absolutely not the whole solution. If you design a process based around connecting with customers on a regular basis and having feedback flow throughout your company, you won’t have to “make perfect features”.
I’ve only worked with a small number of companies since that have understood this. Most I’ve worked with have created communication silos and sit in rooms trying to dream up the perfect set of requirements.
Do you not have frequent one-on-ones and regular conversations with your remote employees? How do you learn about their motivations, goals, current situation, or changes?
If you’re trying to replace human interactions and relationships with your employees with single question surveys, you’re going to miss out on all the important context.
Getting good output isn’t about rushing as fast to a goal as possible and not taking time to make sure things are going well. It’s about setting goals that are measurable, coming up with hypothesis about how to get there, and checking in regularly with an objective lens to see if you’re on track or if there’s a better approach.
I bought a VW GTI 20th anniversary (a numbered car) and the transmission broke on my third drive (wouldn’t shift out of third) which was a catastrophic failure. This was about 50 miles deep. Took a few hours to get towed. VW said they would fly in an expert from Germany to diagnose the issue. I opted to get a different car (new number, same model) and that one worked fine. Did I have massive ill will towards VW? no. This was even a numbered car that technically “could not be replaced”. Errors happen.
Statistically, a few hundred thousand model 3s have been delivered and many are well past 6k miles. I think you should wait and hear what Tesla’s reaction is before writing them off.
Things you should consider are:
- they have sold so many cars in the past 2 years that service center growth is struggling to keep up and wait times are long
- Tesla vehicles cannot be serviced easily by third parties so you have no option if Tesla says something you don’t like or wait times are long
Choosing the one broken car out of hundreds of thousands delivered as a reason to not buy one doesn’t make much sense. All manufacturers deliver defective vehicles. Looking at statistics on how many are defective per capita from each manufacturer would be a more reasonable course of action.
Pretty nuts if this is reproducible. Would be similar to decades of progress in shrinking super computers down to pocket sized phones in a few years time.
That’s understandable. I’d imagine at some point you’ll need to decouple the monolith a bit in order to work effectively as you scale. Best of luck with the challenge.
I think I see where you are coming from. Being as we use different tools, we wouldn’t allow a pull to be merged if it wasn’t up-to-date with master which is similar but a different approach. You’ll have to check at merge time because getting up-to-date could take a while and master could have changed. Jenkins does this and it can be done in other CI/CD systems with a bit of custom code.
I’d imagine at 1,000 developers and with a monolithic codebase, you’re looking to minimize test runs both from a time and cost of runners perspective.
You may also want to look into Zuul or Bazel if cost of test suite runs is a factor in coming to this solution.
Good to hear, that’ll make change management less of a chore.
I think the main thing that was missing for me is the rationale behind building this system rather than building a workflow in one of the existing CI/CD tools. Was there a throughout bottleneck in existing tools? Was there something custom about your workflow that wasn’t supported elsewhere? I may be wrong but the workflow you landed upon seems pretty common so I’m curious as to why the need to build and maintain a tool in house for this?
Would you say this is more of a decision based around the constraints of using GitHub or more of the ideal process for Shopify’s needs?
I’m curious because the article doesn’t mention the core reasons that you chose to write your own CD tool versus the other options that exist. The workflow you describe seems readily available in most tools. Perhaps the throughput was causing other options to break?
Thanks, I was hoping for more of this in the blog post. Since tools are just an expression of process/policy, it’s more interesting to here about the process and why than it is about building “yet another CD tool”. Appreciate the thoughtful and thorough response.
The major pain point I agree with on develop is changing the defaults to merge to that rather than master. It’s a shame this is not easier to do in git/github.
I’m not sure I agree with “develop can still be broken” as an issue that supports a queue. Whether it’s a queue or develop, one should run CI on each change to validate that merging it to master will not cause issues. It’s possible for both to be broken via the same scenarios just as it’s possible for master to be broken. Since CI runs before the branch is merged to develop and upon merge, a failure would “stop the world” and prevent more code from being merged unless that code fixes the failure.
I guess I’m not fully understanding how a queue prevents this. Since you don’t have a full picture of the state of master until something is merged from the queue, how do the CI checks in the queue prevent things that branch-based CI checks wouldn’t prevent in a “develop” branch? With branches and develop, pull requests remain open until they can be assured they merge properly with develop as well.
For clarity, I’m not arguing that a develop branch is the way to go, I think CD is much better.
Maybe I’m missing something big here but using multiple branches is permissible in other setups also. You can cherry pick a bunch of commits to a branch and test permutations but only certain branches get deployed to staging and production based on rules.
I’m glad that Shopify has found tools and a process that works. Honestly, I’m just having trouble comparing and constraining this to the other tools that are out there. The article never speaks about other approaches and whether or not they were considered and why you decided to go with a queue. It’s not clear to me if this was a case of improving the existing queue system because it was already in place or whether or not the queue was specifically chosen again because it was better than other alternatives (and why).
That’s what trying to get at in a “more polite” way. Many people use a “develop” branch to stage changes before master so they can run through CI. I’m curious as to why they used an untested queue instead. It seems like they wanted to cherry pick what commits made it to master rather than going in chronological order.
What’s odd is that they seem to characterize this as a tools issue rather than a process issue. There are plenty of CI/CD tools that allow for a similar or the same workflow as what they created. It’s also kinda scary that there’s not an emphasis on the overall SDLC and how the specific attributes of a branch or commit should/shouldn’t affect the process. You’d think at 1000 developers it’d be very important to define what is “launchable” as well. Haven’t worked with 1,000 on the same project but even with 20+, the standards and practices around development were always more important than the tooling. Tooling was just meant to represent workflows that were already defined.
Interesting. It seems like you have a very flexible process of how to launch code which could contribute to issues with visibility and rollbacks.
I’m curious as to why you had a queue instead of a develop branch before moving to CD? Was this to allow arbitrary commits to be launched to production rather than getting them batched by time?
Rather than being pedantic about the definition, maybe you could share your experiences with why that’s superior to validating each branch? Being dogmatic about a definition rather than experimenting with works best in production at your business seems illogical.