Rebuilding the Shopify Admin: Deleting 28k lines of JavaScript(shopify.com)
shopify.com
Rebuilding the Shopify Admin: Deleting 28k lines of JavaScript
http://www.shopify.com/technology/15646068-rebuilding-the-shopify-admin-improving-developer-productivity-by-deleting-28-000-lines-of-javascript/
15 comments
AFAICT its very rare to find developers with that unique ability to write truly minimalist code.
Too true. I find that the gating factor for development for me is the number of things I have to keep in my head at one time while working with a codebase. So simpler and cleaner abstractions mean that development goes faster with a lower defect rate. Maybe it's because I do have a background in music that I prefer to write minimalist code but I think there are a lot of objective advantages in it.
Too true. I find that the gating factor for development for me is the number of things I have to keep in my head at one time while working with a codebase. So simpler and cleaner abstractions mean that development goes faster with a lower defect rate. Maybe it's because I do have a background in music that I prefer to write minimalist code but I think there are a lot of objective advantages in it.
Well, its not often I say I wish I lived closer to Ottawa, but I fully agree with this post, and it was quite nice to see it written down elsewhere.
From the perspective of someone deeply rooted in UI development for the desktop, seeing stuff like this makes me glad I am deeply rooted in UI development for the desktop. If I had to build a Cocoa app with this messy, complicated shit ... I'd blow my brains out. All over my desk. Maybe I'd leave a note, but I think people who knew what I did for a living would get it and not need me to leave one.
It also makes me wonder, how we've all decided that HTML is the way to go for this class of application, how hacking a document display format to ape a native UI is a good idea.
I have to think that at some point people think that there has to be a better way to deliver a network transparent, portable UI without worrying about strange things like a Shadow DOM, the hack that is CSS, and all of the other weird shit web developers have to put up with to create something that mimics (usually poorly) a desktop app.
Like is there a breaking point where people are like "fuck it" and come up with an alternative browser that is specific to this and similar kinds of applications? I guess not because the gray line between what HTML is good at and what people need to HTML to do despite itself is a pretty big fat line.
It also makes me wonder, how we've all decided that HTML is the way to go for this class of application, how hacking a document display format to ape a native UI is a good idea.
I have to think that at some point people think that there has to be a better way to deliver a network transparent, portable UI without worrying about strange things like a Shadow DOM, the hack that is CSS, and all of the other weird shit web developers have to put up with to create something that mimics (usually poorly) a desktop app.
Like is there a breaking point where people are like "fuck it" and come up with an alternative browser that is specific to this and similar kinds of applications? I guess not because the gray line between what HTML is good at and what people need to HTML to do despite itself is a pretty big fat line.
HTML + CSS + Javascript is obviously far from perfect, but is there any other way TODAY to write a single piece of code that will run on just about every computing device in circulation?
Honestly, I challenge anyone to write a simple UI (lets say draw a circle and put some text on the screen) that just works across ALL major operating systems, screen sizes, versions, etc.
If do actually end up doing it in anything besides "web" languages, its going to be more complicated and take longer as well....
To make a car analogy, The Chevy small block V8 engine was/is never the best, fastest, most efficient engine. It's just that for the past 40+ years they've put the damn thing in every vehicle imaginable. It's pretty ubiquitous in the auto world....It just works and everyone still uses it. Sort of like how you will find a HTML, CSS, Javscript rendering engine in everything today...
Honestly, I challenge anyone to write a simple UI (lets say draw a circle and put some text on the screen) that just works across ALL major operating systems, screen sizes, versions, etc.
If do actually end up doing it in anything besides "web" languages, its going to be more complicated and take longer as well....
To make a car analogy, The Chevy small block V8 engine was/is never the best, fastest, most efficient engine. It's just that for the past 40+ years they've put the damn thing in every vehicle imaginable. It's pretty ubiquitous in the auto world....It just works and everyone still uses it. Sort of like how you will find a HTML, CSS, Javscript rendering engine in everything today...
This is one of those religious debates that will never die as long as the web exists in a similar capacity as it does, today.
The fact of the matter—whether we like it or not—is that these are the tools we have for web development. It's hacks on hacks on hacks, as one might expect in a suite of technologies originally intended for X but arguably powerful enough to perform Y, Z, and Z(2014). Take ECMAScript: the first draft was written up retroactively, quirks and all. Many of the oddities and edgecases were written into the official language spec and vendors were encouraged to implement them as written (thus, intentionally writing engines with quirks).
But the fact remains that no single, "fuck it" has been able to take hold. The technologies we have at our disposal have proven themselves "good enough" to keep that from happening to date. At this point, we've invested ~20 years into these technologies.
And, for what it's worth, there are ways to cut through the cruft. These things can be done moderately cleanly. It is possible to write a web-based application that doesn't make one want to blow their brains out (I'd make the argument that, in a lot of cases if you're trying to write a "desktop app" in the browser, you're focusing on the wrong thing but that's neither here nor there[0]). It's simply that legacy and twisting the language to do what we want and not what it was intended for has caused that to not be terribly straightforward.
I understand you're being hyperbolic but it's been my experience that this mindset ("this is so bad I'm never going to touch it") generally comes from folks who take one look and turn in disgust. That's fine—and I'm not going to argue it's not without it's warts because that would be insanity—but it isn't so bad that you'd need to off yourself, and I think the staying power it has had regardless of quirks, poor design choices, etc., proves that.
[0] - with the right team and the right reasons, it can be done. The Slack desktop app is a pretty good example of when it makes sense and is done quite well.
The fact of the matter—whether we like it or not—is that these are the tools we have for web development. It's hacks on hacks on hacks, as one might expect in a suite of technologies originally intended for X but arguably powerful enough to perform Y, Z, and Z(2014). Take ECMAScript: the first draft was written up retroactively, quirks and all. Many of the oddities and edgecases were written into the official language spec and vendors were encouraged to implement them as written (thus, intentionally writing engines with quirks).
But the fact remains that no single, "fuck it" has been able to take hold. The technologies we have at our disposal have proven themselves "good enough" to keep that from happening to date. At this point, we've invested ~20 years into these technologies.
And, for what it's worth, there are ways to cut through the cruft. These things can be done moderately cleanly. It is possible to write a web-based application that doesn't make one want to blow their brains out (I'd make the argument that, in a lot of cases if you're trying to write a "desktop app" in the browser, you're focusing on the wrong thing but that's neither here nor there[0]). It's simply that legacy and twisting the language to do what we want and not what it was intended for has caused that to not be terribly straightforward.
I understand you're being hyperbolic but it's been my experience that this mindset ("this is so bad I'm never going to touch it") generally comes from folks who take one look and turn in disgust. That's fine—and I'm not going to argue it's not without it's warts because that would be insanity—but it isn't so bad that you'd need to off yourself, and I think the staying power it has had regardless of quirks, poor design choices, etc., proves that.
[0] - with the right team and the right reasons, it can be done. The Slack desktop app is a pretty good example of when it makes sense and is done quite well.
I've just started getting into Cocoa dev after doing iOS for a while and it has more than its own share of warts. KVO is a clunky API that's very inconsistently available and easy to misuse. Interfaces built using auto layout in xibs are very tricky to debug and maintain. There's all kinds of string typing in the APIs which defeats many of the benefits of static typing. And there seem to be a lot of quirks and bugs in the API overall which I'm guessing are a result of Apple shifting its focus to iOS, where the money is.
All in all I'll take the web stack, particularly with the changes coming down the pipe. Javascript with ES6 is a much better language. And if web components really take off then component-based UI development on the web will be pretty competitive with Cocoa.
All in all I'll take the web stack, particularly with the changes coming down the pipe. Javascript with ES6 is a much better language. And if web components really take off then component-based UI development on the web will be pretty competitive with Cocoa.
Things are not that complicated today,really.It's just that there isnt only one way to do things.
> I have to think that at some point people think that there has to be a better way to deliver a network transparent, portable UI without worrying about strange things like a Shadow DOM, the hack that is CSS, and all of the other weird shit web developers have to put up with to create something that mimics (usually poorly) a desktop app
It already exists : GWT,or you can use something like C++ and the emscripten toolchain,and whatever C++ gui framework. So if you dislike all the js/html/css you can definetly avoid it.And users dont care,provided it runs in their browser without plugin.
Most webdevs wont use these because they already know javascript and there are good frameworks out there.
And at the end of the day it is still a webpage,it's not a desktop app.
> I have to think that at some point people think that there has to be a better way to deliver a network transparent, portable UI without worrying about strange things like a Shadow DOM, the hack that is CSS, and all of the other weird shit web developers have to put up with to create something that mimics (usually poorly) a desktop app
It already exists : GWT,or you can use something like C++ and the emscripten toolchain,and whatever C++ gui framework. So if you dislike all the js/html/css you can definetly avoid it.And users dont care,provided it runs in their browser without plugin.
Most webdevs wont use these because they already know javascript and there are good frameworks out there.
And at the end of the day it is still a webpage,it's not a desktop app.
Sun with Java and Java applets wanted to be that. Maybe Flash and Flew dreamed of that too.
Browsers were said to render the OS irrelevant and in a way Chromebooks come close to that.
We were also supposed to not owned & store any software locally and rent & download everything. Office Online, Creative clood are there.
But things are rarely that radical and the web was there and opened. I prefer desktop apps (and I'm a web developer !) but how can you beat the ability to login to any of your web tool from any computer ?
Browsers were said to render the OS irrelevant and in a way Chromebooks come close to that.
We were also supposed to not owned & store any software locally and rent & download everything. Office Online, Creative clood are there.
But things are rarely that radical and the web was there and opened. I prefer desktop apps (and I'm a web developer !) but how can you beat the ability to login to any of your web tool from any computer ?
[deleted]
While it's refreshing to see people talk about the things they did wrong and how they intelligently approached fixing them, building your own javascript framework from scratch (batman) is something I would have said no to from the start. Not for the faint of heart or those who have not suffered such an endeavor before.
If I had to guess, none of the solutions back then sufficed for them. Batman has been around for a while.
You're dead on. We started work on Batman some time ago now - options then were quite limited.
2 years ago I was exploring frameworks for an enterprise project. I tried Batman along with most others, I can easily see why building your own framework would have been an option any earlier than that.
Most frameworks weren't even at 1.0 stable yet. I initially wrote a fairly extensive prototype with Ember, but there were breaking changes so frequently I couldn't keep up with the learning curve. In the end I chose Angular because it had the best documentation, Google dogfooding, and testing was an obvious priority. Worked out so well we're on our 4th major Angular app now.
Most frameworks weren't even at 1.0 stable yet. I initially wrote a fairly extensive prototype with Ember, but there were breaking changes so frequently I couldn't keep up with the learning curve. In the end I chose Angular because it had the best documentation, Google dogfooding, and testing was an obvious priority. Worked out so well we're on our 4th major Angular app now.
Yeah. I've made that kind of decision a few times in the past — do we use NewFramework or do we develop our own.
It's a really tough call. I've been wrong (and right ;-) multiple times going down either path.
The last time I had to make that sort of choice we went the set-based design / real options path and developed both in parallel for as long as we could. More expensive, but lower risk. I'm glad we did coz the decision we made four months in on which one to drop wasn't the one we would have made at the start.
It's a really tough call. I've been wrong (and right ;-) multiple times going down either path.
The last time I had to make that sort of choice we went the set-based design / real options path and developed both in parallel for as long as we could. More expensive, but lower risk. I'm glad we did coz the decision we made four months in on which one to drop wasn't the one we would have made at the start.
Money quote from the article:
The new admin would make a return to more classic architecture with some modernizations. Our approach would be ERB views and server-side rendering, with the use of Turbolinks, and a lightweight custom JavaScript binding system. This allowed us to tackle problems of code duplication and developer productivity in a single blow.
As always, the devil is in the details and I'm looking forward to seeing what their JS binding system looks like, but the pendulum appears to be swinging back towards server-side rendering and models, even amongst the cool kids.
I have been working on a small library for doing HTML partial AJAX programming, using fairly straight-forward HTML attributes and traditional server side rendering (plus some goodies like custom HTTP header support, timers, etc.):
http://intercoolerjs.org
I'm using it successfully in a few projects and very much enjoy the simplicity of the whole approach when contrasted with full MVC systems.
The new admin would make a return to more classic architecture with some modernizations. Our approach would be ERB views and server-side rendering, with the use of Turbolinks, and a lightweight custom JavaScript binding system. This allowed us to tackle problems of code duplication and developer productivity in a single blow.
As always, the devil is in the details and I'm looking forward to seeing what their JS binding system looks like, but the pendulum appears to be swinging back towards server-side rendering and models, even amongst the cool kids.
I have been working on a small library for doing HTML partial AJAX programming, using fairly straight-forward HTML attributes and traditional server side rendering (plus some goodies like custom HTTP header support, timers, etc.):
http://intercoolerjs.org
I'm using it successfully in a few projects and very much enjoy the simplicity of the whole approach when contrasted with full MVC systems.
Can we please drop the "cool kids" cliche? It's a tired generalization.
Clearly at this point both architectures work, it's childish to see this as an argument for server-side rendering. Obviously it works, so do client MVC systems.
The value in this article is the humble detailing of their mistakes that many of us also experience in our careers.
Clearly at this point both architectures work, it's childish to see this as an argument for server-side rendering. Obviously it works, so do client MVC systems.
The value in this article is the humble detailing of their mistakes that many of us also experience in our careers.
I think part of that is a reaction to how many people jumped on to the client-side JS framework bandwagon very early, because it was "the next big thing", but before the tools were ready for production.
I've had to deal with more than my fair share of crappy, hacked-together Ember apps that were developed like this. There have been too many breaking changes to upgrade, performance is bad, an they're full of hacks to work around deficiencies that existed back then.
Even though we actually have production-ready frameworks now, that sort of thing can leave a bad taste in your mouth. Especially when you see the same mistakes made by people jumping onto the Meteor bandwagon, for example. But it's defintiely something to try and get over :)
I've had to deal with more than my fair share of crappy, hacked-together Ember apps that were developed like this. There have been too many breaking changes to upgrade, performance is bad, an they're full of hacks to work around deficiencies that existed back then.
Even though we actually have production-ready frameworks now, that sort of thing can leave a bad taste in your mouth. Especially when you see the same mistakes made by people jumping onto the Meteor bandwagon, for example. But it's defintiely something to try and get over :)
Ember now feels like a complete framework, but fundamentally any Ember app will be tightly coupled to Ember the framework.
In many ways this is the same as Rails Apps have traditionally been very much tied to Rails.
Having worked on many Rails Apps that have grown beyond the scope of the framework, the apps I build today are as decoupled as possible from the framework.
I hope that the Ember community can grow in that direction too, and at a distant look, it feels like Web Components will be that method of decoupling.
Right now I tell everyone that I'd never render HTML (or JS) on the server, and I hope that JS frameworks will keep maturing with my clients' needs, so I don't have to renege on that approach.
In many ways this is the same as Rails Apps have traditionally been very much tied to Rails.
Having worked on many Rails Apps that have grown beyond the scope of the framework, the apps I build today are as decoupled as possible from the framework.
I hope that the Ember community can grow in that direction too, and at a distant look, it feels like Web Components will be that method of decoupling.
Right now I tell everyone that I'd never render HTML (or JS) on the server, and I hope that JS frameworks will keep maturing with my clients' needs, so I don't have to renege on that approach.
Can we please drop the "cool kids" cliche? It's a tired generalization.
Worse, 'cool kids' implies doing something because someone else is doing it rather than because it's the right thing to do in your organisation. It smacks of cargo cult behaviour - copying actions and hoping to get the same end result without understanding why.
Worse, 'cool kids' implies doing something because someone else is doing it rather than because it's the right thing to do in your organisation. It smacks of cargo cult behaviour - copying actions and hoping to get the same end result without understanding why.
I think it's actually the exact phrase we should be using. I work with helping beginner programmers learn to code and if they want to know "javascript" (they don't even know what that means) they are always asking for ember/angular/node and when I ask why they want to use these technologies or what project they are working on that would benefit from that style they blank. It's definitely cool kid syndrome.
>and a lightweight custom JavaScript binding system
I started down a similar path with ember.js and was starting to see the same issues. Primarily the business models duplicated on the client. So, it strikes me that we are still figuring out the right path forward on the client/server balance.
My teammate and I compete to see not how much code we can write, but who can have the highest sum of lines deleted in all commits. Any old hack can write a lot of code, but to remove code while still adding functionality (or at least, not removing it) is the sign of a good developer, imho.
We do that too, in our team. Reminds me of: http://delete-your-code.herokuapp.com
A little bit OOT, i need to connect to my VPN in US to be able to see the post. It is always redirect to my country's TLD (www.shopify.co.id), in which there is no related link (http://www.shopify.co.id/technology/15646068-rebuilding-the-...)
Ah, this is a bug in our redirect system. I will look into it.
fidz: should be fixed now. Thanks for reporting this.
So AFAIK they send html back to the client right? that's my understanding of turbolinks
The problem with this is that it makes client and server tightly coupled.One cannot update the client app without touching the server code.
I understand that's a tradeoff but I think a restful architecture serving only json/xml data is better. And you dont have to duplicate logic that much.If any validation needs to happen clientside,create some validation resource for each model and do it entirely serverside for instance. Even SPAs dont need fat clientside models or a complex service layer on the client.
The problem with this is that it makes client and server tightly coupled.One cannot update the client app without touching the server code.
I understand that's a tradeoff but I think a restful architecture serving only json/xml data is better. And you dont have to duplicate logic that much.If any validation needs to happen clientside,create some validation resource for each model and do it entirely serverside for instance. Even SPAs dont need fat clientside models or a complex service layer on the client.
I don't know that what you've identified there is actually a problem in practice – I'd wager that in a system like Shopify's admin, the client and server are inherently coupled regardless of what architecture you choose. Trying to abstract to a client-side app and JSON API in these situations often results in more complexity for few benefits.
This. We ended up creating models for the exact same stuff in both Ruby and JS. Lots of code duplication, and losing client-side performance due to deserializing JSON all the time.
If you have multiple front ends on top of a shared backend, like a desktop and mobile UI, as well as a bunch of third party integrations, then it makes a lot of sense to build the API first. The thing is that it requires three different use cases before you can make an API generic enough to cover the majority of the needs, and that makes it difficult to build the API first and UI later. I believe in the approach, but it's not the right way to go for everybody.
I guess the key bit of the article is:
"These frameworks work well for highly interactive apps with complex user interface requirements. We only needed a small piece of what a full framework would offer."
As you say its a tradeoff...
"These frameworks work well for highly interactive apps with complex user interface requirements. We only needed a small piece of what a full framework would offer."
As you say its a tradeoff...
But one of the nice things with Angular is that for small things you can get away with almost no code at all, small controllers that isn't really depending on each other for example. It is when you start on a larger system you have to think about splitting your app into services, controllers and templates/views and sending information between all those things.
With all the buzz around SPA frameworks like Angular or Ember it's a worthwhile reminder that there's a still a lot of room in the world for more conventional server-side rendering approaches. Not every web app needs to be an SPA and old-fashioned Rails apps can be a lot simpler to implement.
On the other hand I had to develop a small admin framework and decided to do it with server rendering since I thought it would be easier. I soon found all the small details I needed to think about and that there were a number of server side components/files that I had to create to do what I wanted to do. It would probably have been easier to just do it in the SPA framework I am used to (Angular).
Sometimes I wish I can do more server-side rendering etc.
I have been working on single page applications for awhile now; I miss the ease that is afforded by just capturing state through server-side page transitions.
Specifically, with SPAs (existing codebases now mostly), I run in bugs a lot that involve Backbone events: when are they firing, what part of the codebase is firing them, who is ignoring the global state and firing anyway. Sometimes, looking through a call stack is not even helpful. Every project that I worked on had these problems. The last big bug I dealt with involved, for me, an unfamiliar codebase and about a week-and-a-half of sitting with the debugger tracing every line.
While for new projects, you can work very hard to ensure that firing events is hygienic, existing projects might not be so hygienic.
With a server-centric approach, each page load limits the mental load with state. On page-transition, the page state is clean.
While I think frameworks like Angular is very interesting, I tend to question my own ambition personally. Something like the approach that Turbolinks takes might actually be more appealing.
I have been working on single page applications for awhile now; I miss the ease that is afforded by just capturing state through server-side page transitions.
Specifically, with SPAs (existing codebases now mostly), I run in bugs a lot that involve Backbone events: when are they firing, what part of the codebase is firing them, who is ignoring the global state and firing anyway. Sometimes, looking through a call stack is not even helpful. Every project that I worked on had these problems. The last big bug I dealt with involved, for me, an unfamiliar codebase and about a week-and-a-half of sitting with the debugger tracing every line.
While for new projects, you can work very hard to ensure that firing events is hygienic, existing projects might not be so hygienic.
With a server-centric approach, each page load limits the mental load with state. On page-transition, the page state is clean.
While I think frameworks like Angular is very interesting, I tend to question my own ambition personally. Something like the approach that Turbolinks takes might actually be more appealing.
Yup, also a good reminder that you need to constantly question the design decisions as you go (on an almost daily basis).
Wondering why Shopify redirected my request on that post URL to http://www.shopify.co.id . Ngghhh, i can't read those sweat post :(
I think it is great that they hade the courage to redo something that wasn't working for them anymore.
Duplication and poor documentation made it difficult for developers to make changes to the admin. Once we started questioning Batman, we saw that the evidence strongly indicated it was time to move on to the next chapter.
I don't see a SPA vs Server side rendering debate in the article. Building your own framework was the problem, is not a core competence of Shopify to build JS frameworks. Maybe they could choose other route, like going with React or Angular there.
I don't see a SPA vs Server side rendering debate in the article. Building your own framework was the problem, is not a core competence of Shopify to build JS frameworks. Maybe they could choose other route, like going with React or Angular there.
We actually evaluated other options as well, including Angular, but found them not the best solution for our problem. We documented our experiments and evaluations at the time here: https://gist.github.com/kristianpd/f4c2e0aeb53d09f6def1
I ported one of Shopify's open source projects (their dashboard tool) to Golang a while back, and in the process I also managed to wrangle the Batman.js frontend portion into a Grunt pipeline. It was a challenge, to say the least :p
https://github.com/gigablah/generator-dashing-go/blob/master...
I've been toying with the idea of replacing the whole frontend with something much more lightweight. Perhaps a good time for me to play with Mithril or React...
https://github.com/gigablah/generator-dashing-go/blob/master...
I've been toying with the idea of replacing the whole frontend with something much more lightweight. Perhaps a good time for me to play with Mithril or React...
AFAICT its very rare to find developers with that unique ability to write truly minimalist code. The natural tendency of almost all of the developers I've worked with seems to be to add complexity and to revel in their ability to navigate that complexity. That's why, given the choice, I try to hire or recommend developers who also have a background in music composition and/or visual arts - people who are motivated more by producing an end product that evokes beauty, not by the intricacies of the medium used to produce the work.
Batman was led by someone who previously worked on the Cappuccinno web framework, which is based on its own programming language (Objective-J) and an attempt to port most of the Cocoa framework over to the web environment. Batman was different, but it might say something about its tolerance for complexity. Cross-compilation from 3rd generation language X to 3rd generation language Y has always been a bad idea imho, and seems to be based on a desire to mitigate the effort of learning, much like the tendency to reach for a framework with the most bells and whistles instead of understanding the simplest way to solve a problem. Trying to counter that tendency towards complexity has motivated a few talks I've done at our local JavaScript meetup [2][3][4] (apologies for being overly self-referential).
[1] https://github.com/darrenderidder/talks/tree/master/jswidget...
[2] http://ottawajs.org
[3] http://51elliot.blogspot.com/2012/08/a-simple-intro-to-mvc-p...
[4] http://darrenderidder.github.io/talks/ModulePatterns/#/