Ask HN: How hard is it to build a platform like Netlify?
I'm fascinated by Netlify, by its speed and neat developer experience. I want to learn more about the inner workings of such platforms and thought that the best way to do it is to build something like Netlify myself. Nothing stellar, just a simple proof of concept clone.
Any suggestions on where to start and where to learn about the architecture of platforms like Netlify? I'm a moderately experienced frontend developer, I have some backend and DevOps skills, but it's mostly "street knowledge," and I don't have a CS degree.
13 comments
If I were to do this, I think I would start somewhere along the lines of:
1 - $5 linode/digital ocean machines at different locations (CA, TX, NJ, London, wherever you want the things)
These machines would run probably Nginx serving static up static content. They'd get this data pushed to them over something like rsync/scp from a central place.
It would answer for the same domains, with the same content at each location.
2 - DNS Route53 / GeoDNS / Round-Robin
Servers in all those locations are cool, but we want to get users close to them. I think Route53 does GeoIP DNS - routing users to the geographically closer machine.
3 - $10/$20 Linode at a central location for build/deploy.
I'd probably look at Drone.io as the thing to take gitrepos and then build the website. This could handle JAM stack stuff pretty easily. Once it has the artifacts created, you ship them off to the linodes in step one to be served.
This is kind of the broad view of where I would start for technologies to help with that kind of thing. Then comes a lot of glue and programs to make it easier to use.
The form submission stuff they made seems pretty cool. You could make something that scans the artifacts looking for forms with a given data attribute, and when you find them build a data model to match and stuff in a POST address. Making that would be really fun.
I hope this helps, I also dig this kind of stuff =)
1 - $5 linode/digital ocean machines at different locations (CA, TX, NJ, London, wherever you want the things)
These machines would run probably Nginx serving static up static content. They'd get this data pushed to them over something like rsync/scp from a central place.
It would answer for the same domains, with the same content at each location.
2 - DNS Route53 / GeoDNS / Round-Robin
Servers in all those locations are cool, but we want to get users close to them. I think Route53 does GeoIP DNS - routing users to the geographically closer machine.
3 - $10/$20 Linode at a central location for build/deploy.
I'd probably look at Drone.io as the thing to take gitrepos and then build the website. This could handle JAM stack stuff pretty easily. Once it has the artifacts created, you ship them off to the linodes in step one to be served.
This is kind of the broad view of where I would start for technologies to help with that kind of thing. Then comes a lot of glue and programs to make it easier to use.
The form submission stuff they made seems pretty cool. You could make something that scans the artifacts looking for forms with a given data attribute, and when you find them build a data model to match and stuff in a POST address. Making that would be really fun.
I hope this helps, I also dig this kind of stuff =)
I guess it depends on how far down the rabbit hole you want to go and what you want to learn?
For example, we recently built something to the same effect to deploy our single page applications to AWS. It creates a new Cloudformation Stacks for each site, sets up CodePipeline for continous deployment, build artifacts are placed in a public S3 bucket, and then it's served through CloudFront as the CDN. Beyond that you just need to set up Route53 for your domain and use the Certificate Manger to handle SSL.
I know that's pretty specific to AWS but hopefully it gives you an idea of where you'd start.
Basically to build it from scratch you'd need: - A service that can respond to GitHub webhooks, build projects in an isolated environment based on a config, and push build artifacts out.
- A content distribution network that you can deploy those artifacts to.
- A DNS server to route traffic.
- An SSL service like Let's Encrypt CertBot to issue certificates periodically.
For example, we recently built something to the same effect to deploy our single page applications to AWS. It creates a new Cloudformation Stacks for each site, sets up CodePipeline for continous deployment, build artifacts are placed in a public S3 bucket, and then it's served through CloudFront as the CDN. Beyond that you just need to set up Route53 for your domain and use the Certificate Manger to handle SSL.
I know that's pretty specific to AWS but hopefully it gives you an idea of where you'd start.
Basically to build it from scratch you'd need: - A service that can respond to GitHub webhooks, build projects in an isolated environment based on a config, and push build artifacts out.
- A content distribution network that you can deploy those artifacts to.
- A DNS server to route traffic.
- An SSL service like Let's Encrypt CertBot to issue certificates periodically.
Please do it. It is reasonably simple to build something like that, I've done it many times for over a decade now - its so much fun and truly rewarding. From the beginning you need to focus on the developer experience that means you need to make sure your execution of all the components is done in a way that allows a developer to develop and not have to click buttons or fill in any forms.
To me the interface is the IDE and important changes in source code happen when I merge branches to master. From there the ingredients are a well developed cicd pipeline plus some integrated infrastructure as code (i.e cloudformation, teraform, ansible). The infrastructure as code can get a little complex and you need to know the the underlying components quite well or it might cost you more than you think. You could achieve this with a vps too if you had one. One of the reasons I use Jekyll is I can host in on s3 which allows us to focus our operational capabilities on our main business, Servana does devops and cloud services.
I use Jenkins with a declarative pipeline plus cloudformation which configures all the infra like s3 bucket and cloudfront distribution etc. I kept it simple but still pro with the main goal that a developer does not have to log into anything to trigger updates. The pipelines creates a release and tags master merges.
When I build these workflows the developer experience is very important so I always make the process completely automated with no interrupts if it can be helped. My focus is on one thing, the code and the development experience.
To me the interface is the IDE and important changes in source code happen when I merge branches to master. From there the ingredients are a well developed cicd pipeline plus some integrated infrastructure as code (i.e cloudformation, teraform, ansible). The infrastructure as code can get a little complex and you need to know the the underlying components quite well or it might cost you more than you think. You could achieve this with a vps too if you had one. One of the reasons I use Jekyll is I can host in on s3 which allows us to focus our operational capabilities on our main business, Servana does devops and cloud services.
I use Jenkins with a declarative pipeline plus cloudformation which configures all the infra like s3 bucket and cloudfront distribution etc. I kept it simple but still pro with the main goal that a developer does not have to log into anything to trigger updates. The pipelines creates a release and tags master merges.
When I build these workflows the developer experience is very important so I always make the process completely automated with no interrupts if it can be helped. My focus is on one thing, the code and the development experience.
In my view, it‘a a rather lot of work. The hard parts are probably less visible: combatting abuse of the free build system, sysadmin/scaling stuff, et c. Getting build scripts running and static pages on a webserver, their main function, is relatively straightforward.
If you want to build one just for yourself, that is probably a weekend project. Doing multitenant safely is likely where it gets a lot more complex.
If you want to build one just for yourself, that is probably a weekend project. Doing multitenant safely is likely where it gets a lot more complex.
I built something similar with the same basic functionality although lacking all the polish. Basically I had a cheap vps with docker installed and a small control panel where a user could log into github and pick a repo and add domain names. There was then a number of glue scripts to pull and build the sites and deploy them as Caddy server containers, this handled all the automatic TLS and static file serving. I had all these load balanced with nginx listening to the docker deamon and restarting. In hindsight the whole thing could have been a good opportunity to learn and use k8s, maybe would have been simpler to piece together but I learned a lot.
Have fun :) https://staticdeploy.io/
It depends on how much of it you want to recreate. Like, do you want to create the servers, or are you happy letting a CDN do the heavy lifting? If it's the latter, you can deploy to GitHub pages, and then use CloudFlare's CDN. I don't think I need to post it here since it's HN, but I documented this technique in a comprehensive article once[0].
On the other end, though, you can buy small storage units and put Raspberry Pi 4s all around the world, and to mimic the CDN-ness of it. You'd need a central build server, which doesn't matter where it physically is, but it'd push JAM builds to your CDN network, and then a DNS server that routes requests to the right CDN node depending on the Geolookup of the IP address.
[0]: https://www.toptal.com/github/unlimited-scale-web-hosting-gi...
On the other end, though, you can buy small storage units and put Raspberry Pi 4s all around the world, and to mimic the CDN-ness of it. You'd need a central build server, which doesn't matter where it physically is, but it'd push JAM builds to your CDN network, and then a DNS server that routes requests to the right CDN node depending on the Geolookup of the IP address.
[0]: https://www.toptal.com/github/unlimited-scale-web-hosting-gi...
I use an execution system to run scripts on remote hosts. I create dns via api calls, configure a web server to accept that domain, which points to a parked landing page. When ready I run another script that pulls the latest tag from git and drops it onto the web server . These pages are served until I make a change to git and push deploy. I would like to make a ui to this but my JS skills suck donkey butt.
Basically I did exactly this :) I created a tool that helps you deploy your static (and hybrid!) app, with a few more features like creating environments, specifying remote configuration and staged rollouts. It took me a year part time, all by myself. Check it out here: https://www.stormkit.io. Basically, you gotta have a nice understanding of the cloud providers, some network knowledge, a lot of backend knowledge and a descent frontend knowledge if you want to compete with their awesome UX :) - Obviously, Netlify has a team behind it, but despite that I believe I did a good job.
Recently I've discovered commonshost and it seems like a great way to host a content. It's a free global CDN with < 30ms for most of the regions I've tested. If you want to startup a node in your region it only costs around 200$ for the equipment to run a node
I can't help but I would also love to learn more about it's inner working!
Especially the certificate fetching and distribution across the edge servers part.
Especially the certificate fetching and distribution across the edge servers part.
I recently used Netlify to deploy a Gatsby site, the experience of deployment is so easy: just link a GitHub repo or drop in the project directory. The Netlify community forums are really helpful too. If you’ve any questions please ask!
Wow savage downvoting on that. I wonder why? Was the comment edited?
I’m surprised that a pretty vanilla comment got so much hate, it’s hard to please everyone!
It has absolutely nothing to do with the question, why wouldn't it be downvoted?
The shade of grey was unreadable that implies a lot of downvotes. This is usually reserved for pure trolling style comments.
I totally misread the question, my bad!
Yeah I recently deployed a Gatsby site too, but opted for Github Pages in the end.