Ask HN: Trouble learning things I view as solutions looking for a problem
27 comments
I think a lot of the mis-match comes from scale. Most companies want to be operating at 10X of whatever they currently are doing. So there is a clear bias towards solutions that can auto scale and are easy for growing companies to hire for.
It might make sense for you to bite the bullet and settle down in some out of the way software shop where there is no conceit of needing to scale. State and Local government come to mind. Be prepared for other tradeoffs and inefficiencies.
It might make sense for you to bite the bullet and settle down in some out of the way software shop where there is no conceit of needing to scale. State and Local government come to mind. Be prepared for other tradeoffs and inefficiencies.
one possible way it to realize that it's all re-inventions of the same thing all the way down.
Slack is just IRC with pretty icons.
Reddit is usenet.
Twitter is just AOL AIM away messages.
Etc.
One way that might work is to turn it into a game.
Taking an example from your post, you say: "Why do I need Jenkins and Chef when I can SSH and add scripts and cron jobs."
So work backward and write/post about it.
Starting with SSH, cron, and scripts, here's how you can recreate Jenkins and Chef.
Along the way you'll help other folks who also missed the boat figure out what the new technologies also do as well as give you some fodder for interviews where you can say, I reimplemented from scratch Jenkins using basic technologies and that's how I would use XYZ in this situation.
Or, maybe you could try to focus on jobs where the company is scaling and actually needs to look at whether Chef, Jenkins is the best idea or whether it can be replaced by more optimized technologies that are used to using and have been "battle-tested" to an n-th degree.
Maybe that could help?
Slack is just IRC with pretty icons.
Reddit is usenet.
Twitter is just AOL AIM away messages.
Etc.
One way that might work is to turn it into a game.
Taking an example from your post, you say: "Why do I need Jenkins and Chef when I can SSH and add scripts and cron jobs."
So work backward and write/post about it.
Starting with SSH, cron, and scripts, here's how you can recreate Jenkins and Chef.
Along the way you'll help other folks who also missed the boat figure out what the new technologies also do as well as give you some fodder for interviews where you can say, I reimplemented from scratch Jenkins using basic technologies and that's how I would use XYZ in this situation.
Or, maybe you could try to focus on jobs where the company is scaling and actually needs to look at whether Chef, Jenkins is the best idea or whether it can be replaced by more optimized technologies that are used to using and have been "battle-tested" to an n-th degree.
Maybe that could help?
The famous HN comment on Dropbox. https://news.ycombinator.com/item?id=9224
E.g. "For a Linux user, you can already build such a system yourself quite trivially by getting an FTP account, mounting it locally with curlftpfs, and then using SVN or CVS on the mounted filesystem."
I still don't use dropbox (used it briefly a long time ago), replace ftp with sftp and cvs with git and it might still be true-ish?
Another good thing about re-implementing, often these solutions are like 700 pound gorillas, built fast and trying to move trains not place flowerpots.
Usually the 700 pound gorilla can be gutted, and the danger of not gutting it is that everyone starts to think that having 700 pound gorillas in your backyard is a "normal" part of "modern" living (dystopia living, actually).
Which is how we arrive here: https://spectrum.ieee.org/cloud-computings-coming-energy-cri...
Where arguably the tech industry is fast becoming one of the most wasteful, hurtful things to this planet.
Another good thing about re-implementing, often these solutions are like 700 pound gorillas, built fast and trying to move trains not place flowerpots.
Usually the 700 pound gorilla can be gutted, and the danger of not gutting it is that everyone starts to think that having 700 pound gorillas in your backyard is a "normal" part of "modern" living (dystopia living, actually).
Which is how we arrive here: https://spectrum.ieee.org/cloud-computings-coming-energy-cri...
Where arguably the tech industry is fast becoming one of the most wasteful, hurtful things to this planet.
Right but the value proposition of Dropbox wasn't the tech it was built on, it was the fact they gave you free cloud storage in exchange for an email address.
in fact it was despite the tech.
Most my musician and composer friends were forced to use it because it was the cheapest way to share a few hundred gigabytes of instrument samplers with remote contract workers.
They would reserve a week to deal with botched dropbox client upload/downloads. A freaking week. I convinced a few to just pay postage on a external HDD and bite the bullet if they got damaged. They never looked back.
Most my musician and composer friends were forced to use it because it was the cheapest way to share a few hundred gigabytes of instrument samplers with remote contract workers.
They would reserve a week to deal with botched dropbox client upload/downloads. A freaking week. I convinced a few to just pay postage on a external HDD and bite the bullet if they got damaged. They never looked back.
> it was the fact they gave you free cloud storage in exchange for an email address
This feels rather reductive
This feels rather reductive
I mean, that's what happened. I don't know how else I could phrase this.
I suspect this is fairly common.
But I'm also not sure self study will help unless you're willing to lie on the resume. People often explicitly demand experience with the particular tool.
Which is why we see so much "resume driven development". People know they need experience with tools, so they generate it.
But I'm also not sure self study will help unless you're willing to lie on the resume. People often explicitly demand experience with the particular tool.
Which is why we see so much "resume driven development". People know they need experience with tools, so they generate it.
Keyword is scale, specifically massive scale. If your scope is just a couple of (web)servers, then, yeah, you can stick with your tried-and-tested tools and processes. But once you start managing 10s or 100s of servers, including a requirement to recover (i.e., redeploy) them ASAP after an outage, then your basic tools and processes might not be enough…
[deleted]
I've been in cloud / IT space for about 15ish years now, and at some point in the last few years I became quite jaded with all the new shiny tech things. I had a lot of trouble buying into the K8s ecosystem when I could do magical wonderous things with SSH and Ansible, and most conversations with my colleagues at the time were unconvincing at best - they would say K8s can do blah blah, and I would point to our Ansible playbooks that were already doing the same thing. The problem for me was less about the differences in technology but more about finding the willingness to learn something just because I have to.
I've since learned to recontextualize these things in terms of people. CI/CD isn't good because it's better than SSH, it's good because people who speak English can understand a simple devops pipeline but not my custom SSH wizardry. It's a way of inviting developers and even non-tech mgmt folks into my arcane world of development / production and allowing them to see what's going on under the hood. My incentive now isn't about what CI/CD tech is better or worse, rather does it allow my team / peers to understand what I'm trying to achieve and join in. And ultimately that's what I get paid to do - I don't get paid to do cool tech stuff, I get paid to make other people's work easier, or at least that's how I see devops / CI/CD. I know I can always find easier ways to do things, but will they necessarily understand them?
Just my 2 cents. Hope this helps.
I've since learned to recontextualize these things in terms of people. CI/CD isn't good because it's better than SSH, it's good because people who speak English can understand a simple devops pipeline but not my custom SSH wizardry. It's a way of inviting developers and even non-tech mgmt folks into my arcane world of development / production and allowing them to see what's going on under the hood. My incentive now isn't about what CI/CD tech is better or worse, rather does it allow my team / peers to understand what I'm trying to achieve and join in. And ultimately that's what I get paid to do - I don't get paid to do cool tech stuff, I get paid to make other people's work easier, or at least that's how I see devops / CI/CD. I know I can always find easier ways to do things, but will they necessarily understand them?
Just my 2 cents. Hope this helps.
Very well said.
These tools help businesses move forward without having to involve a programmer for every little change.
Also there are quite a few analytically strong and talented folks out there, who do not know programming. Given the right set of tools, they can make quite an impact.
Definitely true. I’ve also seen even the most technical people benefit greatly from the artifacts produced by CI tools.
When each commit (or each RC etc) has automated regression testing for output quality, performance, or XYZ metric critical for your business, the artifacts/reports from those code changes can be stored and audited when unexpected things happen during a release crunch.
We’ve had teams that shrugged off integrating with our CI tools, then a ”final” manual quality check on their release builds’ output showed significant regressions in completeness/accuracy. They scramble all their troops to run through the merged commits to find the culprit(s), and jeopardize delivery.
A basic CI setup could have posted perf/recall/whatever stats to their PR thread for each merged change, or better yet block the merge at some threshold for regressed metrics.
I definitely had the same view initially of CI, similar to unit test suites or verbose structured logging… things I viewed as complete overkill. What I learned was that these things DO still feel like overkill until the moment you really need the ability to reliably audit how each change affects an evolving large system
When each commit (or each RC etc) has automated regression testing for output quality, performance, or XYZ metric critical for your business, the artifacts/reports from those code changes can be stored and audited when unexpected things happen during a release crunch.
We’ve had teams that shrugged off integrating with our CI tools, then a ”final” manual quality check on their release builds’ output showed significant regressions in completeness/accuracy. They scramble all their troops to run through the merged commits to find the culprit(s), and jeopardize delivery.
A basic CI setup could have posted perf/recall/whatever stats to their PR thread for each merged change, or better yet block the merge at some threshold for regressed metrics.
I definitely had the same view initially of CI, similar to unit test suites or verbose structured logging… things I viewed as complete overkill. What I learned was that these things DO still feel like overkill until the moment you really need the ability to reliably audit how each change affects an evolving large system
"When each commit (or each RC etc) has automated regression testing"
This has a lot in common with old IBM batch processing essentially designed to consume rented resources.
This has a lot in common with old IBM batch processing essentially designed to consume rented resources.
When the teams struggle to get a baseline suite working, adding k8s is not helpful.
> These tools help businesses move forward without having to involve a programmer for every little change.
Guess what — they will hire a programmer to do it anyway. They just can't be arsed as long as there is a warm body to boss around that will do it for them. It also strokes their egos, so why roll up the sleeves?
Same pipe dream as low code/no code tools enabling executives to create workflows on their own, which never happens in real life — it's "I had Kevin do it for me" all the time, every time, where "Kevin" is usually a programmer.
Same pipe dream as imagining everyone contributing to the in-house tools they are using and "collaborating". It just never happens. Everyone is happy to dump their feature request onto Andy who developed them in the first place and whine if the Andy is taking too long to implement them.
No amount of added eye candy and wishful thinking is going to change this. Even discovering Santa mating with Nessie is more likely.
Guess what — they will hire a programmer to do it anyway. They just can't be arsed as long as there is a warm body to boss around that will do it for them. It also strokes their egos, so why roll up the sleeves?
Same pipe dream as low code/no code tools enabling executives to create workflows on their own, which never happens in real life — it's "I had Kevin do it for me" all the time, every time, where "Kevin" is usually a programmer.
Same pipe dream as imagining everyone contributing to the in-house tools they are using and "collaborating". It just never happens. Everyone is happy to dump their feature request onto Andy who developed them in the first place and whine if the Andy is taking too long to implement them.
No amount of added eye candy and wishful thinking is going to change this. Even discovering Santa mating with Nessie is more likely.
You realize how much UX is important in CICD at the same time you realize everything outside of product/IT (and half inside IT) isn't version controlled, change managed, and/or automatically deployed.
More user friendly dev tools are the way to get people to do these things.
More user friendly dev tools are the way to get people to do these things.
For more on these themes, check out Seeing Like a State by James C Scott. What he says about legibility, and especially
how the administrative state necessarily ignores any information which couldn't be made legible to them, helped me understand better how to convey realities to managers working outside the trenches.
https://www.ribbonfarm.com/2010/07/26/a-big-little-idea-call...
https://www.ribbonfarm.com/2010/07/26/a-big-little-idea-call...
It's funny that you say that because no management and barely any devs actually understand k8s, they understand its a resume bullet point and it sounds like the thing so they go for it.
They don't need to care about k8s. They just need to care about the text file I sent them which has HA/DR rules and other things customers might care about which requires their input / signoff. As a trivial example, as long as they understand that "replicas:1" is bad or that increasing the number means we get more processing power, I can reliably talk to them about important customer-impacting things. It's definitely hard to make your entire infrastructure commmunication-friendly but, back to OP's point, some tools let you do this better than others.
The best way to learn any technology is to use it in a project.
If you're between jobs you have more time to spin up weird experimental side-projects than most people.
It's a bit harder to spin up sideprojects for stuff like Kubernetes because it's pretty expensive to run, but there are plenty of related technologies - like Google Cloud Run or GitHub Actions - that are excellent fodder for small experimental projects to kick the wheels on them.
A great thing about side projects is that they're a risk-free environment for experimenting with new technology.
If you're between jobs you have more time to spin up weird experimental side-projects than most people.
It's a bit harder to spin up sideprojects for stuff like Kubernetes because it's pretty expensive to run, but there are plenty of related technologies - like Google Cloud Run or GitHub Actions - that are excellent fodder for small experimental projects to kick the wheels on them.
A great thing about side projects is that they're a risk-free environment for experimenting with new technology.
You should work on developing a growth mindset. It sounds like you have a typical fixed mindset where you believe that the things you learned and developed early in your career are perfectly fine and there's little benefit to extending yourself further. This mindset will not serve you well professionally and I strongly suggest you work on that as a priority.
Specifically to understand the benefits of CI/CD I recommend reading Jez Humble's book on the topic[0]. He's an excellent writer on the topic. You can also google his name and watch some excellent conference talks from him.
The best way to understand CI/CD is through experiencing an organisation that is building and deploying at scale. But you're between a rock and a hard place if you can't find employment without that experience. Jez Humble's books and talks will help you understand the benefits and capabilities of these systems.
[0] https://www.amazon.com.au/dp/0321601912
Specifically to understand the benefits of CI/CD I recommend reading Jez Humble's book on the topic[0]. He's an excellent writer on the topic. You can also google his name and watch some excellent conference talks from him.
The best way to understand CI/CD is through experiencing an organisation that is building and deploying at scale. But you're between a rock and a hard place if you can't find employment without that experience. Jez Humble's books and talks will help you understand the benefits and capabilities of these systems.
[0] https://www.amazon.com.au/dp/0321601912
After working with F100 customers, the number of applications they work with across the organization is simply mindboggling. When you have tens of thousands of applications (not branches) managing this stuff gets out of really quick. Things like license management and tracking what applications are using is no longer just a task that can be manually done.
> It's not that I actually believe most of these things are "useless". But rather that my mind can't draw a good problem scenario for them. So from my POV they look like solutions looking for a problem. Maybe my experience just has no good frame of reference for those things.
For lots of things your frame of reference should probably be sth like replacing very expensive and very limited engineering time with inexpensive tooling. That could be inferior. But I'm far more eng time limited than I am cash limited. Or, particularly for deploys, (1) blue-green deploys are great; and (2) it's easier to hire eng that don't understand how to use rsync or capistrano than it is to hire those that do. Just like it's easier to use github actions to make it very hard to deploy a branch w/ failing tests than it is to just rely on everyone running tests. (Which, in a perfect world, you could do... but here we are.)
For lots of things your frame of reference should probably be sth like replacing very expensive and very limited engineering time with inexpensive tooling. That could be inferior. But I'm far more eng time limited than I am cash limited. Or, particularly for deploys, (1) blue-green deploys are great; and (2) it's easier to hire eng that don't understand how to use rsync or capistrano than it is to hire those that do. Just like it's easier to use github actions to make it very hard to deploy a branch w/ failing tests than it is to just rely on everyone running tests. (Which, in a perfect world, you could do... but here we are.)
Scale, Complexity, Job Specialization, Organizational or Business needs blah blah.
Let me explain.
Imagine you have a team that grows from say 4 to 40 over time. All the initial programmers, business managers etc. have left. One fine day, you get a call from the client saying "The feed has not arrived today, fix it in the next 2 hours or we are going elsewhere (or can you please please please fix it ASAP) ..."
What do you do? How do you prevent this from happening in the first place? What procedures / software do you need to have in place to even know what went wrong etc.
Now let me give examples that these tools help you handle.
1. You need to view all the jobs that run on a daily basis across 400 of your servers. 2. You want to make sure that no changes to anything in production can happen without at least 1 more programmer and 1 more business person signing off on it. 3. A job / script fails partially, how do you assess the impact and move forward (you are not even aware of the existence of this script). 4. You have a backup script / job that needs to run on demand. You want to let your support folks run it or even the business folks, without any programmer intervention (you don't want to get a call every time someone in Japan wants to run this report).
Coming to CI/CD systems, some of their goals are
A. Codify the knowledge into scripts / configs and do not rely in tribal knowledge. B. Ensure that your code is always in a "buildable" state with the new changes (i.e. don't even let bad code in) C. When a build fails, notify the programmers who introduced the bug immediately D. Once a build is complete, send it to other systems for stress testing. ...
Yeah large businesses can have very intricate and complex needs. They rely on generic and configurable software (proven and tested by other large organizations) to meet their operational needs. Also, given that you can hire people that are proficient in these tools, you can focus on the business needs rather then reinventing these tools.
Let me explain.
Imagine you have a team that grows from say 4 to 40 over time. All the initial programmers, business managers etc. have left. One fine day, you get a call from the client saying "The feed has not arrived today, fix it in the next 2 hours or we are going elsewhere (or can you please please please fix it ASAP) ..."
What do you do? How do you prevent this from happening in the first place? What procedures / software do you need to have in place to even know what went wrong etc.
Now let me give examples that these tools help you handle.
1. You need to view all the jobs that run on a daily basis across 400 of your servers. 2. You want to make sure that no changes to anything in production can happen without at least 1 more programmer and 1 more business person signing off on it. 3. A job / script fails partially, how do you assess the impact and move forward (you are not even aware of the existence of this script). 4. You have a backup script / job that needs to run on demand. You want to let your support folks run it or even the business folks, without any programmer intervention (you don't want to get a call every time someone in Japan wants to run this report).
Coming to CI/CD systems, some of their goals are
A. Codify the knowledge into scripts / configs and do not rely in tribal knowledge. B. Ensure that your code is always in a "buildable" state with the new changes (i.e. don't even let bad code in) C. When a build fails, notify the programmers who introduced the bug immediately D. Once a build is complete, send it to other systems for stress testing. ...
Yeah large businesses can have very intricate and complex needs. They rely on generic and configurable software (proven and tested by other large organizations) to meet their operational needs. Also, given that you can hire people that are proficient in these tools, you can focus on the business needs rather then reinventing these tools.
My problem with Jenkins is that it very often leads to a pile of Groovy, that no one really understands and then some things can be only done through Jenkins instead of a developer's machine for example. That's also one of the reasons behind all those YAML-CI/CD configs that are trendy today, which (besides using YAML) I fully agree with. Also I hate that Jenkins pushes secrets through environmental flags.
I understand that for most companies they post a job offer with Jenkins as one of keyword requirements and they are mostly done. That is probably much more important than any actual utility of the tool beyond tens or hundreds of thousands of lines of Groovy that someone has to maintain.
I understand that for most companies they post a job offer with Jenkins as one of keyword requirements and they are mostly done. That is probably much more important than any actual utility of the tool beyond tens or hundreds of thousands of lines of Groovy that someone has to maintain.
Yeah, that's the answer (it's a shame it's currently so low), but I don't think it's in terms the OP will be able to relate to. I'm sure he can achieve all of those examples already.
So I'll try to refine it.
The main reason an organization uses Jenkins or Chief instead of in-house code is because they can just post job positions requiring knowledge on Jenkins or Chief, instead of minutely describing the person's tasks and training newcomers on their internal tools.
So, yeah, those tools do lots of desirable things. And slightly simplify a huge number of tasks. But the main reason one won't find a job without some experience with them is that they act a lot like competence standards on the job market.
And yeah, for a job seeker it does look a lot like they are bullshit. (And they often are - Jenkins and Chief not so much, but others are a lot.) The OP's question is a lot like "why should I learn standard calculus notation if I can calculate everything just fine on my own notation?", and if you replace that with stuff like type theory you will find people actually asking that question. The answer is always the same, it's because you need it to communicate with other people.
So I'll try to refine it.
The main reason an organization uses Jenkins or Chief instead of in-house code is because they can just post job positions requiring knowledge on Jenkins or Chief, instead of minutely describing the person's tasks and training newcomers on their internal tools.
So, yeah, those tools do lots of desirable things. And slightly simplify a huge number of tasks. But the main reason one won't find a job without some experience with them is that they act a lot like competence standards on the job market.
And yeah, for a job seeker it does look a lot like they are bullshit. (And they often are - Jenkins and Chief not so much, but others are a lot.) The OP's question is a lot like "why should I learn standard calculus notation if I can calculate everything just fine on my own notation?", and if you replace that with stuff like type theory you will find people actually asking that question. The answer is always the same, it's because you need it to communicate with other people.
>it's because you need it to communicate with other people.
Yep. Chances are if they use Jenkins, they'll have some vendor plugins for tools they use. Jenkins will download xml/json from the one plugin and upload it to another. Now those systems are coupled via a format and a set of behaviors around it. For a huge portion of corporate users those plugins remove a huge chunk of work they'd have to do internally, and instead of writing something from scratch, they can extend what is already there.
Yep. Chances are if they use Jenkins, they'll have some vendor plugins for tools they use. Jenkins will download xml/json from the one plugin and upload it to another. Now those systems are coupled via a format and a set of behaviors around it. For a huge portion of corporate users those plugins remove a huge chunk of work they'd have to do internally, and instead of writing something from scratch, they can extend what is already there.
Your calculus analogy has me thinking that I may just haven't encountered some pain points where being able to apply certain things that will improve my quality of work. Especially in regards of being a better communicator to be more effective team member, and employee. I also tend to work in silo'd situations.
I mean, I always encounter problems at work, just not the right ones to get me to where I want to be today.
And where I'm in a situation where I'm free to do side projects, it doesn't feel better. I can't effectively come up with new problem scenarios unless they resemble something I encountered in the past.
I mean, I always encounter problems at work, just not the right ones to get me to where I want to be today.
And where I'm in a situation where I'm free to do side projects, it doesn't feel better. I can't effectively come up with new problem scenarios unless they resemble something I encountered in the past.
> And I haven't learned how to use them because I can't come up with problem scenarios where they are a possible solution
Well, you have a problem:
> I have no good answer for them because I haven't really learned how to use them
So if getting a job where that skills is important, learn is your goal.
Well, you have a problem:
> I have no good answer for them because I haven't really learned how to use them
So if getting a job where that skills is important, learn is your goal.
Look up these pieces for some of the why’s of the last decade or two:
“Cattle not pets”
“Html,css,javascript for dinosaurs”
“Hypermedia systems”
Kleppmann’s Data Intensive Apps, and a good Postgres book. Data is the big focus today.
“Cattle not pets”
“Html,css,javascript for dinosaurs”
“Hypermedia systems”
Kleppmann’s Data Intensive Apps, and a good Postgres book. Data is the big focus today.
I used to have the same issue, before it became my job to understand CI/CD tools. Previously, I was a research student for whom Linux, Git and programming were just hobbies; then my interests got me a foot in the door doing something that I found more rewarding: working on HPC and openstack.
Although I'd heard of CI/CD before, I only had vague ideas about what it really meant. Then I had to learn Jenkins, Gerrit and Puppet to do my new job.
I still don't use these tools for my personal side projects, partly because setting them up and maintaining them requires its own overhead. This overhead is easy to justify when you've got a team who is all sharing the resources and you're managing infrastructure with fleets of dozens or hundreds of servers, but it's less easy to justify as a "one man band."
Conclusions:
* Don't worry about it if these tools don't do much for you right now. Not everything is for everyone.
* It might end up making more sense if you get a job where you need to collaborate with other engineers and people are judging your business based on scalability and site reliability metrics.
When you do get a job that requires such tools, this is why the ones you mentioned are important:
* Chef (or Puppet, etc) are necessary to ensure your dozens of hundreds of servers all remain in sync with a consistent and reproducible configuration. We speak of "cattle not pets" -- in part, that means you don't configure machines individually, instead you enrol them into your configuration management system. The configuration management is stored in Git, so Git becomes your source of truth for how your whole fleet is configured.
* Jenkins: It's a featureful service for running a wide variety of automations without needing to rely on third-parties like GitHub runners. Eg when you are collaborating on code, Jenkins can provide automated reviews, like giving you a +1 if the linting checks pass. When you merge the code, Jenkins can run additional tasks like deployment.
Although I'd heard of CI/CD before, I only had vague ideas about what it really meant. Then I had to learn Jenkins, Gerrit and Puppet to do my new job.
I still don't use these tools for my personal side projects, partly because setting them up and maintaining them requires its own overhead. This overhead is easy to justify when you've got a team who is all sharing the resources and you're managing infrastructure with fleets of dozens or hundreds of servers, but it's less easy to justify as a "one man band."
Conclusions:
* Don't worry about it if these tools don't do much for you right now. Not everything is for everyone.
* It might end up making more sense if you get a job where you need to collaborate with other engineers and people are judging your business based on scalability and site reliability metrics.
When you do get a job that requires such tools, this is why the ones you mentioned are important:
* Chef (or Puppet, etc) are necessary to ensure your dozens of hundreds of servers all remain in sync with a consistent and reproducible configuration. We speak of "cattle not pets" -- in part, that means you don't configure machines individually, instead you enrol them into your configuration management system. The configuration management is stored in Git, so Git becomes your source of truth for how your whole fleet is configured.
* Jenkins: It's a featureful service for running a wide variety of automations without needing to rely on third-parties like GitHub runners. Eg when you are collaborating on code, Jenkins can provide automated reviews, like giving you a +1 if the linting checks pass. When you merge the code, Jenkins can run additional tasks like deployment.
The frame of reference to understand these is that they are targeted towards people that are unfamiliar with the preceding technology. The 'new' technology is obfuscated and hyped up in order to differentiate it.
Do you think the people who authored Kubernetes are unfamiliar with Ansible?
They definitely used SaltStack - in the early days there was a bunch of Salt config in the k8s repo. I don't think it lasted long as it all got built around with k8s native infra/functionality.
Maybe... but that would only be because k8s comes out of Google Borg, not Ansible. Ansible wasn't even a thing when Google Borg was already in use at huge scale.
[deleted]
When you get asked questions IMO I would just swap out your experience for equivalent, e.g. do you have Jenkins experience - no, but I have created my own automated CICD processes to build test and push. (Part of the reason to do CICD stuff is to have another computer do that stuff in an org with many people contributing, but it is the same principle as you built locally.)
For cloud I would swap out discussing rsync and SFTP in much the same manner, although that is slightly more of a stretch, most people asking those questions either won't know better or will know idiosyncratic cloud stuff can be learned if you are at that level.
For cloud I would swap out discussing rsync and SFTP in much the same manner, although that is slightly more of a stretch, most people asking those questions either won't know better or will know idiosyncratic cloud stuff can be learned if you are at that level.
> For instance, why do I need Jenkins and Chef when I can already test, SSH and add scripts and cron jobs no problem.
New members of your team will not be familiar with your scripts and cron jobs, but they might know Jenkins and Chef, learn about them from the docs and fix common issues with Stack Overflow.
New members of your team will not be familiar with your scripts and cron jobs, but they might know Jenkins and Chef, learn about them from the docs and fix common issues with Stack Overflow.
Just don’t learn those things. Sounds like your mind simply won’t latch onto knowledge unless it is associated with an authentic problem. My mind also works that way.
People will tell you to suck it up. Don’t. Find interesting problems and then solve them naturally. You will learn lots and it will feel great. I wrote about this in my book Secrets of a Buccaneer-Scholar.
I have used this method for my entire career since I became a professional coder at 16 after dropping out of high school. It may be important to note that I use coding in my work (software testing consultant, trainer, expert witness) but I am independent and don’t do production coding for commercial products. I don’t have the patience for all that. I burned out as a production coder when I was 19, and have been in testing ever since.
In other words— I get to say screw Jenkins— I write my own frameworks and that’s fine.
People will tell you to suck it up. Don’t. Find interesting problems and then solve them naturally. You will learn lots and it will feel great. I wrote about this in my book Secrets of a Buccaneer-Scholar.
I have used this method for my entire career since I became a professional coder at 16 after dropping out of high school. It may be important to note that I use coding in my work (software testing consultant, trainer, expert witness) but I am independent and don’t do production coding for commercial products. I don’t have the patience for all that. I burned out as a production coder when I was 19, and have been in testing ever since.
In other words— I get to say screw Jenkins— I write my own frameworks and that’s fine.
This x100 for me. I absolutely refuse to learn anything new but will go beyond to learn something if its absolutely necessary to solve a painful/interesting problem
I think some comments are missing the points a little bit. It's not just about scale and I think not at all about not needing new addons to the project not wanting to decipher your bash wizardry. Deciphering Jenkins pipeline scripts or Ruby playbooks for Chef is no easier. In fact, fully figuring out what a Chef recipe will do without running it can be quite inscrutable, way more than figuring out a shell script.
But Jenkins and Chef give you servers. The pipelines or recipes are in a central location with access control that can tie into the same sso or ldap controlling access to everything else in your IT infrastructure. They can run automatically without having to install separate cron jobs on every server you own. Your deployment scripts are probably running from your laptop, which is accessible only to you, not always connected to a network, probably not accessible remotely to anyone, may or may not have any of its own backups, might be unmanaged and impossible for a third party to know you're running it securely and it doesn't have malware.
Organizations are trying to avoid having Dennis Nedry control all of the core infrastructure they rely on.
Sure, as an individual consumer, I might have questioned the utility of something like dropbox, and as it stands, I have never personally used it. But I don't question the utility of distributed filesystems existing when you could just have cron run periodic rsyncs between the root filesystems of some arbitrary number of servers. Yeah, ultimately Chef and Jenkins are just doing what your shell scripts are doing. But Chef and Jenkins are real software. They're well-tested, have a formal development and release process that involves peer review, they're used by thousands of other customers, they come with support contracts and SLAs. Your shell scripts don't have any of that.
But Jenkins and Chef give you servers. The pipelines or recipes are in a central location with access control that can tie into the same sso or ldap controlling access to everything else in your IT infrastructure. They can run automatically without having to install separate cron jobs on every server you own. Your deployment scripts are probably running from your laptop, which is accessible only to you, not always connected to a network, probably not accessible remotely to anyone, may or may not have any of its own backups, might be unmanaged and impossible for a third party to know you're running it securely and it doesn't have malware.
Organizations are trying to avoid having Dennis Nedry control all of the core infrastructure they rely on.
Sure, as an individual consumer, I might have questioned the utility of something like dropbox, and as it stands, I have never personally used it. But I don't question the utility of distributed filesystems existing when you could just have cron run periodic rsyncs between the root filesystems of some arbitrary number of servers. Yeah, ultimately Chef and Jenkins are just doing what your shell scripts are doing. But Chef and Jenkins are real software. They're well-tested, have a formal development and release process that involves peer review, they're used by thousands of other customers, they come with support contracts and SLAs. Your shell scripts don't have any of that.
It's your perspective.
The real question is, given all the materials on the internet that can literally tell you why these tools are useful (I mean, surely if you just read the docs you'd find out?!), why have you not done the work?
It feels like you're in your comfort zone and rationalising why you shouldn't do the work. I mean, is finding a job and being employed not motivation enough?
Perhaps look inside yourself and be honest first. It sounds like instead of admitting that you need to do some work and learn some things you'd rather blame outside circumstances for your lack of personal progress.
I don't mean to sound harsh - sometimes we just need a reality check!
Take responsibility, get out there and DO THE WORK. A new world is waiting for you just over the hill.
The real question is, given all the materials on the internet that can literally tell you why these tools are useful (I mean, surely if you just read the docs you'd find out?!), why have you not done the work?
It feels like you're in your comfort zone and rationalising why you shouldn't do the work. I mean, is finding a job and being employed not motivation enough?
Perhaps look inside yourself and be honest first. It sounds like instead of admitting that you need to do some work and learn some things you'd rather blame outside circumstances for your lack of personal progress.
I don't mean to sound harsh - sometimes we just need a reality check!
Take responsibility, get out there and DO THE WORK. A new world is waiting for you just over the hill.
You're correct that I'm in a comfort zone. I can read about why these tools are useful. At the same time, these are tools that solve problems I don't have. How can one practice them like that? My mind is blocked here, I'm unable to come up with situations even "for learning purposes", for side projects, scenarios that would let me understand them better.
I have difficulty coming up with new problem scenarios in my head, unless they resemble something I already encountered in the past. I will have to overcome this, if I were to become a better engineer.
Without having a problem scenario, there's no work to be done. So there's only the job hunt right now.
I have difficulty coming up with new problem scenarios in my head, unless they resemble something I already encountered in the past. I will have to overcome this, if I were to become a better engineer.
Without having a problem scenario, there's no work to be done. So there's only the job hunt right now.
> At the same time, these are tools that solve problems I don't have.
If you're applying for jobs in companies, and those companies want you to have skills in tools x, y, z, then they have a need and they have problem scenarios that will be relevant to those tools.
You can read company blogs - go look at any of the big tech company engineering blogs and they'll go into a lot of depth on the problems they have and how they go about using modern tooling to solve those problems.
The information is out there, if you don't have any ideas, then talk to ChatGPT , read company blogs, buy a book, do a course, put a question into google.
There has never been an easier time to get an answer to a question - you just need to ask it in the first place.
Your job relies on you being a problem solver. Start by solving your own problem, stop with the excuses.
If you're applying for jobs in companies, and those companies want you to have skills in tools x, y, z, then they have a need and they have problem scenarios that will be relevant to those tools.
You can read company blogs - go look at any of the big tech company engineering blogs and they'll go into a lot of depth on the problems they have and how they go about using modern tooling to solve those problems.
The information is out there, if you don't have any ideas, then talk to ChatGPT , read company blogs, buy a book, do a course, put a question into google.
There has never been an easier time to get an answer to a question - you just need to ask it in the first place.
Your job relies on you being a problem solver. Start by solving your own problem, stop with the excuses.
> The real question is, given all the materials on the internet that can literally tell you why these tools are useful (I mean, surely if you just read the docs you'd find out?!), why have you not done the work?
This is it. All of these tools (with a few exceptions) have legitimate use-cases in the tech industry. The fact that OP hasn't clued into what those use-cases are betrays a lack of adaptability and an inability to leave their comfort zone.
I used to be that kind of person so I get it. It's really uncomfortable having to shift your mental model to accomodate a rapidly changing industry like tech but it's absolutely necessary in order to prevent stagnation.
This is it. All of these tools (with a few exceptions) have legitimate use-cases in the tech industry. The fact that OP hasn't clued into what those use-cases are betrays a lack of adaptability and an inability to leave their comfort zone.
I used to be that kind of person so I get it. It's really uncomfortable having to shift your mental model to accomodate a rapidly changing industry like tech but it's absolutely necessary in order to prevent stagnation.
I can read an article explaining how a tool is used.
There are many tools that solve technical and organizational problems I don't have. There is no "work" to be done if I don't have those problems. You're correct that I'm in a comfort zone. When there is no problem, I cannot find agency. The most effective learning I've had is learning on the job, or from being able to draw parallels to past job experiences. Without either of these things my mind is blocked and I feel unable to take action.
I'm unable to create problem scenarios for something that I haven't used before. My perspective doesn't go past- encounter problem first and then see if I should try something else that would work within my constraints.
I wish that mind block could go away.
There are many tools that solve technical and organizational problems I don't have. There is no "work" to be done if I don't have those problems. You're correct that I'm in a comfort zone. When there is no problem, I cannot find agency. The most effective learning I've had is learning on the job, or from being able to draw parallels to past job experiences. Without either of these things my mind is blocked and I feel unable to take action.
I'm unable to create problem scenarios for something that I haven't used before. My perspective doesn't go past- encounter problem first and then see if I should try something else that would work within my constraints.
I wish that mind block could go away.
That's great it will serve you well.
Once you choose a problem to solve, then you'll have the motivation to learn the tools needed to solve it. So find that problem and solve it, and then maybe you won't need to find a job.
Once you choose a problem to solve, then you'll have the motivation to learn the tools needed to solve it. So find that problem and solve it, and then maybe you won't need to find a job.
In a small tech company with a small audience and finite scale, your 2012 approach will still work in 2024
In any company that operates at web scale, where success means suddenly having millions of users, and thousands of employees, you've got to have a robust way to deploy and orchestrate software.
The key point of the cloud, and technologies like Kubernetes, is that they allow you to treat hardware (servers and network infrastructure) as if they were software. Copy and paste a code template, and you have a data centre of your own to play with. To get more servers and route traffic to them, simply change some configuration. And because you're managing your server fleet via configuration, any changes can be easily automated, permissioned, audited, visualised, rolled back, shared, documented etc.
As a solo developer, you could feasibly create a whole company serving an app or website to hundreds of thousands of users before you had to consider employing any more staff. The cloud, and its infrastructure-as-code platform technology makes that possible.
Startups can focus on their core competency, rather than having to build expensive competancies and making huge and risky upfront capital investments in infrastructure.
In any company that operates at web scale, where success means suddenly having millions of users, and thousands of employees, you've got to have a robust way to deploy and orchestrate software.
The key point of the cloud, and technologies like Kubernetes, is that they allow you to treat hardware (servers and network infrastructure) as if they were software. Copy and paste a code template, and you have a data centre of your own to play with. To get more servers and route traffic to them, simply change some configuration. And because you're managing your server fleet via configuration, any changes can be easily automated, permissioned, audited, visualised, rolled back, shared, documented etc.
As a solo developer, you could feasibly create a whole company serving an app or website to hundreds of thousands of users before you had to consider employing any more staff. The cloud, and its infrastructure-as-code platform technology makes that possible.
Startups can focus on their core competency, rather than having to build expensive competancies and making huge and risky upfront capital investments in infrastructure.
[deleted]
Many good responses here about why you actually need those CI/CD tools. From my experience I would summarize it as “standardized, repeatable, automated processes at scale”.
I am surprised though that you are getting those questions about XYZs and how would you use them where XYZ are supposedly CI/CD tools. I personally do not understand the hate standardized LeetCode, system design, and STAR interviews get. It is something well-known that you can prepare for, and ace those interviews even if you never had to (or never will) solve a contest-style problem or design a distributed system. These interviews test your general intelligence, memory, tenacity, ability to learn, and communicate. Who cares if I never used ArgoCD and only used Jenkins or rsync or whatever.
I admit sometimes companies need someone with a specialized knowledge to hit the ground running, but these situations are rare. So if your interview consists mostly of questions like “have you used XYZ” and “how would you use XYZ to do ABC” - those are lazy questions, and an imperfect proxy to measure your professional ability. Usually you don’t want to work for companies that ask those, anyway. Anyone smart and with attitude of getting things done can figure those XYZs if and when needed.
I am surprised though that you are getting those questions about XYZs and how would you use them where XYZ are supposedly CI/CD tools. I personally do not understand the hate standardized LeetCode, system design, and STAR interviews get. It is something well-known that you can prepare for, and ace those interviews even if you never had to (or never will) solve a contest-style problem or design a distributed system. These interviews test your general intelligence, memory, tenacity, ability to learn, and communicate. Who cares if I never used ArgoCD and only used Jenkins or rsync or whatever.
I admit sometimes companies need someone with a specialized knowledge to hit the ground running, but these situations are rare. So if your interview consists mostly of questions like “have you used XYZ” and “how would you use XYZ to do ABC” - those are lazy questions, and an imperfect proxy to measure your professional ability. Usually you don’t want to work for companies that ask those, anyway. Anyone smart and with attitude of getting things done can figure those XYZs if and when needed.
The simple answer is, because that is what is being used.
The complicated answer is there are multiple ways to do things. And just because you know one way doesn't mean you shouldn't learn another. For uploading files you say you need to use one of three things, until you need to upload a file somewhere that doesn't support those things.
You just need to learn some of these technologies, without a frame of reference. Learn a technology to learn a technology.
These solutions do not need to look for a problem. They solve very real pain points. I am old enough that I remember doing the things “ssh to server and rsync” way. Heck I do it sometimes still for my personal projects. Here are the examples of the problems that CI/CD tools solve.
1. You rsync your working directory to a server, ssh to it and restart the server. That’s your only server. While you server restarts, your customers experience downtime. If you used modern tools, your tooling would automatically do rolling deployment or blue green of multiple load balanced instances.
2. Your service does not come up because in your local development environment you used the constructs from Python 3.11 that are not understood by Python 3.8 that is pre-installed on the server. What do you do? Upgrade Python of course. This breaks something else running on the same server. Would not be a problem if you deployed a dockerized app.
3. You figured out the Python version problem (all the while your customers could not access your service). Service starts up and crashes after a couple of minutes. Analyzing the stack trace you realized that this is a bug that would be caught by your unit tests - have you not forgot to run them before the deployment. Would not be a problem if running tests was a pre requisite for deployment.
4. At this point you give up and decide to roll back to the previous version. You go through the manual steps of extracting old version, rsync-ing or ftp-inc it again, restarting the servers (good thing you have only one server, right?). All the while service is still down for your customers. Whereas with modern tooling that would be a push of a button.
5. Your boss calls. They are rightfully angry that customers cannot access the service and asks why you did not ask for permission or a sign off for a prod deployment. You said you confirmed with Sue, verbally, but Sue has left the company and any audit trail of that is lost. Modern tools would give you an audit trail of an approval.
6. Jarred by this experience you decide to deploy to at least two servers. Congratulations - now you have to do twice as much ssh-ing and rsync-ing in a carefully orchestrated manner. Let alone that now you have to deploy some sort of a load balancer too. Modern tools do it for you behind the scenes.
7. Suddenly your product is very successful and you get much more of traffic. Your two servers are barely handling the load. And now you cannot do deployments at all because if you stop one of the servers the other will crash under load immediately. You need more servers. Of course you still use physical machines and you need more of them. You call your procurement. They give you two months of the lead time. On the cloud you would just move a slider in some interface and off you go.
I could continue but hopefully you get the idea.
1. You rsync your working directory to a server, ssh to it and restart the server. That’s your only server. While you server restarts, your customers experience downtime. If you used modern tools, your tooling would automatically do rolling deployment or blue green of multiple load balanced instances.
2. Your service does not come up because in your local development environment you used the constructs from Python 3.11 that are not understood by Python 3.8 that is pre-installed on the server. What do you do? Upgrade Python of course. This breaks something else running on the same server. Would not be a problem if you deployed a dockerized app.
3. You figured out the Python version problem (all the while your customers could not access your service). Service starts up and crashes after a couple of minutes. Analyzing the stack trace you realized that this is a bug that would be caught by your unit tests - have you not forgot to run them before the deployment. Would not be a problem if running tests was a pre requisite for deployment.
4. At this point you give up and decide to roll back to the previous version. You go through the manual steps of extracting old version, rsync-ing or ftp-inc it again, restarting the servers (good thing you have only one server, right?). All the while service is still down for your customers. Whereas with modern tooling that would be a push of a button.
5. Your boss calls. They are rightfully angry that customers cannot access the service and asks why you did not ask for permission or a sign off for a prod deployment. You said you confirmed with Sue, verbally, but Sue has left the company and any audit trail of that is lost. Modern tools would give you an audit trail of an approval.
6. Jarred by this experience you decide to deploy to at least two servers. Congratulations - now you have to do twice as much ssh-ing and rsync-ing in a carefully orchestrated manner. Let alone that now you have to deploy some sort of a load balancer too. Modern tools do it for you behind the scenes.
7. Suddenly your product is very successful and you get much more of traffic. Your two servers are barely handling the load. And now you cannot do deployments at all because if you stop one of the servers the other will crash under load immediately. You need more servers. Of course you still use physical machines and you need more of them. You call your procurement. They give you two months of the lead time. On the cloud you would just move a slider in some interface and off you go.
I could continue but hopefully you get the idea.
This is something that is playing a major part in impeding my progress, blocking my motivation to learning some tech skills that are more in demand. Especially since I currently have no job, so there is no real-world frame of reference to understand why they could be important. Why they could save my butt one day or make my life easier.
At some point, long ago (2012-ish) I was no longer required to do deployment for web, but unfortunately I never caught up because I was never made responsible for production ever again. So I am just familiar with old ways of deployment and missed out on a lot of changes that led to where full-stack web dev is today.
This mostly goes for a lot of skills in CI/CD and cloud that are in demand these days. Some of it comes up more in full-stack job interviews, in terms of, do you have experience with XYZ or how would you use XYZ in some situation. I have no good answer for them because I haven't really learned how to use them. And I haven't learned how to use them because I can't come up with problem scenarios where they are a possible solution. For instance, why do I need Jenkins and Chef when I can already test, SSH and add scripts and cron jobs no problem. Why would I need the cloud when my past jobs and clients got by well with a cheap shared host. For uploading files I just needed to know at least two of these three- Git, rsync, and SFTP.
It's not that I actually believe most of these things are "useless". But rather that my mind can't draw a good problem scenario for them. So from my POV they look like solutions looking for a problem. Maybe my experience just has no good frame of reference for those things.
Discovering a context, a frame of reference to understand these tools and apply them- especially without a job -that's what I want to figure out.