Why Does Developing on Kubernetes Suck?(blog.tilt.dev)
blog.tilt.dev
Why Does Developing on Kubernetes Suck?
https://blog.tilt.dev/2019/08/21/why-does-developing-on-kubernetes-suck.html
86 コメント
I agree with you that many of these are self inflicted problems. I particularly like your answer to #3, which I think I'm going to have to start adopting!
One part that I agree with the author is that local kubernetes clusters or even bare metal clusters are hard. I love being able to just spin up a cluster in GKE and have it just work. It really does just work, especially for ingress and storage. Get one piece working at a time, then you can move it to its final location and do whatever else needs to happen to make that work. It's valuable to have it working in the cloud even if prod is not, since that makes for easier testing/CI.
As for your #4, it is hard, although I've seen some success with running a pod that has tcpdump in it as a sidecar. Example here: https://medium.com/@xxradar/how-to-tcpdump-effectively-in-ku... (haven't used these directions personally, but it looks very similar to what I've seen). Some networking problems will only happen in kubernetes, things like timeout configuration and nginx stuff will just be different, so it's good to be able to do this.
One other note is for logs, just use the helm charts to install kibana/fluentd/elasticsearch into your cluster. It's really easy, and then you don't have to worry about which pod is making the logs, and by default it is tagged with the pod name and other k8s metadata.
One part that I agree with the author is that local kubernetes clusters or even bare metal clusters are hard. I love being able to just spin up a cluster in GKE and have it just work. It really does just work, especially for ingress and storage. Get one piece working at a time, then you can move it to its final location and do whatever else needs to happen to make that work. It's valuable to have it working in the cloud even if prod is not, since that makes for easier testing/CI.
As for your #4, it is hard, although I've seen some success with running a pod that has tcpdump in it as a sidecar. Example here: https://medium.com/@xxradar/how-to-tcpdump-effectively-in-ku... (haven't used these directions personally, but it looks very similar to what I've seen). Some networking problems will only happen in kubernetes, things like timeout configuration and nginx stuff will just be different, so it's good to be able to do this.
One other note is for logs, just use the helm charts to install kibana/fluentd/elasticsearch into your cluster. It's really easy, and then you don't have to worry about which pod is making the logs, and by default it is tagged with the pod name and other k8s metadata.
> that local kubernetes clusters or even bare metal clusters are hard
what's so hard about k3s, kubespray or minikube? the only problem k8s has, that without a linux dev machine you are going through the vm pain. but anything else is just simple.
btw. using mac sucks when trying to develop with local k8s. windows is better/okish, but it's best to run directly on linux.
what's so hard about k3s, kubespray or minikube? the only problem k8s has, that without a linux dev machine you are going through the vm pain. but anything else is just simple.
btw. using mac sucks when trying to develop with local k8s. windows is better/okish, but it's best to run directly on linux.
Why is it worse on the mac than on Windows? Legit question here, I haven't run Windows in many years so wouldn't know.
I find developing on k8s on the mac just fine. Most of the time I either just run the Docker for Desktop k8s feature, or fire up a kind cluster.
kind fits really well with VS Code's Remote - Containers extension for a completely isolated, all batteries included dev experience.
I find developing on k8s on the mac just fine. Most of the time I either just run the Docker for Desktop k8s feature, or fire up a kind cluster.
kind fits really well with VS Code's Remote - Containers extension for a completely isolated, all batteries included dev experience.
Year old experience: minikube requires root if you want to run directly on the host, without a VM layer. Every couple of months it goes into a borked state, which is nigh impossible to debug and sometimes hard to recreate from scratch, possibly due to the root issue above. Got a little luckier with Ubuntu and microk8.
thank you. As for #4, i've heard good things about https://istio.io - supposedly it does provide nice means to gain introspection to the traffic between services.
> 1. Why kubernetes for development? For most cases you don't need the whole orchestration over multiple nodes overhead, docker-compose is just fine. Incidentally, your push/pull problem is solved as you pull the dependencies and BUILD your app locally.
Then it's not hermetic and repeatable. We strive for reproducibility between envs, because it reduces the chance that we're relying on incorrect assumptions about environmental behaviors.
> 2. Want to have someplace to test your kube configurations? That's the ops part of devops. Set up a proper testing cluster instead of trying to cram everything in your devenv. Kube config should be in a separate project anyways.
Is your test env just for you, or is there a chance you could break other engineers? What if they need to leverage the services your team provides in order to test what they're building, but your services are currently broken as a result of your own testing?
Develop in development. Keep staging/test accessible for tests that are minimally intrusive.
> 3. No strace/bash. Look, man, this isn't even kubes' problem. That's how you build your images. Have separate dev images that use prod image as a base. Deploy dev images to staging env. Build dev images locally.
I agree, but it still sucks. We need to get better at this. There should be a means of injecting tooling. Or having it sit on an outer layer only privileged users can execute. But then you also don't want to ship those bytes... It's a problem.
> 4. Network debugging. That's difficult on any system. If you don't have access to the underlying host, you are screwed either way - call your devops for support. Anything else (internal ips, svc endpoints, ingresses) is trivial to debug on kube. However, we're talking about development... just don't use kube for development, period. Use docker-compose. Which, actually solves all your problems... right tools for the right job, or suffer the self-inflicted consequences.
It's dangerous to get away from a real production config. It's easy to work yourself into a world where things don't work the same. Prod could talk over abstract sockets to an envoy control plane with additional smarts for retry or forwarding behavior, and you might miss some of this behavior in development if you swapped out grpc ingress.
Then it's not hermetic and repeatable. We strive for reproducibility between envs, because it reduces the chance that we're relying on incorrect assumptions about environmental behaviors.
> 2. Want to have someplace to test your kube configurations? That's the ops part of devops. Set up a proper testing cluster instead of trying to cram everything in your devenv. Kube config should be in a separate project anyways.
Is your test env just for you, or is there a chance you could break other engineers? What if they need to leverage the services your team provides in order to test what they're building, but your services are currently broken as a result of your own testing?
Develop in development. Keep staging/test accessible for tests that are minimally intrusive.
> 3. No strace/bash. Look, man, this isn't even kubes' problem. That's how you build your images. Have separate dev images that use prod image as a base. Deploy dev images to staging env. Build dev images locally.
I agree, but it still sucks. We need to get better at this. There should be a means of injecting tooling. Or having it sit on an outer layer only privileged users can execute. But then you also don't want to ship those bytes... It's a problem.
> 4. Network debugging. That's difficult on any system. If you don't have access to the underlying host, you are screwed either way - call your devops for support. Anything else (internal ips, svc endpoints, ingresses) is trivial to debug on kube. However, we're talking about development... just don't use kube for development, period. Use docker-compose. Which, actually solves all your problems... right tools for the right job, or suffer the self-inflicted consequences.
It's dangerous to get away from a real production config. It's easy to work yourself into a world where things don't work the same. Prod could talk over abstract sockets to an envoy control plane with additional smarts for retry or forwarding behavior, and you might miss some of this behavior in development if you swapped out grpc ingress.
I think your team is missing a real testing pipeline that validates your changes against a production-like config on a test-kube..
Like someone higher said:
Develop using a docker-compose, when you think you are done -> run a test pipeline with your versions. That pipeline should be able to validate whether you did not break anything.
Dev_Ops means you do both but doesnt exactly mean you have to do both at the same time..
Most of the issues author described were rether self-imposed.
Like someone higher said:
Develop using a docker-compose, when you think you are done -> run a test pipeline with your versions. That pipeline should be able to validate whether you did not break anything.
Dev_Ops means you do both but doesnt exactly mean you have to do both at the same time..
Most of the issues author described were rether self-imposed.
> Then it's not hermetic and repeatable.
in my experience, the core difference between local env and production in kubernetes is configuration of baremetal nodes and the firewall between them. In essence, if your docker-compose does not cover the logical equivalent of your cluster... well, you have architecture problems, and you should probably not solve them just by throwing kubernetes at it. Either that, or, covering your next point
> Is your test env just for you, or is there a chance you could break other engineers?
if your system requires significantly more resources or for some other reason you just can not bring the whole system up via docker-compose and you just really need to have a staging-level-thing-in-development, build a separate cluster for development. Leverage kubernetes namespaces so that each developer could have his/her own little namespace-sandbox-kube. I've seen that working pretty well in practice.
> There should be a means of injecting tooling
i agree that https://github.com/kubernetes/kubernetes/issues/30656 should have been closed long ago, but, again, this is not something kube should (or could) solve. Consider the simple example of your base images - one is using alpine, the other - debian. Injecting tooling is very different on these two. This could be in theory solved by injecting statically-linked executables, but, boy, this is another can of worms i'd rather leave in the hands of end-users...
> It's dangerous to get away from a real production config.
you will get away from a real production config. Your actual cluster nodes have a different configuration already. And, in my experience, that is more of an issue regarding network than additional ingress smarts. And, again, you can run a local version of the ingress as well if you need to replicate its' behavior (ingress is usually just managed nginx anyway, if you're not into exotics like Kong)... And if your control planes' smarts are undocumented and really require a full blown grpc-ingress with all the bells and whistles that kube brings, well... something's rotten in Denmark then...
in my experience, the core difference between local env and production in kubernetes is configuration of baremetal nodes and the firewall between them. In essence, if your docker-compose does not cover the logical equivalent of your cluster... well, you have architecture problems, and you should probably not solve them just by throwing kubernetes at it. Either that, or, covering your next point
> Is your test env just for you, or is there a chance you could break other engineers?
if your system requires significantly more resources or for some other reason you just can not bring the whole system up via docker-compose and you just really need to have a staging-level-thing-in-development, build a separate cluster for development. Leverage kubernetes namespaces so that each developer could have his/her own little namespace-sandbox-kube. I've seen that working pretty well in practice.
> There should be a means of injecting tooling
i agree that https://github.com/kubernetes/kubernetes/issues/30656 should have been closed long ago, but, again, this is not something kube should (or could) solve. Consider the simple example of your base images - one is using alpine, the other - debian. Injecting tooling is very different on these two. This could be in theory solved by injecting statically-linked executables, but, boy, this is another can of worms i'd rather leave in the hands of end-users...
> It's dangerous to get away from a real production config.
you will get away from a real production config. Your actual cluster nodes have a different configuration already. And, in my experience, that is more of an issue regarding network than additional ingress smarts. And, again, you can run a local version of the ingress as well if you need to replicate its' behavior (ingress is usually just managed nginx anyway, if you're not into exotics like Kong)... And if your control planes' smarts are undocumented and really require a full blown grpc-ingress with all the bells and whistles that kube brings, well... something's rotten in Denmark then...
1. Kubernetes is when your service needs google kind of load which 90% of systems don't. Just trade it for simple VM or use LXD containers. Don't jump on next hype cycle. If something works for Google does not mean will work for you.
2. Don't spend too much on it just use traditional knowledge and, you can still use bare-metal, VM or LXD container. Focus on application not on programming a tool designed to solve problems for Google size deployment.
3. Kubernetes is a completely different way of doing things. So if you want to leverage old cluster and distributed system knowledge you can use VM or LXD containers. Kubernetes is an overkill.
4. Abandon kubernetes instead of complaining, just use simple systems.
2. Don't spend too much on it just use traditional knowledge and, you can still use bare-metal, VM or LXD container. Focus on application not on programming a tool designed to solve problems for Google size deployment.
3. Kubernetes is a completely different way of doing things. So if you want to leverage old cluster and distributed system knowledge you can use VM or LXD containers. Kubernetes is an overkill.
4. Abandon kubernetes instead of complaining, just use simple systems.
> Kubernetes is when your service needs google kind of load which 90% of systems don't.
This is the most frustrating but oft repeated nonsense about k8s.
Kubernetes is an amazingly helpful abstraction for any amount of load on any larger than trivially tiny set of services/storage/etc.
When you, like 90% of systems, don't have Google kind of load, you don't need Google number of nodes.
Machine count corresponds to load; abstraction doesn't.
This is the most frustrating but oft repeated nonsense about k8s.
Kubernetes is an amazingly helpful abstraction for any amount of load on any larger than trivially tiny set of services/storage/etc.
When you, like 90% of systems, don't have Google kind of load, you don't need Google number of nodes.
Machine count corresponds to load; abstraction doesn't.
Kubernetes may be a useful framework, but "amazingly helpful" is not a word I'd use for it. To me, it's a half-assed collection of five hundred nearly identical (but all slightly different) yaml files masquerading as "abstraction". And because they're yaml you can't refactor out the common part.
Well, to be fair, yaml itself doesn't exactly have a stellar reputation, but I've never seen fifty lines of yaml that could convey so little information in any other place.
Well, to be fair, yaml itself doesn't exactly have a stellar reputation, but I've never seen fifty lines of yaml that could convey so little information in any other place.
Use a Helm chart (or alternative)?
When you're writing YAML you can absolutely include from elsewhere (it calls it 'aliases', which is... Well, at least it exists) in multuple places - though annoyingly not from other files.
When you're reading (partially) generated YAML, (e.g. kubectl describe or edit) it's IMO a feature that everything is in its place with no indirection. (And as far as I'm aware this is YAML, not kubectl.)
When you're writing YAML you can absolutely include from elsewhere (it calls it 'aliases', which is... Well, at least it exists) in multuple places - though annoyingly not from other files.
When you're reading (partially) generated YAML, (e.g. kubectl describe or edit) it's IMO a feature that everything is in its place with no indirection. (And as far as I'm aware this is YAML, not kubectl.)
> Machine count corresponds to load; abstraction doesn't.
K8s is developed to manage large number of service nodes running docker or oci containers coming from Google borg project. So it was designed for managing google level of service loads. It doesn't help when a service is only for few million users with few hundred or thousand containers. LXD and traditional tools can already do it without learning new way with lots of moving parts.
K8s is developed to manage large number of service nodes running docker or oci containers coming from Google borg project. So it was designed for managing google level of service loads. It doesn't help when a service is only for few million users with few hundred or thousand containers. LXD and traditional tools can already do it without learning new way with lots of moving parts.
> It doesn't help when a service is only for few million users with few hundred or thousand containers.
Well my company switched from VMs to k8s (few hundred containers) and it helped. Proof me wrong.
Well my company switched from VMs to k8s (few hundred containers) and it helped. Proof me wrong.
My company switched from VMs to a managed k8s cluster (gcloud). It made everything harder, the application became about 5x slower and instead of a $300 monthly bill we paid for the VMs now we pay $1300.
So in summary:
* we still have same amount of developers
* we still need same amount of sysadmins
* we introduced more complexity
* the application became slower
* the whole thing became much more expensive (managed k8s price, education, consulting, etc.)
* but! we have the same amount of income every month. Surprisingly our users don't pay more for the same service because we run our stuff on k8s
So in summary:
* we still have same amount of developers
* we still need same amount of sysadmins
* we introduced more complexity
* the application became slower
* the whole thing became much more expensive (managed k8s price, education, consulting, etc.)
* but! we have the same amount of income every month. Surprisingly our users don't pay more for the same service because we run our stuff on k8s
Cannot comment on it unless see it in details that adding additional moving components of k8s helped.
Do you manage Kubernetes, personally? Did you manage the VMs?
Yes, i'm a DevOps engineer. I have to say, we use the Google Cloud and many Google Services, this makes live so much easier. Before that we managed everything ourselves on bare metal with debian VMs.
Not exactly my point. Are you comparing managing bare metal VMs with using a managed Kubernetes cluster? That's hardly a fair comparison.
> Kubernetes is an amazingly helpful abstraction for any amount of load on any larger than trivially tiny set of services/storage/etc.
Abstractions have a cost. You either pay the cloud provider or you pay the Ops team that runs the cluster.
If you're fine with that and have evaluated the costs correctly, great!
Abstractions have a cost. You either pay the cloud provider or you pay the Ops team that runs the cluster.
If you're fine with that and have evaluated the costs correctly, great!
This meme needs to die. Kubernetes is not overkill for non-Google workloads.
In my current work, we run several Kubernetes clusters via GKE on Google Cloud Platform. We're a tiny company — less than 20 nodes running web apps, microservices and search engines — but we're benefiting hugely from the operational simplicity of Kubernetes. Much, much, much better than the old fleet of Puppet-managed VMs we used to run. Having surveyed the competition (Docker Swarm, Mesos/Marathon, Rancher, Nomad, LXD, etc.), I'm also confident that Kubernetes was the right choice.
Kubernetes may be a large and complex project, but the problem it solves is also complex. Its higher-level cluster primitives are vastly better adapted to modern operations than the "simple" Unix model of daemons and SSH and what not. The attraction isn't just the encapsulation that comes with containers, but the platform that virtualizes physical nodes and allows containers to be treated as ephemeral workloads, along with supporting primitives like persistent volumes, services, ingresses and secrets, and declarative rules like horizontal autoscalers and disruption budgets. Given this platform, you have a "serverless"-like magically scaling machine full of tools at your fingertips. You don't need a huge workload to benefit from that.
As for LXD and the other alternatives, these might be great, but Kubernetes has vastly more mainstream adoption these days. That's more important than mere technological superiority, or everyone would be using OpenSolaris with Zones and ZFS these days.
In my current work, we run several Kubernetes clusters via GKE on Google Cloud Platform. We're a tiny company — less than 20 nodes running web apps, microservices and search engines — but we're benefiting hugely from the operational simplicity of Kubernetes. Much, much, much better than the old fleet of Puppet-managed VMs we used to run. Having surveyed the competition (Docker Swarm, Mesos/Marathon, Rancher, Nomad, LXD, etc.), I'm also confident that Kubernetes was the right choice.
Kubernetes may be a large and complex project, but the problem it solves is also complex. Its higher-level cluster primitives are vastly better adapted to modern operations than the "simple" Unix model of daemons and SSH and what not. The attraction isn't just the encapsulation that comes with containers, but the platform that virtualizes physical nodes and allows containers to be treated as ephemeral workloads, along with supporting primitives like persistent volumes, services, ingresses and secrets, and declarative rules like horizontal autoscalers and disruption budgets. Given this platform, you have a "serverless"-like magically scaling machine full of tools at your fingertips. You don't need a huge workload to benefit from that.
As for LXD and the other alternatives, these might be great, but Kubernetes has vastly more mainstream adoption these days. That's more important than mere technological superiority, or everyone would be using OpenSolaris with Zones and ZFS these days.
This misses the genius of k8s. When you adopt it, you’re adopting a completely declarative way of provisioning your system. It’s a total paradigm shift and I will never go back. My big want, is a kube as a service for small workloads - I.e I can just apply a deployment and everything goes up, no cluster considerations or whatever.
It's good if you found it working for you. If your team can grok over all of k8s complexity and moving parts, stick with it, use the tools your team is comfortable with. I tried working with docker since 2013 and later with Kubernetes, I still go back to LXD. In my view less systems and moving parts less complexity, maintenance and cognitive load. On top of it I don't have to give up decades of knowledge I learned regarding cluster and distributed computing on bare-metal and VM as I can apply the same to LXD. Obviously I can do that with Kata containers, but they didn't fit with my team either. Also for most of services with users in the range of 160-180 million LXD work fine.
Managing a VM vs a managed Kubernetes cluster? I'd choose kubernetes without hesitating
Don't manage VM, try LXD containers [1],
they are containers with much better security and you can still use the same tools you learned for last decade or two to manage including the DSL if you like that.
Kubernetes is an Overkill for majority of development and production systems.
[1] https://linuxcontainers.org/
Kubernetes is an Overkill for majority of development and production systems.
[1] https://linuxcontainers.org/
> Managing a VM vs a managed Kubernetes cluster?
So the win is not having to manage something :-) How is that a fair comparison?
So the win is not having to manage something :-) How is that a fair comparison?
> So the win is not having to manage something
Exactly!
> How is that a fair comparison?
Not only is it a fair comparison, but I'd say it's the entire raison d'etre of using any orchestration platform: You get to farm out bits of the infrastructure to others. Getting on k8s enables you to not have to run your own infrastructure, whereas having VMs being your abstraction sorta by definition does not.
Exactly!
> How is that a fair comparison?
Not only is it a fair comparison, but I'd say it's the entire raison d'etre of using any orchestration platform: You get to farm out bits of the infrastructure to others. Getting on k8s enables you to not have to run your own infrastructure, whereas having VMs being your abstraction sorta by definition does not.
for development, i agree. For production/testing/staging - hmmmm, it depends. I have taken a particular liking to the self-healing/HA properties of kubernetes. You just swap the dead nodes out without significant impact on the production... it's awesome.
The trick is to understand that most products don't need to have 99% uptime. Your business will be fine if you are offline once in a while.
Hell, the stock market goes offline every night.
Those last % of uptime are very, very expensive at every level (money, people, logistic...). They better be worth it.
Hell, the stock market goes offline every night.
Those last % of uptime are very, very expensive at every level (money, people, logistic...). They better be worth it.
There's a difference between going offline at a set time for a set amount of time, and going offline randomly for unpredictable amounts of time.
HA is not just about getting that sixth 9, it's also about the peace of mind that you are insulated from a range of problems. Not only would I not worry about our SQL service going down, I also don't have to worry about what I have to do when a node goes down, because that's automated.
HA is not just about getting that sixth 9, it's also about the peace of mind that you are insulated from a range of problems. Not only would I not worry about our SQL service going down, I also don't have to worry about what I have to do when a node goes down, because that's automated.
Even an occasional site crash is ok. I'm not saying it's a good user xp, and avoiding it is both better for the tech team and the marketting one.
But the cost of a 500 is always to be balanced with the cost of going full throttle with micro services.
As perfectionists, it's easy to fall into the trap of choosing the later by default.
But the cost of a 500 is always to be balanced with the cost of going full throttle with micro services.
As perfectionists, it's easy to fall into the trap of choosing the later by default.
What do micro services have to do with it? If anything, HA is harder to achieve with microservices. There's all these weird false dichotomies going around in this thread.
Exactly... HA is about being able to sleep more peacefully each night.
> docker-compose is just fine
Not when part of your dev use macOS and another part uses linux: gotta love how docker manages file permissions between host and containers.
Not when part of your dev use macOS and another part uses linux: gotta love how docker manages file permissions between host and containers.
docker on mac is a pain either way, since it sits in a linux vm and talks to mac fs via nfs. And as for permissions... well, it's been pretty much covered long time now - e.g. https://www.berthon.eu/2018/containers-volumes-and-file-perm...
hope this makes your life a bit easier.
hope this makes your life a bit easier.
> Why Does Developing on Kubernetes Suck ?
IMHO because we are in a phase of transition.
Having worked for years in software industry , I'm convinced we are halfway to a much bigger transformation for Software Engineers / SRE , Developers etc...
I work in a Neobank ( N26 , Revolut, etc...) , we are currently in the process of re-writing our entire Core Banking System with MicroServices on top of Kubernetes with Kafka.
Not a single day pass without having engineers needing to have an exchange about defining basically all of the the terms that exist within the K8/Docker/Kafa world.
- What's a Pod ? How does a pod behave if Kafa goes down ? Do we really need ZooKeeper etc....
Their workflows is insanely complex and requires hours if not a day to deploy a single change... obviously let's not even talk about the amount of work our SRE has in the pipe to "package" the entire stack of 150+ services in K8 through a single YAML file....
I'm sure this complexity is temporary. One tech will automate all of this away and building , deploying and running will be much simpler. In it's current shape , K8 remains me of mainframe ecosystem I used to deal with at brick & mortars banks.
Powerful systems , but they requires a tremendous amount of work and experts to be properly managed and take full advantage of it. Just like mainframes , K8 leaves very little room for "mistakes" or "approximation".
IMHO because we are in a phase of transition.
Having worked for years in software industry , I'm convinced we are halfway to a much bigger transformation for Software Engineers / SRE , Developers etc...
I work in a Neobank ( N26 , Revolut, etc...) , we are currently in the process of re-writing our entire Core Banking System with MicroServices on top of Kubernetes with Kafka.
Not a single day pass without having engineers needing to have an exchange about defining basically all of the the terms that exist within the K8/Docker/Kafa world.
- What's a Pod ? How does a pod behave if Kafa goes down ? Do we really need ZooKeeper etc....
Their workflows is insanely complex and requires hours if not a day to deploy a single change... obviously let's not even talk about the amount of work our SRE has in the pipe to "package" the entire stack of 150+ services in K8 through a single YAML file....
I'm sure this complexity is temporary. One tech will automate all of this away and building , deploying and running will be much simpler. In it's current shape , K8 remains me of mainframe ecosystem I used to deal with at brick & mortars banks.
Powerful systems , but they requires a tremendous amount of work and experts to be properly managed and take full advantage of it. Just like mainframes , K8 leaves very little room for "mistakes" or "approximation".
There is a lot of terminology for sure. We've gone from "is docker ready for production usage?" to "Hey Bob, look at this, a service which isn't running in a container yet" in about 5 years.
In order to work effectively within kubernetes you've got to understand docker, with all of its nomenclature (images, containers, volumes, layers etc). Then you've got the things kubernetes layers on top (pods, ingress, stateful sets, persistent volumes etc).
You then probably layer Helm (adding tiller, deployments, charts, templates) or something else on top of that.
We've still got the complexity from before that, remember when ansible, puppet and chef were all the rage? You're probably still using that to initially configure your bare metal.
You still have to understand all of the quirks of your data store, except now instead of throwing everything in a huge SQL DB you have some things in Redis, MongoDB, Kafka and maybe some legacy stuff in that big DB in the corner.
Let's not forget your CI stack in Jenkins, Travis or maybe codefresh which builds and runs your tests. That probably does some magic with docker too.
Oh, and if anything goes wrong in any of that you'd better be well grounded in general unixy debugging (permissions, networking etc).
It's certainly getting to the point that I feel no one person can be an expert in all of it, especially because when it all works you can more or less let it do its thing. When it breaks though it feels like playing Jenga.
In order to work effectively within kubernetes you've got to understand docker, with all of its nomenclature (images, containers, volumes, layers etc). Then you've got the things kubernetes layers on top (pods, ingress, stateful sets, persistent volumes etc).
You then probably layer Helm (adding tiller, deployments, charts, templates) or something else on top of that.
We've still got the complexity from before that, remember when ansible, puppet and chef were all the rage? You're probably still using that to initially configure your bare metal.
You still have to understand all of the quirks of your data store, except now instead of throwing everything in a huge SQL DB you have some things in Redis, MongoDB, Kafka and maybe some legacy stuff in that big DB in the corner.
Let's not forget your CI stack in Jenkins, Travis or maybe codefresh which builds and runs your tests. That probably does some magic with docker too.
Oh, and if anything goes wrong in any of that you'd better be well grounded in general unixy debugging (permissions, networking etc).
It's certainly getting to the point that I feel no one person can be an expert in all of it, especially because when it all works you can more or less let it do its thing. When it breaks though it feels like playing Jenga.
> Their workflows is insanely complex and requires hours if not a day to deploy a single change... obviously let's not even talk about the amount of work our SRE has in the pipe to "package" the entire stack of 150+ services in K8 through a single YAML file....
One should always keep in mind the famous aphorism:
> "All problems in computer science can be solved by another level of indirection, except for the problem of too many levels of indirection".
You may have just hit the "except" part.
One should always keep in mind the famous aphorism:
> "All problems in computer science can be solved by another level of indirection, except for the problem of too many levels of indirection".
You may have just hit the "except" part.
That's a fantastic quote. Don't know how I haven't come across it before. It definitely sums up our profession in a nutshell.
Something is very wrong at this company if developers need hours or even days to deploy changes in a microservice architecture. I suggest to find an outside consultant with verifiable Kubernetes experience in production to get this checked out. Maybe a bit of outside perspective will help your teams to fix this.
Good luck.
Good luck.
Legacy banking companies have very strict change management systems. These systems usually require many physical eyes on each change. This magnifests into a fear of fully automated change management.
>Their workflows is insanely complex and requires hours if not a day to deploy a single change...
Isn't this what micro-services were supposed to be the solution to? I get the feeling the whole thing is oversold.
Isn't this what micro-services were supposed to be the solution to? I get the feeling the whole thing is oversold.
Isn't it kind of defeating the purpose of micro-services if you deploy every one of them every time?
>I'm sure this complexity is temporary. One tech will automate all of this away and building , deploying and running will be much simpler.
Kubernetes was supposed to be one of those technologies.
Kubernetes was supposed to be one of those technologies.
> I'm convinced we are halfway to a much bigger transformation for Software Engineers / SRE , Developers etc...
I think kubernetes is the intro that gave everyone a taste of whats possible. What's yet to come is an easy abstraction layer or OS integration that makes it possible to use without learning 100 bits of new terminology and config minutiae.
I think kubernetes is the intro that gave everyone a taste of whats possible. What's yet to come is an easy abstraction layer or OS integration that makes it possible to use without learning 100 bits of new terminology and config minutiae.
You need to upgrade to the new Zookeeper 2: Zookeepier!
https://www.youtube.com/watch?v=_F-RyuDLR4o
https://www.youtube.com/watch?v=_F-RyuDLR4o
I think the author is doing it wrong personally. Just spin up a real dev cluster and use that? Why putz around with setups on your laptop? That’s never going to scale team-wise? And if you damn well insist on running your own local k8s cluster for development then the whole thing should be automated infrastructure as code anyways that you start with a command so that other developers will get the same behavior.
Even after local dev the changes should be getting picked up and tested by a testing cluster.
I think leaving it up to devs to come up with their own local testing k8s is asking for bugs.
Even after local dev the changes should be getting picked up and tested by a testing cluster.
I think leaving it up to devs to come up with their own local testing k8s is asking for bugs.
Ability to run locally a production like setup (at smaller scale of course) is a big plus imho. 1st, you know how it works, 2nd smaller iterations/feedaback cycles (restart locally vs. push and wait for tests to run).
I agree with this. But usually you have both. A staging/test environment is nice, but if you have 5 developers and one test env then devs will quickly end up in line waiting for the test env to free up.
Remote debugging is also hard. Much easier when running locally.
Remote debugging is also hard. Much easier when running locally.
The idea is that you have a dev cluster _per developer_. I've ran with something similar way back working for a certain notable Perl shop way before k8s/Docker became popular, and I have to say it has a lot going for it, especially for more complex setups and more annoying database stuff. It was an (on-demand one-click provisioned) _VM_ per service per dev, though, so the biggest annoyance (no code reload, slow fs sync) was nullified by popular preference for either Vim or Emacs — with my current preference for VS Code I'd probably be annoyed by it. Also it's a bit expensive, I guess, but you can stuff a lot of VMs into a single big server.
What an immature and rude way to criticize a young open source project. I’m disappointed that so many in our community appreciate this disrespectful writing style criticizing our k8s supporting peers.
Secondly, developing on Kubernetes “sucks” compared to what? Mesos? Docker Swarm?
Maybe this engineer is still frustrated with Tilt’s failure as a business (https://www.fastcompany.com/3069164/how-tilt-veered-off-cour...) and Airbnb imposing changes on his workflows like how they deploy their apps. If that’s true then I hope he finds more healthy and mature habits to manage his anger.
Secondly, developing on Kubernetes “sucks” compared to what? Mesos? Docker Swarm?
Maybe this engineer is still frustrated with Tilt’s failure as a business (https://www.fastcompany.com/3069164/how-tilt-veered-off-cour...) and Airbnb imposing changes on his workflows like how they deploy their apps. If that’s true then I hope he finds more healthy and mature habits to manage his anger.
> Secondly, developing on Kubernetes “sucks” compared to what? Mesos? Docker Swarm?
My thoughts exactly. I feel like most people who complain about these things are just feeling grumpy about the overall experience, and aren't always quite sure where to place the blame.
It takes a lot of introspection to know exactly where a system should be better than it is... "this yaml is too complex", until you start thinking about how you'd do it better, and then you realize all the problems each of the similar approaches have, and that a lot of it was done for a reason, etc.
Or you start comparing k8s to completely different approaches like just rsyncing some files to a remote webserver and SIGHUP'ing it, which is much simpler but has its own host of reliability/testability/reproducability concerns.
I think in reality, people are overwhelmed by what it really takes to adopt best practices (declarative deployment, CI/CD, health checks, service discovery, etc). Practices that have been hard fought and discovered over many years of people trying things and failing. K8S IMO represents the state of the art in a lot of them, but too often people place blame on k8s when what they're really doing is questioning the best practices themselves. Practices that are also shared across k8s's competitors like mesos and docker swarm.
My thoughts exactly. I feel like most people who complain about these things are just feeling grumpy about the overall experience, and aren't always quite sure where to place the blame.
It takes a lot of introspection to know exactly where a system should be better than it is... "this yaml is too complex", until you start thinking about how you'd do it better, and then you realize all the problems each of the similar approaches have, and that a lot of it was done for a reason, etc.
Or you start comparing k8s to completely different approaches like just rsyncing some files to a remote webserver and SIGHUP'ing it, which is much simpler but has its own host of reliability/testability/reproducability concerns.
I think in reality, people are overwhelmed by what it really takes to adopt best practices (declarative deployment, CI/CD, health checks, service discovery, etc). Practices that have been hard fought and discovered over many years of people trying things and failing. K8S IMO represents the state of the art in a lot of them, but too often people place blame on k8s when what they're really doing is questioning the best practices themselves. Practices that are also shared across k8s's competitors like mesos and docker swarm.
I share the vision of this article. We are working on the same problem at Okteto Inc.
As an ex-docker employee, I love to work with docker-compose and I agree it is usually a good enough abstraction for development.
But it is not only about the tool, I think the future of development is on the cloud and Kubernetes is a perfect fit for ephemeral dev environments:
- You have unlimited hardware and network resources but efficiently shared by all your team.
- You can share endpoints for fast validation and easier integration with external systems and webhooks.
- You reduce k8s integration issues.
- You have access to infra services like service mesh, logs aggregators and metrics from development.
We just need to improve our dev tools to reach a great dev experience on remote Kubernetes clusters. I have been working like this for 2 years now and I can say my dev experience is much better than before. I have replicable dev environments, I can spin a new dev environment on a new namespace for each git branch, switch dev environments with a single command, and deploy my changes instantly.
And 5G is just around the corner and it will make the experience even better.
As an ex-docker employee, I love to work with docker-compose and I agree it is usually a good enough abstraction for development.
But it is not only about the tool, I think the future of development is on the cloud and Kubernetes is a perfect fit for ephemeral dev environments:
- You have unlimited hardware and network resources but efficiently shared by all your team.
- You can share endpoints for fast validation and easier integration with external systems and webhooks.
- You reduce k8s integration issues.
- You have access to infra services like service mesh, logs aggregators and metrics from development.
We just need to improve our dev tools to reach a great dev experience on remote Kubernetes clusters. I have been working like this for 2 years now and I can say my dev experience is much better than before. I have replicable dev environments, I can spin a new dev environment on a new namespace for each git branch, switch dev environments with a single command, and deploy my changes instantly.
And 5G is just around the corner and it will make the experience even better.
The only time I develop on Kubernetes is when I develop something which builds on top of the Kubernetes API. In that case I use skaffold. For all the rest Docker Compose is a better fit.
You don't develop your full CI pipeline on your machine either. So why care about the orchestration?
You don't develop your full CI pipeline on your machine either. So why care about the orchestration?
I'd love to talk if you use skaffold; I (disclaimer: building it) work on Tilt, and suspect it's a better fit. Especially because it can work on either k8s, like skaffold, or docker-compose, with a useful UI on top.
The one thing I have found tedious with k8s clusters is when you need to work on the host system in relation to a specific pod. E.g. to diagnose kernel issues or for profiling/debugging tools that need unrestricted root.
This can require jumping through several hoops. Find the node, get through some bastion host, inject SSH keys into target node, build an SSH chain to get into the host, then escalate from admin to root account.
It would be nice if kubectl offered an escape hatch for these kinds of things, assuming one has sufficient permissions.
This can require jumping through several hoops. Find the node, get through some bastion host, inject SSH keys into target node, build an SSH chain to get into the host, then escalate from admin to root account.
It would be nice if kubectl offered an escape hatch for these kinds of things, assuming one has sufficient permissions.
Run a privileged container with the host's procfs mounted into it via hostPath volume. Then you can use nsenter to break out of the container into the host system.
I already used a host mount to inject SSH keys. I guess I can just skip the rest and automate the steps to pivot fully into the host.
Local development on Kubernetes is in a pretty decent state imo. Minikube + Skaffold. Helm to manage/configure charts. And override resources to lower mem/cpu consumption when running locally. Works just fine for most situations. It's reproducible/consistent across developer environments and is as close as you can get to mapping 1-to-1 with prod.
So you install Minikube on each developer's machines? I thought that Minikube was created solely for running tutorials/excercises for people learning Kubernetes.
If you are developing microservices I think you're much better off with a development Kubernetes cluster, and a single namespace for each developer within it. This way the Kubernetes configuration can be identical to PROD, and you do not need tons of CPU cores and RAM on each of developer's laptops.
If you are developing microservices I think you're much better off with a development Kubernetes cluster, and a single namespace for each developer within it. This way the Kubernetes configuration can be identical to PROD, and you do not need tons of CPU cores and RAM on each of developer's laptops.
I've found https://www.telepresence.io/ to be helpful. It allows to integrate a local running process into a K8s cluster seamlessly allowing for fast iteration and easy debugging.
It’s trying to do too much, just like Google Wave, Angular, Google Web Toolkit, etc. Avoid anything that tries to do everything. Instead, build on an ecosystem of independent services that integrate well together.
If iterations take longer than 10s, then I'm really not interested.
Typical Docker based CIs are ten minutes to build. Nevermind the deployment. A ridiculous waste of time and energy.
Typical Docker based CIs are ten minutes to build. Nevermind the deployment. A ridiculous waste of time and energy.
I refuse to do work on docker because of this.
If you’re working with python or javascript or something similar, the usual solution is to mount a directory from your host OS as a docker volume, so that you don’t have to rebuild image layers on every change. Isn’t it?
If you have a fleet of services and currently work on services A and B you can just start up A - G in docker-compose based on CI-built images but mount your local build output into the A and B containers. Override their commands to watch filesystem changes so they reload themselves when you build.
This can be easily managed with an personal override file on top of a committed compose file used by your team.
This can be easily managed with an personal override file on top of a committed compose file used by your team.
I agree the local dev environment UX is pretty much there. I use volume mounts myself.
But when you're deploying on a remote host. OMG.
Just sharing a Docker image internally is anguish.
Developing as a remote team using Docker / Kubernetes workflow is just crazy. You're way WAY better off using serverless, where some implementations take ~2s to deploy my Go binary.
But when you're deploying on a remote host. OMG.
Just sharing a Docker image internally is anguish.
Developing as a remote team using Docker / Kubernetes workflow is just crazy. You're way WAY better off using serverless, where some implementations take ~2s to deploy my Go binary.
This can be hacked together in a similar way. Add a wrapper that watches for filesystem changes to the container command, then add a kubectl cp as last build step locally. Or push an image, set the deployment to pull on start and kill the pod.
For sharing docker images, assuming you have write access to some dev repository, you can push to a tag tied to your development branch and others can configure their docker setups to pull that on restart.
Dev images can be built faster by just copying build output into an image as last step, that way other steps to prepare the image are cached. No need to use whatever slow things CI is doing in its Dockerfile.
It only takes a few lines of shell or makefile to automate most of these things. Of course local dev is still nicer.
For sharing docker images, assuming you have write access to some dev repository, you can push to a tag tied to your development branch and others can configure their docker setups to pull that on restart.
Dev images can be built faster by just copying build output into an image as last step, that way other steps to prepare the image are cached. No need to use whatever slow things CI is doing in its Dockerfile.
It only takes a few lines of shell or makefile to automate most of these things. Of course local dev is still nicer.
[deleted]
For small setups, it is better to stay away from Kubernetes until it reaches a good level of stability. We jumped early on in the K8S bandwagon (before AWS EKS) and paid a steep price for that. We continue to pay. Our AWS cost has doubled since the time we transitioned to K8S from AWS Elastic Beanstalk.
I really wish I could say what would be a good time to embrace K8S, but your mileage might vary. When we made the switch, our service was at about 80TPS during regular load and would go up to 300TPS during peak load.
I really wish I could say what would be a good time to embrace K8S, but your mileage might vary. When we made the switch, our service was at about 80TPS during regular load and would go up to 300TPS during peak load.
The project that I work on depends directly on the Kubernetes API, admission webhooks and API extensions. I do my daily development on a GKE cluster, because local k8s environments are slow and inconsistent. What I really like about tilt is that it enables the continuous development experience, where code changes are continuously built and deployed to my remote cluster as I made them in my editor. It helps to shorten my feedback loop by replacing the series of `docker build`, `docker push`, `kubectl apply` etc. commands with just a single `tilt up` command.
I can sympathize with what the author said. A while ago, I worked as the only devops engineer on a team of 10 full stack developers. The tl;dr is that management decided to migrate their existing workloads to k8s for reasons. The developers were not too keen about all the new concepts, tools and techniques they needed to learn as a result of that decision. They knew they had to own their code from dev to prod, and couldn't just throw it over the fence (because there was no fence). The issues that were brought up in the post definitely reasonated with some of the concerns the developers brought up. I do think a tool like tilt would have made the transition easier for that team.
Finally, I think the post could have been made simpler if it stays focus on just one tool, and how that tool helps to solve the problems that non-ops developers have, when writing code that needs to run on k8s.
I can sympathize with what the author said. A while ago, I worked as the only devops engineer on a team of 10 full stack developers. The tl;dr is that management decided to migrate their existing workloads to k8s for reasons. The developers were not too keen about all the new concepts, tools and techniques they needed to learn as a result of that decision. They knew they had to own their code from dev to prod, and couldn't just throw it over the fence (because there was no fence). The issues that were brought up in the post definitely reasonated with some of the concerns the developers brought up. I do think a tool like tilt would have made the transition easier for that team.
Finally, I think the post could have been made simpler if it stays focus on just one tool, and how that tool helps to solve the problems that non-ops developers have, when writing code that needs to run on k8s.
What would you recommend for development of some private project on a local server? No cloud cluster, bare metal and great opportunity to learn.
Something on the order of 20-40 services for even the most trivial things. Development speed above all else
Something on the order of 20-40 services for even the most trivial things. Development speed above all else
Minikube works great for me. You can configure your Docker to use it too, e.g. so ‘docker run’ runs a container.
Tilt (the product created by the company hosted by the blog) is an implied solution.
There is also Draft (Microsoft), Skaffold (Google) and Garden.io
Here is comparison of the last 3 https://codefresh.io/howtos/local-k8s-draft-skaffold-garden/
(I have no affiliation with any of those solutions, I just co-authored the comparison article)
There is also Draft (Microsoft), Skaffold (Google) and Garden.io
Here is comparison of the last 3 https://codefresh.io/howtos/local-k8s-draft-skaffold-garden/
(I have no affiliation with any of those solutions, I just co-authored the comparison article)
Hi aloer!
Garden co-founder, so perhaps (likely) a little biased[0].
Depending on your needs, Garden and Tilt are both great solutions, with slightly different philosophies.
It's easy to get started with Tilt if you already have the Kubernetes manifests. If not, Garden might better suit your needs since its "container" module type allows you to provide only a minimal description of your module which Garden then maps into K8s manifests. So you won't have to write the full specs for 20+ services. (Garden also supports Kubernetes manifests and Helm charts if that's your jazz).
Garden will also manage the dependencies between all your services (at a granularity of builds, deploys, tasks, and tests). So if service12 needs to be built before service20, and service18 needs to be deployed before service5, but only after the database migration for service2 is completed, Garden can take care of that. This also improves dev speed since Garden knows exactly what to re-build and re-deploy as you make changes to your source code.
And on the topic of dev speed, Garden supports both hot-reloading and in-cluster building which allows you to share build caches with the rest of you team.
In general, there are a lot of interesting tools in this space and it's evolving rapidly. So even though developing on Kubernetes on your local laptop _can_ suck, developing distributed systems doesn't have to.
[0]: https://github.com/garden-io/garden
Garden co-founder, so perhaps (likely) a little biased[0].
Depending on your needs, Garden and Tilt are both great solutions, with slightly different philosophies.
It's easy to get started with Tilt if you already have the Kubernetes manifests. If not, Garden might better suit your needs since its "container" module type allows you to provide only a minimal description of your module which Garden then maps into K8s manifests. So you won't have to write the full specs for 20+ services. (Garden also supports Kubernetes manifests and Helm charts if that's your jazz).
Garden will also manage the dependencies between all your services (at a granularity of builds, deploys, tasks, and tests). So if service12 needs to be built before service20, and service18 needs to be deployed before service5, but only after the database migration for service2 is completed, Garden can take care of that. This also improves dev speed since Garden knows exactly what to re-build and re-deploy as you make changes to your source code.
And on the topic of dev speed, Garden supports both hot-reloading and in-cluster building which allows you to share build caches with the rest of you team.
In general, there are a lot of interesting tools in this space and it's evolving rapidly. So even though developing on Kubernetes on your local laptop _can_ suck, developing distributed systems doesn't have to.
[0]: https://github.com/garden-io/garden
> For example, a common set up is to only allow your developers to access to create/edit objects in one namespace.
Not best-practice. Best practice is to lock down access to production to grant read-only permissions to developers and force deployments to occur through a controlled pipeline. This controlled pipeline can ensure that changes first go through a staging environment that is essentially identical to production, verify that the changes work there, and only afterwards deploy to production.
> Testing things like NetworkPolicies is also fraught.
Test, staging, pre-production environments. Particularly since the developer won't typically have the entire environment set up on their developer machine, and permissive network access is a working default, usually more restrictive network policies are setup by whoever has responsibility for the larger environment and breaks are caught in later stages of the pipeline.
> ...test ingress changes, but even then changes can take 30 minutes to take effect and can result in inscrutable error messages.
Not Kubernetes's fault. Even then it's only partially the cloud provider's fault (because of the inscrutability of said error messages) - infrastructure doesn't provision itself in a split-second and that reality stands at odds with kubectl's desire to return asynchronously, ironically enough, for a better developer experience compared to staring at a console waiting for the cloud provider to respond within a timeout.
> Maybe in the future, SSH will be as anachronistic as the floppy disk icon, but for now, I want to log in to a container, poke around, see what the state is
Oh man. Please don't. Just don't. Improve your logging and metrics first, then we'll talk.
> It’s reasonable that this image doesn’t have strace, and kind of reasonable that it doesn’t have bash, but it highlights one of the Kubernetes best practices that makes local development hard: keep your images as small as possible.
That's not a Kubernetes issue, that's a containerization issue. And a large part of the philosophy behind why it's OK in containerland is, there are only three things you need to effectively observe containerized stateless applications, and they are logs (including request tracing), exported metrics, and network traffic statistics (which really are a kind of exported metric, just not one exported by the service).
> Even if you can avoid going out to the internet when pushing an image, just building an image can take forever. Especially if you aren’t using multi stage builds, and especially if you are using special development images with extra dependencies.
So... use multi-stage image builds?
> What I want to do is just sync a file up to my pod... if your container is restarted for any reason, like if your process crashes or the pod gets evicted, you lose all of your changes.
By design. Your changes aren't going to propagate to staging, let alone production.
> In dev, I want to tail the relevant logs so I can see what I’m doing. Kubernetes doesn’t make that easy.
https://github.com/wercker/stern
> If we want to empower developers to create end-to-end full stack microservices architectures we need to provide some way to get their hands dirty with networking. Until then that last push to production will always reveal hidden networking issues.
If you need to alter multiple microservices to deliver a single feature, your architecture is probably screwed up, or your development process isn't sufficiently structured to build up individual backend services incrementally and in a serial, rollback-able fashion. You can't fix a poor architecture or SDLC by throwing tooling at the problem.
Not best-practice. Best practice is to lock down access to production to grant read-only permissions to developers and force deployments to occur through a controlled pipeline. This controlled pipeline can ensure that changes first go through a staging environment that is essentially identical to production, verify that the changes work there, and only afterwards deploy to production.
> Testing things like NetworkPolicies is also fraught.
Test, staging, pre-production environments. Particularly since the developer won't typically have the entire environment set up on their developer machine, and permissive network access is a working default, usually more restrictive network policies are setup by whoever has responsibility for the larger environment and breaks are caught in later stages of the pipeline.
> ...test ingress changes, but even then changes can take 30 minutes to take effect and can result in inscrutable error messages.
Not Kubernetes's fault. Even then it's only partially the cloud provider's fault (because of the inscrutability of said error messages) - infrastructure doesn't provision itself in a split-second and that reality stands at odds with kubectl's desire to return asynchronously, ironically enough, for a better developer experience compared to staring at a console waiting for the cloud provider to respond within a timeout.
> Maybe in the future, SSH will be as anachronistic as the floppy disk icon, but for now, I want to log in to a container, poke around, see what the state is
Oh man. Please don't. Just don't. Improve your logging and metrics first, then we'll talk.
> It’s reasonable that this image doesn’t have strace, and kind of reasonable that it doesn’t have bash, but it highlights one of the Kubernetes best practices that makes local development hard: keep your images as small as possible.
That's not a Kubernetes issue, that's a containerization issue. And a large part of the philosophy behind why it's OK in containerland is, there are only three things you need to effectively observe containerized stateless applications, and they are logs (including request tracing), exported metrics, and network traffic statistics (which really are a kind of exported metric, just not one exported by the service).
> Even if you can avoid going out to the internet when pushing an image, just building an image can take forever. Especially if you aren’t using multi stage builds, and especially if you are using special development images with extra dependencies.
So... use multi-stage image builds?
> What I want to do is just sync a file up to my pod... if your container is restarted for any reason, like if your process crashes or the pod gets evicted, you lose all of your changes.
By design. Your changes aren't going to propagate to staging, let alone production.
> In dev, I want to tail the relevant logs so I can see what I’m doing. Kubernetes doesn’t make that easy.
https://github.com/wercker/stern
> If we want to empower developers to create end-to-end full stack microservices architectures we need to provide some way to get their hands dirty with networking. Until then that last push to production will always reveal hidden networking issues.
If you need to alter multiple microservices to deliver a single feature, your architecture is probably screwed up, or your development process isn't sufficiently structured to build up individual backend services incrementally and in a serial, rollback-able fashion. You can't fix a poor architecture or SDLC by throwing tooling at the problem.
My take on it was that the author was talking rather about the DevOps and not only development. And ofcourse he did a pretty poor job at making his DevOps work easy at the company.
With all the love for K8s - this post is spot on in many points.
Especially around accessing containers and observability. But ... That's where modern solutions of virtual-logging come in: e.g https://Rookout.com
When your software connects back to you and you can instrument it on the fly (Add logs line, non-breaking-breakpoints, etc.) most of these pain points are resolved
Especially around accessing containers and observability. But ... That's where modern solutions of virtual-logging come in: e.g https://Rookout.com
When your software connects back to you and you can instrument it on the fly (Add logs line, non-breaking-breakpoints, etc.) most of these pain points are resolved
You guys do realize that Service Fabric solved these issues years ago, right?
Because simplicity was an afterthought.
Srsly? Is this meant to be satire?
So, how can the same story be submitted twice in a span of a few hours?
https://news.ycombinator.com/item?id=20768531
https://news.ycombinator.com/item?id=20768531
1. Why kubernetes for development? For most cases you don't need the whole orchestration over multiple nodes overhead, docker-compose is just fine. Incidentally, your push/pull problem is solved as you pull the dependencies and BUILD your app locally.
2. Want to have someplace to test your kube configurations? That's the ops part of devops. Set up a proper testing cluster instead of trying to cram everything in your devenv. Kube config should be in a separate project anyways.
3. No strace/bash. Look, man, this isn't even kubes' problem. That's how you build your images. Have separate dev images that use prod image as a base. Deploy dev images to testing env. Prod images to staging/prod. Build dev images locally.
4. Network debugging. That's difficult on any system. If you don't have access to the underlying host, you are screwed either way - call your devops for support. Anything else (internal ips, svc endpoints, ingresses) is trivial to debug on kube. However, we're talking about development... just don't use kube for development, period. Use docker-compose. Which, actually solves all your problems... right tools for the right job, or suffer the self-inflicted consequences.