Dynamic Route53 records for AWS auto scaling groups with Terraform(underthehood.meltwater.com)
underthehood.meltwater.com
Dynamic Route53 records for AWS auto scaling groups with Terraform
https://underthehood.meltwater.com/blog/2020/02/07/dynamic-route53-records-for-aws-auto-scaling-groups-with-terraform/
7 comments
You can attach an ELB to the ASG. Then DNS name to ELB.
That's how AWS expects you to use it.
I can see why you might not want to, though. App ELBs charge by usage and can get somewhat expensive (like running another EC2 instance or two). They can also have cold-start performance issues in specific circumstances (traffic spikes).
yup. but that's usually negligible to most companies. and it's probably not worth the extra complexity introduced by production hacks.
That doesn't solve the problem of the hostname on the EC2 instance itself being the same across all instances thereby making it harder to see what logs came from what hosts.
It doesn't solve the problem of allowing you to look at logs and then quickly SSH'ing to a single machine in the ASG.
It doesn't solve the problem of allowing you to look at logs and then quickly SSH'ing to a single machine in the ASG.
Is this not already a solved issue?
Install a log agent on the machine like fluentd. Have it inject the host ip and other contextual meta data in to the logs then forward to your central log system?
When you see the error message in your logs, you get the internal ip and can ssh in.
Persistent internal ip’s/hostnames also means you are not treating hosts as ephemeral. It’s always good in the cloud to get things to a point you can just blow away instances and they auto recreate. It’s even possible with traditional services requiring persistent storage. Put the storage on a seperate volume and have the instance startup scripts discover available volumes and attach as required.
Install a log agent on the machine like fluentd. Have it inject the host ip and other contextual meta data in to the logs then forward to your central log system?
When you see the error message in your logs, you get the internal ip and can ssh in.
Persistent internal ip’s/hostnames also means you are not treating hosts as ephemeral. It’s always good in the cloud to get things to a point you can just blow away instances and they auto recreate. It’s even possible with traditional services requiring persistent storage. Put the storage on a seperate volume and have the instance startup scripts discover available volumes and attach as required.
what you need is probably something like https://github.com/adhocteam/ec2ssh (I never used it, but I have built similar ones) -- and then you tag the log entries with instance id.
so you can do "ec2ssh i-0017c8b3"
imho: Hacking around debugging tools is better (mostly because more reliable) than hacking around production configurations (one problem you will see is that changing route53 records frequently will be subject to API rate-limits).
so you can do "ec2ssh i-0017c8b3"
imho: Hacking around debugging tools is better (mostly because more reliable) than hacking around production configurations (one problem you will see is that changing route53 records frequently will be subject to API rate-limits).
Post author here.
We absolutely put an ELB/ALB in front of these ASGs as well. The post mentions a few use cases where unique hostnames with internal Route53 records are helpful for us.
We absolutely put an ELB/ALB in front of these ASGs as well. The post mentions a few use cases where unique hostnames with internal Route53 records are helpful for us.
This is a pretty sweet Terraform module that I will personally be testing out soon.
I was curious what they are using for the Lambda function, and it turns out is a framework-less Python script[0].
One thing I'm not clear on yet is if using this will imply one such lambda for every autoscaling group or not.
[0] https://github.com/meltwater/terraform-aws-asg-dns-handler/b...
I was curious what they are using for the Lambda function, and it turns out is a framework-less Python script[0].
One thing I'm not clear on yet is if using this will imply one such lambda for every autoscaling group or not.
[0] https://github.com/meltwater/terraform-aws-asg-dns-handler/b...
I just checked with the Jim, the author of the blog post, and module (we both work for Meltwater).
He confirmed "You only need one instance per account, we have always been fine with just one".
I will let him fill in further details here but I figured you would be interested in a short update on this.
He confirmed "You only need one instance per account, we have always been fine with just one".
I will let him fill in further details here but I figured you would be interested in a short update on this.
Great to hear! Please let us know how the testing goes!
Can't this be solved by using IP addresses for hostnames? This can be a part of bootstrap script(which ASG/Launch Configuration already supports via UserData[1])
What I can't understand is -
If your logs are in ELK and metrics in prometheus/grafana - why do you need SSH access? Sounds like thats a good problem to solve
[1] - https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/user-dat...
What I can't understand is -
If your logs are in ELK and metrics in prometheus/grafana - why do you need SSH access? Sounds like thats a good problem to solve
[1] - https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/user-dat...
Post author here.
SSH access is a last resort, but it can be necessary in certain cases. For example, if our log forwarding breaks. SSH is also just one example, it can also be helpful to curl endpoints on the host directly without hitting the ELB/ALB.
The post actually provides the user_data script we use.
SSH access is a last resort, but it can be necessary in certain cases. For example, if our log forwarding breaks. SSH is also just one example, it can also be helpful to curl endpoints on the host directly without hitting the ELB/ALB.
The post actually provides the user_data script we use.
I do this already but from my configuration management system on my instances, but one thing I don't have that I'd love if Route53 would help support is being able to run route53 to handle in-addr.arpa zones for my IP addresses so I can get reverse IP looking for my VPC networks without having to run my own resolver.
We run our reverse zones on route53. It is just a little bit cumbersome but overall works relatively well with vpc private hosted zones.
Are you using RFC1918 networks or are you using public subnets you have actual delegation for?
I wrote and use this regularly, docker app that adds and removes instances from route53, similar to their terraform solution. Similar idea, different implementation.
https://github.com/discobean/route53-sidecar
https://github.com/discobean/route53-sidecar
Why are ASGs 'outdated' in a world with Kubernetes?
Our nodes are in ASGs.
Our nodes are in ASGs.
Post author here.
I brought up that point since I think most developers prefer the user experience of Lambda/Kubernetes where they don't have to manage individual instances in Auto Scaling Groups. They certainly are not 'outdated' for our use cases, and especially not for those responsible for running the underlying infrastructure (when running Kubernetes nodes).
I brought up that point since I think most developers prefer the user experience of Lambda/Kubernetes where they don't have to manage individual instances in Auto Scaling Groups. They certainly are not 'outdated' for our use cases, and especially not for those responsible for running the underlying infrastructure (when running Kubernetes nodes).
I think AWS not having automatic hostnames for ASG instances is a way to lure you into Lambda's, at least that's how I got hooked.
Why should an instance created by an ASG have a host name? These are cattle not pets. I use Serilog for logging with an EC2 enricher that automatically adds the instance Id and the IP address.
Since Serilog does structured logging, I can use either an ElasticSearch or Mongo sink and do complex queries.
If I routinely need to log into an instance to troubleshoot, I need to be capturing data and sending it to a central logging system.
Since Serilog does structured logging, I can use either an ElasticSearch or Mongo sink and do complex queries.
If I routinely need to log into an instance to troubleshoot, I need to be capturing data and sending it to a central logging system.
> Why should an instance created by an ASG have a host name?
It means you can connect to it by just knowing its instance ID.
Adding the IP address everywhere also works.
There can be some nice SSH config options though, like using a particular key for everything *.prod.myaws.com
It means you can connect to it by just knowing its instance ID.
Adding the IP address everywhere also works.
There can be some nice SSH config options though, like using a particular key for everything *.prod.myaws.com
You can also use SSM and session manager to get a similar experience for those instances: https://docs.aws.amazon.com/systems-manager/latest/userguide...
I haven't had to manage SSH keys in a long time ;)
With this I just have a bash function for my various environments (e.g. dev = dssm) where I provide in the instance ID giving me issues if I really need to log into the server.
e.g.
function dssm { aws --region us-west-2 --profile my-dev-profile-name ssm start-session --target $1 }
Then:
dssm i-abcdef123456
And I'm dropped into a shell. SSM Session manager is far from perfect, but it gets the job done, and is fully auditable, gets logged (including commands ran), and best of all works with SAML IAM profiles right out the gate. No more sharing keys, no more managing keys, it's great!
I haven't had to manage SSH keys in a long time ;)
With this I just have a bash function for my various environments (e.g. dev = dssm) where I provide in the instance ID giving me issues if I really need to log into the server.
e.g.
function dssm { aws --region us-west-2 --profile my-dev-profile-name ssm start-session --target $1 }
Then:
dssm i-abcdef123456
And I'm dropped into a shell. SSM Session manager is far from perfect, but it gets the job done, and is fully auditable, gets logged (including commands ran), and best of all works with SAML IAM profiles right out the gate. No more sharing keys, no more managing keys, it's great!
Yes exactly, SSH access is also one of the reasons for building the module, that is mentioned in the blog post.
> Access: When troubleshooting, we save time not having to look up the instance’s internal IP address for SSH access.
> Access: When troubleshooting, we save time not having to look up the instance’s internal IP address for SSH access.
That’s the second part. If I’m troubleshooting by logging into EC2 instances, there is something wrong with my logging infrastructure. That’s actually the larger issue.
Post author here.
SSH access is absolutely a last resort, but can be necessary in certain cases (like when Filebeat breaks...). Turning SSH off completely (i.e. "No SSH") is certainly better for security and something we may pursue.
I mentioned in another comment here that SSH is just one example, we can also easily hit endpoints with curl via hostname.
Also mentioned in the post are other tools (like Grafana dashboards) have an expectation of unique hostnames.
SSH access is absolutely a last resort, but can be necessary in certain cases (like when Filebeat breaks...). Turning SSH off completely (i.e. "No SSH") is certainly better for security and something we may pursue.
I mentioned in another comment here that SSH is just one example, we can also easily hit endpoints with curl via hostname.
Also mentioned in the post are other tools (like Grafana dashboards) have an expectation of unique hostnames.
> If I’m troubleshooting by logging into EC2 instances, there is something wrong with my logging infrastructure.
I suppose it's possible to build enough logging to account for an interactive SSH session for debugging problems...but that would be massive.
I ran out of disk space. Why?
I suppose it's possible to build enough logging to account for an interactive SSH session for debugging problems...but that would be massive.
I ran out of disk space. Why?
If you’re logging to a local disk on ephemeral VMs, that doesn’t make the situation any better.
That’s why you need a central logging facility. If you’re using AWS, you could store your structured JSON logs in S3 and query them with Athena. (https://medium.com/quiq-blog/store-json-logs-on-s3-for-searc...)
Of course there are other ways both using AWS and third party services. Centralized logging is a solved problem.
AWS isn’t going to run out of disk space any time soon. You could also use a lifecycle policy to delete old logs or move them to a lower cost storage depending on your retention policy.
I’m not saying that I have never had to log on to a VM to troubleshoot, but that’s a sign of the need of better logging.
And if my logging infrastructure isn’t good, how pray tell will I troubleshoot my programs running on Lambda or Fargate?
That’s why you need a central logging facility. If you’re using AWS, you could store your structured JSON logs in S3 and query them with Athena. (https://medium.com/quiq-blog/store-json-logs-on-s3-for-searc...)
Of course there are other ways both using AWS and third party services. Centralized logging is a solved problem.
AWS isn’t going to run out of disk space any time soon. You could also use a lifecycle policy to delete old logs or move them to a lower cost storage depending on your retention policy.
I’m not saying that I have never had to log on to a VM to troubleshoot, but that’s a sign of the need of better logging.
And if my logging infrastructure isn’t good, how pray tell will I troubleshoot my programs running on Lambda or Fargate?
I never said your disk was full with logs.
> how pray tell will I troubleshoot my programs running on Lambda or Fargate?
That is indeed a big problem running on Lambda and Fargate.
In my experience, Fargate isn't very commonly used and Lambda is used for only relatively simple things.
> how pray tell will I troubleshoot my programs running on Lambda or Fargate?
That is indeed a big problem running on Lambda and Fargate.
In my experience, Fargate isn't very commonly used and Lambda is used for only relatively simple things.
It’s not a problem at all with lambda or Fargate. Logging can be as simple as printing to the console and they go to CloudWatch.
It’s the same concept. If you’re troubleshooting at any point involves needing to log in to an EC2 instance, you might as well have a few bespoke servers called “Web01” and “Web02”. You’re just using ASG to create pets at scale. We run an ASG in production that scales from 2 to 30 instances based on the number of messages in a queue, lambdas running all of the time, some a Fargate tasks etc. it would be a nightmare to troubleshoot all of those processes without centralized, queryable logs.
In my experience, Fargate isn't very commonly used and Lambda is used for only relatively simple things.
And that experience is representative of the entire AWS ecosystem?
It’s the same concept. If you’re troubleshooting at any point involves needing to log in to an EC2 instance, you might as well have a few bespoke servers called “Web01” and “Web02”. You’re just using ASG to create pets at scale. We run an ASG in production that scales from 2 to 30 instances based on the number of messages in a queue, lambdas running all of the time, some a Fargate tasks etc. it would be a nightmare to troubleshoot all of those processes without centralized, queryable logs.
In my experience, Fargate isn't very commonly used and Lambda is used for only relatively simple things.
And that experience is representative of the entire AWS ecosystem?
I agree, I wouldn't want it any other way nowadays, but back then I had to migrate a lot of legacy system to AWS under pressure.
For one part we had a legacy service needing to connect to the services in the ASG and the best way to implement it was with round-robin DNS. So the lambda would update a DNS record contianing all the ASG host ips.
Also, because we had some had some semi stateful legacy instances that where basically lift and shift to AWS, but I wanted to have them in ASG to keep our environment similar until we could refactor them into real cattle.
For one part we had a legacy service needing to connect to the services in the ASG and the best way to implement it was with round-robin DNS. So the lambda would update a DNS record contianing all the ASG host ips.
Also, because we had some had some semi stateful legacy instances that where basically lift and shift to AWS, but I wanted to have them in ASG to keep our environment similar until we could refactor them into real cattle.
Just out of curiosity, why not just put the ASG behind a load balancer?
I don't remember exactly. We did use elb's for all other services. So it was either cost or it had to do with MX record restrictions in that you're not allowed to use CNAMEs in MX records.
Or a way to get you more familiar with tagging, or the various queries and filters on different api results. It's annoying at first, but it leads to less reliance on the console and more effective scripting. (Instead of naming my instances I just made a script which looks up the instance I want and outputs the IP and username, and put that in an SSH config)
They expect you to front it with a load balancer.