Cloudflare silently deleted my DNS records(txti.es)
txti.es
Cloudflare silently deleted my DNS records
https://txti.es/cloudflare-deleted-my-dns
432 comments
I've put this idea forward a number of times here on HN in regards to other big tech companies.
Technology companies need an "ombudsman" - a contact that customers can go to when the normal tech support processes have failed.
The Ombudsman must not be part of the technology companies ordinary support processes, it must be entirely separate, and have highest level authority to demand action within the company.
To avoid the Ombudsman being overused, you could give it a price of say $20, which is always refunded when the case is resolved.
HN constantly has front page posts from people for whom big tech companies have support processes have failed but there is simply no other recourse unless you have "a friend in the business".
It just doesn't work to have some random Cloudflare person offer their email address as some post disaster issue resolution process on social media. Formalise it with an official Ombudsman and maybe then companies like Cloudflare might avoid HN front page bad publicity.
I had an issue at "one of the biggest tech companies" that went on for days and days in which tech support kept telling me I had set up something wrong, until eventually I emailed one of the top managers who I happen to "know" at that company - it was fixed within hours. That "contact a friend in the business who can actually get things done" is a necessary part of a large support organisation and it simply does not exist yet in any tech company that I know of.
Technology companies need an "ombudsman" - a contact that customers can go to when the normal tech support processes have failed.
The Ombudsman must not be part of the technology companies ordinary support processes, it must be entirely separate, and have highest level authority to demand action within the company.
To avoid the Ombudsman being overused, you could give it a price of say $20, which is always refunded when the case is resolved.
HN constantly has front page posts from people for whom big tech companies have support processes have failed but there is simply no other recourse unless you have "a friend in the business".
It just doesn't work to have some random Cloudflare person offer their email address as some post disaster issue resolution process on social media. Formalise it with an official Ombudsman and maybe then companies like Cloudflare might avoid HN front page bad publicity.
I had an issue at "one of the biggest tech companies" that went on for days and days in which tech support kept telling me I had set up something wrong, until eventually I emailed one of the top managers who I happen to "know" at that company - it was fixed within hours. That "contact a friend in the business who can actually get things done" is a necessary part of a large support organisation and it simply does not exist yet in any tech company that I know of.
> To avoid the Ombudsman being overused, you could give it a price of say $20, which is always refunded when the case is resolved.
What if the cost was put onto the business instead of the consumer and the business just hired support people who are all Ombudsmans by default?
Instead of focusing on copy / pasting boilerplate scripts and answering as many tickets as possible, they should focus on the problem the customer is having by default and do everything possible to reduce the number of incoming questions by fixing bugs, making a better product, improving their docs, etc..
I personally do around the clock email support for 35,000+ people who sign up to my programming courses and support isn't bogging me down. Relative to the number of minutes I'm awake, support is one of the least business related time consuming things I do per day, but I send individually personalized in-depth answers to everyone who asks me questions -- usually within an hour or less.
What if the cost was put onto the business instead of the consumer and the business just hired support people who are all Ombudsmans by default?
Instead of focusing on copy / pasting boilerplate scripts and answering as many tickets as possible, they should focus on the problem the customer is having by default and do everything possible to reduce the number of incoming questions by fixing bugs, making a better product, improving their docs, etc..
I personally do around the clock email support for 35,000+ people who sign up to my programming courses and support isn't bogging me down. Relative to the number of minutes I'm awake, support is one of the least business related time consuming things I do per day, but I send individually personalized in-depth answers to everyone who asks me questions -- usually within an hour or less.
Because requiring the customer to pay a cost fixes the incentives problem. A customer may have an issue that is a minor inconvenience to them, that can either stay as not resolved, is due to error, or is simply not worth resolving to them. By having them pay for fixing the issue it could greatly reduce the number of inbound cases, allowing them to go straight to people that can act on them instead of sitting in triage and getting canned responses for O(days|forever).
I agree that in theory you can accomplish the same thing by making the product foolproof, but I don't think you can accomplish that for consumer facing products, and that doing so may not be a worth while trade-off. Additionally focusing on issues that greatly impact people rather than small things that cause friction with the product may (or may not! if it causes lack of retention) be worth more.
I agree that in theory you can accomplish the same thing by making the product foolproof, but I don't think you can accomplish that for consumer facing products, and that doing so may not be a worth while trade-off. Additionally focusing on issues that greatly impact people rather than small things that cause friction with the product may (or may not! if it causes lack of retention) be worth more.
In terms of games theory; if there's no cost to the company to ignore the customer, then that's an incentive to ignore the consumer's complaint. Especially if that company has a monopoly.
I can understand why GP suggested company pays $20.
I can understand why GP suggested company pays $20.
The suggestion is the customer pays $20 so they don't try and go straight to the ombudsman with ever little issue. The company doesn't and shouldn't pay the customer under any circumstance.
In technical support, one of the largest problems you have to deal with is all of the idiots that don't know how to use the product and won't look up the documentation or learn the product on their own. These people spam the shit out of support all day for the most basic shit. Seriously, I was working support at an anti-virus company and 85% of the calls into the paid business tier support were requests for us to do installs or basic application configuration for them. I know that not everybody knows how to configure a firewall, but "where do I download the install file for x" is literally Google-able.
The idea of an ombudsman is ultimately a bad one in this case because the problem isn't with the product or the support or the documentation. The problem is people Always think their issue is special and the most capable person should help them. Our support tier was paid, so more often than not we did what they asked, but the free customers would just get routed to a sales rep because free customers are even dumber and more entitled than the paying ones.
This whole situation is farcicle to me. Yet another free customer blew a minor issue out of proportion and had to apologize when it turned out the system hadn't failed him, he was just a free loading mooch all along.
In technical support, one of the largest problems you have to deal with is all of the idiots that don't know how to use the product and won't look up the documentation or learn the product on their own. These people spam the shit out of support all day for the most basic shit. Seriously, I was working support at an anti-virus company and 85% of the calls into the paid business tier support were requests for us to do installs or basic application configuration for them. I know that not everybody knows how to configure a firewall, but "where do I download the install file for x" is literally Google-able.
The idea of an ombudsman is ultimately a bad one in this case because the problem isn't with the product or the support or the documentation. The problem is people Always think their issue is special and the most capable person should help them. Our support tier was paid, so more often than not we did what they asked, but the free customers would just get routed to a sales rep because free customers are even dumber and more entitled than the paying ones.
This whole situation is farcicle to me. Yet another free customer blew a minor issue out of proportion and had to apologize when it turned out the system hadn't failed him, he was just a free loading mooch all along.
A company has no incentive to ignore the customer, all else being equal, but they do have costs in servicing the customer and costs from ignoring the customer.
Assume the company has no way of distinguishing "valid" or "high impact" cases from other cases. This means that in order for the company to handle cases they need the sum of fully treating every case to be greater than the sum of the cost of fully treating every case. This is almost certainly never going to be true, unless each of your cases is high cost to ignore (think enterprise support).
So you need to funnel down the cases. Typically this is done with low tier support that tries to suggest fixes and such. You can also offer high value added support to give customers the ability to pay for a support plan. This $20 proposal is like that but on a more ad-hoc basis.
Assume the company has no way of distinguishing "valid" or "high impact" cases from other cases. This means that in order for the company to handle cases they need the sum of fully treating every case to be greater than the sum of the cost of fully treating every case. This is almost certainly never going to be true, unless each of your cases is high cost to ignore (think enterprise support).
So you need to funnel down the cases. Typically this is done with low tier support that tries to suggest fixes and such. You can also offer high value added support to give customers the ability to pay for a support plan. This $20 proposal is like that but on a more ad-hoc basis.
> if there's no cost to the company to ignore the customer
Simplifying, that's "if (false)", since ignoring the customers in the limit gets you a failed business and bankruptcy.
Simplifying, that's "if (false)", since ignoring the customers in the limit gets you a failed business and bankruptcy.
For the non-enterprise user, isn't this the scenario established today, with no failed business?
No, I don't think so. All the major internet companies listen to their non-enterprise users. They may not provide enterprise quality support to each individual user, but user voices in aggregate are considered critically important.
not if you have a monopoly. Which is why customer support for most ISPs in the US is so abysmal.
And even if it is just a sufficiently large company, if the customer is not a large payer and the issue is non-trivial to solve, the the cost of losing that customer could very well be less than the cost of fixing the issue.
And even if it is just a sufficiently large company, if the customer is not a large payer and the issue is non-trivial to solve, the the cost of losing that customer could very well be less than the cost of fixing the issue.
What I was addressing was specifically the notion that companies pay zero cost for ignoring a customer, or the idea that they believe the same. That's false. Now there's no question there are some customers these large companies are better off without, and there are large and small examples of companies firing customers, or ignoring them til they go away. Nothing wrong with that. All the time on this site we talk about firing unprofitable customers. If you do this judiciously it's just good business. Doing that in particular cases is not the same as doing that generally.
NearlyFreeSpeech.net does something kind of like that. It costs money to raise any support ticket (you have to have a special support subscription) but you get refunded your "support points" if it turned out to be a problem on their end.
Because that isn't a useful allocation of resources.
Support that powerful are going to basically be devs. With dev salary expectations.
Support that powerful are going to basically be devs. With dev salary expectations.
I think it's useful and it's very much worth it to give support a salary where they are enthusiastic to be employed.
If someone uses your product or service and has an issue, the first contact they have with your company is through support.
A crappy support experience could easily be the difference between having a life time customer valued at thousands of dollars, or your customer feeling frustrated and going to another company, netting you $0. That has a butterfly effect too because having 100 happy customers who are praising your support could lead to many new sales from organic recommendations. Having a bunch of customers who felt neglected by your support could yield a situation where your company is now on the front page of HN for having bad support or worse.
For whatever reasons, bigger companies focus more on measurable metrics like "tickets closed per hour", where the emphasis is on things that aren't important because measuring customer happiness is pretty subjective and doesn't translate well to employee evaluation scores.
I know if I ever got the point where I would need to delegate support assistance to someone, you can be sure I would pay them amazingly well, at least equal to a developer's salary because I never want anyone to ever feel like they get ignored or have a low quality experience with my products.
If someone uses your product or service and has an issue, the first contact they have with your company is through support.
A crappy support experience could easily be the difference between having a life time customer valued at thousands of dollars, or your customer feeling frustrated and going to another company, netting you $0. That has a butterfly effect too because having 100 happy customers who are praising your support could lead to many new sales from organic recommendations. Having a bunch of customers who felt neglected by your support could yield a situation where your company is now on the front page of HN for having bad support or worse.
For whatever reasons, bigger companies focus more on measurable metrics like "tickets closed per hour", where the emphasis is on things that aren't important because measuring customer happiness is pretty subjective and doesn't translate well to employee evaluation scores.
I know if I ever got the point where I would need to delegate support assistance to someone, you can be sure I would pay them amazingly well, at least equal to a developer's salary because I never want anyone to ever feel like they get ignored or have a low quality experience with my products.
Airbnb is a great example of both. If you call tech support and get India, hang up. If you're lucky they're useless. If you're unlucky, they're going against your direct command and contacting the drug dealers in your house about a complaint. Contrast that to reaching one of the many on-shore agents who do their jobs well. One call, one or two people max and the issue is mostly resolved.
I've got no idea what the cause is. It's probably a mix of poor management, ineffective metrics, low salaries and a work culture in India which isn't synonymous with quality. The bottom line is, even the same company can be doing support wonderfully and terribly at the same time.
I've got no idea what the cause is. It's probably a mix of poor management, ineffective metrics, low salaries and a work culture in India which isn't synonymous with quality. The bottom line is, even the same company can be doing support wonderfully and terribly at the same time.
Yeah AirBnB support in India was awful and I wrote the company off after dealing with them for 20 minutes. Their onshore team actually cared about the huge cockroach infestation in our unit but the Indian guy I first talked to did not give a damn. And that’s who you’re most likely to get at 4 in the afternoon on your first day.
Part of "Wear the Customer's Shoes" at Twilio (when I was there) was that each quarter every developer and product manager was expected to spend a full day doing front line support for free(-ish) tier customers and however long it took to fully resolve the tickets they got during that time. Really let you understand the type of problems people had, how to have empathy, and basically how to make a better product.
I think this happens because it's very very difficult for a manager's manager to know if the frontline employee is spending their time solving real problems or slacking off with no actual regard for the customer.
"Metrics" are an unhappy middleground for everyone.
"Metrics" are an unhappy middleground for everyone.
In my experience companies use not only "tickets closed per hour", but combine those with "average score" of 1-5 stars.
Yeah but at a large company the ratings don't matter too much in the end because it ends up being a spreadsheet game focused on hiring for the lowest cost while maintaining a level of support that's barely 1 notch above "horrible support".
So if you end up hiring someone who produces a bunch of 1-2 star ratings, you just fire them and try again with someone else until you get someone who produces mostly 3s+.
In the end you would think this would result in better support but it never does. You just end up turning over a lot of support employees, and the user experience for the customer is still bad because you need to first get through a robot menu, then talk to an entry level support who will happily let you pour your soul out on the details of the problem for 5 minutes, and then at the end they are like "oh, I can't do that, but I can forward you to someone who can".
And now you get to be put on hold again, anticipating by coincidence you'll probably lose the connection, and if you're lucky now you get someone who is capable of understanding the problem and you get to re-tell the whole situation again.
Before you know it, with wait times included you're 20 minutes into this and you just barely got to the point where you might get help for the issue. Businesses could solve this problem, but they don't. Instead of hiring better support folks for more money, they put the burden on the customer to have to wade through a bunch of BS and essentially train their entry level support for free -- and it's worse than free too because you're paying that company money for their service and you're trading time from your life to do it.
So if you end up hiring someone who produces a bunch of 1-2 star ratings, you just fire them and try again with someone else until you get someone who produces mostly 3s+.
In the end you would think this would result in better support but it never does. You just end up turning over a lot of support employees, and the user experience for the customer is still bad because you need to first get through a robot menu, then talk to an entry level support who will happily let you pour your soul out on the details of the problem for 5 minutes, and then at the end they are like "oh, I can't do that, but I can forward you to someone who can".
And now you get to be put on hold again, anticipating by coincidence you'll probably lose the connection, and if you're lucky now you get someone who is capable of understanding the problem and you get to re-tell the whole situation again.
Before you know it, with wait times included you're 20 minutes into this and you just barely got to the point where you might get help for the issue. Businesses could solve this problem, but they don't. Instead of hiring better support folks for more money, they put the burden on the customer to have to wade through a bunch of BS and essentially train their entry level support for free -- and it's worse than free too because you're paying that company money for their service and you're trading time from your life to do it.
Yeah, even russian social network (200m users) have a bunch of very smart people supporting users from password recovery to psychological help. No way big startup can't afford something like this. You just have to do so.
I like the idea of the Ombudsman, but in this case I'm not sure that it's really applicable. I disagree that the "normal tech support processes have failed".
I wanted to point out something that may have been missed by many - which is that the OP in this case is a free user. i.e. he is NOT a "customer". He is a user of free services that Cloudflare is providing. It even says so in his post that he's a free account holder and that he's not entitled to tech support, hence why he kicked up a fuss online. My company is a Cloudflare Enterprise customer - when we write in, we get a response typically within an hour by their swat team.
The idea of an "Ombudsman" in this case wouldn't be an actual ombudsman in the spirit of advocating for the customer, but would be more of a "one time support fee" kind of deal. Which I think is fair, but in this case I'm not even sure the OP would have paid it. The OP had a viable option which would have been to upgrade to a Business account and get priority support.
I run a popular platform with 1.5 million registered users and the vast majority of them are free users. We also have the same problem. Most of our support is just swamped with free users mostly asking questions about how to do things (and it's in the documentation, they just don't read or search for it, it's too simple to just email support or blast away on Twitter and @ mention us). I've even had to withdraw from Twitter and Facebook entirely just because I was everyone's "ombudsman" (even if they weren't paying customers).
I wanted to point out something that may have been missed by many - which is that the OP in this case is a free user. i.e. he is NOT a "customer". He is a user of free services that Cloudflare is providing. It even says so in his post that he's a free account holder and that he's not entitled to tech support, hence why he kicked up a fuss online. My company is a Cloudflare Enterprise customer - when we write in, we get a response typically within an hour by their swat team.
The idea of an "Ombudsman" in this case wouldn't be an actual ombudsman in the spirit of advocating for the customer, but would be more of a "one time support fee" kind of deal. Which I think is fair, but in this case I'm not even sure the OP would have paid it. The OP had a viable option which would have been to upgrade to a Business account and get priority support.
I run a popular platform with 1.5 million registered users and the vast majority of them are free users. We also have the same problem. Most of our support is just swamped with free users mostly asking questions about how to do things (and it's in the documentation, they just don't read or search for it, it's too simple to just email support or blast away on Twitter and @ mention us). I've even had to withdraw from Twitter and Facebook entirely just because I was everyone's "ombudsman" (even if they weren't paying customers).
If you read his blog you would know that he is very much so a customer. To quote his blog verbatim "Because Cloudflare deleted my domain registration I can't change the status from clientTransferProhibited through their dashboard so I don't think I can even leave."
He registered his domain through Cloudflare Registrar, a paid service. I'd be concerned if my domain registrar told me to head to the forums for support for tld level issues that only the registrar can solve.
He registered his domain through Cloudflare Registrar, a paid service. I'd be concerned if my domain registrar told me to head to the forums for support for tld level issues that only the registrar can solve.
Ok but I still don’t get why the op complains about it so publicly the same day (likely immediately) after submitting the ticket without even giving the company a chance to triage and resolve the issue. (Yes he apologized) He paid $10 for a .org domain, which for Cloudflare is at cost. Goodness... It’s not like Cloudflare is refusing him support, just that he’s not on a tier where someone will respond in real time. In any case, it’s resolved for him now and he got the attention of the CTO (maybe I should try some public outrage sometime instead of paying for ent support heh!)
Also I think there’s an issue with what the expectations for support is for. The Cloudflare service itself - the DNS, DDOS protection, ssl, CDN, etc are all premium services with a free tier for kicking the tires. As far as I understood in the post, the OP was on that free tier but had registered the domain on Cloudflare (which was probably the added complication).
Sidebar - AWS offers no support even if you’re a paying customer - support is an additional product you have to add on and it is a percentage of your overall usage bill, even if you don’t contact support...
Also I think there’s an issue with what the expectations for support is for. The Cloudflare service itself - the DNS, DDOS protection, ssl, CDN, etc are all premium services with a free tier for kicking the tires. As far as I understood in the post, the OP was on that free tier but had registered the domain on Cloudflare (which was probably the added complication).
Sidebar - AWS offers no support even if you’re a paying customer - support is an additional product you have to add on and it is a percentage of your overall usage bill, even if you don’t contact support...
>Ok but I still don’t get why the op complains about it so publicly the same day (likely immediately) after submitting the ticket without even giving the company a chance to triage and resolve the issue.
Well the blog post isn't really primarily about his specific issue, it's about the systems being severely lacking. A ticket won't resolve that issue.
Well the blog post isn't really primarily about his specific issue, it's about the systems being severely lacking. A ticket won't resolve that issue.
This is a really great idea, but I don't think it's possible for this to not get overused for every little issue.
Once it's overused, it becomes useless.
Isn't this the point of the monetary hold? You can just raise it until the amount of entries become manageable. I'm sure higher level orgs would easily put down $10k to talk to a developer at Microsoft. For indie devs, numbers like $100 or $1000 could be manageable, as long as they can trigger the refund and close the case whenever they want.
Arguably this does block out poorer people from receiving "special" customer service, but there are not really other things people are willing to lose (or put up as collateral) for this type of service. I can't really ship Cloudflare my toaster or car until they resolve my case.
Arguably this does block out poorer people from receiving "special" customer service, but there are not really other things people are willing to lose (or put up as collateral) for this type of service. I can't really ship Cloudflare my toaster or car until they resolve my case.
The first person to do that might at least be able to get their attention..
I like the setup my bank's Ombudsman has -- you must first take your issue to first level support, then escalate it with them if not resolved. If the second level of support denies you, then and only then can you reach out to the Ombudsman.
Any requests that haven't gone through the proper process get auto-rejected.
Any requests that haven't gone through the proper process get auto-rejected.
That is the logical way for things to work, but it requires first level tech support letting you escalate, which is not always the case with non-bank industries.
The current go-to move is to tweet a complaint at the company's Twitter account. This is surprisingly effective across multiple industries and actually was something my wife did that helped resolve a time-sensitive AirBnB issue.
The current go-to move is to tweet a complaint at the company's Twitter account. This is surprisingly effective across multiple industries and actually was something my wife did that helped resolve a time-sensitive AirBnB issue.
This may be due to banks having regulatory requirements - they don’t want regulators in. Because of this, resolving issues before regulators are contacted is in everyone’s interest.
Tech companies being inundated with complaints would actually be expected. Making it easy to complain to ombudspersons will cause the complaints to dwindle naturally as tech companies stop the unethical, even criminal, behavior they engage in.
Very few people will complain about their lifetime Google ban after Google employs appropriate personnel to evaluate such cases.
Today very few people complain about Steam's refund policy, after Valve rewrote their refund policy to actually include refunds, after a judge ended their decade-long crime spree that saw an estimated 20,000 Australians robbed and an unknown quantity globally.
Very few people will complain about their lifetime Google ban after Google employs appropriate personnel to evaluate such cases.
Today very few people complain about Steam's refund policy, after Valve rewrote their refund policy to actually include refunds, after a judge ended their decade-long crime spree that saw an estimated 20,000 Australians robbed and an unknown quantity globally.
[deleted]
Here in Australia the ACCC (Australian Competition & Consumer Commission) has suggested tasking the TIO (Telecommunications Industry Ombudsman) with digital platforms.
Basically the way the TIO works like this: 1. The customer must first go through normal support until they get stuck. 2. The TIO sends the Company a nice letter requesting that they get their act together. 3. The Company then has 10 days to fix the issue or the TIO will escalate. 4. Either the problem got solved or the TIO keeps nagging until it is.
Every time the TIO has to mail the company it also charges the company for having to get involved. level 1: $31 level 2: $260 level 3: $475 level 4: $2250
If an issue gets to the end then the TIO can direct companies to implement solutions costing up to $50,000.
This process heavily incentivises companies to actually provide functional support. I don't think your idea of charging the customer $20 to get help with support would achieve anything.
Basically the way the TIO works like this: 1. The customer must first go through normal support until they get stuck. 2. The TIO sends the Company a nice letter requesting that they get their act together. 3. The Company then has 10 days to fix the issue or the TIO will escalate. 4. Either the problem got solved or the TIO keeps nagging until it is.
Every time the TIO has to mail the company it also charges the company for having to get involved. level 1: $31 level 2: $260 level 3: $475 level 4: $2250
If an issue gets to the end then the TIO can direct companies to implement solutions costing up to $50,000.
This process heavily incentivises companies to actually provide functional support. I don't think your idea of charging the customer $20 to get help with support would achieve anything.
I like the ombudsman idea. A lot.
I am a tech executive in my company and my direct e-mail address is at the bottom of every Web site. Yes, this means I deal with routing all sorts of problems BUT I know instantly when there is a problem. Any problem. And I can actually help. But really it helps me with my job. It's win/win/win.
Plus, poor customer services is simply inexcusable -- you have to treat people the way you want to be treated. You're letting yourself and everyone else down otherwise. There is ethics and morals in business and they are important.
It doesn't have to be my way but there are definitely ways to do it, do it well and not break the bank.
I am a tech executive in my company and my direct e-mail address is at the bottom of every Web site. Yes, this means I deal with routing all sorts of problems BUT I know instantly when there is a problem. Any problem. And I can actually help. But really it helps me with my job. It's win/win/win.
Plus, poor customer services is simply inexcusable -- you have to treat people the way you want to be treated. You're letting yourself and everyone else down otherwise. There is ethics and morals in business and they are important.
It doesn't have to be my way but there are definitely ways to do it, do it well and not break the bank.
A properly setup helpdesk negates the need for an Ombudsman. If a tech company cannot get a helpdesk escalation path correct, what makes you think they'll get an Ombudsman scenario correct?
Ombudsman services are able to resolve conflicts where the customer and business don't agree. They're appropriate for regulated businesses to draw a line in the sand as to what is a legitimate complaint by independently settling cases where the customer and business did not agree. If the Ombudsman hears 100 unresolved complaints and upholds only one then probably the ordinary customer services are working as intended. If they uphold ninety nine something is gravely wrong with your customer services and the regulator needs to step in.
This isn't so dissimilar from the method used in pathology to deliver consistency in results across multiple labs and assay methods. You identify some boundary cases. This is definitely normal, this is definitely cancer but this is borderline, repeat a few times. You replicate and get all the labs to mark your samples. Then you can identify labs (or assay techniques) that aren't reliably putting things in the right categories and demand they improve or stop.
This isn't so dissimilar from the method used in pathology to deliver consistency in results across multiple labs and assay methods. You identify some boundary cases. This is definitely normal, this is definitely cancer but this is borderline, repeat a few times. You replicate and get all the labs to mark your samples. Then you can identify labs (or assay techniques) that aren't reliably putting things in the right categories and demand they improve or stop.
I've worked for various technical help desks over the past 20 years and I'm saying that if a help desk is setup properly, ombudsman services become unnecessary.
A big part of this is how empowered you make your various levels. The SLAs I've always worked with, created, and worked under measured time on calls and customer satisfaction was always one of the more important measurements.
A big part of this is how empowered you make your various levels. The SLAs I've always worked with, created, and worked under measured time on calls and customer satisfaction was always one of the more important measurements.
You could also think of it as a "priority lane" for support. Pay nothing, get whoever for support. Pay $x, complaint goes to an engineer. Pay $y, the product manager. Etc. Skips the middleman process for customers that are in a rush.
If a customer is willing to put up 10k to get their issue resolved, it's probably an issue worth resolving.
If a customer is willing to put up 10k to get their issue resolved, it's probably an issue worth resolving.
Some companies, this doesn't work at all. EG: Postmates. Emailed pretty much every one of their executive/management team about a literally brand damaging issue, and received zero bounces, but also zero replies. Some companies have a policy to ignore unsolicited emails, no matter how serious the issue, as to not fuel the idea that doing so will get results in the future.
This is a really nice idea. This already exists for some companies in the form of Twitter accounts. I doubt the people doing the typee-typee actually has any authority over business processes, or demand a change, but I think they at least have the business owners on their speed dial, vs normal support tickets. But having an email is far better.
Amusingly, Twitter itself could use one. My account (@acangiano) has all of its images censored under "sensitive content" even though they are 100% benign images. No amount of tweets to @TwitterSupport has done anything at all to change it. There is basically no recourse. My account is like 12 years old and has 4.5K followers, so it's not like it's a random spam account, either. ¯\_(ツ)_/¯
I find this whole infrastructure, despite my taking advantage of it, to be very flawed. There're many customers who might have alarming issues, who never get attention because they're fearful to be perceived as a "Karen" for bitching on Twitter.
>they're fearful to be perceived as a "Karen" for bitching on Twitter.
On the other side of the equation sometimes I do want to complain for the sake of complaining without being harassed by some support account.
Especially hate getting obviously automated responses for daring to mention company names even if it would actually escalate to a human.
On the other side of the equation sometimes I do want to complain for the sake of complaining without being harassed by some support account.
Especially hate getting obviously automated responses for daring to mention company names even if it would actually escalate to a human.
Yes it's a good point that companies currently have some sort of Twitter presence trying to address bad publicity posts, often they seem to be able to get things done.
The Ombudsman role is there to get things fixed when all else has failed, and before the angry customer posts to social media.
The Ombudsman role is there to get things fixed when all else has failed, and before the angry customer posts to social media.
> Technology companies need an "ombudsman" - a contact that customers can go to when the normal tech support processes have failed.
From what I read this is nothing to indicate the process failed (so far) just that the user decided to skip to the head of the line by writing a blog post internet style to get something resolved and attention. Failed is not 'I didn't get a reply or find what I needed as quickly as I think it should happen so now let me complain publicly so I get a reply'.
> To avoid the Ombudsman being overused, you could give it a price of say $20, which is always refunded when the case is resolved.
In theory nice but first it would be a 'deposit' and also opens up a host of new issues as far as the money being paid back and how that would be done and so on.
From what I read this is nothing to indicate the process failed (so far) just that the user decided to skip to the head of the line by writing a blog post internet style to get something resolved and attention. Failed is not 'I didn't get a reply or find what I needed as quickly as I think it should happen so now let me complain publicly so I get a reply'.
> To avoid the Ombudsman being overused, you could give it a price of say $20, which is always refunded when the case is resolved.
In theory nice but first it would be a 'deposit' and also opens up a host of new issues as far as the money being paid back and how that would be done and so on.
The core of what you say is correct. I posted about this publicly with two goals in mind: getting help from someone at cloudflare and getting advice on how to avoid this sort of issue from the HN community.
From my perspective Cloudflare's process did fail. Assuming I didn't do something insanely dumb and what I think happened did happen, I would consider that a failure on its own even if the support was perfect afterwards.
Copying from elsewhere:
I address this in TFA.
Essentially I felt that this was alright because when I filed a ticket I was informed that I should expect a long wait and that they recommend that their non-business customers post publicly on their support forum for crowdsourced support because that leads to faster replies. I was unable to log into that forum, and I suspect that may be because the way they set up SSO between the forum login and their main login may have failed in Firefox (with all tracking prevention and ad blockers disabled).
I felt that if a company invites me to ask for support publicly on their forum to save on customer support costs it's reasonable to talk about the issue in another public place.
From my perspective Cloudflare's process did fail. Assuming I didn't do something insanely dumb and what I think happened did happen, I would consider that a failure on its own even if the support was perfect afterwards.
Copying from elsewhere:
I address this in TFA.
Essentially I felt that this was alright because when I filed a ticket I was informed that I should expect a long wait and that they recommend that their non-business customers post publicly on their support forum for crowdsourced support because that leads to faster replies. I was unable to log into that forum, and I suspect that may be because the way they set up SSO between the forum login and their main login may have failed in Firefox (with all tracking prevention and ad blockers disabled).
I felt that if a company invites me to ask for support publicly on their forum to save on customer support costs it's reasonable to talk about the issue in another public place.
This is a fair point. If users are directed to post publically for attention for support this user did the right thing.
Ive run into Cloudflare admin pages that fail with ad blocking before. Test your theory in a private session, it works for me when their site "has issues."
(I think in my case it was adding google metrics from the apps page.)
(I think in my case it was adding google metrics from the apps page.)
I would absolutely pay for this as a service. I just ran into an issue with AWS where there was a problem with my account, I was unable to send email from a specific IP in a pool of IP's bound to an ec2 instance.
The machine / IP combination has been working for years and then one day it just stopped. No emails from ec2-abuse, nothing in my support dashboard. After troubleshooting for a while I bit the bullet and ordered business support.
After cycling through three support agents over 48 hours Amazon said that they were filtering the IP because of spam complaints but neither the Abuse department nor my support liaison was able to see the block - it has been applied at the network level and the network team did not communicate the changes to the other departments.
I told them that I would not be paying for a support plan (over $1000) to resolve a problem that I did not have any notifications or alerts or any way to troubleshoot for myself.
It required talking to two manager and almost a month before they finally agreed that the billing should be refunded. After lots of support tickets, emails, and phone calls I got the money back - but man... An ombudsman would have been amazing.
The machine / IP combination has been working for years and then one day it just stopped. No emails from ec2-abuse, nothing in my support dashboard. After troubleshooting for a while I bit the bullet and ordered business support.
After cycling through three support agents over 48 hours Amazon said that they were filtering the IP because of spam complaints but neither the Abuse department nor my support liaison was able to see the block - it has been applied at the network level and the network team did not communicate the changes to the other departments.
I told them that I would not be paying for a support plan (over $1000) to resolve a problem that I did not have any notifications or alerts or any way to troubleshoot for myself.
It required talking to two manager and almost a month before they finally agreed that the billing should be refunded. After lots of support tickets, emails, and phone calls I got the money back - but man... An ombudsman would have been amazing.
I like this idea a lot. Some people below are suggesting that the ombudsman should be the last stop in a support queue if your problem isn't resolved, and that makes sense sometimes, but other times you can't wait that long!
So, all support systems should have a triage type system with a "nurse" having a constant eye on every new case that comes into the support system. When there's an emergency, such as the one associated with this post, then it should be forwarded to the ombudsman or some other emergency team immediately.
So, all support systems should have a triage type system with a "nurse" having a constant eye on every new case that comes into the support system. When there's an emergency, such as the one associated with this post, then it should be forwarded to the ombudsman or some other emergency team immediately.
I think you're incorrectly assuming most companies want to provide good customer support.
That often ends up being the defacto job of a online community manager, if they have one.
> Technology companies need an "ombudsman" - a contact that customers can go to when the normal tech support processes have failed.
All companies should have that!
All companies should have that!
This sounds nice in theory, but you know that tons of people would just go straight to the ombudsman, thinking they can jump the support queue or bypass established process.
The "shit filtering" workload would be tremendous..
The "shit filtering" workload would be tremendous..
The way this works everywhere is you go through the normal support process until you reach top level support if the situation isn't resolve you go to the ombudmen. If you go before you tried support they will direct you to support first.
Yes I understand the process, but as I said that "they will direct you to support first" work would be non-stop..
Not saying it doesn't work, just that a lot of people wouldn't follow the process, and anyone who sets this up should be prepared for a lot of triage work.
Not saying it doesn't work, just that a lot of people wouldn't follow the process, and anyone who sets this up should be prepared for a lot of triage work.
In Amazon's case, Bezos' email is just that. For AWS, Andy Jassy's (CEO) or Charlie Bell's (SVP) might do just as fine.
Source: Someone who emails them from time to time to get impasses resolved.
Source: Someone who emails them from time to time to get impasses resolved.
Just had to chime in here, but fifteen+ years ago, Bezo's email was ombudsman like, with him or a top level person reading it.
Now, it's just another level of very poor, very scripted support.
I'm fairly sure Amazon has taken the (perhaps wise, in a business sense) approach of not caring if a small percentage of users leave, due to support issues.
The cost of keeping customers with certain support issues, greatly outweighing supporting them.
This is why you have to hunt madly around Amazon's webpage to find contact info, why all forms of help point away from contacting a person, including their chat being bots now, until you move outside of their scripts.
Now, it's just another level of very poor, very scripted support.
I'm fairly sure Amazon has taken the (perhaps wise, in a business sense) approach of not caring if a small percentage of users leave, due to support issues.
The cost of keeping customers with certain support issues, greatly outweighing supporting them.
This is why you have to hunt madly around Amazon's webpage to find contact info, why all forms of help point away from contacting a person, including their chat being bots now, until you move outside of their scripts.
Just one data point but, sending Bezos a nice hello and airing my grievances has worked for me every single time.
Obviously, Bezos may not read those emails but his aides and assistants who do have access to his inbox and act on the emails on his behalf do inherit his complete authority.
Some refs:
https://news.ycombinator.com/item?id=16341154
https://news.ycombinator.com/item?id=17193363
https://news.ycombinator.com/item?id=22286350
https://news.ycombinator.com/item?id=9356182
https://news.ycombinator.com/item?id=20782392
https://news.ycombinator.com/item?id=13512106
Obviously, Bezos may not read those emails but his aides and assistants who do have access to his inbox and act on the emails on his behalf do inherit his complete authority.
Some refs:
https://news.ycombinator.com/item?id=16341154
https://news.ycombinator.com/item?id=17193363
https://news.ycombinator.com/item?id=22286350
https://news.ycombinator.com/item?id=9356182
https://news.ycombinator.com/item?id=20782392
https://news.ycombinator.com/item?id=13512106
[deleted]
The problem is that big companies don't care about giving quality support for their products, and for the most part they get away with it. From their perspective there's no problem to solve.
Your solution basically boils down to "companies are failing at escalating support issues well, so they should escalate support issues well."
Your solution basically boils down to "companies are failing at escalating support issues well, so they should escalate support issues well."
jgc is not a "random" Cloudflare person ;)
My point is precisely that to me, "jgc" IS a random person. How the heck do I know who this person is.
It shouldn't matter, and it should not be required, that someone "known and important" within an organisation decides to start doing hands on tech support in social media following a PR disaster.
If "jgc" is actually someone important within this company then maybe after fixing this issue, they can then go fiox their tech support by setting up and ombudsman and get their PR disasters off the front page of HN.
It shouldn't matter, and it should not be required, that someone "known and important" within an organisation decides to start doing hands on tech support in social media following a PR disaster.
If "jgc" is actually someone important within this company then maybe after fixing this issue, they can then go fiox their tech support by setting up and ombudsman and get their PR disasters off the front page of HN.
It’s the CTO according to their HN profile ;)
You're missing the point. It shouldn't be necessary for someone to know who's-who in order to get things resolved.
You may be right, but knowing "who's who" is very largely how general business gets done. Buying services over the internet from an anonymous black box with no support is a recent disruption.
No, normally you didn’t have to know someone in the C suite to “get business done”. That’s totally unscalable.
What’s a recent development is the complete lack of support when shit goes south. Back when you were interacting with real reps you had people that could see when stuff was obviously wrong and escalate appropriately.
What’s a recent development is the complete lack of support when shit goes south. Back when you were interacting with real reps you had people that could see when stuff was obviously wrong and escalate appropriately.
Did I say anything about C-suite? Your point is nearly word for word the same as mine, I'm not sure why you're replying as a refutation.
For a regular generic Cloudflare customer like me, for personal use, jgc is one of the random Cloudflare person. I have started a spreadsheet with his name, email, comment link, and my copy of screenshot of comment; just in case if I need to email him anything in future.
[email protected] worked amazingly for me back when I bought apple products. Both times I literally get a phone call from someone who has already talked to my local store and has his proposed solution set up for me.
But that was like 10 years ago. Not sure if it's still good.
But that was like 10 years ago. Not sure if it's still good.
> Technology companies need an "ombudsman" - a contact that customers can go to when the normal tech support processes have failed.
Pretty much every technology company already have "ombudsman" for their important clients.
> That "contact a friend in the business who can actually get things done" is a necessary part of a large support organisation and it simply does not exist yet in any tech company that I know of.
I can assure you that every tech company has these "friends" available for their most important clients/customers.
Whether it makes sense for companies to make these contacts available to every customer is another matter.
Pretty much every technology company already have "ombudsman" for their important clients.
> That "contact a friend in the business who can actually get things done" is a necessary part of a large support organisation and it simply does not exist yet in any tech company that I know of.
I can assure you that every tech company has these "friends" available for their most important clients/customers.
Whether it makes sense for companies to make these contacts available to every customer is another matter.
> Technology companies need an "ombudsman" - a contact that customers can go to when the normal tech support processes have failed.
At least in the case of telecom companies they all seem to have executive response teams when the usual channels fail. Send an email to any of their Exec team members (CEO, president, CTO, etc) and it nearly always gets assigned to a special team that solves the problem. I can confirm this out of personal experience with at least AT&T, Comcast, and T-Mobile.
At least in the case of telecom companies they all seem to have executive response teams when the usual channels fail. Send an email to any of their Exec team members (CEO, president, CTO, etc) and it nearly always gets assigned to a special team that solves the problem. I can confirm this out of personal experience with at least AT&T, Comcast, and T-Mobile.
Great idea.
Maybe it could be something that is given to someone when their ticket is closed (or maybe after the first tech response... it depends on the company/ corporate structure).
That way the ombudsman has something to work with, and would slow down the barrage that would occur by having a such a public contact point.
I'm never a fan of 'pay then get refunded' for something that's not your fault, and is entirely out of your control.
Maybe it could be something that is given to someone when their ticket is closed (or maybe after the first tech response... it depends on the company/ corporate structure).
That way the ombudsman has something to work with, and would slow down the barrage that would occur by having a such a public contact point.
I'm never a fan of 'pay then get refunded' for something that's not your fault, and is entirely out of your control.
That's actually a really good idea. Bit like a Credit Card Chargeback, except it's charged to the customer and is refundable.
At 20 USD a pop, Google would make a small fortune!
I'd be the first to pay to get them explain to me some of their misterious weirdnesses.
I'd be the first to pay to get them explain to me some of their misterious weirdnesses.
$20 won't stop anyone with using it.
If it is refundable, make it $300. That is high enough that only people and businesses with a showstopper situation will use it.
If it is refundable, make it $300. That is high enough that only people and businesses with a showstopper situation will use it.
$300 becomes a pretty high barrier for a lot of people. There a places with decent connectivity where that is a good weeks wages
[deleted]
I was talking within the US economy context. Prices obviously can differ globally.
But you get it back, the point is to make it a penalty if you are wasting their time.
> But you get it back
You're missing the part where some people, that actually would need this support, literally wouldn't be able to find that money because of the difference in purchasing power of the local currency.
You're missing the part where some people, that actually would need this support, literally wouldn't be able to find that money because of the difference in purchasing power of the local currency.
I was thinking price for business not personal accounts. Maybe youre thinking those are the same thing in freelance.
No, He's thinking about various developing nations that are trying to have a tech boom.
Phillipines is a good example; Average yearly salary is somewhere around 12k USD, Median is 8k USD.
That means if someone is doing a tech startup there, 300$ is somewhere between a quarter and a half of some employee's pay for a -month-. And given the potential for tight margin/cash flow of startups even in the US... it would still price a lot of smaller players out of the market.
Phillipines is a good example; Average yearly salary is somewhere around 12k USD, Median is 8k USD.
That means if someone is doing a tech startup there, 300$ is somewhere between a quarter and a half of some employee's pay for a -month-. And given the potential for tight margin/cash flow of startups even in the US... it would still price a lot of smaller players out of the market.
Prices can be different for different countries/customers.
A single dollar is a high barrier for some people.
You need an adjustable amount that is based on the annual income/revenue of the person/entity making the request.
Make it high enough to be non-trivial, but not so high that it blocks all effective usage of the safety valve.
Now, if you can solve that problem, I've got some bridges for you in Arizona.
You need an adjustable amount that is based on the annual income/revenue of the person/entity making the request.
Make it high enough to be non-trivial, but not so high that it blocks all effective usage of the safety valve.
Now, if you can solve that problem, I've got some bridges for you in Arizona.
I think the only sane thing you can do is price it at the cost it takes to review it. It will still be out of reach of some people, but at least its not arbitory.
Make it $20 but not refundable. Or $300 and not refundable.
This is a good way to filter the full level of importance. Most places that need a problem solved, now, dammit will be willing to pay.
And, if you want to be nice, you can refund it. Or, if the person was a jerk, you can keep it.
This is a good way to filter the full level of importance. Most places that need a problem solved, now, dammit will be willing to pay.
And, if you want to be nice, you can refund it. Or, if the person was a jerk, you can keep it.
If I have a rightful complaint and it costs me $300 to resolve I am done with that business.
> To avoid the Ombudsman being overused, you could give it a price of say $20, which is always refunded when the case is resolved.
So basically you give the ombudsman $20, and they keep the $20 if they fail to resolve the problem?
So basically you give the ombudsman $20, and they keep the $20 if they fail to resolve the problem?
> To avoid the Ombudsman being overused, you could give it a price of say $20, which is always refunded when the case is resolved.
HN actually acts somewhat like a crowd-sourced ombudsman. People who have an issue write a description and post it to HN. If enough people find it compelling, it makes the front page. Once it makes the front page, someone in authority at the involved tech company will see it, and try their best to resolve it.
HN actually acts somewhat like a crowd-sourced ombudsman. People who have an issue write a description and post it to HN. If enough people find it compelling, it makes the front page. Once it makes the front page, someone in authority at the involved tech company will see it, and try their best to resolve it.
So maybe HN should be billing them.
OP here.
You can post updates with any relevant information. Probably goes without saying, but if the issue has to do with my billing or address please don't post specific details without asking me first.
I will link to this comment from TFA for verification. (Edit: added to the bottom. If you need more verification you have my email.)
Edit2: I see that the domain is back in my account and listed as "Pending Nameserver Update". I don't think that's because of something I did.
You can post updates with any relevant information. Probably goes without saying, but if the issue has to do with my billing or address please don't post specific details without asking me first.
I will link to this comment from TFA for verification. (Edit: added to the bottom. If you need more verification you have my email.)
Edit2: I see that the domain is back in my account and listed as "Pending Nameserver Update". I don't think that's because of something I did.
Thanks. Appreciate it. I’ll let the team look into it and communicate with you first.
I do wonder; when a domain is suspended, why don't you send out a courtesy notification e-mail?
what if their email address is @ the domain
Even if cloudflare stopped publishing the correct MX records for a domain, wouldn't Cloudflare still be able to send email to the MX server that had been specified? Seems like email should at worst be down for everyone but Cloudflare.
Well, that's poor planning on the user's part. In this case the OP stated he registered using a gmail address.
If the only contact information the registrar has is the @ of the domain then that likely mean that the registrar is failing the contractual obligations that exist between the registry and registrar. While I have not read the exact contract that exist for .com registrars, I am confident enough to say that you can not do that.
The OP made it clear that he used a gmail account for contact so there was no excuse for him to have not received a message regarding the deletion of his domain records.
I think it's pretty common advice that anything domain-related needs to be a different email. I've heard that, and I've only ever admin'd my personal domain. For exactly that reason I registered for Cloudflare with a gmail account.
Also display a prominent notification in the web interface then.
I appreciate you're looking in to this.
I’ve worked for Cloudflare since it was 24 people. I care a hell of a lot about our customers. I know eastdakota does also.
I can tell. Cloudflare has been a delight to use up to now and I'm grateful I'm getting this kind of support when I've only used your loss-leader services.
This is probably obvious but you're getting this kind of support because they're getting terrible PR by virtue of your post being at the top of HN. Not because he's such a great guy that cares so much about the customers. Not saying he isn't a great guy, or that anyone at cloudflare actually wanted this to be your experience.
If he’d emailed me directly I would have done the same thing I did. I emailed the head of engineering and support and asked for an explanation. I then jumped into the relevant chat room.
I do this sort of thing all the time. Sure, it’s unfortunate this is #1 on HN, but shrug. Fixing the problem and figured out what happened is important.
I do this sort of thing all the time. Sure, it’s unfortunate this is #1 on HN, but shrug. Fixing the problem and figured out what happened is important.
The more important question is what will be done to prevent this from happening again to somebody else (obviously this doesn’t need a response here, I assume an internal investigation is underway and this will be publicly communicated after its complete).
Hi John, your website's contact link is pointing to your twitter account and you don't have messages opened.
What is the best email to reach out to you?
Thank you.
What is the best email to reach out to you?
Thank you.
To be fair this seems to happen with a bunch of tech companies, such as Stripe too. I wish more companies took customer support seriously.
I remember about 4-5 months ago I spent like 2 weeks going back and forth with Stripe's regular email support trying to understand their docs for SCA.
I kept getting a new rep who repeated the same things the previous reps were saying, which also had no bearing on what I was asking. It was basically a copy / paste from a script loop.
Then something negative about Stripe was on the HN front page and I happened to comment about a bad experience with the new SCA docs.
Within a few hours I was put in contact with a lead developer from Stripe who went as far as creating custom flow charts for my use case that wasn't covered in the docs and it was a pleasant experience, where "pleasant" felt like the person receiving the email was reading the words I wrote instead of just skimming them and pasting a boilerplate response.
But it only happened because of the HN comment. If that thread never appeared on HN, I'm not even sure I would still be using Stripe.
I remember about 4-5 months ago I spent like 2 weeks going back and forth with Stripe's regular email support trying to understand their docs for SCA.
I kept getting a new rep who repeated the same things the previous reps were saying, which also had no bearing on what I was asking. It was basically a copy / paste from a script loop.
Then something negative about Stripe was on the HN front page and I happened to comment about a bad experience with the new SCA docs.
Within a few hours I was put in contact with a lead developer from Stripe who went as far as creating custom flow charts for my use case that wasn't covered in the docs and it was a pleasant experience, where "pleasant" felt like the person receiving the email was reading the words I wrote instead of just skimming them and pasting a boilerplate response.
But it only happened because of the HN comment. If that thread never appeared on HN, I'm not even sure I would still be using Stripe.
> was a pleasant experience, where "pleasant" felt like the person receiving the email was reading the words I wrote instead of just skimming them and pasting a boilerplate response.
Crazy how far down the bar has gone.
Crazy how far down the bar has gone.
With many other companies I would agree with you, but in this case it is actually just because he's running a tight ship. I've had friends who've reached out directly via email and received the same level of support.
You mention loss-leader services, but the model is no doubt setup to funnel you towards profitable services.
I switched a personal domain to them, but it won't be long before I get my employer to move their resources over to them, assuming I see this all play out well.
I switched a personal domain to them, but it won't be long before I get my employer to move their resources over to them, assuming I see this all play out well.
John, you guys seem pretty awesome about posting root cause analysis and being transparent, can we expect to see at least a comment (if not a blog post) summarizing what took place here?
Maybe contact support first, figure out what happened and why before writing the blog post?
Maybe it's a cloudflare issue, maybe it's an honest mistake, maybe they are bad at customer service... either way, it's not great to shame a company before you even open a support ticket or talk to someone to find out.
Maybe it's a cloudflare issue, maybe it's an honest mistake, maybe they are bad at customer service... either way, it's not great to shame a company before you even open a support ticket or talk to someone to find out.
I address this in TFA.
Essentially I felt that this was alright because when I filed a ticket I was informed that I should expect a long wait and that they recommend that their non-business customers post publicly on their support forum for crowdsourced support because that leads to faster replies. I was unable to log into that forum, and I suspect that may be because the way they set up SSO between the forum login and their main login may have failed in Firefox (with all tracking prevention and ad blockers disabled).
I felt that if a company invites me to ask for support publicly on their forum to save on customer support costs it's reasonable to talk about the issue in another public place.
Essentially I felt that this was alright because when I filed a ticket I was informed that I should expect a long wait and that they recommend that their non-business customers post publicly on their support forum for crowdsourced support because that leads to faster replies. I was unable to log into that forum, and I suspect that may be because the way they set up SSO between the forum login and their main login may have failed in Firefox (with all tracking prevention and ad blockers disabled).
I felt that if a company invites me to ask for support publicly on their forum to save on customer support costs it's reasonable to talk about the issue in another public place.
Oh, the horror of recommending a public forum. (/s, of course.)
A fair number of questions aren't unique - product questions, how to use an API, etc. Someone may have asked a similar question, in which case you'll find an answer, find it faster than it'll take to hear back from the support team, and it deflects an unnecessary (already answered) ticket. That should be a win all around.
Now if you do have a novel question or something account-specific, by all means, open a ticket. There you'll get replies from people who can look up your account and give you specific answers.
The ombudsman tip in this post doesn't make a whole lot of sense when the normal support process wasn't really given a chance before making the blog post.
A fair number of questions aren't unique - product questions, how to use an API, etc. Someone may have asked a similar question, in which case you'll find an answer, find it faster than it'll take to hear back from the support team, and it deflects an unnecessary (already answered) ticket. That should be a win all around.
Now if you do have a novel question or something account-specific, by all means, open a ticket. There you'll get replies from people who can look up your account and give you specific answers.
The ombudsman tip in this post doesn't make a whole lot of sense when the normal support process wasn't really given a chance before making the blog post.
When your entire site and suite of applications are offline, especially during business hours, waiting who knows how long for a support queue isn't really an option. Outages like this can kill a business or cost millions.
(op here). To be fair to Cloudflare a business with millions on the line should be paying a lot more money to Cloudflare than I, a college student with significantly less on the line, would. If I had the option to pay $1,000 right now and get everything back up instantly I wouldn't take it because it isn't worth that much to me.
I realize that... the issue isn't necessarily tethered to your paying status, and it could just as easily happen to someone with a high value business account.
This is unlikely for a few reasons.
1. If you write into support as an Enterprise customer, you're basically getting an automatic response.
2. Enterprise customers can dial a direct line to support and have this under investigation within minutes.
3. Enterprise customer have a dedicated account manager, and in most cases a Solutions Engineer helped get everything setup. Usually Enterprise domains get locked, so they cannot get caught by linting type services. The likelihood of a domain being removed under those conditions is very low.
4. The most common cause of this type of thing is the Name Servers no longer pointing to Cloudflare, no one noticing for a while. Cloudflare periodically checks to see if a domain is still using CF name servers, and if they aren't they get moved as the assumption is they are no longer on Cloudflare. I don't know that this is what happened here, but it's easily the most common issue on lower tier plans. Enterprise companies often have people monitoring their infra, thus they catch this before Cloudflare conducts the removal of the domain.
Source: used to work on the Cloudflare support team.
1. If you write into support as an Enterprise customer, you're basically getting an automatic response.
2. Enterprise customers can dial a direct line to support and have this under investigation within minutes.
3. Enterprise customer have a dedicated account manager, and in most cases a Solutions Engineer helped get everything setup. Usually Enterprise domains get locked, so they cannot get caught by linting type services. The likelihood of a domain being removed under those conditions is very low.
4. The most common cause of this type of thing is the Name Servers no longer pointing to Cloudflare, no one noticing for a while. Cloudflare periodically checks to see if a domain is still using CF name servers, and if they aren't they get moved as the assumption is they are no longer on Cloudflare. I don't know that this is what happened here, but it's easily the most common issue on lower tier plans. Enterprise companies often have people monitoring their infra, thus they catch this before Cloudflare conducts the removal of the domain.
Source: used to work on the Cloudflare support team.
> Oh, the horror of recommending a public forum. (/s, of course.)
He _tried_ to request help in their recommended public forum, but could not due to login issues.
Seems entirely reasonable to me to then go and ask about out in a different public forum?
He _tried_ to request help in their recommended public forum, but could not due to login issues.
Seems entirely reasonable to me to then go and ask about out in a different public forum?
And then/also submitted a support ticket, unless I'm misunderstanding their post?
> I spent some time thinking about if it was fair for me to post this on the same day as I filed a support ticket with Cloudflare.
> I spent some time thinking about if it was fair for me to post this on the same day as I filed a support ticket with Cloudflare.
I think it is a bit harsh to say that the OP was trying to shame anyone. He did post a support ticket. When your domain is "off-line" it is literally a shit hit the fan moment, and no one likes to be given the run-around.
To combat, hey my issue is always a Sev1 ticket, one can probably institute something like, here is a red button and if you click it, we will charge 100$. If it is indeed an issue that caused you to lose 90%(say) traffic and it was our fault, we will return the money.
To combat, hey my issue is always a Sev1 ticket, one can probably institute something like, here is a red button and if you click it, we will charge 100$. If it is indeed an issue that caused you to lose 90%(say) traffic and it was our fault, we will return the money.
Microsoft used to have this policy (maybe still does) for some support options. It cost $100 to open a case, and if the case was the result of a bug in the product, you got your money back.
Imagine how you would feel though if you were in OP’s shoes, where your DNS entries disappeared and client emails lost, through no fault of your own, but support won’t even listen to you until you pay money.
I think I'd be fine with that?
I would trust Cloudflare to pay me back, and I'm already putting something on the line. If it really was my fault this is going to be very embarrassing.
Edit: In case it isn't clear from context I'm the OP so I'm pretty sure I know what I feel.
I would trust Cloudflare to pay me back, and I'm already putting something on the line. If it really was my fault this is going to be very embarrassing.
Edit: In case it isn't clear from context I'm the OP so I'm pretty sure I know what I feel.
Except you can bet that a lot of cases will end up with a battle of back and forth emails arguing over whose fault it was and who should pay the $100.
The vast majority of support tickets are the customers 'fault'. They'll be things like setting up DNS records incorrectly, believing they don't need to pay a renewal with their old registrar just because they moved their DNS over to cloudflare, frustrations over cached resources, etc.
The vast majority of support tickets are the customers 'fault'. They'll be things like setting up DNS records incorrectly, believing they don't need to pay a renewal with their old registrar just because they moved their DNS over to cloudflare, frustrations over cached resources, etc.
(op here) My belief is that I completely transfered everything to cloudflare, paid in advance, and didn't touch it. If this was as much my fault as any of the common tickets you listed I will deeply regret having posted this and will apologise to Cloudflare.
In the blog post he mentions that Cloudflare recommends posting your problem to their open community forum to try and get things resolved. He couldn't access their forum, so he posted it on a different one.
And maybe not too.
Please explain something to me.
For me and my of my customers, having your "entire cloud deleted" is like... the #1 nightmare scenario.
So why does this capability/function even exist for active accounts at CloudFlare? It sounds like the OP fell victim to what is essentially a regular process.
Or to put it another way: No amount of explanation or assurance is ever going to make me feel comfortable with my doctor having a handgun as one of his medical instruments.
For me and my of my customers, having your "entire cloud deleted" is like... the #1 nightmare scenario.
So why does this capability/function even exist for active accounts at CloudFlare? It sounds like the OP fell victim to what is essentially a regular process.
Or to put it another way: No amount of explanation or assurance is ever going to make me feel comfortable with my doctor having a handgun as one of his medical instruments.
It doesn't sound like it actually deleted anything other than the DNS entries. It's not like any actual data got lost.
I dunno, but DNS entries are actual data.
I certainly don’t know what all the DNs entries on my 15 domains are, so if they disappear it’s not going to be easy to get it all back.
I certainly don’t know what all the DNs entries on my 15 domains are, so if they disappear it’s not going to be easy to get it all back.
(OP here)
It appeared to me as though they deleted my domain registration from their system. That's mostly why I freaked out. I wasn't sure if I'd even be able to transfer my registration easily.
However, on the advice of someone from HN, I found that if I went through the flow of linking an external domain to my cloudflare account the registration details came back.
It appeared to me as though they deleted my domain registration from their system. That's mostly why I freaked out. I wasn't sure if I'd even be able to transfer my registration easily.
However, on the advice of someone from HN, I found that if I went through the flow of linking an external domain to my cloudflare account the registration details came back.
Why can you delete DNS records? Well it's pretty common to want to do so (usually followed by adding some new ones).
Your doctor has a syringe as a medical instrument. They can kill if not used correctly.
Your doctor has a syringe as a medical instrument. They can kill if not used correctly.
This is not the user manually deleting records.
This is the doctor seeing a patient that hasn't moved in a few hours and executing them:
https://community.cloudflare.com/t/does-cloudflare-automatic...
Oh wait, he was just sleeping? Ah, well... "oops".
There is exactly one reason you'd delete customer data without their active involvement: They're overdue on their bill and have ignored all warnings and communications. That's it. There are no other reasons.
Feel free to disable things if they're a security risk, but Do. Not. Delete. Customer. Data.
Ever.
This is the doctor seeing a patient that hasn't moved in a few hours and executing them:
https://community.cloudflare.com/t/does-cloudflare-automatic...
Oh wait, he was just sleeping? Ah, well... "oops".
There is exactly one reason you'd delete customer data without their active involvement: They're overdue on their bill and have ignored all warnings and communications. That's it. There are no other reasons.
Feel free to disable things if they're a security risk, but Do. Not. Delete. Customer. Data.
Ever.
Ugh! I may have interpreted the comment I responded to wrong, and replied to my own strawman. I think you need the power for the user to delete records (obviously) but this could mean a bug in the system could cause the records to be deleted. But it sounds from what you are saying that they sort of "intentionally" deleted the records (not sure if a human had the intent or an algorithm?) but if intentionally deleted yes that sucks, there is no reason for that I can think of, given what they do is about handling DDOS.
Unless the mechanism exists for legal reasons, i.e. they are forced to take some sites down by the government or law suits.
Unless the mechanism exists for legal reasons, i.e. they are forced to take some sites down by the government or law suits.
Having the CTO offer up support in this way is heartening. Especially because infrastructure suppliers are getting more and more centralized, which leads to possible SPOFs even with individual company distributed environments.
The problem is that big companies don't care about giving quality support for their products, and for the most part they get away with it. From their perspective there's no problem to solve.
Your solution basically boils down to "companies are failing at escalated support issues well, so they should escalate support issues well."
Your solution basically boils down to "companies are failing at escalated support issues well, so they should escalate support issues well."
I posted this under the wrong parent comment, moved but cant delete.
Context for the lazy, jgrahamc is the longtime CTO of Cloudflare.
Please do update if possible.
It's likely a good learning for all.
It's likely a good learning for all.
Of course, I just don’t want to promise something when it might be revealing information about an individual account.
He gave me an incredibly detailed and fast response to an issue I was fact-checking.
First, you are awesome, really :)
Second, a bunch of honest questions:
Did you consult to your supervisor (or anyone with authority) to be able to bypass the support process (if there is any) like this? If so what was the response? If the response was negative, how did you convince people? After things resolve, can you kindly post how many spam or unrelated emails you receive so that it will be an example to the industry?
I'd like to put my skepticism on hold and blindly believe that your post is a reflection of pure concern and not just a PR stunt for damage control.
Second, a bunch of honest questions:
Did you consult to your supervisor (or anyone with authority) to be able to bypass the support process (if there is any) like this? If so what was the response? If the response was negative, how did you convince people? After things resolve, can you kindly post how many spam or unrelated emails you receive so that it will be an example to the industry?
I'd like to put my skepticism on hold and blindly believe that your post is a reflection of pure concern and not just a PR stunt for damage control.
The CTO at Cloudflare has to ask his supervisor before trying to help someone over at hacker news? :)
I didn't know which probably slightly proves some point other people made.
jgrahamc is the CTO of Cloudflare. His only supervisor is eastdakota/ Matthew Prince, the CEO of Cloudflare.
Then I'd like to know the CEO's opinion then.
CTO of any company doesn't need the CEO's permission to make a customer relations post.
It's not about permission but cooperation of two disciplines.
It's hard to understand criticism when it is not put into words.
From my perspective, a developer from the company the article is about arrives in the comment section. I ask an honest question about the communication process internal to the company. I am told this developer is the CTO and is supervised by the CEO. Then I state the fact that I would appreciate if we also knew what the CEO thinks about this situation.
I don't literally expect answers from the people I addressed and that's OK, they are probably too busy working on the subject matter. If they take time and write one, that is really admirable and I'd be thankful.
What I don't understand is why it is considered inappropriate to be curious about these topics, as judged by the amount of downvotes I received.
Maybe the tone I intended for my post got lost in translation, since I'm not a native English speaker.
From my perspective, a developer from the company the article is about arrives in the comment section. I ask an honest question about the communication process internal to the company. I am told this developer is the CTO and is supervised by the CEO. Then I state the fact that I would appreciate if we also knew what the CEO thinks about this situation.
I don't literally expect answers from the people I addressed and that's OK, they are probably too busy working on the subject matter. If they take time and write one, that is really admirable and I'd be thankful.
What I don't understand is why it is considered inappropriate to be curious about these topics, as judged by the amount of downvotes I received.
Maybe the tone I intended for my post got lost in translation, since I'm not a native English speaker.
It comes across as entitled, and also just strange and unnecessary. Like, why would you need or expect to hear directly from the CEO of Cloudflare when we're already hearing from the CTO? I'm not sure why your reply here got downvoted as well though, as I found it helpful that you explained you were simply interested in the CEO's position rather than actually expecting a response from him. So, just trying to illustrate how the initial comment came across to me, and perhaps others.
Thanks for the feedback.
"I'd like to hear CEO's opinion" probably doesn't mean "I wonder what the CEO thinks about public communication by his/her colleagues inside internet forums". I'm reminded English is hard even when I think I have a basic understanding of common speech patterns (parlance?).
"I'd like to hear CEO's opinion" probably doesn't mean "I wonder what the CEO thinks about public communication by his/her colleagues inside internet forums". I'm reminded English is hard even when I think I have a basic understanding of common speech patterns (parlance?).
Correct. In fact, even the second phrasing, "I wonder what the CEO thinks..." could easily be interpreted as expecting a response. It seems you'd have to really bend over backwards to avoid that, perhaps explicitly saying something like, "Obviously I wouldn't expect him to respond here, but I'm curious whether the CEO shares this view." Or something like that.
This reply might be what you're looking for: https://news.ycombinator.com/item?id=22407399
Cloudflare is a solid company with a good reputation and has done a lot for general welfare of the internet. The staff there generally care about customers and it seems like they're trying to figure out what went wrong here, and of course preventing negative PR is always a bonus.
There's really not much of a process when you're a top officer at a company, you have the freedom and responsibility to exercise your judgement and do whatever it takes for the good of the organization.
Cloudflare is a solid company with a good reputation and has done a lot for general welfare of the internet. The staff there generally care about customers and it seems like they're trying to figure out what went wrong here, and of course preventing negative PR is always a bonus.
There's really not much of a process when you're a top officer at a company, you have the freedom and responsibility to exercise your judgement and do whatever it takes for the good of the organization.
> BTW If you, dear reader, ever find yourself so frustrated with Cloudflare that you feel like your only recourse is a blog post... my email is [email protected] and I’m happy to hear from people.
I know that people will think it's great that you are doing this and I also know that you think it's good (for you) to have a feel for the issues that frustrate every day users. But I think it's not a great use of a company execs time and I am not even sure it's a good way to deploy resources at Cloudflare.
The reason is people will tend to (as a rule) do as little as they can themselves but then use as a hammer the court of public opinion to get something resolved.
You say 'ever find yourself so frustrated with Cloudflare' but you know that in itself is different for different people. What will happen is you will get people using you as a help desk and then after you don't help them as quickly as they think you should they will then follow up with a post, comment or story about how you did nothing.
Separately if someone is posting publicly about an issue (as this person is) and if you can verify that it's actually coming from the customer (I mean who says it is actually?) I don't think you need them to say it's ok to resolve online. In fact to me it's the opposite. You take the time to reach out publicly and you take what follows good or bad even calling you out (the customer yes you can do that by the way) if you think they didn't put the appropriate effort into finding an answer.
I know that people will think it's great that you are doing this and I also know that you think it's good (for you) to have a feel for the issues that frustrate every day users. But I think it's not a great use of a company execs time and I am not even sure it's a good way to deploy resources at Cloudflare.
The reason is people will tend to (as a rule) do as little as they can themselves but then use as a hammer the court of public opinion to get something resolved.
You say 'ever find yourself so frustrated with Cloudflare' but you know that in itself is different for different people. What will happen is you will get people using you as a help desk and then after you don't help them as quickly as they think you should they will then follow up with a post, comment or story about how you did nothing.
Separately if someone is posting publicly about an issue (as this person is) and if you can verify that it's actually coming from the customer (I mean who says it is actually?) I don't think you need them to say it's ok to resolve online. In fact to me it's the opposite. You take the time to reach out publicly and you take what follows good or bad even calling you out (the customer yes you can do that by the way) if you think they didn't put the appropriate effort into finding an answer.
I disagree with you. It matters enormously that people like me are available. Sure there are time wasters who’ll send me email. But I don’t care. Dealing with a small number of real customers doesn’t take a lot of my time and matters.
Everyone optimizes for the worst case. They think “if I give out my email I’ll get tons of useless email”. I can assure you I get 10x the crap sent to me on LinkedIn than via direct emails from customers or others.
Everyone optimizes for the worst case. They think “if I give out my email I’ll get tons of useless email”. I can assure you I get 10x the crap sent to me on LinkedIn than via direct emails from customers or others.
Thank you, this is a good attitude to have and I wish more company exec’s thought like this. We might have fewer post’s like OP’s story.
My little firm is nothing like the scale of CF but I do the same and have dropped my email address around a fair bit. I do get some pretty interesting missives that make it past the filters but signal to noise is very high in my inbox.
I deleted my Linkedin account well over a year ago and I still get emails from them saying my profile is being viewed. Tossers.
I deleted my Linkedin account well over a year ago and I still get emails from them saying my profile is being viewed. Tossers.
If I ever have someones direct email I don’t think I would use it in any situation that I wouldn’t appreciate getting one myself.
> I know that people will think it's great that you are doing this and I also know that you think it's good (for you) to have a feel for the issues that frustrate every day users. But I think it's not a great use of a company execs time and I am not even sure it's a good way to deploy resources at Cloudflare.
As a counter example, Jeff Bezos (whose time may be worth more than anybody else's) famously audits his email for customer complaints and occasionally derails an organization for a day or two in order to figure out what happened. He stands behind this practice and has said that he often picks out cases where the anecdotal complaint is counter to data that he's been presented, and that more often than not the anecdotes are correct and find a shortcoming in the data. IMO it also demonstrates a culture of caring and following up about anecdotes to others whose time is worth less than his own.
As a counter example, Jeff Bezos (whose time may be worth more than anybody else's) famously audits his email for customer complaints and occasionally derails an organization for a day or two in order to figure out what happened. He stands behind this practice and has said that he often picks out cases where the anecdotal complaint is counter to data that he's been presented, and that more often than not the anecdotes are correct and find a shortcoming in the data. IMO it also demonstrates a culture of caring and following up about anecdotes to others whose time is worth less than his own.
> As a counter example, Jeff Bezos (whose time may be worth more than anybody else's) famously audits his email for customer complaints and occasionally derails an organization for a day or two in order to figure out what happened.
Is that why Amazon is growing more and more notorious for selling fraudulent items over the years?
Is that why Amazon is growing more and more notorious for selling fraudulent items over the years?
Both of those things could be true without one being causal or even related.
Maybe someone needs to email Jeff about that.
[deleted]
Now that the OP has given a go ahead to go public, We will eagerly wait for your update jgc
martin1975(1)
Occasions are rare where I get to say "hey, I built a thing that might help here!" - so forgive me as I take this opportunity with both hands.
Whether this was a bug or a rare protective mechanisme, there will be times when your DNS provider makes a mistake and removes records. You mentioned in your post your DNS isn't hard to reproduce, but how certain are you that _all_ records are restored? How long do you have to fight DNS issues before it's OK?
I built DNS Spy [1] for this exact occasion. It monitors your DNS for any changes made, keeps a version of all DNS records (current & former) and allows you to restore/download a BIND9 zone file for your zone. You can easily import this into any commercial DNS provider or in your own BIND9/PowerDNS setup.
I would love to hear feedback on how DNS Spy could be improved when DNS disasters like these occur!
[1] https://dnsspy.io/
Whether this was a bug or a rare protective mechanisme, there will be times when your DNS provider makes a mistake and removes records. You mentioned in your post your DNS isn't hard to reproduce, but how certain are you that _all_ records are restored? How long do you have to fight DNS issues before it's OK?
I built DNS Spy [1] for this exact occasion. It monitors your DNS for any changes made, keeps a version of all DNS records (current & former) and allows you to restore/download a BIND9 zone file for your zone. You can easily import this into any commercial DNS provider or in your own BIND9/PowerDNS setup.
I would love to hear feedback on how DNS Spy could be improved when DNS disasters like these occur!
[1] https://dnsspy.io/
(OP here). That looks really useful. If I was running a real service I would definitely look into it. Because this is just the personal website and email of a college student I don't think I could justify the expense when using something like Uptime Robot to monitor if a single record points to a web server would probably give me close to the same reliability.
Oh I absolutely agree!
If you're a business, whether it's a SaaS or "just" a marketing website for your brick & mortar store, I think it's crucial to have back-ups. Most people think of backups as files, database dumps, previous versions, etc of their website. But the configuration data (in the form of DNS) isn't often considered.
You're tech-savvy and can restore your DNS records because you know yoru servers' IP address and your MX records, but who else could do the same?
If you're a business, whether it's a SaaS or "just" a marketing website for your brick & mortar store, I think it's crucial to have back-ups. Most people think of backups as files, database dumps, previous versions, etc of their website. But the configuration data (in the form of DNS) isn't often considered.
You're tech-savvy and can restore your DNS records because you know yoru servers' IP address and your MX records, but who else could do the same?
[deleted]
The issue I see with this is that
1) You can't use it after the fact.
2) It's very specialized.
People are not going to set up dozens and dozens of services to monitor for really rare things. It should be part of general purpose monitoring suite.
1) You can't use it after the fact.
2) It's very specialized.
People are not going to set up dozens and dozens of services to monitor for really rare things. It should be part of general purpose monitoring suite.
How do you get all records for domains that don’t allow zone transfer (most don’t)? I’ve always thought it was impossible to get 100% accurate results with normal DNS queries (ex: ALL).
YOU as a customer import them. This is not a DNS spider service like https://dnsdumpster.com/ which also will not be 100% accurate in most cases, but finds a lot more for most domains.
>but how certain are you that _all_ records are restored?
A solution to this is to keep DNS under version control with eg Terraform and deployed by CI. master is then authoritative
A solution to this is to keep DNS under version control with eg Terraform and deployed by CI. master is then authoritative
Nice! A few years ago I whipped up something simple using dig, diff, cron and some bash scripts. It's handy to get alerted when something changes, and I've definetly caught a few unintentional changes.
Just wanted to point out that the menu doesn't work properly on mobile in case you want to fix it.
https://imgur.com/a/TcibAKs
https://imgur.com/a/TcibAKs
Looks really useful and fulfills a very important purpose. What good is all your backups if you can't get your services back up due to missing DNS configuration.
DNS Spy costs money? wow the art of sysadmin really is dead. having this kind of tight control of DNS used to be a given.
Just signed up - it's pretty nice. Any plans to add SSHFP or other record types?
Do you use the cloudflare api to get them all? Otherwise you'll miss some
There's a number of Cloudflare folks who are HN regulars, so hopefully you'll get some answers. Hopefully it's something they can reverse.
But as a general reminder to everyone (I think this is an unfortunately common problem from a number of companies): If this is how your company handles account issues, you're probably wrong. Whether it's automated or manual, a user should be able to access all of their own information even when you decide to no longer provide them service. And you should test and retest the ability for people who you now deny service to transfer out.
But as a general reminder to everyone (I think this is an unfortunately common problem from a number of companies): If this is how your company handles account issues, you're probably wrong. Whether it's automated or manual, a user should be able to access all of their own information even when you decide to no longer provide them service. And you should test and retest the ability for people who you now deny service to transfer out.
FWIW I recently evaluated a few DNS companies after Namecheap ballsed up our MX records in a similar way.
I actively looked for someone we could pay money to, so we are their customer (as opposed to being a free tier user, effectively a cost)
The winner was DNSimple[1], who do exactly 1 thing, and they do it extremely well. And they are small enough to not take themselves too seriously[2], which I really appreciate.
Oh and their normal support channel is email, and everyone in the company takes a turn. I tested out their support before signing up and quickly heard back from a competent engineer, so they passed that test too.
[1] https://dnsimple.com [2] https://dnsimple.com/dnsound <— bonkers
I actively looked for someone we could pay money to, so we are their customer (as opposed to being a free tier user, effectively a cost)
The winner was DNSimple[1], who do exactly 1 thing, and they do it extremely well. And they are small enough to not take themselves too seriously[2], which I really appreciate.
Oh and their normal support channel is email, and everyone in the company takes a turn. I tested out their support before signing up and quickly heard back from a competent engineer, so they passed that test too.
[1] https://dnsimple.com [2] https://dnsimple.com/dnsound <— bonkers
Digitalocean has free dns service with an api; it's good and reliable.
Running my own dns looks more and more reasonable though.
Running my own dns looks more and more reasonable though.
Perhaps, but Digital Ocean also host spammers / scammers and doesn't do anything about them when they're reported.
I am sure AWS, OCI, GCP, etc. all host scam websites with varying degrees of removal efficiency. What cases are you referring to specifically? Did they state they were not going to take these sites down or what was the context that you object to?
In this context, that sounds like an endorsement, honestly. If we're discussing providers that are willing to kill your services too easily, then saying that a provider is unwilling to cut service even to problem customers sounds like an amazing reason to use them.
I don't see how this is relevant.
You listed their good points, the other poster listed some counterpoints. The one post is no less relevant than the other in a discussion about possible DNS hosting options IMO.
Though I think the post would benefit from some citations to improve its relevance/usefulness otherwise it is little better than personal opinion/conjecture.
Unless you are specifically questioning the relevance of hosting spammers, on which case: If that is true (again, some examples would be helpful here) and you intend to host your own mail servers via their services not just the MX records pointing to other mail services, you could find yourself blocked by association at some point. False positives are a big problem in this area and can be much admin to clear up.
Though I think the post would benefit from some citations to improve its relevance/usefulness otherwise it is little better than personal opinion/conjecture.
Unless you are specifically questioning the relevance of hosting spammers, on which case: If that is true (again, some examples would be helpful here) and you intend to host your own mail servers via their services not just the MX records pointing to other mail services, you could find yourself blocked by association at some point. False positives are a big problem in this area and can be much admin to clear up.
I use no-ip as dyndns for my home ip, so I can log in at home from outside. Recently at work my putty failed to connect, so I figured my internet line was down, it happens.
Came home, internet works fine. Everything looked just fine.
Back at work next day still can't connect. So I tried pinging, and I immediately see that the ip my home hostname resolves to is not what my ISP has. So I go to nslookup and try a DNS server I know (another local ISP), and it resolves to what I expected.
A bit of checking later I find that at work they've started using OpenDNS, and OpenDNS has blocked all of no-ip due to malware and spam.
So yeah, could be relevant.
Came home, internet works fine. Everything looked just fine.
Back at work next day still can't connect. So I tried pinging, and I immediately see that the ip my home hostname resolves to is not what my ISP has. So I go to nslookup and try a DNS server I know (another local ISP), and it resolves to what I expected.
A bit of checking later I find that at work they've started using OpenDNS, and OpenDNS has blocked all of no-ip due to malware and spam.
So yeah, could be relevant.
Then you probably shouldn't be recommending DNS providers.
Are you saying if you use their dns you will get spammed/scammed?
I think they're mad that DigitalOcean's IP range shows up in their ssh logs with failed authentication. A lot of people think that it's the ISP's job to regulate all traffic on their network, judging from the comments here, DigitalOcean at one time or another has failed to do that.
I host all my personal stuff there, including something that updates their DNS via an API. They've been great to me.
I host all my personal stuff there, including something that updates their DNS via an API. They've been great to me.
NS1 could be another one to look at. I have never used their services (directly), but I've noticed Netlify uses them for their DNS services.
I've used ns1 for a few years, they've been great!
We used ns1 at my last job, they were indeed great to us. We moved from self-hosting DNS because the DNS servers would randomly become unresponsive and would start returning fake records. After switching to ns1 and getting our first bill, we realized that a lot of our network equipment apparently did a DNS lookup for every log line. This resulted in an exceedingly large bill, which ns1 happily reversed (we did fix our stuff ;).
I thought self-hosting DNS would be easy, since it is 40 year old tech now.
Even now, like you, I get the occasional (ie. once per year) named segfaults or it randomly stops responding over TCP.
Few SaaS products are less effort than one reboot per year, but still worth it IMO.
Even now, like you, I get the occasional (ie. once per year) named segfaults or it randomly stops responding over TCP.
Few SaaS products are less effort than one reboot per year, but still worth it IMO.
Since then I've started using CoreDNS, which does seem easier to monitor. (I don't know if it's faster or more reliable, but it has a lot of ways to figure out what's going wrong. As it turns out, DNS causes people a lot of trouble these days when they use it for service discovery, and their services discover each other hundreds of times a second. So that's why DNS servers grew APIs and observability features.)
Do you know what the reliability of their free plan is like?
I'm on their free plan and have never had an issue. They also send me warnings letting me know I'm at 50% of my free usage, which is a nice touch.
I did the same after getting tired of NC's DNS interface. I host a few client sites with Netlify[1] anyways and moving over to their DNS (NS1) has been a breath of fresh air. It is free but they do have some paid options and the is UI dead simple which should be a requirement. Feel fairly confident I can rely on them to not muck up DNS records as this is critical to mail systems, websites, etc.
Two years ago there was a moment where I was close to working for them too so I always try to use their products where I see fit. :)
[1] https://docs.netlify.com/domains-https/netlify-dns/
Two years ago there was a moment where I was close to working for them too so I always try to use their products where I see fit. :)
[1] https://docs.netlify.com/domains-https/netlify-dns/
Thank you. Looks like I'll just have to pay more. Any recommendations for a registrar?
Hurricane Electric has a pretty solid free DNS offering. I've been using it for like 10 years.
https://dns.he.net/
I haven't needed to talk to them much, but one time I tried to add a .ninja domain, and there backend wouldn't handle it. I emailed them to report the problem at 4:49 p.m. I got an email at 7:09 p.m. the same day (2 hours 20 minutes later later) asking me to try adding it again. [1] When a free service fixes your problem in a few hours, they get +1 gold star from me.
[1] I just checked my email to look up the actual times. This was on Mar 15, 2017.
https://dns.he.net/
I haven't needed to talk to them much, but one time I tried to add a .ninja domain, and there backend wouldn't handle it. I emailed them to report the problem at 4:49 p.m. I got an email at 7:09 p.m. the same day (2 hours 20 minutes later later) asking me to try adding it again. [1] When a free service fixes your problem in a few hours, they get +1 gold star from me.
[1] I just checked my email to look up the actual times. This was on Mar 15, 2017.
Also, HE DNS will "secondary" from your own server.
For example, you can run your own DNS server on a VPS or something, and HE will AXFR the zones from your VPS and serve them authoritatively.
This allows you to run a hidden master, for example, which I can imagine some HN folks being interested in.
For example, you can run your own DNS server on a VPS or something, and HE will AXFR the zones from your VPS and serve them authoritatively.
This allows you to run a hidden master, for example, which I can imagine some HN folks being interested in.
Doing exactly this with HE. Running Knot DNS [1] on Digital Ocean.
[1]: https://www.knot-dns.cz/
[1]: https://www.knot-dns.cz/
I don't know what you just said, but it sounds awesome. I bookmarked this comment to review the next time I mess with DNS :)
+1 for Hurricane Electric. Right now I'm giving CloudFlare's DNS a try, but HE gave me solid service (including dynamic DNS) for years.
Dnsimple is a registrar as well. I have my personal domains there.
I’ve been using iwantmyname for many, many years and have been super satisfied with them.
Have you considered Route53 ?
Wrapped with git; https://dns-api.com/
Love the idea of this. I was about to switch providers then discovered that they are 3-4 times more expensive than my current provider for small-mid sized websites (currently on dnsmadeeasy.com)
Last i checked rackspace has good free dns
Be wary of being part of something that is a cost center for the company instead of a profit center.
CloudFlare is selling domains at cost. That means they are not making any money from being a domain registrar, which means they will do everything to keep the cost of doing it as low possible to themselves. This means lack of customer service and use of ML dragnets for "anomalous" behavior.
CloudFlare is selling domains at cost. That means they are not making any money from being a domain registrar, which means they will do everything to keep the cost of doing it as low possible to themselves. This means lack of customer service and use of ML dragnets for "anomalous" behavior.
.com has a price floor of $7.85. Most registrars seem to target anywhere from the $9.99 - $14.99 range for registration because, as far as I can tell, there is no real differentiation outside of price.
Sure, I could spend $lots to get a dedicated account rep from MarkMonitor or CSC but that's not really feasible for my personal site.
Are there really any registrars that hit a reasonable price point for individuals and offer service beyond bargain basement? Because if so I'm doing some transfers this weekend.
Sure, I could spend $lots to get a dedicated account rep from MarkMonitor or CSC but that's not really feasible for my personal site.
Are there really any registrars that hit a reasonable price point for individuals and offer service beyond bargain basement? Because if so I'm doing some transfers this weekend.
Namesilo has been a great, cheap registrar for me for many years and has always had privacy included for free.
I tried several of the lower price registrar's back in the day, and they all sucked in their own way, despite me not needed anything except the thing to just stay registered.
One or two would change the price of their domain privacy, most renew the privacy for like 3 dollars and then send you the renewal email that your domain needs to be renewed, one of them used to charge me separately like 80 cents from some weird Canadian shell company...
I actually have a domain still with probably the biggest "cheap" provider, and they now have a thing where you are supposed to keep a deposit in your account to cover automatic renewals. Just charge my damn credit card guys, please.
So I'm saying namesilo all the way. Only one that hasn't ever pulled any shenanigans on me.
I tried several of the lower price registrar's back in the day, and they all sucked in their own way, despite me not needed anything except the thing to just stay registered.
One or two would change the price of their domain privacy, most renew the privacy for like 3 dollars and then send you the renewal email that your domain needs to be renewed, one of them used to charge me separately like 80 cents from some weird Canadian shell company...
I actually have a domain still with probably the biggest "cheap" provider, and they now have a thing where you are supposed to keep a deposit in your account to cover automatic renewals. Just charge my damn credit card guys, please.
So I'm saying namesilo all the way. Only one that hasn't ever pulled any shenanigans on me.
What do you consider reasonable price point?
I'll speak for myself and say that all my domains have been with Hover for well over a decade now, and the times I've had to deal with their customer service, they've been excellent.
In fact, I even had to call them once, and I got a human almost immediately, and that human was able to resolve my issue while I was on the phone.. I don't recall the exact issue and I'm sure it wasn't anything major, but it was still nice.
So yeah, Hover. They're nice. And I think their prices are decent?
I'll speak for myself and say that all my domains have been with Hover for well over a decade now, and the times I've had to deal with their customer service, they've been excellent.
In fact, I even had to call them once, and I got a human almost immediately, and that human was able to resolve my issue while I was on the phone.. I don't recall the exact issue and I'm sure it wasn't anything major, but it was still nice.
So yeah, Hover. They're nice. And I think their prices are decent?
> .com has a price floor of $7.85.
Not true. CF claims it is $8.03. Let’s say it’s $7.85 + .18 tax or something.
You don’t deal with the registry. You deal with a registrar. Just as they can charge a markup, they can give a discount.
Registration could be free. It would mean a loss of $7.85/$8.03 per year. Maybe they make it up by selling ads on your domain. Maybe they use it as a loss leader service.
There is no floor.
Even CF’s at cost pricing loses them money. It’s not free to maintain the services.
Not true. CF claims it is $8.03. Let’s say it’s $7.85 + .18 tax or something.
You don’t deal with the registry. You deal with a registrar. Just as they can charge a markup, they can give a discount.
Registration could be free. It would mean a loss of $7.85/$8.03 per year. Maybe they make it up by selling ads on your domain. Maybe they use it as a loss leader service.
There is no floor.
Even CF’s at cost pricing loses them money. It’s not free to maintain the services.
One of the reasons I've stuck with GoDaddy is 24/7, American phone support. Their .com pricing is closer to $20 for renewal at this point, but I've called them at 2 AM before and gotten help.
From previous research, at least, most domain registrars have ticket support at best. I did move all my "less important" domains to Cloudflare for cost savings recently, but they have my most important domains.
From previous research, at least, most domain registrars have ticket support at best. I did move all my "less important" domains to Cloudflare for cost savings recently, but they have my most important domains.
DNS has become frighteningly unreliable. Here are previous stories that show how it is possible to lose access to your domain for no fault of yours:
(1) https://news.ycombinator.com/item?id=21700139 - Sinkholed
(2) https://news.ycombinator.com/item?id=19322966 - I lost my domain and everything that goes with it
No different than this story where the author's DNS records were deleted because of so called "anomaly".
Here are so many more stories: https://news.ycombinator.com/item?id=21710939
DNS was a good idea but now there are organizations that have the power to arbitrarily take control and even remove your domain names and records. We really need to come up with a peer-to-peer solution and take back control of the naming system from these authorities.
(1) https://news.ycombinator.com/item?id=21700139 - Sinkholed
(2) https://news.ycombinator.com/item?id=19322966 - I lost my domain and everything that goes with it
No different than this story where the author's DNS records were deleted because of so called "anomaly".
Here are so many more stories: https://news.ycombinator.com/item?id=21710939
DNS was a good idea but now there are organizations that have the power to arbitrarily take control and even remove your domain names and records. We really need to come up with a peer-to-peer solution and take back control of the naming system from these authorities.
>DNS has become frighteningly unreliable. Here are previous stories that show how it is possible to lose access to your domain for no fault of yours:
The second story you posted is about a user who forgot to renew their domain and did not wish to pay the overly-inflated fee to re-register it while it was in the grace period.
I hold no love for any registrar that jacks up rates for getting back an expired domain and agree that they should have sent a reminder email, but describing this as someone "losing their domain through no fault of their own" is, frankly, incredibly misleading.
The user:
1) forgot to renew their domain 2) had full right to recover their domain but objected to the price 3) had full right to transfer the domain out to another registrar for the original 15EUR price and 4) eventually got back full control of the domain
The second story you posted is about a user who forgot to renew their domain and did not wish to pay the overly-inflated fee to re-register it while it was in the grace period.
I hold no love for any registrar that jacks up rates for getting back an expired domain and agree that they should have sent a reminder email, but describing this as someone "losing their domain through no fault of their own" is, frankly, incredibly misleading.
The user:
1) forgot to renew their domain 2) had full right to recover their domain but objected to the price 3) had full right to transfer the domain out to another registrar for the original 15EUR price and 4) eventually got back full control of the domain
I looked into self-hosting DNS and it doesn't seem like that big of a deal as long as you can ensure uptime to be honest. If you set up the two first on different hosts and possibly have #3/4 being cloud providers I think you're pretty good.
Does anyone here have experience with running their own DNS servers for their domains?
Does anyone here have experience with running their own DNS servers for their domains?
I've been self hosting DNS for 20+ years. It's easy as pie. I have a couple name servers on my home network (business cable) and another on a VPS.
Cool, do you have any software to recommend for a casual power user who would like something a bit more lean than BIND from a configuration perspective? pdns seems nice, and there's an official terraform provider for it as well.
I use BIND. It is actually fairly lean. I used to run it on 386 systems with 16 megs of RAM.
I've been self hosting for years. Currently using online.net secondary DNS service as my 3rd or 4th backup NS. They've lost my 10€/month box once (shitty cheap intel avoton hardware with everything soldered on I suspect) but the domain still resolved fine. I had backups and restored it in a day. You can also use a VPS image to self host DNS. Some providers offer automatic or manual snapshots. Hetzner comes to mind. They've annoyingly asked for a copy of my id card (welcome to Germany), but their services are fine.
You don't even need multiple servers (especially if both your website and mail run on the same server), it's a misconception debunked by the author of djbdns:
http://cr.yp.to/djbdns/third-party.html
http://cr.yp.to/djbdns/third-party.html
These are some valid arguments against third-party providers. Assuming nobody's perfect, I still see benefit in having redundancy for cases of downtime. Could just be a duplicated setup on a separate physical network (if all your DNS records point to the same network your DNS is on, I guess it's pointless with network separation, but I don't think that is very common).
That talks about whether you have servers on multiple networks. If it debunks the idea that you should have at least two servers I can't find where.
Well, maybe because DJB takes it for granted that you don't actually need two servers in the first place, so, there's not much to debunk?
% dig ns yp.to +short
uz5jmyqz3gz2bhnuzg0rr0cml9u8pntyhn2jhtqn04yt3sm5h235c1.yp.to.
%
% dig @tonic.to yp.to ns
…
;; AUTHORITY SECTION:
yp.to. 86400 IN NS uz5jmyqz3gz2bhnuzg0rr0cml9u8pntyhn2jhtqn04yt3sm5h235c1.ns.yp.to.
yp.to. 86400 IN NS uz5jmyqz3gz2bhnuzg0rr0cml9u8pntyhn2jhtqn04yt3sm5h235c1.yp.to.
;; ADDITIONAL SECTION:
uz5jmyqz3gz2bhnuzg0rr0cml9u8pntyhn2jhtqn04yt3sm5h235c1.yp.to. 86400 IN A 131.193.32.108
uz5jmyqz3gz2bhnuzg0rr0cml9u8pntyhn2jhtqn04yt3sm5h235c1.ns.yp.to. 86400 IN A 131.193.32.109The question is whether you're violating the standard or doing something unreasonable. Clearly DNS can't prevent you from using one server, just like it can't prevent you from using one network you own.
That would solve the problem of losing DNS records. What do you do when you lose access to the domain name in the first place?
The main problem which people seem to have is that their domain name registrar decides to pull their domain. Luckily, there is ample competition in this space, my place of employment included, which should make it reasonable to pick a place which 1. doesn’t do that and 2. has reasonable real-live-person support.
Of course, if the registry (i.e. the TLD) wants your domain gone, you are out of luck whatever you do. If this is a concern then you should pick a TLD with what you consider reasonable management. There are a lot of ccTLDs and gTLDs to choose from.
Therefore, what you absolutely shouldn’t do is to pick whatever domain registrar is either cheapest or largest, and pick whatever domain name which happens to look cool and be available. Both are recipies for potential disaster.
Of course, if the registry (i.e. the TLD) wants your domain gone, you are out of luck whatever you do. If this is a concern then you should pick a TLD with what you consider reasonable management. There are a lot of ccTLDs and gTLDs to choose from.
Therefore, what you absolutely shouldn’t do is to pick whatever domain registrar is either cheapest or largest, and pick whatever domain name which happens to look cool and be available. Both are recipies for potential disaster.
Indeed.
I am curious to see what comes out of attempts at decentralizing this such as Handshake[0] and ENS[1]. I think I saw something similar with prominent backers come up here on HN the other week but can't recall it now. Namecoin[2] was very early on this.
[0]: https://www.namebase.io/
[1]: https://ens.domains/
[2]: https://bit.namecoin.org/
[0]: https://www.namebase.io/
[1]: https://ens.domains/
[2]: https://bit.namecoin.org/
Odd comment to make a throwaway for, not very controversial (unless you work for Cloudflare?)
I don't work for Cloudflare but I work for another large company that also manages domain names and DNS records. I don't want to risk the possibility that my comment could be interpreted by my employer as conflict the interest.
I had this exact thing happen to me as well, but wrote it off to having been compromised (fortunately I was only using Cloudflare as secondary DNS servers on a non-production account and am not using them as a registrar, so I only noticed months after the fact). I think a major reason going with someone like Cloudflare for DNS in the first place is reliability and availability and this does not speak to that.
Zero communication in my case as well.
Zero communication in my case as well.
Can't think of a reason this domain was touched (I don't work for CF) but I'd recommend reading the threads related to this search:
https://community.cloudflare.com/search?q=127.0.0.1%20audit
Every related incident seems to be due to either nameservers temporarily/incidentally chanced away from CF (and CF's service not re-checking it perhaps) or the registration billing failing (which doesn't look to be the case since registration expires 2021[0]). The latest change to the domain was about a week ago[0], so if that was when it was transferred to CF, it might be the first scenario.
> Because Cloudflare deleted my domain registration I can't change the status from clientTransferProhibited through their dashboard so I don't think I can even leave.
Unless something else happened, deleting the zone from your account doesn't affect the registration. Re-adding the domain will instantly allow you to view the registration info and likely transfer away; this would only not work if the zone is banned for some reason.
0: https://who.is/whois/danielzfranklin.org
https://community.cloudflare.com/search?q=127.0.0.1%20audit
Every related incident seems to be due to either nameservers temporarily/incidentally chanced away from CF (and CF's service not re-checking it perhaps) or the registration billing failing (which doesn't look to be the case since registration expires 2021[0]). The latest change to the domain was about a week ago[0], so if that was when it was transferred to CF, it might be the first scenario.
> Because Cloudflare deleted my domain registration I can't change the status from clientTransferProhibited through their dashboard so I don't think I can even leave.
Unless something else happened, deleting the zone from your account doesn't affect the registration. Re-adding the domain will instantly allow you to view the registration info and likely transfer away; this would only not work if the zone is banned for some reason.
0: https://who.is/whois/danielzfranklin.org
OP here.
> Every related incident seems to be due to either nameservers temporarily/incidentally chanced away from CF (and CF's service not re-checking it perhaps) or the registration billing failing (which doesn't look to be the case since registration expires 2021[0]).
The changes a week ago involves adding and deleting TXT and A records only. Cloudflare manages the nameservers I use as my registrar and I never changed them from the default. I just confirmed all of that in the Cloudflare audit log.
> Unless something else happened, deleting the zone from your account doesn't affect the registration. Re-adding the domain will instantly allow you to view the registration info and likely transfer away; this would only not work if the zone is banned for some reason.
Thank you so much! Trying that now.
> Every related incident seems to be due to either nameservers temporarily/incidentally chanced away from CF (and CF's service not re-checking it perhaps) or the registration billing failing (which doesn't look to be the case since registration expires 2021[0]).
The changes a week ago involves adding and deleting TXT and A records only. Cloudflare manages the nameservers I use as my registrar and I never changed them from the default. I just confirmed all of that in the Cloudflare audit log.
> Unless something else happened, deleting the zone from your account doesn't affect the registration. Re-adding the domain will instantly allow you to view the registration info and likely transfer away; this would only not work if the zone is banned for some reason.
Thank you so much! Trying that now.
i think the parent poster meant changes done by or for a registrar like shown in whois. Zone changes wont show up there.
Cloudflare is my register and manages my DNS. I would expect anything them to log any significant changes to either in their audit log.
i think that it would be rather unusual to update timestamps in whois (which i guessed the parent poster was referring to) based on updates to in-zone data. A transfer would be an example of something that would change information in whois and thus update the timestamp noted there for the latest update. it is sometime possible to infer the date of the latest change of in-zone data because the serial of the zone is often constructed by using a date and a counter. But that is actually just convention and not reliable. Its also unlikely the parent poster was referring to this.
Huh. I definitely didn't make any such changes within the last two weeks. Maybe the whois date got changed because of something opaque and internal to Cloudflare?
yes that is most likely what happened. A change of the nameservers with authority for your zone for example or updates of DNSEC keys would trigger that too, i think. But most commonly it probably happens when the domain gets a renewed registration period or the contact details for some person changed.
btw here are the dates referred in whois for your domain:
btw here are the dates referred in whois for your domain:
Updated Date: 2020-02-24T20:32:34Z
Creation Date: 2015-06-09T15:34:37Z
Registry Expiry Date: 2021-06-09T15:34:37Z
I believe the latest change happened afterwards when on judge2020's advice I re-added the domain to cloudflare which I think triggered a NS change back to them.
> Re-adding the domain will instantly allow you to view the registration info
"Your domain registration configuration depends on your DNS zone configuration" is a very strange way to do things.
"Your domain registration configuration depends on your DNS zone configuration" is a very strange way to do things.
> However, I'm unable to log in to their community forum. When I click the login button I'm redirected to my dashboard, and when I then click Support on the dashboard I'm redirected back to the forum without being logged in. I suppose it's possibly an issue with Firefox blocking cookies (although I disabled tracking prevention) so it's possible this part is partly a problem on my end.
I'm into issues like this more and more, where you run into some strange behavior on a website and you wonder "How did this ever make it into production?", then you open the website in Chrome and the flows work fine. I worry that Firefox is becoming less and less viable.
I'm into issues like this more and more, where you run into some strange behavior on a website and you wonder "How did this ever make it into production?", then you open the website in Chrome and the flows work fine. I worry that Firefox is becoming less and less viable.
This is not Firefox becoming less and less viable. This is developers caring less and less about supporting older browsers, less capable hardware and, I guess, long-term maintenance in general.
Just had a similar case today: My Mom tried to order something online on her old Android tablet - and it didn't work. She blamed the tablet for it, saying "It's just too old, it doesn't work correctly anymore! I used to be able to order stuff on this website". I had to explain to her that her tablet is still working fine, it's just the website that is broken because it's not supporting her device (or browser) anymore. Shockingly, she listed quite a few websites, which she has used for years, which have stopped working for her in the past few months and years; all of these she mentioned as evidence that the problem must be her tablet - not the websites. When I opened two of the sites she mentioned, I wasn't too surprised to find very shiny, very modern single-page applications (with service workers registered and even WebAssembly used on one of them)..
So when you are creating a modern web app, please don't just test in Chrome on your new MacBook Pro. Think about your Mom. Ask yourself: "Is this still gonna work on her crappy old device?"
Just had a similar case today: My Mom tried to order something online on her old Android tablet - and it didn't work. She blamed the tablet for it, saying "It's just too old, it doesn't work correctly anymore! I used to be able to order stuff on this website". I had to explain to her that her tablet is still working fine, it's just the website that is broken because it's not supporting her device (or browser) anymore. Shockingly, she listed quite a few websites, which she has used for years, which have stopped working for her in the past few months and years; all of these she mentioned as evidence that the problem must be her tablet - not the websites. When I opened two of the sites she mentioned, I wasn't too surprised to find very shiny, very modern single-page applications (with service workers registered and even WebAssembly used on one of them)..
So when you are creating a modern web app, please don't just test in Chrome on your new MacBook Pro. Think about your Mom. Ask yourself: "Is this still gonna work on her crappy old device?"
Well, it's also a problem of device manufacturers dropping support for devices too quickly. There are still android 4.1 devices sold on amazon, and you really can't expect web developers to support that.
The manufacturer should be required to support it for the full lifetime of the device. Especially since your mom uses it to order stuff, which usually includes some pretty security sensitive information. I think you are putting the burden on the wrong party.
The manufacturer should be required to support it for the full lifetime of the device. Especially since your mom uses it to order stuff, which usually includes some pretty security sensitive information. I think you are putting the burden on the wrong party.
Well, I haven't analysed the exact technical reason for why submitting the order failed. But I'm pretty certain that submitting a HTML form is a solved problem in web development.. Or at least it should be. I haven't tried submitting a form with an async fetch from a web worker that communicates with a redux store implemented in WebAssembly yet (or whatever that web app is doing..).
If the order site is just submitting an HTML form in the old way with credentials stored in a cookie (also the old way) that would probably be open to trivial CSRF attacks.
If it is somehow checking for support for SameSite, Secure, CSP or any of the other mechanisms that have been implemented in the last years then it might fail. Or they might be using mechanisms that work around the problem that those three are supposed to help since they are not available in older clients, but just don't have the resources to test the random android 4.12 version that you use. I think it should have a proper error message if that is the case.
But I feel like you are pointing the finger in the wrong direction. I try to build my apps without extraneous fads, but keeping a webapp secure (in other words keeping up to date with the latest protections) does not mean "submitting a form", and it does not mean letting any old client lacking the required protections through.
It also does not mean doing "WASM compiled redux reducers in ES6 module workers authenticating over JWT to send gRPC commands to a kafka broker talking with ingressrouting over anycast and a internal service mesh with mTLS3.9 auth using curve9999.9, token binding and Wireguard to secure internal communications over a VPC-less multi-cloud k8s cluster that uses Multi-Raft, Single-Paxos to have a single, distributed, disputably non-consistent CRDT-consensus algo over blockchain RS-232".
So, yeah, I'm not for fads over usability in tech. But I'm also not for supporting insecure clients just because the manufacturer of those clients doesn't give a shit.
If it is somehow checking for support for SameSite, Secure, CSP or any of the other mechanisms that have been implemented in the last years then it might fail. Or they might be using mechanisms that work around the problem that those three are supposed to help since they are not available in older clients, but just don't have the resources to test the random android 4.12 version that you use. I think it should have a proper error message if that is the case.
But I feel like you are pointing the finger in the wrong direction. I try to build my apps without extraneous fads, but keeping a webapp secure (in other words keeping up to date with the latest protections) does not mean "submitting a form", and it does not mean letting any old client lacking the required protections through.
It also does not mean doing "WASM compiled redux reducers in ES6 module workers authenticating over JWT to send gRPC commands to a kafka broker talking with ingressrouting over anycast and a internal service mesh with mTLS3.9 auth using curve9999.9, token binding and Wireguard to secure internal communications over a VPC-less multi-cloud k8s cluster that uses Multi-Raft, Single-Paxos to have a single, distributed, disputably non-consistent CRDT-consensus algo over blockchain RS-232".
So, yeah, I'm not for fads over usability in tech. But I'm also not for supporting insecure clients just because the manufacturer of those clients doesn't give a shit.
CSRF doesn't require any javascript to defend against. Having a nonce inside of the HTML resolves that completely. SameSite and Secure cookies also don't require any javascript, just some extra HTTP headers. I don't think security justifies this.
> Well, it's also a problem of device manufacturers dropping support for devices too quickly. There are still android 4.1 devices sold on amazon, and you really can't expect web developers to support that.
Are you kidding me? If you're looking for shiny stuff to add to your resume, yeah, you can't possibly support those! If you're an HTML5 game developer, yeah, gotta use the latest and greatest. But if you're in the business of selling shoes, why do you need anything newer than Android 4.1 in order to process the transactions?!
Are you kidding me? If you're looking for shiny stuff to add to your resume, yeah, you can't possibly support those! If you're an HTML5 game developer, yeah, gotta use the latest and greatest. But if you're in the business of selling shoes, why do you need anything newer than Android 4.1 in order to process the transactions?!
The latest version of Android 4.1 was released in 2012. The latest version of chrome for that seems to be around 23. A lot of android 4.1 shipped without TLS 1.1 by default. So you are stuck with SSLv3 or TLSv1. Those are just examples, there's a lot in security that has changed since 2012.
A lot of basic security was missing back then, asking anyone to enter any security-sensitive info on those devices now is like telling someone to use WinXP and IE8 for online banking. Besides that the great variety of devices that shipped on versions v4.1-v4.4 and with their different hacks/customizations it is simply not feasible to ask developers to support them years after many of them cannot be had to even test a website on, let alone debug it.
A lot of basic security was missing back then, asking anyone to enter any security-sensitive info on those devices now is like telling someone to use WinXP and IE8 for online banking. Besides that the great variety of devices that shipped on versions v4.1-v4.4 and with their different hacks/customizations it is simply not feasible to ask developers to support them years after many of them cannot be had to even test a website on, let alone debug it.
A couple of things..
- The tablet is ~5 years old. Even if it was 8 years old, I think you should still be able to order a pizza online.
- TLSv1 is far from plaintext. But in any case, the TLS version doesn't matter in this case. The site was loading properly.
- Even if the TLS version was outdated, are we now banning people from using the internet for using an older device?
- Submitting the form didn't work, the navigation was partially broken. It was probably a JS issue, but I don't know.
- The moral of the story, to me, is that, as a web developer, you should also care about those $few% that don't have all the green boxes on caniuse.com. Depending on how many users your site has, this may just be a few people or it might be millions - among them, possibly, your Mom.
If a service doesn't function properly without tracking I wouldn't blame it on a privacy respecting browser.
This is frightening. I just started the process of moving all ~60 of my domains from Amazon Registrar + Google Cloud DNS to Cloudflare, and will definitely wait until somebody from Cloudflare chimes in here to clarify what's going on.
> moving all ~60 of my domains from Amazon Registrar + Google Cloud DNS to Cloudflare
You're very brave considering that Cloudflare doesn't even have U2F yet Google and Amazon do.
You're very brave considering that Cloudflare doesn't even have U2F yet Google and Amazon do.
Great point! And let's just say that the migration project is now on-hold :)
Are you using physical U2F keys for your Google or Amazon accounts?
Cloudflare does support standard TOTP-based 2FA like most people use for Amazon and Google. So whether or not the lack of U2F support should matter depends on whether you actually use it elsewhere anyways.
Cloudflare does support standard TOTP-based 2FA like most people use for Amazon and Google. So whether or not the lack of U2F support should matter depends on whether you actually use it elsewhere anyways.
U2F is under active development. My team is actually working on it as we speak
They have TOTP 2 factor however.
Amazon doesn't allow multiple U2F. D-:
I've been planning too soon, am also now going to wait to see where this goes. DNS is obviously a critical system and I don't know if I can trust Cloudflare now. I'm not a big fish that can make noise. I'm an easy victim.
Don't put all your eggs in one basket, ie. don't just use one provider.
Also for your core domains, do not let the registrar and dns provider be the same entity.
Also, don't decide on not migrating just because of one bad experience. None of them are perfect, though vigilance is wise.
(I know am probably preaching to the choir :) )
Also for your core domains, do not let the registrar and dns provider be the same entity.
Also, don't decide on not migrating just because of one bad experience. None of them are perfect, though vigilance is wise.
(I know am probably preaching to the choir :) )
This is what I do. Domains are at Porkbun and domain registration is the ONLY thing I use them for. DNS and caching are at Cloudflare. Hosting is wherever makes sense.
Consolidating domain registration, DNS hosting, and site hosting under the same account is a terrible idea and a big risk IMO. Always ask yourself “what if this account gets banned?” All of the big tech companies have automated systems that could turn you into an outlier with no support.
Consolidating domain registration, DNS hosting, and site hosting under the same account is a terrible idea and a big risk IMO. Always ask yourself “what if this account gets banned?” All of the big tech companies have automated systems that could turn you into an outlier with no support.
OP here. I'm considering moving to Amazon Registrar. Why are you leaving?
I was only moving to Cloudflare because they do registrar services at-cost, which would be cheaper than Amazon Registrar.
How many of your sixty domains are business-critical?
Cloudflare's Domains service is new, and some of it's management tools are lacking, but I also moved most of my domains to it over the last year for cost savings. I'm thrilled with it, but I'm still keeping a few of my most critical domains with GoDaddy. (Hate them all you want, but GoDaddy hasn't screwed up my domains in well over a decade.)
You may be able to save a lot of money without risking your primary domain that you route email through.
Cloudflare's Domains service is new, and some of it's management tools are lacking, but I also moved most of my domains to it over the last year for cost savings. I'm thrilled with it, but I'm still keeping a few of my most critical domains with GoDaddy. (Hate them all you want, but GoDaddy hasn't screwed up my domains in well over a decade.)
You may be able to save a lot of money without risking your primary domain that you route email through.
My takeaway:
1. Setup up monitoring on your critical domains. UptimeRobot and Hetrixtools are good starters with generous free tier. You should know when your website/email/dns isn't working.
2. Don't tie your domain registration with your DNS provider. You lose everything if something goes wrong with your account.
3. Be able to jump ship easily, have backups of your zone, already know where you will transfer to.
1. Setup up monitoring on your critical domains. UptimeRobot and Hetrixtools are good starters with generous free tier. You should know when your website/email/dns isn't working.
2. Don't tie your domain registration with your DNS provider. You lose everything if something goes wrong with your account.
3. Be able to jump ship easily, have backups of your zone, already know where you will transfer to.
> UptimeRobot and Hetrixtools are good starters with generous free tier
Are there any open source status pages/monitor programs that have build-in checks for HTTPS, DNS records (ipv4/6), arbitrary port checks, etc? I'd rather just setup a status page/alert app on a $5 minimal DO/Vultr node and self-host/support/contribute to a FOSS program than use a commercial provider.
Are there any open source status pages/monitor programs that have build-in checks for HTTPS, DNS records (ipv4/6), arbitrary port checks, etc? I'd rather just setup a status page/alert app on a $5 minimal DO/Vultr node and self-host/support/contribute to a FOSS program than use a commercial provider.
I wrote a scalable system for this:
https://github.com/skx/overseer/
Handles SSL-checks, DNS-checks, SMTP-checks, & etc. Runs a thousand-checks every two minutes for me, give or take. Pluggable output via a redis-queue.
https://github.com/skx/overseer/
Handles SSL-checks, DNS-checks, SMTP-checks, & etc. Runs a thousand-checks every two minutes for me, give or take. Pluggable output via a redis-queue.
Here is a good list: https://github.com/n1trux/awesome-sysadmin#monitoring
Maybe one fits what you are looking for.
Maybe one fits what you are looking for.
<opinion class="unpopular">
Nagios. Or its descendant with a better configuration language, Icinga2. They're fairly easy to do a minimal install and configure in a container or on a VM.
</opinion>
Nagios. Or its descendant with a better configuration language, Icinga2. They're fairly easy to do a minimal install and configure in a container or on a VM.
</opinion>
[deleted]
You need to host across several nodes in different geographic locations and data centers to resist network splits. Then you need some way to slowly roll out upgrades to your monitoring platform over time.
I'm just talking about my personal infrastructure. If I host my crap in Vultr or Linode, I should be able to buy one cheap node on another provider just to run a simple status app: something with celary or sidekiq jobs to check my other stuff and intervals and generate a page with some red/yellow/green dots.
How do you know if the monitoring node goes down at the same time as the other servers?
Remember that Linode/Vultr/etc don't run their own datacenters, they share datacenters and sometimes downtime events can exist outside of datacenters.
Remember that Linode/Vultr/etc don't run their own datacenters, they share datacenters and sometimes downtime events can exist outside of datacenters.
If you want email or text message alerts I would assume that's a complicated enough system you would want uptime alerts on it, and so on recursively ad infinitum.
If you can set up nagios (which one would probably consider an interesting evening challenge if you were already willing to go for your own monitoring droplet) setting up pushover or amazon sns (for sms) should be easy enough.
FWIW, a lot of cellular providers have an email gateway for delivering SMS messages. There's also paid SMS gateways, and options for providing arbitrary push notifications to smartphones.
I'm pretty sure the free email gateways have no posted SLA. Plus, that requires a reliable email server, which would also need its own monitoring.
This was a few (3) years past, but they accepted root@localhost sendmail messages just fine in most cases, and delivered alerts within a minute or two of sending. We didn't rely on this long term, but it was a "good enough" first pass.
I'd probably recommend using one of the gateways (or a more fully-featured service like Pagerduty) for more serious businesses, but for personal use (or where an outage detected the next day isn't crippling), it's remarkably useful.
I'd probably recommend using one of the gateways (or a more fully-featured service like Pagerduty) for more serious businesses, but for personal use (or where an outage detected the next day isn't crippling), it's remarkably useful.
I would try to set up a completely open source monitoring setup just for fun, but once I'm paying for SNS I personally would rather just pay epsilon more and buy/rent the whole system. I get that may just be personal taste. I absolutely don't trust myself to run my own highly-reliable mailserver to send status alerts.
check out https://github.com/hunterlong/statping - thats what i'm using.
> Setup up monitoring on your critical domains. UptimeRobot and Hetrixtools are good starters with generous free tier. You should know when your website/email/dns isn't working.
Lesson learned :)
> Don't tie your domain registration with your DNS provider. You lose everything if something goes wrong with your account.
I don't see how that helps. How do I recover from my registrar deleting/disabling my account even if DNS is somewhere else? I think there's still only one failure point and the lesson is that I need to pay that failure point more money.
> Be able to jump ship easily, have backups of your zone,
Luckily I have that
> already know where you will transfer to.
Any suggestions? Ironically I recently moved from Google Domains to Cloudflare because I was worried about issues with opaque support. I've learned my lesson picking based on cost alone, but I'm a college student who can't afford something too heavy-duty.
Lesson learned :)
> Don't tie your domain registration with your DNS provider. You lose everything if something goes wrong with your account.
I don't see how that helps. How do I recover from my registrar deleting/disabling my account even if DNS is somewhere else? I think there's still only one failure point and the lesson is that I need to pay that failure point more money.
> Be able to jump ship easily, have backups of your zone,
Luckily I have that
> already know where you will transfer to.
Any suggestions? Ironically I recently moved from Google Domains to Cloudflare because I was worried about issues with opaque support. I've learned my lesson picking based on cost alone, but I'm a college student who can't afford something too heavy-duty.
>I don't see how that helps. How do I recover from my registrar deleting/disabling my account even if DNS is somewhere else? I think there's still only one failure point and the lesson is that I need to pay that failure point more money.
Your outage was a DNS outage, not a registrar outage. If you still had control of the domain you could update your name servers to another provider, import your backed up records and get the site back online without talking to CloudFlare.
Your outage was a DNS outage, not a registrar outage. If you still had control of the domain you could update your name servers to another provider, import your backed up records and get the site back online without talking to CloudFlare.
> Your outage was a DNS outage, not a registrar outage. If you still had control of the domain you could update your name servers to another provider, import your backed up records and get the site back online without talking to CloudFlare.
I believe it was both.
If I have a registrar outage I'm hosed. If I don't have a registrar outage and do have a DNS outage I can recover with a little work. But in the only case I can recover my registrar was reliable, so why didn't I just have them do DNS as well?
I believe it was both.
If I have a registrar outage I'm hosed. If I don't have a registrar outage and do have a DNS outage I can recover with a little work. But in the only case I can recover my registrar was reliable, so why didn't I just have them do DNS as well?
> But in the only case I can recover my registrar was reliable, so why didn't I just have them do DNS as well?
Because they have just proved being uncapable of doing it? Because redundancy? Because you shouldn't keep all your eggs in the same basket.
I've been self hosting for at least 15 years and did not have any huge problems like the domain becoming non resolvable. I would never host my DNS on my registrar's infrastructure. It's being sloppy and lazy and it gets you embarassed.
Because they have just proved being uncapable of doing it? Because redundancy? Because you shouldn't keep all your eggs in the same basket.
I've been self hosting for at least 15 years and did not have any huge problems like the domain becoming non resolvable. I would never host my DNS on my registrar's infrastructure. It's being sloppy and lazy and it gets you embarassed.
This happened to me with AWS somewhat recently[0], and I never found out exactly what happened. I just chalk it up to some dev made a mistake and didn't tell anyone. It's pretty alarming when things like this happen though.
0. https://news.ycombinator.com/item?id=21326014
0. https://news.ycombinator.com/item?id=21326014
I've been involved in using Route 53 to manage thousands of DNS zones, and haven't come across something like that. I'd recommend putting in a support request via the account that was affected to ensure that it gets looked at.
If you haven't already, you might consider checking the CloudTrail logs for the account in question to see if there were any API commands related to the zone.
If you haven't already, you might consider checking the CloudTrail logs for the account in question to see if there were any API commands related to the zone.
Although not DNS related, I have had weird things happen on AWS, such as spikes of 5xx errors reported from CloudFront which was backed by ELB/EB, but the ELB is showing no errors. Even after contacting AWS support they couldn't resolve it, said they required application logs, but there is no logs because the requests never reached the application servers.
Edit: the dns record export/import functionality is hidden behind the advance search drop down for some reason.
Ignore this entire comment.
From reading the linked helpdoc, apparently your entire domain can get removed from cloudflare if your register stops reporting cloudflare's servers for the ns records.
The mere idea of having to re-enter all hundred or so of our dns records using cloudflares 1.2 second delay at every step of the way add dns record interface because namecheap bugged out for a few seconds is horrifying.
There is no way to export all of these, there is no way to import or mass add and i don't think they can lean on the api to save them here.
Dns records are data, dns records are sometimes important unbacked up customer data. Cloudflare does not offer a way for customers to back this data up, nor a way to restore or recover from a backup but it acts very callous with this data, deleting it in automated systems based on data from 3rd party providers.
Not a good look.
From reading the linked helpdoc, apparently your entire domain can get removed from cloudflare if your register stops reporting cloudflare's servers for the ns records.
The mere idea of having to re-enter all hundred or so of our dns records using cloudflares 1.2 second delay at every step of the way add dns record interface because namecheap bugged out for a few seconds is horrifying.
There is no way to export all of these, there is no way to import or mass add and i don't think they can lean on the api to save them here.
Dns records are data, dns records are sometimes important unbacked up customer data. Cloudflare does not offer a way for customers to back this data up, nor a way to restore or recover from a backup but it acts very callous with this data, deleting it in automated systems based on data from 3rd party providers.
Not a good look.
I googled "cloudflare dns import" and "cloudflare dns export" and the first result both times was an apparently official support article giving step-by-step instructions on how to do so. I myself have used this function about six years ago, so this is not new or untested functionality.
Never clicked that because i have no reason to look for a dns export/import button in the advance search gui... hmm. good to know about, not sure why they put it there.
You can export a zone file and the auto deletion takes days. You’re site would be offline before the zone gets deleted. I don’t like it a ton, but it’s not even close to what you’re saying.
I only have the help article to go on and the fact that their export functionality is for some reason hidden in the advance search drop down? as i've just found out...
Anywho, the help article does not state it it "takes days", so it does not have to "take days". Otherwise known as listening to the documentation not the observed behavior. Programming 101 here.
Anywho, the help article does not state it it "takes days", so it does not have to "take days". Otherwise known as listening to the documentation not the observed behavior. Programming 101 here.
as much as i like cloudflare (and i like them a lot), it's kind of absurd that this kind of thing can happen. a lot of red flags that, if true, would mean that their infrastructure require a lot more care (127.0.0.1 as the source of an audit event? no email when DNS records are deleted? no 1-to-1 message due to this happening?).
I had an issue with them recently where a SRV record pointing to “.” (meaning “service unavailable”) was being rewritten to the string “false”. It didn’t take them too long to fix it, but it made me wonder how they managed to push a bug like that to production without some sort of automated test catching it.
IIRC, if you're on a free plan you get exposed to code changes a little faster than their paying customers.
Simple. They don't give a damn about doing what we've all been doing properly for a quarter of a century. Apparently these large companies are above owning O'Reilly books.
At the very least, this sort of lack of good process is definitely what happens when Google decides to cut you off (and another person just commented a similar experience with Amazon), but I suspect it's likely the case for a much larger number of companies and services than people realize. It's fundamental internet architecture, and often little more thought goes into account termination than what you'd do to ban someone from your mid-2000s phpBB forum.
So much business focus goes into the onboarding experience, and since you assume all of the people your service terminates are "probably bad people anyways", not a lot of thought goes into offboarding, or ideally, appeals.
So much business focus goes into the onboarding experience, and since you assume all of the people your service terminates are "probably bad people anyways", not a lot of thought goes into offboarding, or ideally, appeals.
Had the same thing happen to me some years ago. Had a (not so important) domain with Gandi, which pointed to the Cloudflare nameservers, and after some time, the domain was gone from the CF dashboard together with all DNS entries. The NS records were still pointing to CF and there also weren't any anomalies with renewal of the domain.
I didn't give much thought to it, as I wasn't using CF for anything in production at the time, but sad to see that it also seems to happen to other people.
I didn't give much thought to it, as I wasn't using CF for anything in production at the time, but sad to see that it also seems to happen to other people.
Unrelated issue but sometimes Cloudflare docs/communications are not in sync with their actual system which is immensely frustrating. I was bitten a few times.
For instance, a while back I forgot to renew one of my side project domains so it briefly expired for maybe a day or two. Got this email from Cloudflare saying
> Your DNS records will be completely removed from our system in 7 days.
> ...
> Once you have completed this change, click the “Recheck Nameservers” button in your Cloudflare dashboard to ensure your domain stays active on Cloudflare.
I promptly renewed, except there's no "Recheck Nameservers" button anywhere, and the dashboard still read "Moved" for maybe a day. Eventually the problem was just gone, but the communication worried me that entire time.
(I do appreciate Cloudflare's service, though.)
For instance, a while back I forgot to renew one of my side project domains so it briefly expired for maybe a day or two. Got this email from Cloudflare saying
> Your DNS records will be completely removed from our system in 7 days.
> ...
> Once you have completed this change, click the “Recheck Nameservers” button in your Cloudflare dashboard to ensure your domain stays active on Cloudflare.
I promptly renewed, except there's no "Recheck Nameservers" button anywhere, and the dashboard still read "Moved" for maybe a day. Eventually the problem was just gone, but the communication worried me that entire time.
(I do appreciate Cloudflare's service, though.)
> Your DNS records will be completely removed from our system in 7 days.
This sounds like a plot of a japanese horror movie.
This sounds like a plot of a japanese horror movie.
Also don't forget: Cloudflare breaks many second and third world countries' Internet with their DNS captchas because they think the good guys live only in first world countries (maybe look up the word discrimination in your dictionary cloudflare) and force them to install extensions like PrivacyPass because they think "we are so big and know what is right for the world".
That's CDN captcha, not DNS. If you use Cloudflare solely as a DNS provider, your users don't see the captcha. If you route your traffic through their servers, then they do.
you're right, it's their CDN not their DNS. Nevertheless many site owners choose Cloudflare (paid or not paid) and use Cloudflare's default settings and maybe they also never check their sites from second or third world countries. Result is that the Internet is utterly broken on many Cloudflare hosted sites (and that's a lot of sites) outside of first world countries.
OP here. You're right, and even in the US I still get endless CAPTCHAs because I browse on Firefox on Linux with tracking prevention.
My website was down for yak shaving when this happened, but before then I had DDOS protection turned off.
My website was down for yak shaving when this happened, but before then I had DDOS protection turned off.
Stories like these scare the hell out of me. What do you do if one of the big internet corporation deletes some resource or account that is critical to your business? What happens when support isn’t responsive and you don’t have contacts in the company or your HN post doesn’t get visibility?
I get it - these are free services. You should factor that into every decision. But the risk is real even if you pay for an account. I’ve been slowly moving away from Gmail to a custom domain, but something like loosing DNS records and not being able to restore them quickly is even worse.
Back up everything that can be backed up, don’t rely on a single provider and always have a continuity plan!
I get it - these are free services. You should factor that into every decision. But the risk is real even if you pay for an account. I’ve been slowly moving away from Gmail to a custom domain, but something like loosing DNS records and not being able to restore them quickly is even worse.
Back up everything that can be backed up, don’t rely on a single provider and always have a continuity plan!
Cloudflare considered harmful: https://www.devever.net/~hl/cloudflare
Cloudflare Entrprise sales pissed me off so much the last time I dealt with them that I literally registered "cancelcloudflare.com/net/org". We didn't want to renew there $25k enterprise support for a website that got 50 hits a month. It was embarrassing (for Cloudflare) how incompetent their Enterise Sales rep was. I would rather pay to host my own cache layer then ever have to deal with them again, free or otherwise.
That is... quite scary. Why would you EVER have a way for auto deletion of domains?
> Does anyone know what might have caused Cloudflare to delete my domain? Any ideas for how I could transfer my domain away from Cloudflare sooner?
I don't get the point of 'shoot first ask questions later' type approach. Obviously it would pay to get some kind of affirmative reply from Cloudflare prior to a post which everyone here with incomplete information speculates and wastes time on (like I am doing).
Also Cloudflare did not 'delete my (the) domain. It deleted the dns records. There is a difference and no I am not being pedantic either. How would 'the internet' know why this was done there could be any number of good or bad reasons.
Lastly the domain is not expired and as such the registrar is required (per ICANN) to supply an auth code so someone can transfer out. Or to allow the customer to change the primary and secondary dns to another dns provider. There is zero (legitimately) that allows cloudflare as either a dns provider or a registrar to lock the domain up pretty much (other than for a legal court order) just for some reason they might decide to do that.
I don't get the point of 'shoot first ask questions later' type approach. Obviously it would pay to get some kind of affirmative reply from Cloudflare prior to a post which everyone here with incomplete information speculates and wastes time on (like I am doing).
Also Cloudflare did not 'delete my (the) domain. It deleted the dns records. There is a difference and no I am not being pedantic either. How would 'the internet' know why this was done there could be any number of good or bad reasons.
Lastly the domain is not expired and as such the registrar is required (per ICANN) to supply an auth code so someone can transfer out. Or to allow the customer to change the primary and secondary dns to another dns provider. There is zero (legitimately) that allows cloudflare as either a dns provider or a registrar to lock the domain up pretty much (other than for a legal court order) just for some reason they might decide to do that.
OP here.
> Also Cloudflare did not 'delete my (the) domain. It deleted the dns records. There is a difference and no I am not being pedantic either.
Thanks. You're absolutely right. I meant delete their record of the domain as it shows up in the UI of their dashboard.
> How would 'the internet' know why this was done there could be any number of good or bad reasons.
For many reasons luckily HN isn't 'the internet'. I've already gotten some good suggestions.
> Lastly the domain is not expired and as such the registrar is required (per ICANN) to supply an auth code so someone can transfer out. Or to allow the customer to change the primary and secondary dns to another dns provider. There is zero (legitimately) that allows cloudflare as either a dns provider or a registrar to lock the domain up pretty much (other than for a legal court order) just for some reason they might decide to do that.
I know. Again, I guess I was insufficiently specific. Cloudflare has warned me to expect long wait times before I can talk to a customer support rep. My question was if there's a way to transfer out without needing to wait on a slow support loop.
> Also Cloudflare did not 'delete my (the) domain. It deleted the dns records. There is a difference and no I am not being pedantic either.
Thanks. You're absolutely right. I meant delete their record of the domain as it shows up in the UI of their dashboard.
> How would 'the internet' know why this was done there could be any number of good or bad reasons.
For many reasons luckily HN isn't 'the internet'. I've already gotten some good suggestions.
> Lastly the domain is not expired and as such the registrar is required (per ICANN) to supply an auth code so someone can transfer out. Or to allow the customer to change the primary and secondary dns to another dns provider. There is zero (legitimately) that allows cloudflare as either a dns provider or a registrar to lock the domain up pretty much (other than for a legal court order) just for some reason they might decide to do that.
I know. Again, I guess I was insufficiently specific. Cloudflare has warned me to expect long wait times before I can talk to a customer support rep. My question was if there's a way to transfer out without needing to wait on a slow support loop.
> I don't get the point of 'shoot first ask questions later' type approach.
At first I thought you were talking about Cloudflare shooting first, but apparently not.
At first I thought you were talking about Cloudflare shooting first, but apparently not.
Out of this very fear, when Evolving Web started using CloudFlare for DNS, we wrote this backup script that runs on cron and pushes our settings to a git repo. https://github.com/evolvingweb/cloudflare-dns-backup-tool
We need a neutral, third-party service that monitors other service providers. That is, checks periodically for such things as a host being reachable, DNS working, certificates working, etc., which notifies both the service provider company and the end-user when there is a discrepancy, but most importantly, acts as a source of records in the case of a dispute -- in other words, they hold a record of what the service should be and should do. That is, when the service provider creates a service record, they would cryptographically sign it, and forward it to this future third-party service. The same for deletions.
Now we can track, record, and audit that service provider's promises...
That way, the service provider can't use an all-too-easy excuse like "we can't find that record in our database -- so you must not have ever created one..."
Now we can track, record, and audit that service provider's promises...
That way, the service provider can't use an all-too-easy excuse like "we can't find that record in our database -- so you must not have ever created one..."
[deleted]
Reminds me of Rackspace - clown IT overrpcied. They provisioned email incorrectly and swore that once we changed nameservers to them it would work. We pointed out it wouldn't using a few basic tools and asked for an escalation, they refused and we were stuck with level 1 tech support. After a few days I found someone on Twitter who esclated, they admitted they fucked it up, but the damage was done, I was frustrated and called it day - I proceeded to cancel, they then had to cheek to say "let's all be professional here"... I can deal with occasional shit support but I can't deal with escalation refusal, or endless cycles of support tennis or being redirected to irrelvant knowledgebae articles.
Cloudflare once deleted one of my domains because the NS records were set in the wrong order.
What do you mean by wrong order? Do you mean the NS records in the zone file were after a delegation / referral? What RFC was your zone breaking?
Ignore me, I just went to look again, and one of the NS records was misspelled.
We've just been dealing with this for my company as well. Cloudflare has repeatedly deleted our DNS and cannot provide a reason why it happened. Last time thousands of dollars of PPC Ads were running uselessly.
Had the same or very similar issue .
My domain registrar dropped all name server settings on an 'update'/whois update requirmet change
Luckily I had an old export of DNS records so most were there but not all the settings were retained like DNS only or full proxy.
Those were not mission critical, so not that bad but I bet some emails went to the void because there was no real warning from both my domain registration service provider and cf so I found it very 'upsetting' (I was pissed on them both)
I've Cloudflare delete an entire zone before, and I could never get an answer as to what happened. They said it was deleted because the NS were changed on the domain...but they never were.
Funny that this is coming up. I just transferred over from Namecheap to Cloudflare a few days ago and had a similar issue. One of my A records (out of about 20) were missing after the transfer.
I noticed that if you don't unfocus the input field by focusing somewhere else on the page it may not save. That may be what happened to you.
[deleted]
Last week, I have had an issue where a number of domains were purged from the 2nd tier registrar (Claranet) with exactly the same symptoms (domains suspended, zone-files blown away)... and Network Solutions are to blame.
An assumption of false-payment led to them suspending "300-500" accounts (mostly UK based). I am still of the opinion something far more sinister is at play... and this doesn't comfort me.
An assumption of false-payment led to them suspending "300-500" accounts (mostly UK based). I am still of the opinion something far more sinister is at play... and this doesn't comfort me.
I had the same thing happen to me with the same domain - twice. It just disappeared from Cloudflare without any notice. I run multiple domains on the same CF account, yet this has only happened with that specific one, which uses a somewhat unusual TLD (.do).
same thing happened to me on GoDaddy for multiple domains when I got a call from a client that their emails stopped working. All the zones were factory reset, and no backup of the zones apparently existed at GoDaddy. I was on the call with them for hours refusing to hang up until it was resolved or they would lose the remainder of my business. After 2.5 hours of no valid reason that multiple domains when back to default DNS values and no log of access to my account for moths, I let them go.
That's when I moved the couple of handfuls of domains I had left at GoDaddy over to Hover. It's more expensive, but the Hover interface is better, and I trust Hover (Tucows) more (well, I trust GoDaddy less).
That's when I moved the couple of handfuls of domains I had left at GoDaddy over to Hover. It's more expensive, but the Hover interface is better, and I trust Hover (Tucows) more (well, I trust GoDaddy less).
This is Google-tier lack of support and general 'customer' gaslighting.
Is it really all that surprising when a big company that claims to be good but hosts phishing content in the name of free speech does whatever they want, including breaking things and not explaining why?
I don't trust Cloudflare one bit, and I think everyone should question whether their attempt to re-centralize everything is beneficial to the planet.
There are two major problems here: one, the problem itself, which is the deletion of DNS for apparently no good reason, and two, which is the bigger problem, is that it's incredibly difficult to talk to a human about what happened, so there's no assurance it won't happen again.
If people want things to be reliable, we've got to stop using companies with which we cannot communicate.
I don't trust Cloudflare one bit, and I think everyone should question whether their attempt to re-centralize everything is beneficial to the planet.
There are two major problems here: one, the problem itself, which is the deletion of DNS for apparently no good reason, and two, which is the bigger problem, is that it's incredibly difficult to talk to a human about what happened, so there's no assurance it won't happen again.
If people want things to be reliable, we've got to stop using companies with which we cannot communicate.
IMHO (and I know the parent post includes significant difficulties getting back out of Cloudflare), services like Cloudflare may be crucial to decentralization. I can't deal with something like my blog post being frontpaged on HN if my website is hosted in my house, unless I have a good CDN.
As a self-hosting enthusiast, something like Cloudflare is one of the best chances of having a plan that competes with "just hosting it in the cloud".
As a self-hosting enthusiast, something like Cloudflare is one of the best chances of having a plan that competes with "just hosting it in the cloud".
I hear you, but their DDoS services are painful to the rest of the world and to people who want or need to use Tor, and others.
I'm talking about their rather political move to re-centralize DNS by shoehorning themselves in to Firefox via DoH, for instance. Their unwillingness to be transparent makes this all the more frightening. Add to that their blatant desire to make money at the cost of doing the right thing (and I'm talking about unambiguous things - is someone going to argue that freedom of speech allows people run a phishing site of your bank?), and you've got a scenario where once they reach critical mass, they will be exercising their position to the detriment of everyone who isn't paying them, similarly to how Gmail, through doing and not communicating, say "screw you" to many small email services.
When people who don't use large providers have email issues with Gmail, lots of people have knee-jerk reactions saying that everything should move to the big providers, that people and small businesses should not host their own email, and so on. This is NOT the way the Internet should work, and we should never allow Gmail to just arbitrarily do whatever they want, then accept it as the new normal.
If you have more than a dozen megabits of outgoing bandwidth, you can easily host a blog from your home network which can handle a front paging here. Just don't expect to dynamically generate a new copy of the site for every visitor, and if your bandwidth is tight, then host your images on a static server off of your network. Cloudflare is not necessary - perhaps it's easier, but it isn't necessarily best to blindly trust a company that wants to become a monopoly.
I'm talking about their rather political move to re-centralize DNS by shoehorning themselves in to Firefox via DoH, for instance. Their unwillingness to be transparent makes this all the more frightening. Add to that their blatant desire to make money at the cost of doing the right thing (and I'm talking about unambiguous things - is someone going to argue that freedom of speech allows people run a phishing site of your bank?), and you've got a scenario where once they reach critical mass, they will be exercising their position to the detriment of everyone who isn't paying them, similarly to how Gmail, through doing and not communicating, say "screw you" to many small email services.
When people who don't use large providers have email issues with Gmail, lots of people have knee-jerk reactions saying that everything should move to the big providers, that people and small businesses should not host their own email, and so on. This is NOT the way the Internet should work, and we should never allow Gmail to just arbitrarily do whatever they want, then accept it as the new normal.
If you have more than a dozen megabits of outgoing bandwidth, you can easily host a blog from your home network which can handle a front paging here. Just don't expect to dynamically generate a new copy of the site for every visitor, and if your bandwidth is tight, then host your images on a static server off of your network. Cloudflare is not necessary - perhaps it's easier, but it isn't necessarily best to blindly trust a company that wants to become a monopoly.
I don't know. Sure, maybe not on your barely-broadband DSL connection, but I'm pretty sure you can run most things on a average shared webhosting, even if you're using WP. You just need to make sure that you have a working caching system in place. It's a gigantic difference even on an apache to just read & send a plain file and invoke PHP & run all the costly code. I don't believe HN should bring down any site that can essentially be cached as static HTML.
Cloudflare will only help once your server has gone down with their "Always On" thingy, if you have that enabled. They don't cache HTML by default.
Cloudflare will only help once your server has gone down with their "Always On" thingy, if you have that enabled. They don't cache HTML by default.
There's alternatives to Cloudflare that offer affordable-but-not-free CDNs which has always felt less risky to me. I'd rather know I'm the customer instead of the product.
> I can't deal with something like my blog post being frontpaged on HN if my website is hosted in my house, unless I have a good CDN.
IPFS can work as a CDN, at least for static content that users are willing to seed. This is especially relevant to the "blog post hits frontpage on HN" case. Of course, dynamic content is not quite as easy.
IPFS can work as a CDN, at least for static content that users are willing to seed. This is especially relevant to the "blog post hits frontpage on HN" case. Of course, dynamic content is not quite as easy.
This is why you need name servers from 2 different companies and dns monitoring. It doesn't matter who your provider is. Errors happen and waiting half a week to fix it is insane.
> I registered for Cloudflare with a Gmail address specifically so that I could receive notifications from them if there were issues with my email setup.
good idea
good idea
First time i've seen a https://txti.es posted on here. nice.
i am using api to download/backup zone every week (by cron) to gdrive (fuse drive / cheap solution)
i do this for all the domains i use/manage
this post has been a good reminder to check them :)
imho about audit log: since they "delete" everything, nothing is left in the zone/domain.
thus, initial log (127.0.0.1/creation) comes up. kind of feature of the bug/logic error.
i do this for all the domains i use/manage
this post has been a good reminder to check them :)
imho about audit log: since they "delete" everything, nothing is left in the zone/domain.
thus, initial log (127.0.0.1/creation) comes up. kind of feature of the bug/logic error.
I hate this paid support model. I don’t want to have to pay to tell a company I think they screwed up.
Suspending can be a mistake, that's fine. Deleting DNS records and no notification is absurd.
I wonder how many people backup their DNS settings so they can recreate them quickly...
after they began to turn on their own customers i moved all my domains and closed my account. looks like i have not missed out much.
imagine if one day your bank decided to close your entire bank account without telling you...lol.
imagine if one day your bank decided to close your entire bank account without telling you...lol.
Cloudflare is pretty trash regardless, but putting all your eggs in one basket (no matter which provider) is just a terrible idea.
so what was the issue?
dana321(3)
djsumdog(2)
Cloudflare lost my support when they started de-platforming people for holding opinions they didn't agree with. Censorship outside of strictly legal bounds should not be tolerated from a company as powerful as Cloudflare.
At least in some cases, those people were claiming that because they hadn't been removed, Cloudflare supported them. I don't see what other option Cloudflare had at that point.
Except that Matthew Prince, Cloudflare's CEO, made that up out of thin air. There's no point where Daily Stormer said that Cloudflare supported their ideology. There's no record of this on the Internet. Can't find it, because nobody at the site ever made such claims. Stormer was kicked off of dozens of domain registrars and registries (GoDaddy, Google, Namecheap, Dreamhost, several national cctlds) in the same period -- none of them had to come up with a fake excuse like "people will think we support their ideology". Cloudflare does do infrastructure plenty of pedophile and Islamic terrorist sites, so now we can assume that they actually do support those as they aren't removing them from the service.
Cloudflare also didn't even bother telling that lie anymore when the dozens of sites they censored afterwards including 8chan were systematically barred from basic commerce.
Cloudflare also didn't even bother telling that lie anymore when the dozens of sites they censored afterwards including 8chan were systematically barred from basic commerce.
Is it censorship if they refuse their money for a service? Pretty sure that's just business. Are they stopping you having a website?
Yes, it is censorship. The entire history of First Amendment jurisprudence was set around the idea that powerful people were not allowed to stop political and religious speech. Marsh v. Alabama is a great example: a company town owned sidewalks that they didn't want religious prosyletizers on. The courts ruled that the fact that they owned the sidewalks and roads is irrelevant. For the entire history of my country powerful people were not allowed to buy up the public square and prevent the little guy from speaking. Everyone had a right to enumerate their grievances in a free and open marketplace of ideas. This has of course changed in the age of the Internet, where a bunch of scheming Stanford grads have bought up the courts, wrested control of the key Internet infrastructure away from the public who funded its creation, and sit there and grin as they take the role of arbiter over all speech on the Internet. The wealth and power disparity between the rich and poor is at its height, and it is clear that there will be no legal or democratic solution to the concentration of power in the hands of a handful of Silicon valley billionaires.
How is it censorship?, they're not stopping you publishing anything, they're just not helping you do it.
Would you be OK with your phone provider or imternet provider to stop doing business with because you said some unsavory thing to a friend or have blog supporting the wrong candidate?
Yes, because there's always another option.
I'd switch to an ISP that promised they'd take no money from Nazis
I'd switch to an ISP that promised they'd take no money from Nazis
Why do you think you have a right to host with them? You don’t, you have a privilege that’s extended by them. You’re welcome to host your own thing somewhere else.
>You don’t
You don't now.
Why do successful technology companies have a right to have a proportionally large influence on the public political debate? Is it good for society to allow successful technology companies to have such a large degree of control over something so incredibly vital, merely because they were effective at running a particular type of business?
You don't now.
Why do successful technology companies have a right to have a proportionally large influence on the public political debate? Is it good for society to allow successful technology companies to have such a large degree of control over something so incredibly vital, merely because they were effective at running a particular type of business?
What sites have they de-platformed outside of legal bounds?
Daily Stormer, archive.ph, 8chan
> archive.ph
When/why did they remove that one? Have you got any source?
When/why did they remove that one? Have you got any source?
BTW If you, dear reader, ever find yourself so frustrated with Cloudflare that you feel like your only recourse is a blog post... my email is [email protected] and I’m happy to hear from people.