RethinkDNS Resolver That Deploys to CF Workers, Deno Deploy, Fastly, Fly.io(github.com)
github.com
RethinkDNS Resolver That Deploys to CF Workers, Deno Deploy, Fastly, Fly.io
https://github.com/serverless-dns/serverless-dns
4 comments
Indeed, then someone rewrites it in a compiled language and we get yet another "40x improvement" blog post.
> 40x improvement
Could WASM AoT bring about those? ;)
> rewrites ... compiled language
With stuff like AutoFDO on Android, we've come a full circle with folks "JITing" compiled blocks... https://lwn.net/Articles/995397/
Could WASM AoT bring about those? ;)
> rewrites ... compiled language
With stuff like AutoFDO on Android, we've come a full circle with folks "JITing" compiled blocks... https://lwn.net/Articles/995397/
We have, yet
> Propeller requires specific software and hardware support to do its job
WASM AoT is nothing special, we have had plenty of those since UNCOL (1958).
Two of the oldest systems using bytecode as distribution format are Burroughs from 1961, nowadays Unisys ClearPath MCP, and IBM AS/400 from 1988, nowadays IBM i.
This while not bothering to actually list all the products and attempts between 1958 and 2025.
Naturally on the magpie developer culture it is easier to sell shinny new, than having people caring about our history.
> Propeller requires specific software and hardware support to do its job
WASM AoT is nothing special, we have had plenty of those since UNCOL (1958).
Two of the oldest systems using bytecode as distribution format are Burroughs from 1961, nowadays Unisys ClearPath MCP, and IBM AS/400 from 1988, nowadays IBM i.
This while not bothering to actually list all the products and attempts between 1958 and 2025.
Naturally on the magpie developer culture it is easier to sell shinny new, than having people caring about our history.
> Rethink's server processed DNS requests in 40-60ms, which could potentially make a new webpage render up to 50ms slower the first time, while the system default took just 5-10ms
You can choose "system default" in the Rethink Android app from Configure -> DNS -> System DNS. You don't have to use Rethink's DoH (DNS over HTTPS) / DoT (DNS over TLS) servers. You can setup any DNSCrypt, Oblivious DoH, or plain old DNS endpoint with Rethink to forward DNS requests to.
> I chose AdAway and have stuck to that choice for now
A decent choice.
A word of caution when using it in non-root mode, though: AdAway borrows code from another (now discontinued but pretty solid) project viz. DNS66. From when I looked at the code, DNS66 never handled DNS over TCP. You can test this with Termux: dig +tcp example.com A.
> doubt their choice of implementing something so performance-critical as a DNS server in JavaScript
For a remotely-hosted DNS resolver, network latency & concurrency is likely to dominate than performance of the code itself. You're right that JS is slow (as compared to Rust, say), but looking at metrics from serving 300m to 3bn queries/day on Cloudflare, our median (CPU time) spent processing a DNS request was consistently less than 2ms, and IO wait (there are no "disks" so, this is mostly hitting the caches or upstream resolvers) was estimated to be less than 7ms.
Also, we employ a bunch of optimizations on client & the server, as appropriate (if we're missing any we should implement, let us know).
- Coalesce multiple requests into one upstream request: For example, if there's 40 clients all wanting to resolve the same domain (say, ipv4only.arpa) within milliseconds of each other, only one request is upstreamed.
- Utilize in-process LFU cache and Cloudflare's Cache API to avoid upstreaming where possible.
- Connection pool egress to avoid reconnect overheads.
- Race response for a query from multiple upstreams in parallel.
- Support TLS session resumption.
- Prefer the lighter TLS_AES_128_GCM_SHA256 cipher suite.
- Dynamically adjust the TLS frame size. DNS queries aren't that big.
- Disable Nagling on TCP.
- Rehydrate responses for top domains stored in the in-process cache in the background.
- Admission Control (load shedding) & queuing disciplines (CoDel etc)
---
> curl -w "DNS Time: %{time_namelookup}s\nConnect: %{time_connect}s\nTotal: %{time_total}s\n" -H "accept: application/dns-message" -H "content-type: application/dns-message" --data "<binary-data>" -o nul -s 1.1.1.1
Shouldn't the switch "-s" be "https://one.one.one.one/dns-query"? For DoH perf specifically, I use https://github.com/ameshkov/godnsbench as it supports parallel and cache-busting (random) queries.
Ex:
Rethink on Workers:
You can choose "system default" in the Rethink Android app from Configure -> DNS -> System DNS. You don't have to use Rethink's DoH (DNS over HTTPS) / DoT (DNS over TLS) servers. You can setup any DNSCrypt, Oblivious DoH, or plain old DNS endpoint with Rethink to forward DNS requests to.
> I chose AdAway and have stuck to that choice for now
A decent choice.
A word of caution when using it in non-root mode, though: AdAway borrows code from another (now discontinued but pretty solid) project viz. DNS66. From when I looked at the code, DNS66 never handled DNS over TCP. You can test this with Termux: dig +tcp example.com A.
> doubt their choice of implementing something so performance-critical as a DNS server in JavaScript
For a remotely-hosted DNS resolver, network latency & concurrency is likely to dominate than performance of the code itself. You're right that JS is slow (as compared to Rust, say), but looking at metrics from serving 300m to 3bn queries/day on Cloudflare, our median (CPU time) spent processing a DNS request was consistently less than 2ms, and IO wait (there are no "disks" so, this is mostly hitting the caches or upstream resolvers) was estimated to be less than 7ms.
Also, we employ a bunch of optimizations on client & the server, as appropriate (if we're missing any we should implement, let us know).
- Coalesce multiple requests into one upstream request: For example, if there's 40 clients all wanting to resolve the same domain (say, ipv4only.arpa) within milliseconds of each other, only one request is upstreamed.
- Utilize in-process LFU cache and Cloudflare's Cache API to avoid upstreaming where possible.
- Connection pool egress to avoid reconnect overheads.
- Race response for a query from multiple upstreams in parallel.
- Support TLS session resumption.
- Prefer the lighter TLS_AES_128_GCM_SHA256 cipher suite.
- Dynamically adjust the TLS frame size. DNS queries aren't that big.
- Disable Nagling on TCP.
- Rehydrate responses for top domains stored in the in-process cache in the background.
- Admission Control (load shedding) & queuing disciplines (CoDel etc)
---
> curl -w "DNS Time: %{time_namelookup}s\nConnect: %{time_connect}s\nTotal: %{time_total}s\n" -H "accept: application/dns-message" -H "content-type: application/dns-message" --data "<binary-data>" -o nul -s 1.1.1.1
Shouldn't the switch "-s" be "https://one.one.one.one/dns-query"? For DoH perf specifically, I use https://github.com/ameshkov/godnsbench as it supports parallel and cache-busting (random) queries.
Ex:
Rethink on Workers:
./godnsbench -a https://sky.rethinkdns.com/dns-query -p 10 -c 300 -t 1 -q {random}.dnsleaktest.com
godnsbench with the following configuration:
{
"Address": "https://sky.rethinkdns.com/dns-query",
"Connections": 10,
"Timeout": 1,
"QueriesCount": 300,
}
The test results are:
Elapsed: 6.120078941s
Average QPS: 49.018394
Processed queries: 300
Average per query: 20.400765ms
Errors count: 0
./godnsbench -a https://cloudflare-dns.com/dns-query -p 10 -c 300 -t 1 -q {random}.dnsleaktest.com
godnsbench with the following configuration:
{
"Address": "https://cloudflare-dns.com/dns-query",
"Connections": 10,
"Timeout": 1,
"QueriesCount": 300,
}
The test results are:
Elapsed: 6.780474724s
Average QPS: 44.244383
Processed queries: 299
Average per query: 22.677588ms
Errors count: 1I was able to reproduce your results with the provided command. In that case, Rethink DNS is 1-2 ms faster than Cloudflare DNS. However, the provided command actually runs 300 queries in a really short timeframe (specified by -c 300), which is very different from real-world use cases — in most cases, we look up a domain only once, and the result is cached by the OS.
If we set the count parameter to 1 (closer to the mentioned real-world use cases), we can reproduce the result in my comment above. Rethink DNS actually takes 450 ms to return a response.
- https://check-host.net/check-ping?host=https://sky.rethinkdn...
- https://check-host.net/check-ping?host=https://cloudflare-dn...
If CPU time isn't the bottleneck of response time, any idea what the reason is behind the difference in response time?
If we set the count parameter to 1 (closer to the mentioned real-world use cases), we can reproduce the result in my comment above. Rethink DNS actually takes 450 ms to return a response.
./godnsbench -c 1 -q google.com -a https://sky.rethinkdns.com/dns-query
...
2025/05/04 23:22:47 [info] Average per query: 452.3948ms
./godnsbench -c 1 -q google.com -a https://cloudflare-dns.com/dns-query
...
2025/05/04 23:22:49 [info] Average per query: 34.4343ms
- https://check-host.net/check-ping?host=https://sky.rethinkdn...
- https://check-host.net/check-ping?host=https://cloudflare-dn...
If CPU time isn't the bottleneck of response time, any idea what the reason is behind the difference in response time?
> However, the provided command actually runs 300 queries in a really short timeframe (specified by -c 300)
Our servers see 2500-25000 requests per second. So, my intention was to test the server, not the client (original post points out issues with the server).
> which is very different from real-world use cases
Not really. A decent DoH/DoT/ODoH client would pool TLS/HTTP connections (on top of mutliplexing HTTP/2 on TCP) & have its own caches (probably hydrate the top domains in the background, even).
> As sky.rethinkdns.com is hosted on Cloudflare, there shouldn't be any notable network differences between Rethink DNS and Cloudflare DNS.
Cloudflare likely co-locates its DNS resolvers with all major ISPs, which might explain the speed. For instance, ping 1.1.1.1 reports ridiculous numbers (<3ms) on some networks.
> With Cloudflare DNS, it takes only 35 ms, even when using DNS over HTTPS
In my experiments, for (non-random) repeat queries, the fastest Cloudflare response is about 20% faster with "-c 1" than Max and 30% faster than Sky.
You should try if any other non-BigTech public DNS posts comparable numbers for "-c 1": Like ControlD, AdGuard, NextDNS, CleanBrowsing, DNSFilter, Quad9 etc. In my experiments, these aren't significantly better than Max or Sky. And imo, concurrent "random" (cache-busting) queries are better test of a remote resolver's capabilities.
For Max specifically, caches speed up response times significantly, even if Max is deployed to only 30+ locations (as compared to Cloudflare's 300+) running 40+ low-powered servers on Fly.io's network. For Sky, a lot is under the control of Cloudflare: How they route & shape our traffic and just where they run our Workers. For the most part though, Sky is comparatively pretty fast. If you use Rethink (enable Configure -> DNS -> DNS Booster to turn on optimizations), you should be able to see per-query round-trip time in Configure -> Logs -> DNS.
Our servers see 2500-25000 requests per second. So, my intention was to test the server, not the client (original post points out issues with the server).
> which is very different from real-world use cases
Not really. A decent DoH/DoT/ODoH client would pool TLS/HTTP connections (on top of mutliplexing HTTP/2 on TCP) & have its own caches (probably hydrate the top domains in the background, even).
> As sky.rethinkdns.com is hosted on Cloudflare, there shouldn't be any notable network differences between Rethink DNS and Cloudflare DNS.
Cloudflare likely co-locates its DNS resolvers with all major ISPs, which might explain the speed. For instance, ping 1.1.1.1 reports ridiculous numbers (<3ms) on some networks.
> With Cloudflare DNS, it takes only 35 ms, even when using DNS over HTTPS
In my experiments, for (non-random) repeat queries, the fastest Cloudflare response is about 20% faster with "-c 1" than Max and 30% faster than Sky.
You should try if any other non-BigTech public DNS posts comparable numbers for "-c 1": Like ControlD, AdGuard, NextDNS, CleanBrowsing, DNSFilter, Quad9 etc. In my experiments, these aren't significantly better than Max or Sky. And imo, concurrent "random" (cache-busting) queries are better test of a remote resolver's capabilities.
For Max specifically, caches speed up response times significantly, even if Max is deployed to only 30+ locations (as compared to Cloudflare's 300+) running 40+ low-powered servers on Fly.io's network. For Sky, a lot is under the control of Cloudflare: How they route & shape our traffic and just where they run our Workers. For the most part though, Sky is comparatively pretty fast. If you use Rethink (enable Configure -> DNS -> DNS Booster to turn on optimizations), you should be able to see per-query round-trip time in Configure -> Logs -> DNS.
Really appreciate both of you commenting here.
Could the ~430ms be startup time for a sleeping worker?
Or is it the cost for a cache miss?
Ed: hm, I suppose google.com should pretty much always be a cache hit... Unless they have some really strange TTLs?
Could the ~430ms be startup time for a sleeping worker?
Or is it the cost for a cache miss?
Ed: hm, I suppose google.com should pretty much always be a cache hit... Unless they have some really strange TTLs?
> Could the ~430ms be startup time for a sleeping worker? Or is it the cost for a cache miss?
One won't really gain much insight for DoH / DoT resolver performance with one query per connection, I don't think.
For something as lightweight as domain name resolution, network latency / round-trip time dominate so much (for a public DNS resolver anyway) that the CPU-bound optimizations are almost always pointless if you don't first optimize your network deployment: For example, not running servers closer to your users wouldn't move the needle much regardless of how big your caches are, or the language your resolver is written in (C or Rust or hand-rolled assembly or SIMD/SIMT built-ins etc). A good indicator of performance of a pubic DNS resolver would be to monitor it over a period of time, to effectively rule out noise introduced by variances.
One won't really gain much insight for DoH / DoT resolver performance with one query per connection, I don't think.
For something as lightweight as domain name resolution, network latency / round-trip time dominate so much (for a public DNS resolver anyway) that the CPU-bound optimizations are almost always pointless if you don't first optimize your network deployment: For example, not running servers closer to your users wouldn't move the needle much regardless of how big your caches are, or the language your resolver is written in (C or Rust or hand-rolled assembly or SIMD/SIMT built-ins etc). A good indicator of performance of a pubic DNS resolver would be to monitor it over a period of time, to effectively rule out noise introduced by variances.
But as far as I understand - these tests are run on hosts with comparable distance/ping?
And 400ms ping is remarkably bad for a client that can otherwise do 20ms to a nearby server?
And 400ms ping is remarkably bad for a client that can otherwise do 20ms to a nearby server?
godnsbench isn't testing pings but DoH / DoT (and DoH is expensive due to the TCP, TLS, and HTTP overhead [0]). OP did share a ping report (which paints a different picture): https://check-host.net/check-ping?host=https://sky.rethinkdn...
Sending one DNS query per HTTP connection (which is what the OP claims is "real world") is not how "real world" DoH clients are supposed to be written or work. So it is unlikely to be representative of "real world" delays.
In fact, in my tests, other public DNS resolvers don't fair any better with one query per connection, either.
[0] For example, a TLS handshake transfers 1.5KB to 4KB, while a typical DNS query is 30B to 70B & responses are 70B to 150B.
Sending one DNS query per HTTP connection (which is what the OP claims is "real world") is not how "real world" DoH clients are supposed to be written or work. So it is unlikely to be representative of "real world" delays.
In fact, in my tests, other public DNS resolvers don't fair any better with one query per connection, either.
[0] For example, a TLS handshake transfers 1.5KB to 4KB, while a typical DNS query is 30B to 70B & responses are 70B to 150B.
Actually, 450 ms was already a cache hit. The response time would be 530 ms on the first query of google.com:
./godnsbench -c 1 -q google.com -a https://sky.rethinkdns.com/dns-query
...
2025/05/05 02:50:14 [info] Average per query: 532.1144ms
./godnsbench -c 1 -q google.com -a https://sky.rethinkdns.com/dns-query
...
2025/05/05 02:50:48 [info] Average per query: 468.5749ms
./godnsbench -c 1 -q google.com -a https://sky.rethinkdns.com/dns-query
...
2025/05/05 02:55:04 [info] Average per query: 472.1229ms
./godnsbench -c 1 -q google.com -a tls://max.rethinkdns.com
...
2025/05/05 02:50:26 [info] Average per query: 290.4762ms
./godnsbench -c 1 -q google.com -a tls://max.rethinkdns.com
...
2025/05/05 02:50:51 [info] Average per query: 145.1868ms
So how is it fast in the multiple lookup case?
Is it all per request overhead and gets eaten by a single large request doing multiple lookups?
Is it all per request overhead and gets eaten by a single large request doing multiple lookups?
Cool!
"Pi-hole -esque" is a nice descriptor.
Tangent: Bunny.net is my new favorite CDN / cloud service provider. They have scriptable DNS too.
Tangent: Bunny.net is my new favorite CDN / cloud service provider. They have scriptable DNS too.
Bunny requires a credit card for its free trial and it seems that it would only give 7 days free trial which is really weird.
So I am off to cloudflare workers free tier.
So I am off to cloudflare workers free tier.
FWIW Bunny added free credits (totaling $50) so my free trial is effectively MUCH longer than 7d. They're the fastest CDN, have awesome / intuitive UX, and very robust featureset.
Just fyi (not talking down this project, more providing alternatives) - dnsdist (part of OpenDNS "suite") which is mostly "sold" as a DNS load balancer - is also a perfect stub resolver, that can do DoH/DoT on both ends, that can block certain websites, and is very easy on resources.
Not that many people talk about its usage as a stub resolver, and I don't know why!
Not that many people talk about its usage as a stub resolver, and I don't know why!
The Rethink DNS mobile app hasn't seen a release in years. Is the project still alive?
https://github.com/celzero/rethink-app
https://github.com/celzero/rethink-app
>in years.
not even past 1 year.
See https://github.com/celzero/rethink-app/discussions/1884
not even past 1 year.
See https://github.com/celzero/rethink-app/discussions/1884
> the development happens on the lead developer's feature branch
Oof. This makes me cringe so hard. I once took over a project (but the developer didn't know they were getting fired) and the guy was doing everything on his laptop, from his laptop. Deployments and builds were from his laptop. Even dependencies weren't checked into the code (using global installs of them on unknown versions). The owner had me come in because after talking to several people realized he was in a bad place.
It took me ~2 months to learn everything and document all the things. Then the owner fired him. That guy kept development back for so long by simply not documenting/sharing code and configuration. Now there's an entire team with a healthy development flow. But wow, I had some flashbacks reading that...
Oof. This makes me cringe so hard. I once took over a project (but the developer didn't know they were getting fired) and the guy was doing everything on his laptop, from his laptop. Deployments and builds were from his laptop. Even dependencies weren't checked into the code (using global installs of them on unknown versions). The owner had me come in because after talking to several people realized he was in a bad place.
It took me ~2 months to learn everything and document all the things. Then the owner fired him. That guy kept development back for so long by simply not documenting/sharing code and configuration. Now there's an entire team with a healthy development flow. But wow, I had some flashbacks reading that...
When I was looking for ad-block solutions on Android, Rethink DNS was actually on the top of my list. However, when I found out that their server was written in JavaScript, I did some benchmarking.
Rethink's server processed DNS requests in 400-500ms, which could potentially make a new webpage render up to half a second slower the first time:
While the Cloudflare's server took just 5-10ms, as seen below:
In the end, I chose AdAway and have stuck to that choice.