ztorkelson

قبل سنتينdiscuss

It’s concatenation.

قبل 3 سنواتdiscuss

> Given a long-term token, can I create short-term, auto-expiring tokens?

I suspect the answer is: yes, via attenuation. :)

ztorkelson

قبل 4 سنواتdiscuss

Is (non-)determinism really the right concern here? I’m aware that most hash tables do not have generally predictable iteration orders, but I nevertheless understood them to be deterministic.

ztorkelson

قبل 4 سنواتdiscuss

Is this still true? I understood SHA256 to be faster than SHA512 due to hardware acceleration on current CPUs; dedicated instructions exist for the former but not the latter.

ztorkelson

قبل 4 سنواتdiscuss

Items could be placed at any pixel location within the bounds of the container. The game used the same X and Y (integer) variables which otherwise denoted the item’s location in the world.

This gave players more degrees of freedom to arrange their inventory as compared to a one-dimensional slot-based system.

ztorkelson

قبل 5 سنواتdiscuss

Sequential UUIDs don’t start at 0. They are a 128-bit composite of two integers; a temporal component in the high order bits and a random component in the low order bits.

ztorkelson

قبل 7 سنواتdiscuss

You appear to suggest that programmers with “rudimentary knowledge of transactions” should prefer lower isolation levels which sacrifice correctness for performance. If anything is “grossly irresponsible” here, it’s that.

Such isolation levels are notoriously difficult to reason about—even for experienced practitioners—and their misuse can and does introduce persistent data anomalies that can be costly to remediate.

Generally speaking, performance issues are significantly easier to diagnose and resolve than data anomalies, and they may be address in a targeted fashion as the need arises.

There’s no substitute for thinking. But if I had to prescribe general advice, it’d be this:

(1) When given the choice, select a modern database system which supports scalable and efficient serializable and snapshot transaction isolation levels.

(2) Use serializable isolation, by default, for all transactions.

(3) In the event that your transactions are not sufficiently performant, stop and investigate. Profile the system to identify the bottleneck.

(4) If the bottleneck is contention due to the transaction isolation level, stop. Assess whether the contention is inherent or whether it is incidental to the implementation or data model.

(5a) If the contention is incidental, do not lower the isolation level. Instead, refactor to eliminate the contention point. Congratulations; you are now done.

(5b) Otherwise, lower the isolation level—only for one or more of the transaction(s) in question—by a single step. Carefully assess the anomalies you have now introduced and the ramifications on the system as a whole. Look for other transactions which could intersect concurrently in time and space. Implement compensatory controls as necessary to accommodate the new behavior.

(6) Repeat only as necessary to achieve satisfaction.

ztorkelson

قبل 8 سنواتdiscuss

Clover | Sunnyvale, CA | ONSITE | Full-Time | Senior Software Engineer (Backend) | USD $180k+

Our Team

We are a small team of experienced software engineers tasked with ensuring that Clover’s rapid growth is sustainable over the long term. Our team solves for cross-cutting non-functional requirements like the security, scalability, and fault tolerance of Clover’s backend services. Together we design and develop the core architectural components, libraries, frameworks, tooling, and distributed systems at the heart of our global payment platform.

Our Work

We recently completed a project to horizontally shard our OLTP cluster, which had grown to 10+ TB in size. Next up is building a fully autonomous service for rebalancing merchant data across the shards to distribute load and eliminate hot spots.

We just finished moving our production infrastructure from private data centers to the public cloud in an effort to streamline our global expansion. Now we’re revisiting our architecture, processes, and tooling in order to better take advantage of the cloud environment.

We are actively working on the design, development, and deployment of data pipeline infrastructure to support richer analytics and reporting for our merchants and internal business needs. Our focus is on its security, scalability, reliability, and performance.

We already have a comprehensive suite of functional unit and integration tests, and are now focused on improving our automated stress tests and supporting infrastructure. That involves building the tools to spin up full production-scale environments, synthesize load, perform fault injection, and to collect, analyze, and surface test results to help drive continual improvement of performance and availability.

Our Stack

  - Java for backend services.
  - Python for integration and stress tests.
  - MySQL for OLTP. Snowflake for OLAP.
  - Kafka for stream processing.
  - Memcached for caching (duh).
  - Redis for ephemeral shared data structures.
  - Wavefront and ELK for operational visibility.
  - Google (GCP) as our cloud service provider.
  - Docker for building containers. Kubernetes for running them.
  - Netty for speaking HTTP, behind HAProxy for load balancing.

This is the Clover of today. You can help shape the Clover of tomorrow.

Contact: zac at clover dot com (and mention you saw this in HN!)

More info: https://www.clover.com/job-post?gh_jid=1461732

ztorkelson

قبل 8 سنواتdiscuss

Right. I was waiting for the author to get to this point, and much to my surprise they never did.

The takeaway shouldn’t be “strings are hard”. Strings are hard, but that’s not the problem here. The problem is using inappropriate data types for the task at hand, and the takeaway should be that representation matters.

Representing an IP endpoint as a string only really makes sense at the human-computer interface. In general, the first thing a program should do with such a string is convert it into a more suitable data type. And as noted elsewhere, more often than not there will already be a library function to do so.

Strings are especially pernicious because they are ubiquitous, in that just about anything can be represented as a string, but the operations one can do on a string representation of an object do not generally correspond with the operations one would want to do on the object itself. This disparity is the source of many bugs, which the article exemplifies (though falls short of directly addressing).

Similarly problematic are assumptions that converting an object to a string and a string to an object (i.e. `parse` and `format` functions) are bijective. This is not generally the case. (Wise practitioners might choose such a mapping, though, when the opportunity presents.)

ztorkelson

قبل 8 سنواتdiscuss

Congratulations on the launch! I’ve heard great things about Outlands and have been meaning to log in and check it out.

It’s been a long time since I was involved in the RunUO community, but as I recall, one of the biggest limiting factors on scalability was activated NPC/AI. Range queries and movement were the two big pieces, so I’m sure your improvements would have a big impact there.

We ran load tests on Hybrid with over 10k clients (not just idle, mind you, but moving and talking and whatever else we were able to throw in to the load generators), and the server was able to keep up just fine. That was on mid-2000 era hardware too, but then again, RunUO wasn’t built to really take full advantage of multiple cores.

There are a lot of things I’d do differently if I had the opportunity to go back and redesign it from the ground up, but the simple single-threaded concurrency model is not something I’d want to change without great care. For all the scalability problems RunUO had, I think the concurrency model contributed significantly to its approachability, and I’d be very cautious of making any changes which would complect game logic with concurrency control.

I’ve heard quite a few stories (and now I’m hearing a few more) of folks whose path into software development started by tinkering around with RunUO. In fact, a few of my closest friends (and some now colleagues) took that same path. I am filled with a weird mix of pride and abject humility whenever I have the opportunity to see how the project has touched people all over the world, often in ways I could never have anticipated.

Please do share your changes back. I’d love to take a look at them, even after so many years.

-krrios

ztorkelson

قبل 8 سنواتdiscuss

Clover | Sunnyvale, CA | ONSITE | Full-Time | Senior, Distributed Systems | USD $180k+

Who we are

We’re a small team of experienced software engineers tasked with ensuring that Clover’s rapid growth is sustainable over the long term. We solve for cross-cutting non-functional requirements like the security, scalability, and fault tolerance of Clover’s back-end platform and services. We design and develop the core architectural components, libraries, frameworks, tooling, and distributed systems at the heart of Clover’s global payment platform and services.

What we're doing

We recently completed a project to horizontally shard our transactional (payment) database cluster, which had grown to 10+ TB in size. Now we’re building a fully autonomous service for rebalancing merchant data across the shards to distribute load and eliminate hot spots.

We recently completed a project to lift-and-shift our production infrastructure from private data centers to public cloud in order to streamline our global expansion. Now we’re revisiting our architecture, processes, and tooling in order to more fully leverage the cloud environment.

We are actively working on the design, development, and deployment of data pipeline infrastructure to support richer analytics and reporting for our merchants and internal business needs. Our focus is on its security, scalability, reliability, and performance.

We are paving the road towards a decomposed architecture of horizontal core services (like authentication, authorization, placement, tokenization, and inter-device replication) and vertical product services (like device management, merchant boarding, inventory propagation, and customer loyalty programs).

We already have a healthy suite of functional unit and integration tests, and we are actively enhancing our automated stress tests and supporting infrastructure. That involves building the tools to spin up full production-scale environments, synthesize load, perform fault injection, and to collect, analyze, and surface test results in order to help drive continual improvement of performance and availability.

What we use

  - Java for back-end services.
  - Python for integration and stress tests.
  - MySQL for OLTP. Snowflake for OLAP.
  - Kafka for stream processing.
  - Memcached for caching (duh).
  - Wavefront and ELK for operational visibility.
  - Redis for ephemeral shared data structures.
  - Google (GCP) as our cloud service provider.
  - Docker for building containers. Kubernetes for running them.
  - Netty for speaking HTTP, behind HAProxy for load balancing.

This is the Clover of today. Come help shape the Clover of tomorrow.

Contact: zac at clover dot com (please mention you saw this in HN)

ztorkelson

قبل 8 سنواتdiscuss

Yes. It's durable storage with a RAM interface. Byte-addressable through conventional CPU load/store instructions. It follows the usual cache coherence protocols for memory, but once it gets flushed, it's stable—just like conventional storage devices.

ztorkelson

قبل 8 سنواتdiscuss

The benefits of persistent memory, as I understand it, are principally improved performance (higher throughput, lower latency) and secondarily ease of programming. (Edit: another obvious benefit is cost per gigabyte; this stuff is cheaper than RAM.)

The benefits of having access to that technology on GCP (or another cloud) are the usual: reduced operational burden, increased availability, flexible pricing structure, elastic scalability, etc.

Your question about pets vs cattle is a non sequitur. Nothing about this announcement or the underlying technologies are suggestive of how they should be used (or misused). It's just a new tool in the toolbox—and as a distributed systems engineer specializing in database technology, having easy access to this hardware at scale is extremely compelling.

ztorkelson

قبل 8 سنواتdiscuss

Clover | Sunnyvale, CA | ONSITE | Full-Time | Senior, Distributed Systems | USD $180k+

We’re a small group of experienced software engineers tasked with ensuring that Clover’s continued growth is sustainable over the long term. Clover recently shipped its one-millionth device and we process over USD $60B in annualized GPV.

If you're interested in scalability, either at the operational level (e.g. 10,000 TPS this year to 20,000 TPS the next) or at the institutional level (e.g. growing from 150 engineers to 300), then please don’t hesitate to reach out. Some recent projects we’ve been working on:

  - Decomposing a monolithic Java backend into a set of modular services.
  - Strategic migration from on-premises infrastructure to public cloud. We use both AWS and GCP.
  - Design and development of a high-throughput, low-latency OLAP pipeline, backed by Kafka and Snowflake.
  - Design and development of a horizontally-scalable MySQL database layer for sharding tens of terabytes of tenant data.
  - Tooling for load test automation and analysis within our SDLC. We use Python and Locust to synthesize the workload.

zac at clover dot com

ztorkelson

قبل 8 سنواتdiscuss

That's not a very charitable interpretation of the GP. He's not criticizing hash tables because they aren't radix trees. He's criticizing this and other hash table implementations because they have undesirable performance characteristics with respect to his needs, and noting an alternative data structure which he has found to be more suitable. I think that's fair game.

Note that there exist hash table implementations which support incremental resizing.

I do agree, though, that it's a leap to go from "hash tables with this property are inappropriate under these constraints" to "hash tables with this property should not be considered further", precisely because there is such a broad spectrum of data structure requirements across the industry.

ztorkelson

قبل 8 سنواتdiscuss

Right. When you ask "how could we support deletion of nodes", am I right to assume you mean "how could we reclaim memory of deleted nodes"?

Conventionally, a free list would be used, though admittedly that sacrifices locality which was one of the original motivating factors. To the extent that deletion is a significant component of the workload, then yeah, this doesn't help.

ztorkelson

قبل 8 سنواتdiscuss

Gotcha. Yes, while there are implementation techniques that can help (e.g. arena allocation, prefix compression, adaptive node sizes), the data structure is fundamentally a tree, and so one can expect a higher number of random memory accesses (when compared with suitably optimized hash tables, like those discussed in the article).

In some scenarios, radix trees can overcome that with better space efficiency (i.e. implicit key storage), but I think it's fair to say that there are many scenarios (if not most!) where hash tables outperform radix trees.

Of course, that all hinges on how you measure performance. As Salvatore pointed out, if you're looking to minimize variance, hash tables might not be suitable due to their growth characteristics. Radix trees also give you efficient sorting and range scans out of the box, and are a viable immutable persistent data structure as well. If none of those are meaningful to your use cases, then hash tables could certainly be a better fit.

ztorkelson

قبل 8 سنواتdiscuss

I've found myself reaching for radix trees more and more for the same reason you listed: competitive and predictable performance.

That said, I can see two counter-arguments:

1. Rehashing can be avoided if the table size is known in advance. 2. Rehashing is immaterial if the data structure is write-once, read-many.

In those cases, we might reasonably expect a hash table to perform better.

ztorkelson

قبل 8 سنواتdiscuss

Would you mind elaborating? I'm not quite sure what you mean by segmented here.

ztorkelson

قبل 8 سنواتdiscuss

I actually don't have much hands-on experience with Postgres, so I can't speak to how it performs in practice.

My comment was highlighting that Postgres' implementation of SSI is at least a better starting point than most other purely pessimistic implementations of the serializable isolation level. It does not surprise me, however, to hear that there are still performance deficiencies in practice. For example, the Postgres implementation is imprecise, which will result in spurious transaction aborts (those will presumably be retried, but it comes at a cost to latency and throughput). And even though reads are optimistic (which will perform better than pessimistic concurrency control, if contention rates are sufficiently low), maintaining transaction dependency metadata can still have a non-negligible cost.

I also think Postgres suffers from other more dubious choices--like a lack of clustered indexes--which can exacerbate things.

My work in this area has focused on revisiting various architectural and implementation design decisions which have contributed to the current state of affairs. I think the relational data model is generally the right choice, but most relational databases have some godawful characteristics around performance, scalability, reliability, and programmability which limit their efficacy in practice.

ztorkelson

Submissions

Beyond Blockchain: Convergent Consensus (Strange Loop 2022)

Amazon Aurora: Avoiding Distributed Consensus for I/Os, Commits, and Membership

comments