Let’s talk about usernames(b-list.org)
b-list.org
Let’s talk about usernames
https://www.b-list.org/weblog/2018/feb/11/usernames/
429 comments
> So if you’re enforcing unique email addresses, or using email addresses as a user identifier, you need to be aware of this and you probably need to strip all dot characters from the local-part, along with + and any text after it, before doing your uniqueness check. Currently django-registration doesn’t do this, but I have plans to add it in the 3.x series.
Please don't "normalize" email addresses like this. Not all mail systems are Gmail, and many do treat "[email protected]" and "[email protected]" as different identities. And even if we are talking about Gmail - it's not your identity system's job to deduplicate different logical addresses for the same physical inbox.
Please don't "normalize" email addresses like this. Not all mail systems are Gmail, and many do treat "[email protected]" and "[email protected]" as different identities. And even if we are talking about Gmail - it's not your identity system's job to deduplicate different logical addresses for the same physical inbox.
> And while I could write this as one of those “falsehoods programmers believe about X” articles, my personal preference is to actually explain why this is trickier than people think, and offer some advice on how to deal with it, rather than just provide mockery with no useful context.
Thank you, so much.
I just wanted to highlight that for anyone who looked at the comments to decide whether or not to read the article.
Thank you, so much.
I just wanted to highlight that for anyone who looked at the comments to decide whether or not to read the article.
In my 20 years of experience validating email addresses, I've found one thing that works every time without fail:
Send email to it
That is literally the only way to validate an email address. There is no regular expression or algorithm that can validate and/or deduplicate an email address.
You must simply treat every email as unique until you send an email to it and that person proves otherwise.
That being said, this article brings up a lot of important things about confusables that everyone should definitely be aware of, especially if you're going to have public identities.
Send email to it
That is literally the only way to validate an email address. There is no regular expression or algorithm that can validate and/or deduplicate an email address.
You must simply treat every email as unique until you send an email to it and that person proves otherwise.
That being said, this article brings up a lot of important things about confusables that everyone should definitely be aware of, especially if you're going to have public identities.
> So if you’re enforcing unique email addresses, or using email addresses as a user identifier, you need to be aware of this and you probably need to strip all dot characters from the local-part, along with + and any text after it, before doing your uniqueness check. Currently django-registration doesn’t do this, but I have plans to add it in the 3.x series.
Isn't this a really dangerous game to play? Just because some major MTA's assign a class of addresses to each user doesn't make each member of that class not a unique identifier in general. Is it worth the headache to maintain a list of various email systems' policies rather than just treating them all as unique?
Isn't this a really dangerous game to play? Just because some major MTA's assign a class of addresses to each user doesn't make each member of that class not a unique identifier in general. Is it worth the headache to maintain a list of various email systems' policies rather than just treating them all as unique?
> Many systems ask the username to fulfill all three of these roles, which is probably wrong.
What system with any non-trivial level of use uses the text username as (1) the FK in the database, as opposed to the generated or auto-incremented ID in the db; (2) the login name; and (3) the publicly-displayed displayed "name" of the user for others to see?
Plenty of forums etc use the login name for #2 and #3, and I'm not convinced by this article that that's the wrong way to do it. I haven't ever seen a single professional product that uses the text username that a user logs in with as the actual DB-level foreign key. That's grade school level database design.
What system with any non-trivial level of use uses the text username as (1) the FK in the database, as opposed to the generated or auto-incremented ID in the db; (2) the login name; and (3) the publicly-displayed displayed "name" of the user for others to see?
Plenty of forums etc use the login name for #2 and #3, and I'm not convinced by this article that that's the wrong way to do it. I haven't ever seen a single professional product that uses the text username that a user logs in with as the actual DB-level foreign key. That's grade school level database design.
> Uniqueness is harder than you think
Discord has a very interesting solution to this. They have user names and user ids. User IDs are tied to emails and the user's name seems to just be a random text identity for displaying to users. I assume most of their backend code used a unique, sequential or random, integer ID to identify and talk about users while their frontend just makes the ID to a "user name". As long as you slap account creation behind a verification email and don't mind one user being able to sign up for multiple accounts you side step many of the larger problems that come from choosing user names because, in effect, you are choosing the "Real" username and you can make any guarantees that make writing all of your other software easy.
Discord has a very interesting solution to this. They have user names and user ids. User IDs are tied to emails and the user's name seems to just be a random text identity for displaying to users. I assume most of their backend code used a unique, sequential or random, integer ID to identify and talk about users while their frontend just makes the ID to a "user name". As long as you slap account creation behind a verification email and don't mind one user being able to sign up for multiple accounts you side step many of the larger problems that come from choosing user names because, in effect, you are choosing the "Real" username and you can make any guarantees that make writing all of your other software easy.
This should be exhibit number one why you should always favor open source libraries rather than writing your own plumbing functionality, especially around authentication and authorization. The onus should be on the developer to explain why the open source library isn't a fit, rather than defaulting to 'roll your own'.
The edge cases discussed don't pop up that often unless you have lots of folks using your software or are really diligent about fuzzing and testing edge cases. If you roll your own, say, username system, you probably aren't going to fall into either of those two cases. Which means you're vulnerable.
The edge cases discussed don't pop up that often unless you have lots of folks using your software or are really diligent about fuzzing and testing edge cases. If you roll your own, say, username system, you probably aren't going to fall into either of those two cases. Which means you're vulnerable.
I agree with everything here except the email addresses example. Yes I do want to register [email protected] and [email protected]. Those are different accounts, please don’t mess with that.
Hey community, shameless plug: For the purpose mentioned in the article to disallow certain usernames, I created this GitHub Repo sometime back. Feel free to submit a pull request :)
https://github.com/dsignr/disallowed-usernames
https://github.com/dsignr/disallowed-usernames
> So if you’re enforcing unique email addresses, or using email addresses as a user identifier, you need to be aware of this and you probably need to strip all dot characters from the local-part, along with + and any text after it, before doing your uniqueness check.
Please don't do this, lots of people (including myself) use the '+' hack to separate accounts for different contexts (business/personal, different projects/clients, etc).
Please don't do this, lots of people (including myself) use the '+' hack to separate accounts for different contexts (business/personal, different projects/clients, etc).
My Linkedin username: https://www.linkedin.com/in/Αdmin/
If you're going to allow unicode usernames then you should casefold them rather than lowercasing them before normalizing as NFKC.
You should ideally also store a second copy of the username in the original casing and normalized as NFC for display purposes, as some users care a lot about seeing their username exactly as they entered it. (And in fact not allowing this may be seen as culturally insensitive in some cases, much like not supporting unicode.) The same applies to the user's first and last name, which you can store in NFC for display purposes and casefolded into NFKC for string comparison (e.g. search) purposes.
That said, most sites limit usernames to ASCII characters so that they can be (easily) used in URLs. In this case you don't need to casefold or normalize, just converting to lowercase is enough.
You should ideally also store a second copy of the username in the original casing and normalized as NFC for display purposes, as some users care a lot about seeing their username exactly as they entered it. (And in fact not allowing this may be seen as culturally insensitive in some cases, much like not supporting unicode.) The same applies to the user's first and last name, which you can store in NFC for display purposes and casefolded into NFKC for string comparison (e.g. search) purposes.
That said, most sites limit usernames to ASCII characters so that they can be (easily) used in URLs. In this case you don't need to casefold or normalize, just converting to lowercase is enough.
These "battle hardened" articles are fascinating. It is the output of years of experience and learning from real problems. It's building the best practices guides, and building the tools to scan for the edge cases. A great read!
Worth mentioning Spotify's account hijacking problem when using unicode
https://labs.spotify.com/2013/06/18/creative-usernames/
When it comes to email address normalization, it sounds like we could do with a standardized way for a domain to express normalization policies.
Could be as simple as publishing a set of regular expression subsitution rules, specifying (for example):
* render to lower case (because this particular domain is case insensitive)
* drop periods (because this domain treats them like gmail does)
* drop '+' and any subsequent characters (because this domain treats them like gmail does)
* ASCII only (because mail software is old, and doesn't support unicode)
Etc.
Each domain could then publish their own rule, perhaps in a DNS txt record, and anyone needing to check if two email addresses alias to the same could run the correct checks.
Could be as simple as publishing a set of regular expression subsitution rules, specifying (for example):
* render to lower case (because this particular domain is case insensitive)
* drop periods (because this domain treats them like gmail does)
* drop '+' and any subsequent characters (because this domain treats them like gmail does)
* ASCII only (because mail software is old, and doesn't support unicode)
Etc.
Each domain could then publish their own rule, perhaps in a DNS txt record, and anyone needing to check if two email addresses alias to the same could run the correct checks.
My strategy is to think of a name so unique that no one else would think of using it. The one on my HN account I originally cooked up in Nov. 2014, and I never had to extend it with numbers to get it accepted on various forums (yes, most of them were fine with changing the nick). My biggest gripe is that since YT changed their username system in October of that year, my most popular channel is stuck on an old username despite being a few months older.
Aside from the authors library, django-registration, what other similar libraries for Python and other languages have taken all or some of this into consideration?
Excellent read by the way. Many things I have never considered or even worried about before.
Excellent read by the way. Many things I have never considered or even worried about before.
> 3. Public identity, suitable for displaying to other users
Many sites-- like HN-- may not even need that. If you have system and login identity you can just display "dingus" as the name of every single user and the system should still work the same.
Many sites-- like HN-- may not even need that. If you have system and login identity you can just display "dingus" as the name of every single user and the system should still work the same.
I don't follow the initial premise.
>Well, it’s easy until we start thinking about case. If you’re registered as john_doe, what happens if I register as JOHN_DOE? It’s a different username, but could I cause people to think I’m you? Could I get people to accept friend requests or share sensitive information with me because they don’t realize case matters to a computer?
Just this month we fixed this issue by using a citext column in postgres. So yes, it is easy. Maybe I'm missing an edge case here?
https://www.postgresql.org/docs/9.1/static/citext.html
>Well, it’s easy until we start thinking about case. If you’re registered as john_doe, what happens if I register as JOHN_DOE? It’s a different username, but could I cause people to think I’m you? Could I get people to accept friend requests or share sensitive information with me because they don’t realize case matters to a computer?
Just this month we fixed this issue by using a citext column in postgres. So yes, it is easy. Maybe I'm missing an edge case here?
https://www.postgresql.org/docs/9.1/static/citext.html
The entire concept of usernames that are unique and permanent is stupid and even "cruel". The reality is that a relatively small handful of privileged early adopters get good usernames that match their identities, and everyone else gets screwed. These identifiers then act like tatoos that you got a long time ago and are stuck with for the rest of your life: people end up reminded every day of a sport they can no longer play due to an injury ("hockeystar") or loves lost ("iheartjessie"), attached to a joke that is no longer funny or to a thought that they found adorable as a 13 year old (when you are legally asked to "choose a username": a modern era coming of age scenario) but which adults find inane, or to a nickname that means something different than you realized to some people and now can't change.
The reality is that there are almost ten billion people on this planet and they live for upwards of a century. You are simply deluding yourself if you think it is reasonable to build a system with unique, permanent usernames. Nothing in the real world works like that, including trademarks. And it just helps enforce the very problem that people try to trust usernames and then get tricked by people who sniped usernames that are tied to other peoples' well-known identities (leading to abused "verified" badge systems and legal challenges and expensive hostage scenarios... it just sucks).
And for what? To make it easier to hand-type a URL? Does anyone even do that? I am super technical and I barely even do that in 2018, as if nothing else there are too many websites in existence to remember all of their one-off URL schemes. Like almost everyone, I either use the site's built-in search feature or I do a search on Google to find people, and let a combination of page rank and personalized results guide me to the right destination. Some web browsers don't even show URLs anymore!
Here is a great example of where it is completely insane: Facebook. There is absolutely no good reason for that website to have usernames for regular users, and they frankly shouldn't have usernames for businesses either. It isn't even clear to me that the app--which most users are using, not the website--even has a way to show people's usernames, which means this is an identifier which somehow everyone knows must be chosen and must be unique and is nigh-unto permanent but which somehow is also simultaneously meaningless but is also a horrible point of contention? What?
I am lucky. I spent a bunch of time in 1994 to select a username, and despite being 13, I was mature enough to come up with something that wouldn't ever come to cause me complex problems. People ask me what it means, and it essentially doesn't mean anything: it has only a positive connotation to me when I hear it, it is entirely neutral, and it had no existing usage I could find. Yet, I also still got screwed, as I am semi-famous, and everyone knows me as this username. I have kids who look up to me enough to want to take my name as a show of support and I have to essentially be the big bad asshole about it because in a world of unique and permanent usernames, people then assume the kid is really me. On the other side, I have been asked to rename myself by moderators of various forums as they couldn't believe the real saurik got an account on their site, and it was "confusing" people.
And so in the end we all have to deal with the worst-case scenario anyway: unless you do nothing but sign up for random sites rumored to be interesting constantly (which I seriously tried to do), you eventually will succumb to needing a way to prove who you are on multiple sites and tie together those identifies. And for most users... as in virtually all "normal users", that moment comes when they are using only two websites, as their username was probably something like jay.freeman.178 as everything that was even remotely interesting to them was taken a decade earlier by literally a different generation of humans, so they let the website automatically generate one.
In a world where everyone is having to solve the worst-case problem anyway, every site should just have numbers as unique identifiers, at most have some kind of trust score for degrees of separation on the site (so you can get a feeling for "is this the saurik that I met?"), and everyone should be trained "names don't matter and if you see someone with that name it doesn't even slightly mean that they are the same person you met last week".
The reality is that there are almost ten billion people on this planet and they live for upwards of a century. You are simply deluding yourself if you think it is reasonable to build a system with unique, permanent usernames. Nothing in the real world works like that, including trademarks. And it just helps enforce the very problem that people try to trust usernames and then get tricked by people who sniped usernames that are tied to other peoples' well-known identities (leading to abused "verified" badge systems and legal challenges and expensive hostage scenarios... it just sucks).
And for what? To make it easier to hand-type a URL? Does anyone even do that? I am super technical and I barely even do that in 2018, as if nothing else there are too many websites in existence to remember all of their one-off URL schemes. Like almost everyone, I either use the site's built-in search feature or I do a search on Google to find people, and let a combination of page rank and personalized results guide me to the right destination. Some web browsers don't even show URLs anymore!
Here is a great example of where it is completely insane: Facebook. There is absolutely no good reason for that website to have usernames for regular users, and they frankly shouldn't have usernames for businesses either. It isn't even clear to me that the app--which most users are using, not the website--even has a way to show people's usernames, which means this is an identifier which somehow everyone knows must be chosen and must be unique and is nigh-unto permanent but which somehow is also simultaneously meaningless but is also a horrible point of contention? What?
I am lucky. I spent a bunch of time in 1994 to select a username, and despite being 13, I was mature enough to come up with something that wouldn't ever come to cause me complex problems. People ask me what it means, and it essentially doesn't mean anything: it has only a positive connotation to me when I hear it, it is entirely neutral, and it had no existing usage I could find. Yet, I also still got screwed, as I am semi-famous, and everyone knows me as this username. I have kids who look up to me enough to want to take my name as a show of support and I have to essentially be the big bad asshole about it because in a world of unique and permanent usernames, people then assume the kid is really me. On the other side, I have been asked to rename myself by moderators of various forums as they couldn't believe the real saurik got an account on their site, and it was "confusing" people.
And so in the end we all have to deal with the worst-case scenario anyway: unless you do nothing but sign up for random sites rumored to be interesting constantly (which I seriously tried to do), you eventually will succumb to needing a way to prove who you are on multiple sites and tie together those identifies. And for most users... as in virtually all "normal users", that moment comes when they are using only two websites, as their username was probably something like jay.freeman.178 as everything that was even remotely interesting to them was taken a decade earlier by literally a different generation of humans, so they let the website automatically generate one.
In a world where everyone is having to solve the worst-case problem anyway, every site should just have numbers as unique identifiers, at most have some kind of trust score for degrees of separation on the site (so you can get a feeling for "is this the saurik that I met?"), and everyone should be trained "names don't matter and if you see someone with that name it doesn't even slightly mean that they are the same person you met last week".
One of the reasons I made my own bad username lookup. https://github.com/flurdy/bad_usernames
Its a simple json file of usernames to disallow.
It does not address many of the other things higlighted in this post but it is a start, at least for my services.
It does not address many of the other things higlighted in this post but it is a start, at least for my services.
> So if you’re enforcing unique email addresses, or using email addresses as a user identifier, you need to be aware of this and you probably need to strip all dot characters from the local-part, along with + and any text after it, before doing your uniqueness check. Currently django-registration doesn’t do this, but I have plans to add it in the 3.x series.
This is needlessly user-hostile. If users wish to use mailbox extensions to have multiple unique accounts, that's their right. They can always get multiple different email accounts, after all.
He doesn't mention the one thing he ought to do, which is to strip email addresses of comments before checking them: (foo)[email protected], jdoe(bar)@example.com, [email protected], jdoe@(home)example.com & (a (nested (comment)))jdoe(more)@example.com(all done) are all the same email address.
This is needlessly user-hostile. If users wish to use mailbox extensions to have multiple unique accounts, that's their right. They can always get multiple different email accounts, after all.
He doesn't mention the one thing he ought to do, which is to strip email addresses of comments before checking them: (foo)[email protected], jdoe(bar)@example.com, [email protected], jdoe@(home)example.com & (a (nested (comment)))jdoe(more)@example.com(all done) are all the same email address.
I'm surprised nobody has mentioned PRECIS - the framework for Preparation, Enforcement, and Comparison of Internationalized Strings in Application Protocols[0].
It defines a (small) set of profiles to validate and compare various types of string, including "Username" (in both case folded and case prepared variants) and "Nickname".
Want to compare two usernames for equality? Run the two strings through the comparison steps for the UsernameCaseMapped[1] profile.
It won't solve all of your problems, but it's a good place to start.
[0] https://tools.ietf.org/html/rfc8264
[1] https://tools.ietf.org/html/rfc8265#section-3.3
It defines a (small) set of profiles to validate and compare various types of string, including "Username" (in both case folded and case prepared variants) and "Nickname".
Want to compare two usernames for equality? Run the two strings through the comparison steps for the UsernameCaseMapped[1] profile.
It won't solve all of your problems, but it's a good place to start.
[0] https://tools.ietf.org/html/rfc8264
[1] https://tools.ietf.org/html/rfc8265#section-3.3
For PHP, see the Spoofchecker class for similar functionality to the Python class discussed in the article.
http://php.net/manual/en/class.spoofchecker.php
http://php.net/manual/en/class.spoofchecker.php
> What we really want in terms of identifying users is some combination of:
> System-level identifier, suitable for use as a target of foreign keys in our database
> Login identifier, suitable for use in performing a credential check
> Public identity, suitable for displaying to other users
Some sites want a fourth one:
Public Identity, suitable for other users to use to refer to each other.
Like on Twitter: "Discussed this with @bob and @jane yesterday, you'll find ..."
Now, you don't need a unique username to be able to meet this requirement - StackOverflow is an example of a site that handles this I think? But having a unique username is a common pattern that many sites use to solve this so it seems worth mentioning.
> System-level identifier, suitable for use as a target of foreign keys in our database
> Login identifier, suitable for use in performing a credential check
> Public identity, suitable for displaying to other users
Some sites want a fourth one:
Public Identity, suitable for other users to use to refer to each other.
Like on Twitter: "Discussed this with @bob and @jane yesterday, you'll find ..."
Now, you don't need a unique username to be able to meet this requirement - StackOverflow is an example of a site that handles this I think? But having a unique username is a common pattern that many sites use to solve this so it seems worth mentioning.
With all the unicode problems I'm surprised so few languages include ICU which has everything unicode-related. Things mentioned in the article:
- Normalization: http://userguide.icu-project.org/transforms/normalization
- Confusables: http://icu-project.org/apiref/icu4j/com/ibm/icu/text/SpoofCh...
(and there are so many more things)
- Normalization: http://userguide.icu-project.org/transforms/normalization
- Confusables: http://icu-project.org/apiref/icu4j/com/ibm/icu/text/SpoofCh...
(and there are so many more things)
> Django’s auth system doesn’t enforce case-insensitive uniqueness of usernames
Their routing (for URLs) is also not case-insensitve. The whole framework by default is case sensitive. Honestly, kind of annoying.
Their routing (for URLs) is also not case-insensitve. The whole framework by default is case sensitive. Honestly, kind of annoying.
It is certainly slower but what i always did even back around 1999-2000 when i first learned web programming, was to simply query the user db to see if any user exists with the requested username before even doing anything else. Also at some point i decided to also store a "broken down" version of the username with symbols removed, Os replaced with zeroes, etc and check against that.
Also i never allowed less than two letters and characters that weren't numbers, latin letters, a space and a few punctuation symbols.
Also i never allowed less than two letters and characters that weren't numbers, latin letters, a space and a few punctuation symbols.
I'm currently working on an service and have put a lot of thought into about seven tenths of what is said in this article.
This is a very good read and one I have bookmarked to share with colleagues.
This is a very good read and one I have bookmarked to share with colleagues.
Sorry what? That seems pretty unneccessary. A third party system to dictate how a third party system handles it local alias system for emails? I can't see any benefit to that.
Whether a mail server handles '+' in a standard way is not guaranteed, and surely it is up to the user how they use that feature if enabled.