Why input validation could cost you a lot(medium.com)
medium.com
Why input validation could cost you a lot
https://medium.com/because-we-love-software/1b9dcc107a27
17 comments
One thing I did on my website was being quite lenient in what I accepted and then re-formulating what I understood to the user http://i.imgur.com/NiPtDEX.png
I'm still not sure it's a good idea since I've never seen it anywhere else.
This is an amazing idea, thanks.
It is also one of the best ways to check if you are being understood IRL - just ask your interlocutor to explain in his/her words what you just told.
I've seen something similar (well, now I know it's simmilar, it didn't occur to me until I saw your example) - though I assumed it would be unhandy. It was a way to show your webshop cart as a text saying something like: You have bought X, We will send it to (Addres here). It should arrive @ (date). Will cost (some)$. (it didn't look as awkward as my example though).
I've seen something similar (well, now I know it's simmilar, it didn't occur to me until I saw your example) - though I assumed it would be unhandy. It was a way to show your webshop cart as a text saying something like: You have bought X, We will send it to (Addres here). It should arrive @ (date). Will cost (some)$. (it didn't look as awkward as my example though).
frankly I haven't done any research into that (the way to present it or the effectiveness of the idea), I might have found the idea somewhere else, like in the "About Face" book or in one of D.Norman's books, but I don't remember.
I strongly believe in showing the future in users before they click. They are afraid to click and explore, that way they know what to expect.
It's the same with the date field, you can type a lot of stuff in it and it will tell you what it decides to understand: the first letters of "Today", "Yesterday", a number will be understood as the last day with this number(this month or last month), a full date, etc.
I don't think this is a good argument against input validation, though it's a good argument against incorrect input validation, of course.
A few things that seem missing from this critique is the idea that shipping and payment aren't just human mediated processes.
With most ordering systems payments and shipping are directly integrated using external APIs, and if you pass garbage data out to those then you will fall into the trap that professionals call "really fucking everything up", which is undesirable. Processing payments typically requires a name and an address, as does shipping. That's not much of an excuse for not doing proper input validation or accepting non-ascii input but it does help to explain why the problem isn't a trivial one. If users are just sending you cash (or, say, some sort of gift-card code which might as well be digital cash) and you're just hand labeling packages and taking them to the local post office that's one thing, but that's not the way even very tiny businesses work today.
A few things that seem missing from this critique is the idea that shipping and payment aren't just human mediated processes.
With most ordering systems payments and shipping are directly integrated using external APIs, and if you pass garbage data out to those then you will fall into the trap that professionals call "really fucking everything up", which is undesirable. Processing payments typically requires a name and an address, as does shipping. That's not much of an excuse for not doing proper input validation or accepting non-ascii input but it does help to explain why the problem isn't a trivial one. If users are just sending you cash (or, say, some sort of gift-card code which might as well be digital cash) and you're just hand labeling packages and taking them to the local post office that's one thing, but that's not the way even very tiny businesses work today.
Agreed, especially when shipping, and invalid address can be expensive. We were charged $$ for each address which was "invalid"...even as simple as missing a "NE" on the street. An address lookup service was put in place, but it was done non-obtrusively. The user was prompted if the address was completely invalid, but they were allowed to ignore and continue anyway instead of stopping the order.
Bad-validation certainly doesn't justify no-validation.
as long as one does not prevent the user from ordering, the process can be ok but still it is not working in so many cases.
Really, what cases are these? We found the address lookups to be very accurate, and saved money.
Last week I tried to fund something over KickStarter. The form wouldn't accept my card's expiry date. The expiry month field went red. No explanation about why, it just went red and I couldn't submit.
The expiry month is this month, but it expires at the end of the month, not the start. Obviously an OBOE, right? So I wrote to them and explained.
They wrote back that the problem was that my card would have expired by the end of the campaign (I'm still waiting for a new one because of the postal service).
Form validation is normally syntactic rather than semantic, but KickStarter mixed their validation types to something I didn't expect and then didn't tell me why when it went wrong.
It's either surprisingly easy to get wrong or surprising that KickStarter got it wrong. I'll be generous and let you choose which.
The expiry month is this month, but it expires at the end of the month, not the start. Obviously an OBOE, right? So I wrote to them and explained.
They wrote back that the problem was that my card would have expired by the end of the campaign (I'm still waiting for a new one because of the postal service).
Form validation is normally syntactic rather than semantic, but KickStarter mixed their validation types to something I didn't expect and then didn't tell me why when it went wrong.
It's either surprisingly easy to get wrong or surprising that KickStarter got it wrong. I'll be generous and let you choose which.
I've seen the last month of expiry happen before. Although annoying it might be a security for the seller. If you enter your card number on Sep 29th and for some reason the bank only transfers the money on Oct 1st, it might not work?
I don't care what the business rules are. I just want them to explain why my card isn't being accepted.
Nope, you should still validate your input. The correct thing to do is the validate that the input is semantically valid rather than syntactically valid. If they enter an address, try looking it up. If they enter an email, try sending a confirmation email. If they enter a phone number at least see if you can find it in a phone number DB.
Customers and users do have an interest in getting their shipping details correct. Help them get it right rather than telling them they're wrong.
Customers and users do have an interest in getting their shipping details correct. Help them get it right rather than telling them they're wrong.
Nope, you should still validate your input. The correct thing to do is the validate that the input is semantically valid rather than syntactically valid. If they enter an address, try looking it up.
We rented a newly constructed house. The old houses in that area were demolished, but the same street names were kept. It was very annoying to have web forms doing such validation telling me that the address was invalid (you cannot have that house number in that street).
Another pet-peeve: I have an ë in my name. When I purchase some software through a web form that accepts that character, please don't just remove it or replace it with garbage in my license key ;).
We rented a newly constructed house. The old houses in that area were demolished, but the same street names were kept. It was very annoying to have web forms doing such validation telling me that the address was invalid (you cannot have that house number in that street).
Another pet-peeve: I have an ë in my name. When I purchase some software through a web form that accepts that character, please don't just remove it or replace it with garbage in my license key ;).
I had the same problem getting internet to my rented office space. The ISP didn't have an updated db with the building's new office numbers.
I had to call and talk to someone put in a work-around. That meant I lost the discount for doing it over the web. It also meant that it took months and several phone calls for the bill to start coming to the right place.
Perhaps the message should be- "please double-check, we can't find a record of that <> in our database." But accept it if the customer insists.
I had to call and talk to someone put in a work-around. That meant I lost the discount for doing it over the web. It also meant that it took months and several phone calls for the bill to start coming to the right place.
Perhaps the message should be- "please double-check, we can't find a record of that <> in our database." But accept it if the customer insists.
this looks a bit different from the other side.
When I worked at in a big retail shop we had to have forms (ie. zip/city) validation.
There was a time when validation was as simple as proposed in the article (checking for presence of data, not the meaning).
You wouldn't believe the things people are able to type into forms.
One of the most common errors was a bad combination of zip/city (ie. zip code for a different country area) which required a live person on the phone asking customer about proper data - it costed money.
The most unusual one was when a person managed to type gibberish in every field avaliable and than add to it a proper adress, name, zip etc... all in a three or four letter wide house-number box.
VALIDATE ALL THE THINGS!
One of the most common errors was a bad combination of zip/city (ie. zip code for a different country area) which required a live person on the phone asking customer about proper data - it costed money.
The most unusual one was when a person managed to type gibberish in every field avaliable and than add to it a proper adress, name, zip etc... all in a three or four letter wide house-number box.
VALIDATE ALL THE THINGS!
> a three or four letter wide house-number box
That sounds problematic to begin with. I grew up in a house with a 5-digit street number.
That sounds problematic to begin with. I grew up in a house with a 5-digit street number.
Well exactly this is the total wrong approach. This does not work at all. For example you will find myself not in address nook neither is my mobile phone number in any public phone book. A customer knows exactly how his address is written and furthermore mail services are most of the time able to figure out where the delivery belongs to
If the address is not clear. As mentioned by the other comments above, there are so many exceptions. This is a pretty good article about zip code: http://www.mjt.me.uk/posts/falsehoods-programmers-believe-ab...
There are databases which allow you to lookup addresses simply to find out if they are valid, not to reverse lookup the name. Your address is in there. Similarly, there are databases which allow you to lookup whether a phone number exists but not the name. Your phone number is in there, too.
Not being in the phone book or having a "private" number has nothing to do with your address and phone number not being in various lookup databases.
Not being in the phone book or having a "private" number has nothing to do with your address and phone number not being in various lookup databases.
Where are those databases?
There are lots of providers that do phone verification on unlisted numbers. Here's a recent release from a quick google search: http://cardnotpresent.com/news/cnp-news-may13/TNS_Leverages_...
Tons of places do address verification and you don't need to be in any phone book for it to work. Even the USPS does address verification both paid (for businesses) and free (for non-profits).
It's pretty funny to get downvoted to zero on a true post just because someone really wishes it were true that their phone number and address can't be verified.
Tons of places do address verification and you don't need to be in any phone book for it to work. Even the USPS does address verification both paid (for businesses) and free (for non-profits).
It's pretty funny to get downvoted to zero on a true post just because someone really wishes it were true that their phone number and address can't be verified.
Your suggestions are nice, but only if not finding the phone number in any database, or not being able to find the address when looking it up shows nothing more than a notice saying exactly that.
That should be the behaviour of all such forms, even when some seemingly important information is missing.
I can't count the amount of forms in which I had to come up with some validation-passing-yet-not-really-valid postcode so I could get my order shipped to my place with no postcode, while not giving a postcode that would get the postal service to ship to the wrong place (say, 00000 or 99999 don't work, but 10000 would indicate that I want the package shipped to Troyes, when I really want it to some overseas territory - so sometimes 11111 validates and is clearly enough not a real postcode, etc).
Also... I routinely receive packages from the US with the address printed with missing or mangled accents, sometimes just for my name, sometimes just for my city, sometimes both... come on, Unicode isn't some newfangled technology anymore, you should be able to get accents right! And these accents aren't even outside the latin1 charset.
That should be the behaviour of all such forms, even when some seemingly important information is missing.
I can't count the amount of forms in which I had to come up with some validation-passing-yet-not-really-valid postcode so I could get my order shipped to my place with no postcode, while not giving a postcode that would get the postal service to ship to the wrong place (say, 00000 or 99999 don't work, but 10000 would indicate that I want the package shipped to Troyes, when I really want it to some overseas territory - so sometimes 11111 validates and is clearly enough not a real postcode, etc).
Also... I routinely receive packages from the US with the address printed with missing or mangled accents, sometimes just for my name, sometimes just for my city, sometimes both... come on, Unicode isn't some newfangled technology anymore, you should be able to get accents right! And these accents aren't even outside the latin1 charset.
Trust but verify. The back end charges could end up eating all your profit too.
http://www.ups.com/content/br/en/shipping/cost/additional.ht...
http://www.fedex.com/us/2013rates/surcharges-and-fees.html
http://www.ups.com/content/br/en/shipping/cost/additional.ht...
http://www.fedex.com/us/2013rates/surcharges-and-fees.html
I think this exposes a need for a service that does just that: localized input validation. It knows the right ZIP code for the right country, it supports every kind of name imaginable and lifts another weight from the shoulders of thousands of developers.
We use libphonenumber from Google to do just that. Have to stay on top of the latest library version, though, to avoid pissing off customers when their country's phone format changes, which happens surprisingly often. I subscribed to the ITU-T newsletter for a while, and there are phone number format changes somewhere in the world every month. Kind of amazing.
But for what advantage? What are the chances that I type my mail, phone and address completly wrong so that the sender can't contact me at all? I don't see any advantages in that validation.
Sometimes these arbitrary pseudo-technical restrictions are just infuriating. Once I came across an online store that wouldn't let you ship to one country and pay with credit card of another one (say, you try to order something to the US and pay with Canadian credit card), like no one ever moves between countries. Or Elance.com won't check your address in Belarus because they don't 'support' Belarus (I guess it means they didn't want to find anyone who can read Belarussian), preventing anyone from Belarus from using their service.
It's great the author brought this problem to everyone's attention.
It's great the author brought this problem to everyone's attention.
This craziness with CC is the norm rather than the exception whenever the company has international breadth.
It sadly due to international treaties, weird taxation and bureaucracy rather than coding.
three examples:
Apple store
Amazon
Playstation Store
I agree with you that this is ludicrous, but sadly there's not much you can do. (I keep my french bank account on only to keep my ps games.)
I agree with you that this is ludicrous, but sadly there's not much you can do. (I keep my french bank account on only to keep my ps games.)
Is it really a norm? Most of online stores don't attempt to 'validate' your CC number in this way. Entire reshipping businesses are built on this premise.
for my experience with ps store and apple store:
ITA CC FR Account: No ps store, no apple;
FR CC US Account: No ps store, no apple;
US CC FR Account: No ps store;
other combinations not tested.
I remember having issues with amazon, but not for the CC, but for shipping outside the country.
for what i saw, the issue with foreigners is very bad in france and us. in france your country is usually "constant" and even the gov.fr does not allow you to specify a new residence outside france. US are even worst. someone with no SSN almost does not exist. Or the eventuality of foreigners is not even considered possible. It is sad that then we go around bragging about "globalization" when it is impossible to keep something like the credit score if you move country.
Things to watch out for when validating people's names: http://www.kalzumeus.com/2010/06/17/falsehoods-programmers-b...
Even following a spec for email validation could be trouble: https://github.com/swiftmailer/swiftmailer/issues/71
Even following a spec for email validation could be trouble: https://github.com/swiftmailer/swiftmailer/issues/71
One place I've seen a lot of web developers really screw up on this when they try to write something for mobile, they make tons of fields required that they really don't need. Meanwhile people really hate typing and entering stuff on mobile and it is really tough on the users. So much so that many just ditch. I guess novice developers just see they can mark things required in some library trivially, so they do so without thinking through the consequences.
Well, while I am all for pushing back on bad design and I'm all for showing folks how many people abandon their forms, it's probably not so much that novice folks think it's okay to mark everything required because it's easy to do with a library... what is easy for novice folks to do is mark everything required because your boss/director/client demands that it be done.
I hope web developers/designers learn this lesson -- quickly. Perhaps we need to re-think how we do validation. For example, SmartyStreets is trying to make the validation of street addresses painless, allowing you to type it basically however you want: http://smartystreets.com/kb/liveaddress-api/website-forms
There's nothing inherently wrong with input validation. The real problem here is twofold: Validation rules that make no sense, and what action is being taken by the system when input fails validation.
Instead of just rejecting the form submission outright, why not just ask the user for explicit confirmation that their input is not mistaken?
Instead of just rejecting the form submission outright, why not just ask the user for explicit confirmation that their input is not mistaken?
The example the author gives is of poorly executed form validation. His solution is to to tear down the firehouse because a firetruck hit a single pedestrian -- "Always try to avoid input validation wherever it is possible and not required."
If you have received a large sample from, say, 1,000,000 users a certain % of those entries will be invalid. The number of users is too large to manually fix or attempt to contact the user to fix the entry. If the contact information is invalid, you can not contact the user. If the contact information is invalid but you think you may be able to correct it, e.g., add a .com to the end of "user@gmail", you may not even legally have permission to contact that person.
The goal is to reduce the invalid % to as low as possible. There is no 100% perfect rate. You are going to sacrifice 1 user for 10.
If you have received a large sample from, say, 1,000,000 users a certain % of those entries will be invalid. The number of users is too large to manually fix or attempt to contact the user to fix the entry. If the contact information is invalid, you can not contact the user. If the contact information is invalid but you think you may be able to correct it, e.g., add a .com to the end of "user@gmail", you may not even legally have permission to contact that person.
The goal is to reduce the invalid % to as low as possible. There is no 100% perfect rate. You are going to sacrifice 1 user for 10.
I like what Amazon is doing in that regard. When entering an address they try to verify it and offer you a looked-up address, asking you whether that's probably better.
this is exactly why Ive started arguing with my clients for using a freeform text field for the address. Why have separate fields for state and city and street? I dont ever need that information separately. And the user will have a far better idea of how to format their address than I will. So let them type the whole thing into one textbox and I never have to worry about it.
Germany has a five-digit postcode system yes, but it doesn't start at 10000.
I don't understand why it complains for ö but not é or ü. o.O
Error: you have an accent in your name