Symantec throws up hurdles to independent software developers(codeandweb.com)
codeandweb.com
Symantec throws up hurdles to independent software developers
http://www.codeandweb.com/blog/2012/06/23/how-symantec-ruins-independent-developers
14 comments
Your steps sound nice but in reality it doesn't work out this way (speaking from multiple experiences with your reputation system).
First of all, EVERY piece of desktop software my company delivers to users is signed via a known and trusted authority. We knew that would be important and took steps before ever releasing our first piece of desktop software.
Secondly, the error message that users are presented with SCARES THEM. It's not clear why the software is being blocked, and in most cases the user just abandons the software instead of calling us to let us know there was an issue.
When we finally did discover the issue, it wasn't clear what to do. It took us quite a while to figure out where that "false positive" link was, and we weren't even sure that it was the right place to send it to. Even worse, you claim that they "should" have the programs whitelisted within a few business days. This is patently false and never happens that quickly. It took a month before the executable we submitted was whitelisted and you know what? It didn't help one bit.
Symantec seems to not take into account the fact that the executable will be updated, so by the time our first submission was whitelisted we had published 2 updates adding features and fixing bugs. Those updates were blocked even after the initial executable was whitelisted.
You may not be "out to squish the little guy" but honestly that doesn't mean you haven't done quite a bit of damage with your lack of clear messages to your users about why a piece of software is being blocked, and not allowing someone to easily choose to ignore your suggestion that something might not be safe just because symantec hasn't seen it before. (By the time our first symantec using user installed our software we had an installed base of over one hundred users)
First of all, EVERY piece of desktop software my company delivers to users is signed via a known and trusted authority. We knew that would be important and took steps before ever releasing our first piece of desktop software.
Secondly, the error message that users are presented with SCARES THEM. It's not clear why the software is being blocked, and in most cases the user just abandons the software instead of calling us to let us know there was an issue.
When we finally did discover the issue, it wasn't clear what to do. It took us quite a while to figure out where that "false positive" link was, and we weren't even sure that it was the right place to send it to. Even worse, you claim that they "should" have the programs whitelisted within a few business days. This is patently false and never happens that quickly. It took a month before the executable we submitted was whitelisted and you know what? It didn't help one bit.
Symantec seems to not take into account the fact that the executable will be updated, so by the time our first submission was whitelisted we had published 2 updates adding features and fixing bugs. Those updates were blocked even after the initial executable was whitelisted.
You may not be "out to squish the little guy" but honestly that doesn't mean you haven't done quite a bit of damage with your lack of clear messages to your users about why a piece of software is being blocked, and not allowing someone to easily choose to ignore your suggestion that something might not be safe just because symantec hasn't seen it before. (By the time our first symantec using user installed our software we had an installed base of over one hundred users)
> Secondly, the error message that users are presented with SCARES THEM.
Antivirus customers are they type of users that are scared, they are typically users that don't know what to trust. I feel you, I really do.
> It took us quite a while to figure out where that "false positive" link was.
This was a huge peeve of mine, the form is pretty impossible to find unless you use a search engine.
> When we finally did discover the issue, it wasn't clear what to do. It took us quite a while to figure out where that "false positive" link was, and we weren't even sure that it was the right place to send it to. Even worse, you claim that they "should" have the programs whitelisted within a few business days. This is patently false and never happens that quickly. It took a month before the executable we submitted was whitelisted and you know what? It didn't help one bit.
Normal turn around time is a few days, it shouldn't take a month. Was this around Christmas? Were the files served via https? Are files unique between downloads? Were the files mirrored to different domains? Did the team have actual executables to vet?
I understand your frustration and I am sorry it feels like Symantec is working against you. Please continue to fill out false positive reports, the team takes those seriously. With false positives, it shows the system is flawed and they'll take a deeper look at fixing the fundamental problems, otherwise they think the system is working perfectly.
Antivirus customers are they type of users that are scared, they are typically users that don't know what to trust. I feel you, I really do.
> It took us quite a while to figure out where that "false positive" link was.
This was a huge peeve of mine, the form is pretty impossible to find unless you use a search engine.
> When we finally did discover the issue, it wasn't clear what to do. It took us quite a while to figure out where that "false positive" link was, and we weren't even sure that it was the right place to send it to. Even worse, you claim that they "should" have the programs whitelisted within a few business days. This is patently false and never happens that quickly. It took a month before the executable we submitted was whitelisted and you know what? It didn't help one bit.
Normal turn around time is a few days, it shouldn't take a month. Was this around Christmas? Were the files served via https? Are files unique between downloads? Were the files mirrored to different domains? Did the team have actual executables to vet?
I understand your frustration and I am sorry it feels like Symantec is working against you. Please continue to fill out false positive reports, the team takes those seriously. With false positives, it shows the system is flawed and they'll take a deeper look at fixing the fundamental problems, otherwise they think the system is working perfectly.
We are having the very same experience. We signed our software with a code signing cert from a reputable issuer (DigiCert). We discovered that Norton 360 was automatically quarantining the downloaded installer and most users had no idea how retrieve it from quarantine. We got whitelisted and then released a bug-fix update and voila the our app started getting quarantined again. We are losing potential users and it have been damaging to our company!
Is it actually possible for everyone to sign their executables? Last time I did it, I had to fork over something like $250/yr for a signing key along with providing copies of various documents. This seems a little high for someone who just, say, wants to make free utilities available to the world.
In the case of Firefox, one would think it would be possible for you guys to do something about it on your end, because you're the ones who added this reputation system that's causing users grief. Record known-good SHA or MD5 sums of unsigned apps like Firefox that you know are okay, for example. Or just not default this feature on.
In the case of Firefox, one would think it would be possible for you guys to do something about it on your end, because you're the ones who added this reputation system that's causing users grief. Record known-good SHA or MD5 sums of unsigned apps like Firefox that you know are okay, for example. Or just not default this feature on.
So, I also worked on this team (and am good friends with mjard). At some point there was a big decision made to heavily bias the engine against "unknown" executables. I said it was a bad idea then, and I still think it is now. The only way it can "know" about an executable is via its source, or its signature, or if it is on other users machines. This creates the obvious huge problem for "the little guy" distributing software. They actually think this is no big deal, and when it steps on toes, the distributor can just use their dispute system and eventually they will fix it. And if you don't like it, you can just get your software signed. I was of the opinion that this was bad behavior and unreasonable. They really liked what it did for review scores (surprise, we detect everything!). I lost.
Create a harmless helloworld.exe and put it on a random website. Download and run it. If things haven't changed since I left, it will get flagged as malware.
What I can say is that this has nothing to do with trying to crush the little guy or malice. With some exceptions, there is a general attitude there of not caring, or caring about the wrong things. Hanlon's Razor a little bit.
Create a harmless helloworld.exe and put it on a random website. Download and run it. If things haven't changed since I left, it will get flagged as malware.
What I can say is that this has nothing to do with trying to crush the little guy or malice. With some exceptions, there is a general attitude there of not caring, or caring about the wrong things. Hanlon's Razor a little bit.
> Is it actually possible for everyone to sign their executables? Last time I did it, I had to fork over something like $250/yr for a signing key along with providing copies of various documents. This seems a little high for someone who just, say, wants to make free utilities available to the world.
No, it's not. If you sell programs, then forking over the $250 a year makes sense. If you give away programs, well, is it a loss if a user is scared off? (Serious question)
> In the case of Firefox, one would think it would be possible for you guys to do something about it on your end.
In the end, this is what we did. But it is impossible to do this for everybody.
> because you're the ones who added this reputation system that's causing users grief.
Oh boy. We did keep stats on this. Files that the reputation system scored to be "bad" and were later vetted. All in all, the reputation system works really well. There are some false positives and those do cause grief, but a majority of the time, the system blocks legitimately bad software.
No, it's not. If you sell programs, then forking over the $250 a year makes sense. If you give away programs, well, is it a loss if a user is scared off? (Serious question)
> In the case of Firefox, one would think it would be possible for you guys to do something about it on your end.
In the end, this is what we did. But it is impossible to do this for everybody.
> because you're the ones who added this reputation system that's causing users grief.
Oh boy. We did keep stats on this. Files that the reputation system scored to be "bad" and were later vetted. All in all, the reputation system works really well. There are some false positives and those do cause grief, but a majority of the time, the system blocks legitimately bad software.
It's "easy" to build a program with the same length and md5 code than another program. The md5 hash is useful as a protection against accidental collision, but it is not secure enough as a protection against an attack.
http://en.wikipedia.org/wiki/MD5#Collision_vulnerabilities
The fact, that MD5 is broken has nothing to do with building database of known-good binaries, because anyone would just use some other hash.
And also, current practical attacks on MD5 (and also on MD4) can be used to find two different strings with same digest (which is not so useful in this case), not string with same digest as some other predetermined string. For this to be exploitable, same organization would have to create two binaries, one trustworthy and one not. There is no reason to do that, as such code is mostly deemed trustworthy on the basis of who published it, not what it does.
And also, current practical attacks on MD5 (and also on MD4) can be used to find two different strings with same digest (which is not so useful in this case), not string with same digest as some other predetermined string. For this to be exploitable, same organization would have to create two binaries, one trustworthy and one not. There is no reason to do that, as such code is mostly deemed trustworthy on the basis of who published it, not what it does.
What about the user, how can they run an untrusted executable? I never figured this out when I ran into this problem.
This is the reason I dropped Norton 360, a product I was given for free from work. I'm just using MS security essentials now which is free and less robust but it actually allows me to run programs on my computer, which is a nice feature.
This is the reason I dropped Norton 360, a product I was given for free from work. I'm just using MS security essentials now which is free and less robust but it actually allows me to run programs on my computer, which is a nice feature.
It's complicated because Symantec assumes that it recognizes more malware than the user (which is almost certainly true). But it is possible if you jump through some hoops. You could tell Symantec to ignore everything in a specific folder, like a build folder for code that's mistakenly quarantined. If you want to recover a specific file it's more complicated. http://community.norton.com/t5/Norton-Internet-Security-Nort...
> This goes directly to the team and they should have your programs whitelisted within a few business days.
50% of my users are from USA - Symantec claims that they have a 60% market share in USA - this means that I loose 30% of my income. Day by day until it is fixed. Who is going to compensate for this?
Is there a way to seed things up?
> Second: sign your executables. Well - now I know. But I just leaned it because I got complaints from my users.
> Third: don't change domains. This wiped out your known reputation. Thanks for that information. I switched my internet provider and domain. I also use Amazon Cloud Front for fast software distribution to allow users outside Europe a fast download. I've also added subdomain cdn.codeandweb.com because I thought this might solve the problem.
So you say that it will help 1) To place the downloads on the "old" domain? 2) Sign the executables 3) Move the downloads back to the new domain / cdn?
> Second: sign your executables. Well - now I know. But I just leaned it because I got complaints from my users.
> Third: don't change domains. This wiped out your known reputation. Thanks for that information. I switched my internet provider and domain. I also use Amazon Cloud Front for fast software distribution to allow users outside Europe a fast download. I've also added subdomain cdn.codeandweb.com because I thought this might solve the problem.
So you say that it will help 1) To place the downloads on the "old" domain? 2) Sign the executables 3) Move the downloads back to the new domain / cdn?
> Is there a way to speed things up?
Afraid not, the team will manually vet your executables so it takes some time. Fill out the forms, I wouldn't be surprised if they responded by monday. Make sure you list the possible locations to download your software.
I'd mention on your download page that Norton is currently vetting your software, so a WS.Reputation.1 message can be expected. Users are able to pull the file out of quarantine, if they read the popup message, so it'd help if they are expecting it.
>> Second: sign your executables. Well - now I know. But I just leaned it because I got complaints from my users.
ALL DEVELOPERS SHOULD SIGN THEIR EXECUTABLES / PACKAGES: it's not just for antrivirus products, its a way for your customers to verify they are getting the expected product from you.
> So you say that it will help 1) To place the downloads on the "old" domain? 2) Sign the executables 3) Move the downloads back to the new domain / cdn?
I wouldn't change anything at this point, you have the downloads working as you want, let Symantec fix its reputation for how your system works.
Afraid not, the team will manually vet your executables so it takes some time. Fill out the forms, I wouldn't be surprised if they responded by monday. Make sure you list the possible locations to download your software.
I'd mention on your download page that Norton is currently vetting your software, so a WS.Reputation.1 message can be expected. Users are able to pull the file out of quarantine, if they read the popup message, so it'd help if they are expecting it.
>> Second: sign your executables. Well - now I know. But I just leaned it because I got complaints from my users.
ALL DEVELOPERS SHOULD SIGN THEIR EXECUTABLES / PACKAGES: it's not just for antrivirus products, its a way for your customers to verify they are getting the expected product from you.
> So you say that it will help 1) To place the downloads on the "old" domain? 2) Sign the executables 3) Move the downloads back to the new domain / cdn?
I wouldn't change anything at this point, you have the downloads working as you want, let Symantec fix its reputation for how your system works.
Just to be clear, Mozilla has always signed beta builds, which were never a problem as far as I recall (I do a lot of outreach for Mozilla). We did get false positive complaints from nightly users who also used Symantec software, and we now sign nightlies, in part because of the reputation cred given to valid, signed binaries.
All releases and betas are signed with the Mozilla Corp. cert, there's a separate cert for nightly and Aurora builds, and a third cert for dev builds. I agree that we did have false positives with nightlies in the past, but I don't think that's happened for a while. If I'm mistaken, it'd be great to hear about them (I'll ping my contacts there, as well).
All releases and betas are signed with the Mozilla Corp. cert, there's a separate cert for nightly and Aurora builds, and a third cert for dev builds. I agree that we did have false positives with nightlies in the past, but I don't think that's happened for a while. If I'm mistaken, it'd be great to hear about them (I'll ping my contacts there, as well).
How can a developer check his or her applications against this without needing to run Norton Internet Security or AV.
If this isn't possible, is there a cheap(er) option that I can get my hands on that still includes this reputation check?
If this isn't possible, is there a cheap(er) option that I can get my hands on that still includes this reputation check?
Afraid not, which is pretty annoying. There are free trial versions, if you are willing to install NIS or 360.
It's not just Symantec.
I've had issues with multiple AV companies that pertained to binary-string signatures in my code. The AV companies I've dealth with all seem to have online ticketing systems that allowed for rapid correction of these situations.
A few months ago, I found that a command-line screen-capture tool that I publish was flagged as malware by multiple AV products due to behavioral characteristics.
In ScreenKap, I was experimenting with obfuscation of text-strings used by the code. I removed the obfuscation from the code and resubmitted to VirScan.org. I received a clean bill of health.
Note that I did not formally pursue this with any of the AV companies as the string obfuscation was an experiment and was nothing that needed to remain an integral part of my product. If my assumption is correct ( please note that it is an assumption ), we might be restricted to coding in the way the AV companies think we should code.
I've had issues with multiple AV companies that pertained to binary-string signatures in my code. The AV companies I've dealth with all seem to have online ticketing systems that allowed for rapid correction of these situations.
A few months ago, I found that a command-line screen-capture tool that I publish was flagged as malware by multiple AV products due to behavioral characteristics.
In ScreenKap, I was experimenting with obfuscation of text-strings used by the code. I removed the obfuscation from the code and resubmitted to VirScan.org. I received a clean bill of health.
Note that I did not formally pursue this with any of the AV companies as the string obfuscation was an experiment and was nothing that needed to remain an integral part of my product. If my assumption is correct ( please note that it is an assumption ), we might be restricted to coding in the way the AV companies think we should code.
Norton has caused a large amount of frustration for our GitHub for Windows users - Symantec will basically block any EXE using MSys, because of its use of the CreateRemoteThread API. There is no way I am going to submit all of the 200+ EXEs that comprise MSysGit to that web form, though we will try signing all of the EXEs.
Argh, that's rough, submit a few mention there are more, the team will be able to vet the entire collection.
We have Symantec AntiVir at work.
Few months ago I was researching way to make DLL's behave like OSX/linux - e.g. while they are loaded, they can get replaced. This is doable with the compiler option /SWAPRUN:CD,NET - e.g. if your dll/exe was running from CD or Network, and the media went down, it should still work. This somehow pulls the whole data somewhere (I guess in the page file), and it can be replaced.
Anyway, as soon I as started using this Symantec started reporting virus reports - not for everything - but few were enough for me to stop.
Few months ago I was researching way to make DLL's behave like OSX/linux - e.g. while they are loaded, they can get replaced. This is doable with the compiler option /SWAPRUN:CD,NET - e.g. if your dll/exe was running from CD or Network, and the media went down, it should still work. This somehow pulls the whole data somewhere (I guess in the page file), and it can be replaced.
Anyway, as soon I as started using this Symantec started reporting virus reports - not for everything - but few were enough for me to stop.
Suppose Symantec started a program where companies were allowed to pay for their apps to be white listed would and precluded from this check. Could this be considered a protection racket under anti-trust rules?
No. There's nothing remotely resembling a monopoly here, just a garbage product.
A garbage product that is unfortunately pre-installed on hundreds of millions of windows PCs. Its actually quite difficult to remove all of its components; its not just a one click operation as it should be.
I have had some success with http://www.symantec.com/nrt .
You don't need to be a monopoly to be a racketeer. They're completely different.
Then prosecute them under Rico act. It has real teeth too.
They do have this program, though you are allowed to pay other CAs if you want. Symantec sells certificates (they bought Verisign), which will whitelist you, but it is not required that you get your certificate from them.
Is there even much point in using AV software?
I ask this because I have never installed any on my computer (including on Windows) and I have only ever knowingly been infected once in the last 10 years (I think this happened because I didn't update Windows Media Player and it was still associated with a file type and somehow a rogue media file streamed from a website attacked it).
On the other hand people I know who have things like Norton etc installed seem to have way more problems with their computers than me (including fairly tech savvy people). For example programs randomly breaking, tracking cookies being flagged as "malware" , general slowness of the system , nonsensical warning messages etc. Besides that they still seem to end up infected with malware more often than me and usually re-format their systems once every few months.
On that one occasion that I did end up infected , I had to install 3 different AV programs and do full scans before it was even detected.
Mac and Linux users never bother having AV installed and as far as I am aware there is nothing inherently more secure about either of these systems than there is Windows 7.
If you are running a network , surely it would be simpler just to disallow any executable files apart from those explicitly whitelisted and to make sure security patches are installed?
I ask this because I have never installed any on my computer (including on Windows) and I have only ever knowingly been infected once in the last 10 years (I think this happened because I didn't update Windows Media Player and it was still associated with a file type and somehow a rogue media file streamed from a website attacked it).
On the other hand people I know who have things like Norton etc installed seem to have way more problems with their computers than me (including fairly tech savvy people). For example programs randomly breaking, tracking cookies being flagged as "malware" , general slowness of the system , nonsensical warning messages etc. Besides that they still seem to end up infected with malware more often than me and usually re-format their systems once every few months.
On that one occasion that I did end up infected , I had to install 3 different AV programs and do full scans before it was even detected.
Mac and Linux users never bother having AV installed and as far as I am aware there is nothing inherently more secure about either of these systems than there is Windows 7.
If you are running a network , surely it would be simpler just to disallow any executable files apart from those explicitly whitelisted and to make sure security patches are installed?
In the past, on Windows and Macs (pre-OSX), it was pretty much a requirement unless your machine was really stand-alone and you never dealt with files from any untrusted sources.
These days, if you keep your system patched, use an unprivileged account for your normal activity, use a local firewall and/or NAT, and stay away from shady websites you are probably pretty safe.
I have similar experiences with a friend who's constantly getting malware even though he's running Windows 7 and Microsoft Security Essentials. The main vector seems to be PDF files; he deals with a lot of them via email as part of his job, and he's very much in the habit of just opening PDFs in email before he even really looks at who the sender is.
I agree that many AV programs slow the sytem way down, and in general cause problems, and don't seem to really guarantee that you won't get infected. And FUD is a huge part of how it's marketed. Even Windows itself will nag you with ominous warnings if you don't have any AV software installed.
These days, if you keep your system patched, use an unprivileged account for your normal activity, use a local firewall and/or NAT, and stay away from shady websites you are probably pretty safe.
I have similar experiences with a friend who's constantly getting malware even though he's running Windows 7 and Microsoft Security Essentials. The main vector seems to be PDF files; he deals with a lot of them via email as part of his job, and he's very much in the habit of just opening PDFs in email before he even really looks at who the sender is.
I agree that many AV programs slow the sytem way down, and in general cause problems, and don't seem to really guarantee that you won't get infected. And FUD is a huge part of how it's marketed. Even Windows itself will nag you with ominous warnings if you don't have any AV software installed.
There is possibly a problem with user education here.
Running software on your computer that is not set to automatically pull down and install security patches to me seems like a far bigger problem than not running AV software. Windows does warn you if you turn automatic updates off , but afaik there is rarely such a warning about third party software.
In your friends case it would seem that he is not getting updates from adobe, since viewing a PDF file should not cause third party code execution so he must be getting PDFs that are exploiting his PDF reader (presumably adobe fixes these quickly as they arise).
Running software on your computer that is not set to automatically pull down and install security patches to me seems like a far bigger problem than not running AV software. Windows does warn you if you turn automatic updates off , but afaik there is rarely such a warning about third party software.
In your friends case it would seem that he is not getting updates from adobe, since viewing a PDF file should not cause third party code execution so he must be getting PDFs that are exploiting his PDF reader (presumably adobe fixes these quickly as they arise).
Why isn't this grounds for a product-disparagement lawsuit?
I once worked for a company that ran into this same problem, hence, I have a whole lot of sympathy. However, I also sympathize with Symantec.
The biggest problem with the AV world is that it tends to be reactive. A criminal releases a piece of malware, it infects computers and then there is a fix released. The problem is that there is a gap between release and fix and criminals exploit this gap to steal information.
Reputation analysis is one possible solution. Alas, when it fails, it fails big (and hurts primarily independent developers).
The biggest problem with the AV world is that it tends to be reactive. A criminal releases a piece of malware, it infects computers and then there is a fix released. The problem is that there is a gap between release and fix and criminals exploit this gap to steal information.
Reputation analysis is one possible solution. Alas, when it fails, it fails big (and hurts primarily independent developers).
Hence why I don't install rubbish virus checkers. It also gets
confused on those 4k demos (due to the packers they use probably)
Software that does this should become illegal. This is technical slander.
They are not even trying to explain what this means, the reason for this is simple: they want to show off, how many times they "protected" their customers, so that they are fooled to believe that AV products actually have value in them.
They are not even trying to explain what this means, the reason for this is simple: they want to show off, how many times they "protected" their customers, so that they are fooled to believe that AV products actually have value in them.
[deleted]
Are your executables signed?
Signing doesn't always solve the problem.
Signing should allow reputation from previous versions to apply to updates, if implemented correctly by the antivirus vendor.
The product is out for about 1.5 years now but I moved to a new domain (from texturepacker.com to codeandweb.com). This is when the trouble started.
I have not signed it yet - just learned about the issue some days ago.
Companies pay to get stuff listed. Someone paid to get rid of your product.
First: file a false positive report at https://submit.symantec.com/false_positive/ . (Options: "When downloading a file", "Norton Internet Security 2012 or Norton AntiVirus 2012", "Download Insight")
This goes directly to the team and they should have your programs whitelisted within a few business days.
Second: sign your executables. This goes a long way. And no, it doesn't have to be Verisign.
Third: don't change domains. This wiped out your known reputation. (Would have been acceptable if your binaries were signed)
Symantec is not out to squish the little guy. Sometimes you do have a few more hoops that you are required to hop through. Symantec should have better transparency on how this process works, it's something I pushed for pretty heavily but never had the power to get done.
Don't worry, you're not alone. Example: We weren't able to get Mozilla to sign their beta or developer builds that are shared on multiple mirrors (domains not related to mozilla). We'd get lots of angry (understandably) reports of reputation issues on these builds.
If anybody has any questions within reason, I'll be glad to answer them.