I accidentally built a nudity/porn platform(elazzabi.com)
elazzabi.com
I accidentally built a nudity/porn platform
https://elazzabi.com/2020/08/11/the-day-i-accidentally-built-a-nudity-porn-platform/
338 comments
I've been maintaining a community forum for more than a decade. We had some abusive users, so we introduced pre-moderation. Meaning, any new user is on probation for a few posts, and all anonymous posts have to be manually approved by an admin to be posted publicly. This has pretty much completely stopped the visible abuse.
However, for about 10 years, there are bots registering every day, some of them even make realistic accounts with cool and unique username, email, even description. Strangely, the email and username never match... And they make post with html links embedded. They even know to actually select the 'HTML' content type, which is an extra select input. Some bots even make a few innocent posts, before the link spam. But just too few to get past probation. Obviously, the spam posts never get approved, and accounts never get out of pre-moderation queue and yet they're still trying every day... Not intelligent enough to make more than 2 dumb posts.
Similarly, I have some work projects that have user registration with a human on-boarding process, where another person has to add the user to their group for them to share any data, none of which is ever public. So these bots are tirelessly registering, and staying in limbo forever. Thousands of useless accounts.
It boggles the mind how much energy is wasted, but I guess it must my profitable enough.
However, for about 10 years, there are bots registering every day, some of them even make realistic accounts with cool and unique username, email, even description. Strangely, the email and username never match... And they make post with html links embedded. They even know to actually select the 'HTML' content type, which is an extra select input. Some bots even make a few innocent posts, before the link spam. But just too few to get past probation. Obviously, the spam posts never get approved, and accounts never get out of pre-moderation queue and yet they're still trying every day... Not intelligent enough to make more than 2 dumb posts.
Similarly, I have some work projects that have user registration with a human on-boarding process, where another person has to add the user to their group for them to share any data, none of which is ever public. So these bots are tirelessly registering, and staying in limbo forever. Thousands of useless accounts.
It boggles the mind how much energy is wasted, but I guess it must my profitable enough.
"Take a competitor product, remove all features you don’t need, and make it crazy fast."
Seems to me there are hundreds of lifestyle businesses just waiting to happen by following this formula. So many good ideas out there could be made so much better by reducing them to their essentials, but making them elegant and "crazy fast".
Seems to me there are hundreds of lifestyle businesses just waiting to happen by following this formula. So many good ideas out there could be made so much better by reducing them to their essentials, but making them elegant and "crazy fast".
Yup. If you build a communications platform, it will be used for spam. If you build a hosting platform, it will be used for porn. If you build a linking platform, it will be used to spam links for porn.
Anything else requires a constant uphill battle of content filtering and deletion. You could call it censorship, but it's a necessary reality.
Anything else requires a constant uphill battle of content filtering and deletion. You could call it censorship, but it's a necessary reality.
We were running a privacy focussed Chat-App-Network dating platform in 2017 which was accelerated by Facebook[1].
i.e. A network between, Messenger<->Viber<->Telegram<->Line App.
By design no media sharing was allowed(to prevent pornography) and the user profile images were received from the platform itself. But we soon faced unique challenge of people from certain countries using their children picture as profile picture(often just the children), there were people with group photo as profile image and then there were people who using explicit images as profile pictures.
So we integrated Amazon Rekognition to identify children, group and explicit images. Those using explicit images were banned immediately and those with children/group photo image(Face detection not Facial recognition) were asked to change their profile image(Their profile was not shown to anyone until they changed their profile picture with just them). We were processing >200,000 profile images per month as people change their profile images often.
But, as we very well know Amazon Rekognition or for the fact any such ML solution is not 100% accurate, we faced issues with people with darker skin color(Amazon told me that they were working to fix the issue; exactly why this type of half baked tech shouldn't be used for things which can cause harm) and so we had to reduce the confidence levels to such an extent that anything resembling a child would be flagged by the system(False positives are better in this case than false negatives).
[1]https://hitstartup.com/about/#FindDate
i.e. A network between, Messenger<->Viber<->Telegram<->Line App.
By design no media sharing was allowed(to prevent pornography) and the user profile images were received from the platform itself. But we soon faced unique challenge of people from certain countries using their children picture as profile picture(often just the children), there were people with group photo as profile image and then there were people who using explicit images as profile pictures.
So we integrated Amazon Rekognition to identify children, group and explicit images. Those using explicit images were banned immediately and those with children/group photo image(Face detection not Facial recognition) were asked to change their profile image(Their profile was not shown to anyone until they changed their profile picture with just them). We were processing >200,000 profile images per month as people change their profile images often.
But, as we very well know Amazon Rekognition or for the fact any such ML solution is not 100% accurate, we faced issues with people with darker skin color(Amazon told me that they were working to fix the issue; exactly why this type of half baked tech shouldn't be used for things which can cause harm) and so we had to reduce the confidence levels to such an extent that anything resembling a child would be flagged by the system(False positives are better in this case than false negatives).
[1]https://hitstartup.com/about/#FindDate
A market platform I recently worked on allowed users (free sign up) to create multiple wishlists and then send those wishlists to arbitrary email addresses. The user could set a custom title, limited to 100 characters or so.
We soon discovered a similar problem to OPs - bot accounts (mostly @qq.com addresses) were registering by the hundreds per day to create wishlists and then send those wishlists to other @qq.com addresses. They were setting the titles to arbitrary code blocks.
I found it fascinating, if terribly inefficient. Some colleagues and I were speculating on the purpose, perhaps someone experimenting some kind of laundered botnet control path?
We tried all kinds of measures to prevent it but ultimately we blocked all @qq.com accounts and eventually disabled the wishlist feature altogether as it had such little real usage.
We soon discovered a similar problem to OPs - bot accounts (mostly @qq.com addresses) were registering by the hundreds per day to create wishlists and then send those wishlists to other @qq.com addresses. They were setting the titles to arbitrary code blocks.
I found it fascinating, if terribly inefficient. Some colleagues and I were speculating on the purpose, perhaps someone experimenting some kind of laundered botnet control path?
We tried all kinds of measures to prevent it but ultimately we blocked all @qq.com accounts and eventually disabled the wishlist feature altogether as it had such little real usage.
We've had multiple spammer attacks over the years. Our platform allows users to create and publish their own content. Our primary target is education, teachers and students. But naturally it's being abused by spammers. It's been an interesting cat and mouse game to counter them.
- One time, they used our platform to publish links to their streaming websites for the quarter finals of the 2018 Champions League. Suddenly we ended up being first result on Google for "arsenal v barcelona". It was fifteen minutes before the game so you can imagine that we got a lot of traffic. On the one hand it was kind of flattering that the SEO ranking of our domain was so strong. On the other hand, it wasn't great nor beneficial for the platform to be abused like that. As a counter-measure, we decided to block indexing of project pages for 24 hours when they're first made public. The spammers never came back.
- Another time, we got an email from AWS that our SES bounce rate was 15%, and it was rising fast. Being blocked from sending emails by AWS would have been a disaster. It turned out that our invitation system was abused. A creator of a project can invite an external person by email. That person receives an email saying "John Doe invites you to collaborate on 'A nice project about the 2018 Champions League'" with a link to the project. Replace "A nice project about the 2018 Champions League" by a Chinese ad and you've got yourself spammers who are sending thousands of emails a second to a random collection of email addresses. Naturally a lot of these bounce, which caused AWS to warn us. So we had to start verifying the MX validity of invited email addresses and throttle the system to a maximum of 100 emails in a window of 24 hours.
- We still get a lot of spammers publishing obvious spammy projects. One thing that has helped is the Clearbit Risk API. You send them an email address and it comes back with an assessment of how spammy the address is. We use it for certain domains (protonmail.com, yandex.com,...) and it frequently flags someone as a spammer right after signup. They can still use the platform but can't make stuff public, completely defeating the purpose of them being spammy.
I'm sure they'll keep finding creative ways to get around the limitations we put in. The toughest is to find a way to counter them without it hampering the experience for all the other users.
- One time, they used our platform to publish links to their streaming websites for the quarter finals of the 2018 Champions League. Suddenly we ended up being first result on Google for "arsenal v barcelona". It was fifteen minutes before the game so you can imagine that we got a lot of traffic. On the one hand it was kind of flattering that the SEO ranking of our domain was so strong. On the other hand, it wasn't great nor beneficial for the platform to be abused like that. As a counter-measure, we decided to block indexing of project pages for 24 hours when they're first made public. The spammers never came back.
- Another time, we got an email from AWS that our SES bounce rate was 15%, and it was rising fast. Being blocked from sending emails by AWS would have been a disaster. It turned out that our invitation system was abused. A creator of a project can invite an external person by email. That person receives an email saying "John Doe invites you to collaborate on 'A nice project about the 2018 Champions League'" with a link to the project. Replace "A nice project about the 2018 Champions League" by a Chinese ad and you've got yourself spammers who are sending thousands of emails a second to a random collection of email addresses. Naturally a lot of these bounce, which caused AWS to warn us. So we had to start verifying the MX validity of invited email addresses and throttle the system to a maximum of 100 emails in a window of 24 hours.
- We still get a lot of spammers publishing obvious spammy projects. One thing that has helped is the Clearbit Risk API. You send them an email address and it comes back with an assessment of how spammy the address is. We use it for certain domains (protonmail.com, yandex.com,...) and it frequently flags someone as a spammer right after signup. They can still use the platform but can't make stuff public, completely defeating the purpose of them being spammy.
I'm sure they'll keep finding creative ways to get around the limitations we put in. The toughest is to find a way to counter them without it hampering the experience for all the other users.
I later discovered that Instagram banned all mylink.fyi links from the platform. A customer also confirmed to me via email that Snapchat started blocking links. Heh, I’m banned by Instagram and Snapchat!
[...]
If you’re interested in acquiring the domain name, and/or the app, let’s talk.
That domain name must have a negative value now?
That domain name must have a negative value now?
Linktree, the service the author aimed to replicate had similar problems according to Wikipedia [1].
> In 2018, Instagram banned the site due to "spam," although it was lifted and Instagram issued an apology.
> Rumors circulate that Instagram will issue another ban.
[1] https://en.wikipedia.org/wiki/Linktree
> In 2018, Instagram banned the site due to "spam," although it was lifted and Instagram issued an apology.
> Rumors circulate that Instagram will issue another ban.
[1] https://en.wikipedia.org/wiki/Linktree
Paying for bouncers is an understood part of the cost model for a nightclub. If you provide any sort of real or virtual venue that allows unvetted participants, you have to factor in the cost of dealing with bad actors. If you have a virtual venue, you have to factor in the cost of dealing with automated bad actors.
It's just how human nature works. 90% of people are great, but 10% can do a lot of damage.
It's just how human nature works. 90% of people are great, but 10% can do a lot of damage.
Here's an example of a similar problem - that I created for a website I help a friend with.
We had outbound referral links, basically to monitor the number of times that a visitor clicks out of the website. The URL pattern was something like: example.com/out.php?url=outbound.com
The out.php script would just simply (naively) redirect the user to the url specified. We never validated if the outbound link was to an authorized reference.
The result is the same. Eventually spammers figured out the above link and then just started posting their spam using the site's redirect script to any number of social media sites, embedded in email, etc.
What's interesting too is that we would see multiple redirects embedded into a single request. e.g. out.php?url=another.link.service/link=spammer.com
Obviously in hindsight this was stupid, but when we built it (some 10 years ago), the idea seemed pretty sound if not maybe a little naive. The solution would have been to only allow redirect links to authorized outbound sites, which works when those links are relatively static and not open ended.
We had outbound referral links, basically to monitor the number of times that a visitor clicks out of the website. The URL pattern was something like: example.com/out.php?url=outbound.com
The out.php script would just simply (naively) redirect the user to the url specified. We never validated if the outbound link was to an authorized reference.
The result is the same. Eventually spammers figured out the above link and then just started posting their spam using the site's redirect script to any number of social media sites, embedded in email, etc.
What's interesting too is that we would see multiple redirects embedded into a single request. e.g. out.php?url=another.link.service/link=spammer.com
Obviously in hindsight this was stupid, but when we built it (some 10 years ago), the idea seemed pretty sound if not maybe a little naive. The solution would have been to only allow redirect links to authorized outbound sites, which works when those links are relatively static and not open ended.
There is an easier way to accidentally build a porn website:
1) register domain
2) don't renew
Not completely unrelated either, since it seems like OP's domain is now bust too...
What is also interesting is that probably half of what FB or <insert nonexistant competitor> does is moderation of sorts. This is why FB is becoming a commerce/community page website and why they need Instagram for social media.
1) register domain
2) don't renew
Not completely unrelated either, since it seems like OP's domain is now bust too...
What is also interesting is that probably half of what FB or <insert nonexistant competitor> does is moderation of sorts. This is why FB is becoming a commerce/community page website and why they need Instagram for social media.
I run a mediawiki site. I got to play the cat and mouse game as well. For a while I was getting hundreds of spam page creations per day despite implementing as many defenses as I could. I was at least once a day going through and deleting the spam posts using the Smite Spam plugin. It's finally calmed down and I get one or two posts maybe every couple of weeks. I think I may have finally been removed from the site list of whatever "packaged" software the spammers use.
I've come across these kinds of prototype link sharing sites in my battles with SEO spam.
On the face of it SEO spam is simple: the bad guys bung a load of links into your wiki or blog comments in the hope of gaining google rankings for their "SEO clients", but...
Increasingly they started bouncing links all over the place in complex spamdexing webs. Link sharing sites often get thrown in the mix, along with cross-linking other blogs/wikis where they've got their spam to stick. This all makes it easier to evade any filtering by a particular blog/wiki admin, but maybe also makes it harder for google to filter and down-rank the baddies, and finally depending on how complex a spamdexing web is, it offers protection to their clients because it ends up being impossible to see which end websites the spammer is actually aiming to promote (the links tucked away amidst the randomised cross-linked chaos)
But maybe that was a game of a few years ago, and now spam bots are mostly just trying to push porn on social media.
But maybe that was a game of a few years ago, and now spam bots are mostly just trying to push porn on social media.
I wonder how one can know the history of a domain name before buying it. Imagine making a new website and realizing that your domain name is banned from all social networks!
> Other solutions include: requiring credit cards for trial periods, ban all adult content from the platform… But they all require me to put extra effort in the project. And I don’t have time for that.
There is people who spend their entire life trying to build a working business, and those who just walk away from what would possibly be a pretty lucrative business because of lack of time.
Humans, strange and exciting beings, really.
There is people who spend their entire life trying to build a working business, and those who just walk away from what would possibly be a pretty lucrative business because of lack of time.
Humans, strange and exciting beings, really.
Link hiding tools are unfortunately a security threat due to all the bad actors in the world.
Every time I see somebody identify moderation as censorship, I remember my experiences with public wikis, forums, etc. It's always spam and porn. They'll spin up more fake users than you'll ever have real users, every damned time. It's a vicious cycle, as real users won't stick around if you don't filter.
I struggled with it myself with Twicsy, except Twicsy was just a window into Twitter and made it much easier to find the porn and (gag) child porn. I remember the first time I found a network of child porn, it made me sick to my stomach. Tooks me hours to get rid of it, report it to Twitter, and to the authorities. It was a problem for a long time, I felt like I was doing all of Twitter's dirty work that they wouldn't do themselves. I made tools for people to report it easily, and tools to eliminate it en masse. Twitter took way way way way way too long cleaning up their act in this respect. They also often just deleted tweets and not pictures, pictures could linger for months without being deleted from their content servers, and still appear in Google.
IMO the better way to design a product like this to avoid abuse would be to simply force them to sign in using each platform they want to link to. At that point all you're building is a way for people to say, "These 5 social media accounts are all me".
You could allow them to select one of their accounts to source information and a photo for their combined profile. At that point you're not storing anything besides links to social media profile pages.
In effect you get to piggy-back on those sites' abuse mitigation strategies (though of course you're stuck with the lowest common denominator). Your biggest decision at that point is which social media platforms to allow onto your service.
You could allow them to select one of their accounts to source information and a photo for their combined profile. At that point you're not storing anything besides links to social media profile pages.
In effect you get to piggy-back on those sites' abuse mitigation strategies (though of course you're stuck with the lowest common denominator). Your biggest decision at that point is which social media platforms to allow onto your service.
Was it legal "to take a closer look and see the links they are sharing"?
Thinking about this when working on a social network (yes, I know) as a pet project at spare time, I've got a question: are there any tools for handling improper text/images automatically?
I think there are some lists of "bad words", but not sure if they are available for most widely used languages.
But are there libs/SDKs/online services that I can feed a picture and they will tag it as potentially improper, for example porn or some swastikas, so I can pre-moderate manually only such images?
Looks like it could be a nice and useful service.
Ow boy, my first real comment on HN, please be gentle!
Okay, so I have something related to this.
two paragraph back-story was cut...
So I built SASRip[1], an Open Source website with an API that allows you to download audio or video from any web-page that is supported by youtube-dl (I use youtube-dl and ffmpeg to do muxing/transcoding). I also built a browser addon for it called Media Reaper[2] (chromium version available on SASRip's website[3]).
Now, I wanted to build a no BS, no tracking website, so all I have is internal logs, these logs keep incoming requests like so: Time, URL, ID string, success/fail, the ID is just to tell where the request came from, web site, API call or the browser addon, I keep no IP data or anything like that, and boy do people download the nasty stuff, there is all kinds of nasty stuff, taboo stuff, feet stuff and stuff I didn't even know existed.
Now I live in constant fear that someday, looking trough those logs I will find CP and I am not sure what I should do, I know implementing tracking methods goes against both my morals and the philosophy of the service, but at the same time I am not sure if I can keep on going, knowing I could do something about it but I am not.
Ultimately I think it is very likely that I will shut the service down if I find CP on it, with no way of tracking the person down, perhaps leave a message with why I shut down.
----------------------------------------
P.S. I make no money on this service, it's purely donation based.
P.P.S I know I can do muxing and transcoding via some really cool JS libraries, but I wanted to sharpen some of other skills with this project.
----------------------------------------
[1] - https://sasrip.cf/
[2] - https://addons.mozilla.org/en-US/firefox/addon/media-reaper/
[3] - https://sasrip.cf/Home/MediaReaper
Okay, so I have something related to this.
two paragraph back-story was cut...
So I built SASRip[1], an Open Source website with an API that allows you to download audio or video from any web-page that is supported by youtube-dl (I use youtube-dl and ffmpeg to do muxing/transcoding). I also built a browser addon for it called Media Reaper[2] (chromium version available on SASRip's website[3]).
Now, I wanted to build a no BS, no tracking website, so all I have is internal logs, these logs keep incoming requests like so: Time, URL, ID string, success/fail, the ID is just to tell where the request came from, web site, API call or the browser addon, I keep no IP data or anything like that, and boy do people download the nasty stuff, there is all kinds of nasty stuff, taboo stuff, feet stuff and stuff I didn't even know existed.
Now I live in constant fear that someday, looking trough those logs I will find CP and I am not sure what I should do, I know implementing tracking methods goes against both my morals and the philosophy of the service, but at the same time I am not sure if I can keep on going, knowing I could do something about it but I am not.
Ultimately I think it is very likely that I will shut the service down if I find CP on it, with no way of tracking the person down, perhaps leave a message with why I shut down.
----------------------------------------
P.S. I make no money on this service, it's purely donation based.
P.P.S I know I can do muxing and transcoding via some really cool JS libraries, but I wanted to sharpen some of other skills with this project.
----------------------------------------
[1] - https://sasrip.cf/
[2] - https://addons.mozilla.org/en-US/firefox/addon/media-reaper/
[3] - https://sasrip.cf/Home/MediaReaper
I really like the design of the page - very clean and easy to understand the value prop. I could see this being useful for influencers and bloggers. Shame to hear about the mis-use :-(
Everyone just want to be horny on main and I think the internet needs to get over / accept this so we can focus and making that profitable and safe.
I recall in the early days of mobile apps downloading an ipad app which had this neat idea that kids could share their drawings, made in the app, with each other in a sort of random way. It did not take long for me to realize that meant seeing an endless stream of inappropriate, or sometimes potentially harmful content (e.g. from adults interested in exploiting children).
I created a site intended for family photo/video sharing and it did not take long for people to start uploading penis pics. The weird thing is they don't even have a reason or recipient. They just want their penis out there on the internet in the hopes someone might stumble upon it.
I wanted a long time ago to make a free speech forum and platform. As much as I believe it would be good to have place you can say what you want, the hassle of spam and plain nastiness is just what always stopped me from doing this.
I'm building a platform that could be abused in similar ways. Does anyone have any suggested resources I could read/use to avoid this problem, preferably without employing captchas? Akismet might be helpful?
I don't understand what the problem is with using the service to provide links to porn, and also don't understand why Instagram and Snapchat would care about links to lists of links to porn?
- anything that allows anonymous file upload -> childporn + all of the above.
- anything that allows communications -> spam, harassment, bots
- anything that measures something -> destruction of that something (for instance, google, the links between pages)
- any platform where the creator did not think long and hard about how it might be abused -> all of the abuse that wasn't dealt with beforehand.
- anything that isn't secured -> all of the above.
Going through a risk analysis exercise and detecting the abuse potential of whatever you are trying to build prior to launching it can go a long way towards ensuring that doesn't happen. Reacting very swiftly to any 'off label' uses for what you've built and shutting down categorically any form of abuse and you might even keep it alive. React too slow and before you know it your real users are drowned out by the trash.
It's sad, but that's the state of affairs on the web as we have it today.