Japanese peer-to-peer (2008)(jmoiron.net)
jmoiron.net
Japanese peer-to-peer (2008)
http://jmoiron.net/blog/japanese-peer-peer/
8 comments
What I found fascinating about this is not only technological aspect of it, but the social aspect of it.
Apparently, there's some sort of hierarchical social structure, especially prevalent in anime uploads; basically people with higher "caste" would upload more popular anime.
I have heard of stories where those who are not complying these systems, or people trying to get out of the system (especially higher ranked ones) are shunned, either by giving anonymous tips to police, etc. (It has been rumored sporadic arrests being made by police, despite larger user base, are because of this.)
Apparently, there's some sort of hierarchical social structure, especially prevalent in anime uploads; basically people with higher "caste" would upload more popular anime.
I have heard of stories where those who are not complying these systems, or people trying to get out of the system (especially higher ranked ones) are shunned, either by giving anonymous tips to police, etc. (It has been rumored sporadic arrests being made by police, despite larger user base, are because of this.)
Pirate Anime Yakuza?
Sounds like an actual anime show.
Almost like that!
Maybe worse... These people can get pretty nasty. After all, you are dealing with people who would try to locate you from a set of pictures. (Some people's obsession goes so far locating someone's residence from a set of pictures, examining time it was taken, and angle of shadow, etc.)
Maybe worse... These people can get pretty nasty. After all, you are dealing with people who would try to locate you from a set of pictures. (Some people's obsession goes so far locating someone's residence from a set of pictures, examining time it was taken, and angle of shadow, etc.)
Pretty sure this article is totally wrong on all it's assumptions on security being broken.
All arrest to my knowledge were from people posting hashes in a public way.
Statements like this are definitely wrong, this is NSA style stuff, if possible at all. I though there are only a handful (One?) md5 collisions found.
"The relative weakness of md5 has led to hash spoofing attacks on the WinNY network."
All arrest to my knowledge were from people posting hashes in a public way.
Statements like this are definitely wrong, this is NSA style stuff, if possible at all. I though there are only a handful (One?) md5 collisions found.
"The relative weakness of md5 has led to hash spoofing attacks on the WinNY network."
There's this really annoying ip filter that blocks non japanese ip from connecting though....
On the upside, western internet users mostly ignored Usenet in favor of BitTorrent, which made Usenet a real pleasure to download from for years. Copyright holders are starting to catch on, though, so this is going away.
What's with japan and tripcodes?
It's no surprise that Japan has evolved these anonymous networks, considering the legal pressure its media companies applies to its filesharers. Even Perfect Dark's users have been arrested, implying its security has been broken.[1] Maybe we elsewhere can draw some lessons. Reposting a comment of mine:
-----
I actually submitted the Perfect Dark wiki article to HN after reading about Isamu's passing:
http://en.wikipedia.org/wiki/Perfect_Dark_(P2P)
Screenshots: https://www.google.com/search?biw=1097&bih=609&tbm=isch&sa=1...
Basically, you dedicate some space on your hard drive to the program, which then stores chunks of unidentified files there and constantly uploads to other clients - it's true decentralized file storage with a degree of plausible deniability. You can search for files you want, and after you've downloaded them, you can vote on them to extend their lifespan or bury them.
It was a cool setup when I tried it back in '08 or so, giving off a very cyberpunk[2] feel with its anonymity/tripcodes, spartan plain-text-on-black-background UI, and the scrolling chatstream of Japanese comments (often with SJIS text art) posted by other users. In the end I didn't actually use it for anything, but it felt like a futuristic BBS.
It would be interesting to see a global equivalent emerge, maybe built upon bittorent DHTs/magnet links. Maybe one already has, e.g. on Tor?
----
[1] The Japanese network security firm NetAgent announced last year that it was able to decrypt Perfect Dark's security and thus be able to compromise the anonymity of the network. http://www.animenewsnetwork.com/news/2010-06-10/2nd-man-arre...
[2] Considering that PD v1.02 is codenamed "Stand Alone Complex", a reference to the fictional emergent phenomenon of stand-alone individuals spontaneously acting in concert without coordination (coined by the Ghost in the Shell TV anime), its cyberpunk inspirations are quite evident.
-----
I actually submitted the Perfect Dark wiki article to HN after reading about Isamu's passing:
http://en.wikipedia.org/wiki/Perfect_Dark_(P2P)
Screenshots: https://www.google.com/search?biw=1097&bih=609&tbm=isch&sa=1...
Basically, you dedicate some space on your hard drive to the program, which then stores chunks of unidentified files there and constantly uploads to other clients - it's true decentralized file storage with a degree of plausible deniability. You can search for files you want, and after you've downloaded them, you can vote on them to extend their lifespan or bury them.
It was a cool setup when I tried it back in '08 or so, giving off a very cyberpunk[2] feel with its anonymity/tripcodes, spartan plain-text-on-black-background UI, and the scrolling chatstream of Japanese comments (often with SJIS text art) posted by other users. In the end I didn't actually use it for anything, but it felt like a futuristic BBS.
It would be interesting to see a global equivalent emerge, maybe built upon bittorent DHTs/magnet links. Maybe one already has, e.g. on Tor?
----
[1] The Japanese network security firm NetAgent announced last year that it was able to decrypt Perfect Dark's security and thus be able to compromise the anonymity of the network. http://www.animenewsnetwork.com/news/2010-06-10/2nd-man-arre...
[2] Considering that PD v1.02 is codenamed "Stand Alone Complex", a reference to the fictional emergent phenomenon of stand-alone individuals spontaneously acting in concert without coordination (coined by the Ghost in the Shell TV anime), its cyberpunk inspirations are quite evident.
> It would be interesting to see a global equivalent emerge, maybe built upon bittorent DHTs/magnet links. Maybe one already has, e.g. on Tor?
GNUNet [1] is very similar in technology to Perfect Dark (at least from what I can tell based on the description in the linked article). I'd even think that the design of Perfect Dark is based on the same research that GNUNet is based on. See for example [2].
[1] https://gnunet.org/
[2] https://gnunet.org/svn/GNUnet-docs/WWW/download/ecrs.pdf
GNUNet [1] is very similar in technology to Perfect Dark (at least from what I can tell based on the description in the linked article). I'd even think that the design of Perfect Dark is based on the same research that GNUNet is based on. See for example [2].
[1] https://gnunet.org/
[2] https://gnunet.org/svn/GNUnet-docs/WWW/download/ecrs.pdf
BitTorrent over I2P exists (and is popular). A torrent client is bundled out of the box. A custom spec exists for this also[1].
As for Tor.. Tor and BitTorrent do not mix; the bandwidth is not available (BitTorrent chokes Tor relays due to model used between them) as well as the amount of data which can be pushed overall. For the most part (though this would not be relevant to a formal BitTorrent over Tor design), BitTorrent is not anonymous as most intend over Tor[2].
[1] http://www.i2p2.de/bittorrent.html
[2] https://blog.torproject.org/blog/bittorrent-over-tor-isnt-go...
As for Tor.. Tor and BitTorrent do not mix; the bandwidth is not available (BitTorrent chokes Tor relays due to model used between them) as well as the amount of data which can be pushed overall. For the most part (though this would not be relevant to a formal BitTorrent over Tor design), BitTorrent is not anonymous as most intend over Tor[2].
[1] http://www.i2p2.de/bittorrent.html
[2] https://blog.torproject.org/blog/bittorrent-over-tor-isnt-go...
AFAIR Bittorrent transmits IP-addresses in plaintext and thus hurts the privacy of any Tor-node that runs a BitTorrent client over Tor:
http://arstechnica.com/tech-policy/2011/04/not-anonymous-att...
http://arstechnica.com/tech-policy/2011/04/not-anonymous-att...
It does not hurt any nodes used for the connection's anonymity, it does hurt their connectivity and performance. Tor nodes are publicly known and are not the same as Tor users. One of the many problems of BitTorrent or any high bandwidth application over Tor is the lack of capacity in the network (but there are others, as mentioned in my previous comment).
BitTorrent has to provide an IP address; it is part of the protocol. The issue is that some, most clients even if configured to use Tor will leak the true IP address of the user and not use a Tor exit's IP. Not good in any case.
BitTorrent has to provide an IP address; it is part of the protocol. The issue is that some, most clients even if configured to use Tor will leak the true IP address of the user and not use a Tor exit's IP. Not good in any case.
For you I2P users, What are you using for P2P software, I2PSnark, iMule, or I2Phex? I tried it a few years ago, but didn't have much success in finding anything. Any tips for setting up, searching, performance tweaks?
Why don't these P2P apps store the file chunks in your hard drive's free space? The fact that commands like `shred` exist means that you can readily read/write from it, and your OS will automatically overwrite the cache contents when you need to use space (and it doesn't matter because P2P networks are fault tolerant anyway).
That way users don't have to face the idea of "giving up" their disk space to be part of the network.
That way users don't have to face the idea of "giving up" their disk space to be part of the network.
There is no generalized way to access the free space of a mounted and mutable file system.
What you'd be better off doing on Unix type OSes is creating a large empty file, deleting it but keeping the file handle, and reading/writing that. That's a traditional way to make a self erasing scratch file.
It doesn't work on Windows, though, because Windows won't let you delete a file that's open.
What you'd be better off doing on Unix type OSes is creating a large empty file, deleting it but keeping the file handle, and reading/writing that. That's a traditional way to make a self erasing scratch file.
It doesn't work on Windows, though, because Windows won't let you delete a file that's open.
If you do that the space is still occupied. And the file deletes itself whenever the service is restarted, which is also undesirable.
Storage is getting so cheap so quickly that this is a disappearing problem. Also, glossing over the barriers to implementation, your solution to the stated issue makes the overall fidelity of files on the network much less predictable, particularly for those with fewer up-votes. To compensate, the minimum number of copies of each part of each file that the network would have to cache to be reasonably certain of its ability to completely deliver to new downloaders the files riding the minimum up-vote threshold would have to go up. This would in turn push the minimum up-vote threshold up and make it more difficult for obscure content to survive on the network.
Shred doesn't operate on free space at all. It overwrites the file, and assumes that the underlying filesystem will overwrite the data in place.
http://www.gnu.org/software/coreutils/manual/html_node/shred... "Please note that shred relies on a very important assumption: that the file system overwrites data in place. This is the traditional way to do things, but many modern file system designs do not satisfy this assumption. Exceptions include:
Log-structured or journaled file systems, such as those supplied with AIX and Solaris, and JFS, ReiserFS, XFS, Ext3 (in data=journal mode), BFS, NTFS, etc., when they are configured to journal data. File systems that write redundant data and carry on even if some writes fail, such as RAID-based file systems. File systems that make snapshots, such as Network Appliance's NFS server. File systems that cache in temporary locations, such as NFS version 3 clients. Compressed file systems."
http://www.gnu.org/software/coreutils/manual/html_node/shred... "Please note that shred relies on a very important assumption: that the file system overwrites data in place. This is the traditional way to do things, but many modern file system designs do not satisfy this assumption. Exceptions include:
Log-structured or journaled file systems, such as those supplied with AIX and Solaris, and JFS, ReiserFS, XFS, Ext3 (in data=journal mode), BFS, NTFS, etc., when they are configured to journal data. File systems that write redundant data and carry on even if some writes fail, such as RAID-based file systems. File systems that make snapshots, such as Network Appliance's NFS server. File systems that cache in temporary locations, such as NFS version 3 clients. Compressed file systems."
Both Freenet and GNUnet were released years before Perfect Dark, so the name is probably a reference to all three being "darknets". http://en.wikipedia.org/wiki/Darknet_%28file_sharing%29
> Even Perfect Dark's users have been arrested, implying its security has been broken.[1]
What can one expect, with such closed software and flagrant violations of Kerckhoffs's principle?
The real question I'd like to see answered by these articles on Winny and Perfect Dark is why the Japanese are insane enough to use these when they're known to be broken - so they're less secure than alternatives like Freenet and less convenient than things like BitTorrent.
What can one expect, with such closed software and flagrant violations of Kerckhoffs's principle?
The real question I'd like to see answered by these articles on Winny and Perfect Dark is why the Japanese are insane enough to use these when they're known to be broken - so they're less secure than alternatives like Freenet and less convenient than things like BitTorrent.
In my own opinion, there's not much "direct" Japanese content in "western" systems like BitTorrent - my girlfriend's Japanese and constantly asks me for recordings of Japanese day-time TV, which you can't easily get on BitTorrent (there's exactly one tracker for Japanese TV-shows, lots for Anime, I think one tracker offers lesser-known Japanese movies, and that one has a ratio-system). Big western trackers like PirateBay rarely store these TV-shows.
I'm pretty sure that these "Japanese" systems are so successful not because of their technical features, but because they offer Japanese content.
I'm pretty sure that these "Japanese" systems are so successful not because of their technical features, but because they offer Japanese content.
So the solution to easily-used systems like BitTorrent which lack Japanese content... is to not start uploading content there, but to invent entirely new networks without any kind of content whatsoever and start uploading there?
Your explanation makes no sense.
Your explanation makes no sense.
I don't think BitTorrent ever got popular in Japan - this survey [1] says that only 6% of file-sharing users used BitTorrent in 2006. I think these entirely new systems were natural developments out of other, older systems from which everyone then moved, like WinMX and Limewire.
[1] http://www.infoworld.com/d/developer-world/number-file-share...
[1] http://www.infoworld.com/d/developer-world/number-file-share...
> I think these entirely new systems were natural developments out of other, older systems from which everyone then moved, like WinMX and Limewire.
But then why didn't they move to BitTorrent or Freenet? Those are equally natural moves, and superior in some respects. (Look at the system requirements for Perfect Dark, and reflect that that doesn't even buy you security if the firm's claims are true.)
But then why didn't they move to BitTorrent or Freenet? Those are equally natural moves, and superior in some respects. (Look at the system requirements for Perfect Dark, and reflect that that doesn't even buy you security if the firm's claims are true.)
It's the combined effects of
- their unawareness of BitTorrent (the Galapagos effect)
- the unwillingness to/difficulty of using English-language torrent sites
- a lack of Japanese champions of BT
BT simply never took off there, and Freenet hasn't taken off anywhere from what I know. I recall coming across a single BT site that was being used by a small but significant number of Japanese, which was surprising, but everything we heard indicated that WinNY and Share dominated there. Similarly, BT isn't a major player in Korea (which has its own networks), but dominates in China. The Internet is a series of interconnected but independently evolving regions.
Also, people will keep using a risky network if the chance of arrest is low and the benefits substantial enough. Humans are lazy, after all.
- their unawareness of BitTorrent (the Galapagos effect)
- the unwillingness to/difficulty of using English-language torrent sites
- a lack of Japanese champions of BT
BT simply never took off there, and Freenet hasn't taken off anywhere from what I know. I recall coming across a single BT site that was being used by a small but significant number of Japanese, which was surprising, but everything we heard indicated that WinNY and Share dominated there. Similarly, BT isn't a major player in Korea (which has its own networks), but dominates in China. The Internet is a series of interconnected but independently evolving regions.
Also, people will keep using a risky network if the chance of arrest is low and the benefits substantial enough. Humans are lazy, after all.
I think one thing to consider might be a language-gap - not many Japanese speak English, and read English pretty slow, to the point that Japanese Internet is sometimes its own island. BitTorrent started in the English-speaking world and had Japanese support a while later (edit: and there are still few torrenting sites in Japanese), maybe too late and the other new clients had already moved in?
[deleted]
Japan, welcome to http://retroshare.sourceforge.net/
Freenet is a similar global equivalent, but less popular and seemingly much less focused on copyright infringement.
On top of all that, Perfect Dark's author is an anonymous figure known as The Chairman.
It looks like a great app and something I'd use (though the majority of content available likely to be Japanese?).
The only other down-side I can see off the bat is that it runs only on Windows machines.
http://en.wikipedia.org/wiki/Share_(P2P)
Anyone aware of an OSX version?
The only other down-side I can see off the bat is that it runs only on Windows machines.
http://en.wikipedia.org/wiki/Share_(P2P)
Anyone aware of an OSX version?
It seems like protocol is pretty much deciphered (for forensic purpose, more likely.) I never heard any existence of it outside of Windows.
I've once tried to run it on Wine while ago, it looks like it incorporates quite a bit of environment checking (so to prevent someone from modifying network parameters?) and I remember it was having problems running -- not sure if it's still the case with later version of Wine.
I've once tried to run it on Wine while ago, it looks like it incorporates quite a bit of environment checking (so to prevent someone from modifying network parameters?) and I remember it was having problems running -- not sure if it's still the case with later version of Wine.
article is pretty much spot on. we started out on winmx, with huge userlists. the fun thing about winmx it downloaded in order, it didnt grab multiple pieces of random parts of the file.
i could start a download, startup my FXP ftp client and start uploading immediately (would have to cap it so it didnt complete the upload before the download completed).
after a few years winmx started to die out, the familiar reputable uploaders we kept eyes are gone. (probably arrents and some people retiring).
then came winny, it was a little harder to use but then we figured out you can just search the hash for the uploader and started grabbing all their content with ease. it was pretty nice.
used share, didnt like it as much as winny.
winny eventually came to an end, well the uploaders were gone like winmx.
by the time perfect dark came out, we already invested in local japanese residents to cap the streams directly and we no longer needed to use japanese p2p software as a source of some of our caps as we could cap anything in hd.