Z-library were in it for commercial gain (you could access a certain number of books but to get more you had to pay for a subscription). They started out as a fork of library genesis, whose mission has always been strictly non-commercial and about providing free access to everyone without limits.
Hopefully this will encourage more people to go back to the original libgen. I suspect Z-library's popularity was because of a better interface and larger collection, but I think lots of people didn't realize libgen offers all its books/papers for free without limits.
Picture this: 3 million books, at least one CID each (in practice it's often multiple, since the libgen collection uses a chunk size of 256kb). Section 3.1 of the paper talks about content publication - for each CID, a provider record is published on up to 20 different peers. Because the CIDs are derived from a high-quality hash function, they are evenly distributed. So this means that a node with a sufficient number of items ends up connecting to every single node on the network. For 3 million CIDs * 20 publication records, this means sending out 60 million publication records, every 12 hours, i.e. an average of 1388 publication records per second (assuming one CID per file, which is conservative). This is just to announce to the network "hi... just wanted to let you know I still have the same content I did yesterday". And every full replica of libgen is doing this.
Another major flaw derives from the way bitswap works as discussed in Section 3.2, which states "before entering the DHT lookup, the requesting peer asks all peers it is already connected to for the desired CID". I'm not sure if that actually means all the machines the node has any type of connection to, or only those connections over which it is using bitswap. Even so, asking every peer you're connected to, even if it's only the subset for which you have a bitswap connection established, is inefficient.
Compare this to bittorrent:
- First there is a much coarser level of granularity; in the libgen case the torrents contain 1,000 books each, so there's much fewer announcements to the bittorrent DHT and trackers. Of course the tradeoff with this is that you can't look up the identifier of an individual book. Instead if you know the magnet link (which can be constructed from the torrent hash) and the name of the file within that torrent.
- Secondly, a node hosting a large number of torrents (and a large number of active connections) will only send out want lists to the peers that it knows are also hosting that torrent. Peers also exchange have lists, and I think one or both can be represented as a bitfield for efficiency (rather than a list of CIDs/hashes). With bitswap you can end up asking every connected peer, just in case they have it.
On a practical note, hosting the torrents is quite practical with adequate hardware, but even with fairly powerful machines IPFS (at least in it's current main implementation, go-ipfs aka Kubo) really struggles and can bring a machine to its knees, even when hosting only a portion of the full collection. In terms of scalability, bittorrent and IPFS are in completely different leagues. Scalability is the main reason the archive of papers from sci-hub (over 80 million) isn't available via IPFS yet, because it's just not going to be able to handle that at all in its current state.
Having said all this, I should state my knowledge of the protocol and go-ipfs is incomplete, as I've only used it but not done any development work on it or dived deeply into the code. I'm happy to be corrected if I've misunderstood anything mentioned above. Also, bittorrent has more than 20 years of implementation experience and i'm sure with further work IPFS can be made to scale better. I don't have the answers as to how to achieve the granularity you get with IPFS vs bittorrent (which is a major point of difference, and something that sets IPFS apart in a significant way). But it's something that definitely has to be fixed to be truly capable of achieving it's stated goals.
All of library genesis is already available on IPFS (see https://freeread.org/ and https://libgen.fun/dweb.html). Hopefully someone will import this collection into libgen and then these books can be on IPFS too.
Torrents are simpler and more efficient for distribution, but IPFS is better for accessing individual files.
This is nice and all, but there's no valid justification for any of the material in the digital library to be locked behind a paywall in the first place. Most of the research published by ACM was paid for by taxpayers, and authors have to either sign over copyright or grant exclusive publishing rights. To be fair, ACM's fees are far more reasonable than the big publishing companies, and there are open access options available (at a cost to authors).
They opened the whole thing up for unlimited access for a brief period during the pandemic, but decided to walk that back after just three months. If you know where to look, there's a 500gb torrent floating round with the 480k+ papers that were accessible as of June 2020.
It's sad that in this day and age, particularly with the widespread acceptance of open source, most academic publications are still behind a paywall. We shouldn't even be having discussions about "open access"; the "open" part should just be implicit.
The reason why they've stopped temporarily is due to an ongoing court case in India initiated by Elsevier. I'm not sure this is the best article on the case but basically sci-hub agreed not to post any new articles for a period of time (which has been extended) while the case is ongoing:
They did release a bulk issue of 2.7 million articles a few months ago (as part of the torrent collection available from libgen), but nothing new since then.
The academic world is too entrenched in the established order, and while many inside it would agree with you, the incentive structures in place often work against individual action like this because the first priority is usually publishing in high impact-factor journals, open access second.
Despite some laudable efforts, academia as a collective has proven themselves incapable of fighting the publishing cartels. As a result, it's turned out that the only viable pathway to true open access is large-scale civil disobedience on the part of sci-hub and its users.
This is a very interesting case to watch though as it could finally provide a legal basis on which sci-hub can operate, even if only in India.
Correct. Library genesis itself (see https://libgen.rs/ or https://libgen.fun/) is run by volunteers on a strictly non-commercial basis and does not charge for access or accounts.
The gateways are indeed centralized (though there are several different public gateways). But anyone who installs IPFS on their computer can access the content directly. Also some browsers support IPFS natively, e.g. https://brave.com/ipfs-support/
Next up is the scimag collection (also maintained by library genesis), which is a backup of all articles on sci-hub.
The torrents are already widely replicated, but whether or not IPFS is going to be able to scale to the level required for 85M articles is still an unknown.
There's really a need though for more developers to get involved with building tools for more easily searching and working with the collection, ideally with a nice UI and integration with things like crossref. This is a massively valuable data set and it would be great to see what people can come up with. Lots of awesome potential for data mining too.
If it weren't for the legal issues (publishers using copyright law to restrict access to literature they got for free since they never pay authors for their work), there's no shortage of projects that could utilize this data and be enormously beneficial for the scientific community and humanity in general. Unfortunately such work can only be done in the shadows right now, which greatly limits the number of people/institutions likely to do so.
This project is in it's early stages and the documentation has quite some way to go, but the index that's part of the release contains all the necessary information. This tool also contains the code necessary to produce the index files if you have a local copy of the zips.
Each torrent contains 100,000 files, comprised of 100 zip files with 1,000 PDFs each. They are named by DOI. There's a database dump at (http://libgen.rs/dbdumps/) (scimag.sql.gz) which has the id -> DOI mapping and other information. The specific torrent and zip file can be determined based on the id; torrent = id/100000 and zip = id/1000.
It depends on the circumstances and the publisher. In many cases publishers permit authors to host the accepted version of the paper (but not the final version that includes revisions based on reviewer feedback) on their personal/institutional website and to email copies of the paper on an individual basis to people who request them.
It's all PDF files, which have their own compression, so it's unlikely there would be substantial gain from additional compression. Each torrent has 100 zip files, and each zip file has 1000 PDFs, but the files are stored uncompressed within the zips (i.e. using the STORE method).
Yes they're copyrighted - albeit not by the authors who actually wrote them, but by the publishers who require copyright assignment for the privilege of having your work hosted on their website.