OpenReview is building software for transparent scientific peer review, advanced review-paper matching systems, and knowledge bases for scientific communities. We are seeking an experienced full stack developer to join the dynamic team of software developers working across universities and nonprofits. We aim to revolutionize scientific peer review working with machine learning researchers, open access advocates, and a broad community of open source developers. Are you excited to build user-facing applications and inspired to develop new tools for research? Join us! Reviewing applications now!
Thanks for your feedback and support! We're trying to push forward the specification work to support other implementations like https://datrs.yoshuawuyts.com/.
We created a protocol working group and spec are being updated at https://www.datprotocol.com/deps/. But definitely still some need to improve docs all around! We have an upcoming grant to support this :).
Ya, we've thought about that. Dat's storage is pretty flexible and we have a content-addressed storage library. A lot of our users do not want or need to storage data in IPFS though so it adds unnecessary complexity to do that by default.
Someone could built a storage that uses IPFS, similar to our dat-http storage [1].
Dat allows you to store any type of data. Yes, I think you could do what you're suggesting (though I don't follow 100%). You can use it like you would Dropbox, except only one person can write at the moment.
Under the hood, there are two implementations: hyperdrive and hypercore. Hyperdrive is a filesystem for storing any kinds of files. Hypercore can store any kind of data and is really great for streaming data (hyperdrive is built on top of it).
> If I uploaded a bunch of data that was obtained illegally there is nothing stopping me from doing that right?
The hosting provider is responsible for removing illegal content. Dat itself doesn't track any content. The datproject.org is more of a registry, not a host.
> Also, is your peer to peer network able to be attacked by nefarious users like a sybil attack? Is there a situation where I could alter or forge data?
No, only authorized people can write to each dat key (currently only the owner, but multi-writer is coming soon). All the writes are signed with the writers private key and then verified whenever content is downloaded.
We have a few approaches to the disappearing data.
First, we are working with libraries, universities, or other groups with large amounts of storage/bandwidth. They'd help provide hosting for datasets used inside their institutes or other essential datasets.
Second, we started to work on at-home data hosting with Project Svalbard[1]. This is kind of a SETI@home idea where people could donate server space at home to help backup "unhealthy" data (data that doesn't have many peers).
Finally, for "published" data (such as data on Zenodo or Dataverse), we can use those sites as a permanent HTTP peer. So if no data is available over p2p sites then you can get it directly from the published source.
As others said, decentralization is an approach but not a solution. It gives you the flexibility to centralize or distribute data as necessary without being tied to a specific service. But we still need to solve the problem!
We'd love to work with them! We really like what they've done and we have a few partners that use Globus.
Underneath all the transfers in Globus use GridFTP. Using Dat could help distribute bandwidth and speed up transfers. It'll also add version control for free, which (I think) Globus does not have yet.
Ya, there are a few other related questions below. Resilio is BitTorrent-based. But I'm not 100% familiar with how Resilio differs from BitTorrent.
The core difference is in our approach. We're all open source and a non-profit. We're also really focused on the research data use case where BitTorrent is less easily deployed.
We hope that making an open and easy to use p2p protocol will enable other developers to build applications on top, and something like Resilio could be one.
We use TCP or UDP for all the connections. UDP is especially helpful for hole-punching as well.
The direct HTTP support is not fully supported yet. But we're excited for that because it'll allow you to use S3 or other static file servers as peers.
Dat works over any protocol, so it's just a matter of implementing it.
Ah cool! I hadn't seen quilt before. You are spot on with the differences.
Dat is targeting similar users to Quilt. But we are also looking more broadly at libraries, labs, or other larger academic/gov't organizations managing data. There are a lot of data publishing tools in the sciences such as Zenodo. We'd love for it to be easier to download/publish data to those places. Dat is decentralized, so it really fits well in integrating other data tools.
You can use Dat to replace file transfer software like rsync, so it is a bit more general purpose.
Another difference not mentioned is that Dat really starts at the protocol level while Quilt is more software-focused. Dat protocol is a peer-to-peer protocol for syncing files, modeled off Git and BitTorrent. We built the data management software on top of the protocol.
Edit: I should mention we don't offer any hosting right now, all the data up there is temporarily cached. There is public hosting via Hashbase[1] from the Beaker team. The cool part about Dat being p2p is that it's really easy to switch hosts or use multiple hosts.
Hi everyone, I'm one of the core contributors to Dat, @joeahand. Happy to answer any questions. It's an interesting time to see this posted because I've been working on a new datproject.org site recently =).
Dat Project started with a focus on increasing access to research & public data. To support the data tools we built a peer-to-peer protocol. People are doing some really cool stuff on top of Dat (such as Beaker Browser), we're really excited about it and want to make sure to support all the neat use cases.
We'll be launching an updated site soon to highlight more of the work around the protocol and what the community is building. Our main use case will still be data management but most of what you see on the current site will shift to a new domain.
Yes, good point! Those are all important as well. I did not intend in my comment to emphasize a binary identity for authors. Seek out books by authors from any lived experience dissimilar from yours to maximize your grasp of our world.
Identity of the author matters because a writer's work is informed by their experience in the world. As a white male, I will view & experience the world differently from Jane Jacobs. Even in a book about facts, a person's relation to those facts reflects their lived experience.
This is especially important in fiction books where you are immersed in the experience of main characters. If the authors identity mirrors yours, then the experience of the main character will not build my understanding of how other people live. At this point, you are reading only for entertainment (which is fine, but you must acknowledge and accept that).
Books, whether fiction or nonfiction, are stories. They present information in a narrative. That narrative will be determined by who writes it and how they've lived their life.
Me either, and that's exactly the issue. Look at any top books list and it is mostly white males. Either men have an easier time getting published or men are inherently better writers. I believe the former is the case. By seeking out female writers and other diverse voices we can start to tip the balance.
Historically, men were the only ones that had the opportunity to write, so any books from older centuries are going to be by men. But that doesn't have to be the case now.
ps. Love Jane Jacobs! After you read that, if you are interested, you may want to check out The Power Broker to see the other side. It is one of my top nonfiction books. (warning, white male author ;)).
If you struggle to get diverse voices inside your field, maybe you should read from diverse voices outside your field. I understand it is hard and I also gravitate towards comfortable reading, usually meaning from people that look like me. This is the problem. Start being accountable to yourself about it.
2 of 19 the books in the list are by female authors (both recommended by women). Looking for some good books from women? Check out these:
Nonfiction:
* Radical Acceptance, Tara Brach
* The Death and Life of Great American Cities, Jane Jacobs
OpenReview is building software for transparent scientific peer review, advanced review-paper matching systems, and knowledge bases for scientific communities. We are seeking an experienced full stack developer to join the dynamic team of software developers working across universities and nonprofits. We aim to revolutionize scientific peer review working with machine learning researchers, open access advocates, and a broad community of open source developers. Are you excited to build user-facing applications and inspired to develop new tools for research? Join us! Reviewing applications now!
OpenReview is a fiscally sponsored project of Code for Science & Society, a nonprofit supporting public interest technology. Read more about OpenReview + CS&S here: https://blog.codeforscience.org/welcome-and-contratuations-o...
To apply, please send a cover letter and resume to [email protected].