Dataset of 13 Billion Clicks available for research(cnets.indiana.edu)
cnets.indiana.edu
Dataset of 13 Billion Clicks available for research
http://cnets.indiana.edu/groups/nan/webtraffic/click-dataset
17 comments
The data doesn't seem to contain enough information to cluster unrelated URLs (e.g. by User Agent). Each record contains 1) a timestamp, 2) the requested URL, 3) the referring URL, 4) a boolean classification of the user agent (browser or bot), 5) a boolean flag for whether the request was generated inside or outside IU.
Although you could reconstruct who's looking at Facebook from inside IU, it'd be difficult to further associate them with some other website like Hacker News. On the other hand, the timestamps do leak some information which could potentially be used to identify patterns of activity.
Although you could reconstruct who's looking at Facebook from inside IU, it'd be difficult to further associate them with some other website like Hacker News. On the other hand, the timestamps do leak some information which could potentially be used to identify patterns of activity.
You've seen that bit where the user-agent is fairly unique to a particular user, right?
https://www.eff.org/deeplinks/2010/01/tracking-by-user-agent
https://www.eff.org/deeplinks/2010/01/tracking-by-user-agent
Yes, but they aren't sharing the user agent. They're just sharing a boolean derived from the user agent which says whether they think it was a bot or a browser.
I'm really surprised anyone would take the risk to release data like this, even with their security protocols in place. It just doesn't seem worth it:
- The potential upside is a few citations in research papers.
- The potential downside is a widescale invasion of privacy of IU students and staff, and a huge PR disaster.
- The potential upside is a few citations in research papers.
- The potential downside is a widescale invasion of privacy of IU students and staff, and a huge PR disaster.
This shit is going to be available on TPB before I can even click 'add comment'.
Marc Smith at Microsoft Research had a Usenet dB for research porpoises created about 6 years ago or so, and provided it to any researchers who wanted it. Although I didn't care about Usenet for my stuff, it was a good and useful offering for various researchers, and I hope this newer dB also proves useful! Thanks to Indiana for going to the trouble.
How did they collect this data without someone raising privacy flags? Releasing this data is almost certainly a bad idea, since it will likely reveal who the people are who made those requests. Anonymized data usually isn't.
> Additionally, while the dataset has been approved by the Indiana University IRB for “non-human subjects research” (protocol 1110007144), it might potentially contain bits of stray personal data. Therefore we require that you follow these instructions to request the data. You will have to sign a data security agreement.
> Data Transfer: If your request is approved, you will send a blank 3TB hard drive to the address below.
> We will return the loaded drive to the address specified in your request within 10 business days of receiving it. The data on the drive will be encrypted using TrueCrypt. The password to decrypt the data will be emailed to address that you specified in your request. It will be your responsibility to install and configure TrueCrypt (http://www.truecrypt.org/) on the system where you will be accessing the data.
> 4.) Data Removal: When you have finished using the data, you are responsible for securely and permanently removing the data including the drive that was used to transfer the data. For more information about secure data removal please see: (https://protect.iu.edu/cybersecurity/data/secureremoval) .
> I have read and agree to abide by all University data security practice related to access to University confidential data. To the best of my ability, I will comply, keep secure, or return all information provided to me.
TL:DR they're just crossing their fingers and hoping for the best.
> Data Transfer: If your request is approved, you will send a blank 3TB hard drive to the address below.
> We will return the loaded drive to the address specified in your request within 10 business days of receiving it. The data on the drive will be encrypted using TrueCrypt. The password to decrypt the data will be emailed to address that you specified in your request. It will be your responsibility to install and configure TrueCrypt (http://www.truecrypt.org/) on the system where you will be accessing the data.
> 4.) Data Removal: When you have finished using the data, you are responsible for securely and permanently removing the data including the drive that was used to transfer the data. For more information about secure data removal please see: (https://protect.iu.edu/cybersecurity/data/secureremoval) .
> I have read and agree to abide by all University data security practice related to access to University confidential data. To the best of my ability, I will comply, keep secure, or return all information provided to me.
TL:DR they're just crossing their fingers and hoping for the best.
Serious question here. What's the point of all this?
If you are interested in this kind of data, it's worth noting that there are some older, but, in a sense, more manageable datasets at the Internet Traffic Archive [1]---the data there can be downloaded and does not require being physically shipping through the post.
The largest dataset consists of 1.3 billion requests (for the 1998 World Cup website).
[1] http://ita.ee.lbl.gov/html/traces.html
The largest dataset consists of 1.3 billion requests (for the 1998 World Cup website).
[1] http://ita.ee.lbl.gov/html/traces.html
From a research standpoint this data set is much less interesting than a bunch of students/faculty/bots/apps clicking and surfing their way around the whole Internets.
Can someone actually post the real data.
This would be a direct violation of the Data Security Agreement (https://protect.iu.edu/system/files/Data-Security-Access-Agr...) that all persons requesting the data must agree to. Furthermore, they seem to indicate that they will only be releasing the data to researchers or large organizations.
That being said, feel free to request the data from them.
http://carl.cs.indiana.edu/data/webtraffic/click-dataform.pd...
That being said, feel free to request the data from them.
http://carl.cs.indiana.edu/data/webtraffic/click-dataform.pd...
Let's get easy access in honor of Aaron's recent actions.
This dataset is even worse since it includes both referral and the destination.
Keep in mind websites often put the usernames within the URL
Eg: http://www.facebook.com/Your.Name
http://www.reddit.com/user/USERNAME/
http://slashdot.org/~USERNAME
http://news.ycombinator.com/user?id=USERNAME
So no matter how much you think you have it anonymized, a person's browsing history could reveal a lot more than you think.