[I'm one of the Microsoft Research people who worked on this]
Thanks for your questions! We have thought of many heuristics but we didn't want to constrain the dataset release on some heuristic that we picked, possibly ruining the dataset. Participants in the challenge should feel free to apply additional filters as they see fit. For example, this [1] work could be useful as a filtering method.
Unfortunately, we do not have the budget to provide any compute resources to help with running the models at this time. Note that any techniques developed with this dataset will be owned by those who develop them and it's up to them how/if they will make them available/open-source.
[I'm one of the Microsoft Research people who worked on this]
That's certainly true for simple use cases. Our goal here is to eventually also capture the long-tail of queries about a codebase. Often, within the domain of a project there is a set of natural language terms/jargon that describe complex tasks specific to the domain. Imagine for example a developer joining a mid-sized project and trying to find how to achieve some simple but project/domain-specific task.
[I'm one of the Microsoft Research people who worked on this]
We did consider adding StackOverflow questions. Some of our queries in the CodeSearchNet challenge do actually come from StackOverflow (via StaQC [1]). It's certainly interesting to see how all other SO data can be useful for this task. Thanks for the suggestion!
The reason we didn't try this at this point:
Many people in research have tried working with SO data. In my experience I have observed an interesting problem with the data: it's deduplicated! This is great for users but bad for machine learning, since the data looks "sparse" (roughly, each concept appears once). Sparsity is an obstacle, since it's hard for most existing machine learning methods to generalize from sparse data. In contrast, in natural language there are (e.g.) multiple articles describing the same event more or less.
I do not expect them to decrease military expenditures. Their coalition partners (the far-right "Independent Greeks" that will be "holding" the ministry of defense) will not allow that easily.
There is some recent research has shown that programming languages present some "naturalness":
Hindle, Abram, et al. "On the naturalness of software." Software Engineering (ICSE), 2012 34th International Conference on. IEEE, 2012.
http://macbeth.cs.ucdavis.edu/natural.pdf
Although it is interesting, I think that the data should be normalized for life expectancy. A country with a lower life expectancy will necessarily have a younger population. So instead of 14 years old, it would make sense to show the age of the (e.g.) first percentile of the population.
From a machine learning perspective (i.e. theoretically according to my view) this is useful because source code is highly structured with very complex constraints and tons of data available (e.g. every project in GitHub). This means that machine-learning methods for handling such problems need to be developed. Such methods may eventually be useful in other application.
Now on the applied side (software engineering, programming languages), such methods (probabilistic machine learning and probabilistic/statistical models) can handle uncertainty with a principled way and provide useful tools to software engineers that exploit the amounts of data that is available both in internal and external codebases. This is something that is not 100% possible with formal tools, that usually require some form of human knowledge to be embedded. For example, you will see on the list above, tools that can do autocompletion, others that suggest "reasonable" renamings or others that help migration of source code between languages and all thanks to data.
Hopefully, at some point these methods will be so advanced that they will be able to learn (i.e. trained) from every piece of code that is available online and (e.g.) spot bugs in your code, semi-automatically refactor your code etc.
This is a new and very interesting area in machine learning and software engineering. Anyone interested might also find the following papers interesting too
Hindle, Abram, et al. "On the naturalness of software." Software Engineering (ICSE), 2012 34th International Conference on. IEEE, 2012.
Tu, Zhaopeng, Zhendong Su, and Prem Devanbu. "On the Localness of Software."
Nguyen, Tung Thanh, et al. "A statistical semantic language model for source code." Proceedings of the 2013 9th Joint Meeting on Foundations of Software Engineering. ACM, 2013.
Campbell, Joshua Charles, Abram Hindle, and José Nelson Amaral. "Syntax errors just aren't natural: improving error reporting with language models." Proceedings of the 11th Working Conference on Mining Software Repositories. ACM, 2014.
Allamanis, Miltiadis, and Charles Sutton. "Mining source code repositories at massive scale using language modeling." Mining Software Repositories (MSR), 2013 10th IEEE Working Conference on. IEEE, 2013.
Movshovitz-Attias, Dana, and William W. Cohen. "Natural Language Models for Predicting Programming Comments." ACL (2). 2013.
Allamanis, Miltiadis, Earl T. Barr, and Charles Sutton. "Learning Natural Coding Conventions." arXiv preprint arXiv:1402.4182 (2014).
Allamanis, Miltiadis, and Charles Sutton. "Mining Idioms from Source Code." arXiv preprint arXiv:1404.0417 (2014).
This is a stopper. I don't want to pollute the filenames with this information. Using a sqlite db or something even simpler would have been more preferable imho
Thanks for your questions! We have thought of many heuristics but we didn't want to constrain the dataset release on some heuristic that we picked, possibly ruining the dataset. Participants in the challenge should feel free to apply additional filters as they see fit. For example, this [1] work could be useful as a filtering method.
Unfortunately, we do not have the budget to provide any compute resources to help with running the models at this time. Note that any techniques developed with this dataset will be owned by those who develop them and it's up to them how/if they will make them available/open-source.
[1] https://arxiv.org/abs/1806.04616