Nearly a third of new code on GitHub is written with AI help(axios.com)
axios.com
Nearly a third of new code on GitHub is written with AI help
https://www.axios.com/copilot-artificial-intelligence-coding-github-9a202f40-9af7-4786-9dcb-b678683b360f.html
19 comments
There is no way 33% of code is written with Copilot. I want to see the data backing that claim up.
I work for GitHub (although writing in an entirely personal capacity) and the segment in the GitHub Universe Keynote today said that 30% of Python code (written by users with copilot extension active) was written with copilot help.
I don’t know about how the data was gathered, I was not involved, but the segment is 4 minutes in so you can see what the actual announcement claims here: https://youtu.be/etMvd9IKPH4
I have used copilot and watched a colleague writing comments and frequently getting great go code suggestions without loads of edits. I personally am sincerely very impressed.
I don’t know about how the data was gathered, I was not involved, but the segment is 4 minutes in so you can see what the actual announcement claims here: https://youtu.be/etMvd9IKPH4
I have used copilot and watched a colleague writing comments and frequently getting great go code suggestions without loads of edits. I personally am sincerely very impressed.
> the segment in the GitHub Universe Keynote today said that 30% of Python code (written by users with copilot extension active) was written with copilot help.
The gap between this statement and what's in the OP title and text is too big. It's just false misleading bad journalism.
I need to write an article titled "People who install Copilot only use it for 30% of the code".
The gap between this statement and what's in the OP title and text is too big. It's just false misleading bad journalism.
I need to write an article titled "People who install Copilot only use it for 30% of the code".
That video says: "Since we started, 30% of all code written by early adopters come from copilot suggestions."
That is a far different headline than the one on this post. WTH is wrong with journalists now days? FFS.
That is a far different headline than the one on this post. WTH is wrong with journalists now days? FFS.
So 30% of code written by users with the extension installed was written with some extension assistance?
How exactly does that translate into the claim being made? Is github asserting the 100% of users have the extension installed?
My guess is that some small number of trial users did their homework with it then ran to stack overflow when it did something other than what they hoped.
How exactly does that translate into the claim being made? Is github asserting the 100% of users have the extension installed?
My guess is that some small number of trial users did their homework with it then ran to stack overflow when it did something other than what they hoped.
Thanks for sharing. So in other words it's the journalism that is wrong.
People writing code outside of using GitHub are likely more skeptical towards Copilot. We shouldn't forget that other, FOSS Git forges exist.
That doesn't seem to be the claim. They say one third was written with AI help.
Doesn't say how much of the new code was generated by Copilot, just that it was used in the process of writing it.
edit: I read the article wrong, skimming it. I understood it as 30% of new code written by GitHub engineers uses AI help, which sounds much more reasonable.
Doesn't say how much of the new code was generated by Copilot, just that it was used in the process of writing it.
edit: I read the article wrong, skimming it. I understood it as 30% of new code written by GitHub engineers uses AI help, which sounds much more reasonable.
Even if you do the "% of users using Copilot" approach, Copilot was limited to both Visual Studio Code (until today) and an invitation-only program.
The fact that the combo reached 30% of newly generated code doesn't make sense.
The fact that the combo reached 30% of newly generated code doesn't make sense.
The specific claim in the video was % of code being written by early adopters. I can't tell from the video how "early adopters" was defined or how they measured % of code so it's ambiguous. But story should have made it clear this isn't a percentage of all new code.
I hadn't seen the video quote until it was posted above: I assume "early adopters" is those who were accepted into the program and installed it, but yes it's still ambigious.
Ya, it is clearly a lie or the journalist was clueless
Yep! I was just watching how clang-tidy can now automatically apply code fixes in realtime. Is this AI?
https://www.youtube.com/watch?v=EfYMKOMc738
https://www.youtube.com/watch?v=EfYMKOMc738
I mean, Google search uses AI, and developers use Google to find things, so technically you could say the vast majority of code is written with "AI help", but that's not exactly a noteworthy observation to make...
Maybe it's bots doing it? Someone might be trying to fill up Github with autocompleted code.
Maybe they're just measuring lines committed (total guess)?
If you count white space, comments, variable and function declarations, and boiler plate code like creating loops, then maybe I could see how they got there.
Kind of related, but if I think about autocomplete, it probably writes about 10%-20% of the individual characters in my code. While it is totally misleading, it wouldn't be factually incorrect to say that autocomplete writes X percentage of my code. I wouldn't be surprised if GitHub is using some "misleading, but technically true" metric for this.
If you count white space, comments, variable and function declarations, and boiler plate code like creating loops, then maybe I could see how they got there.
Kind of related, but if I think about autocomplete, it probably writes about 10%-20% of the individual characters in my code. While it is totally misleading, it wouldn't be factually incorrect to say that autocomplete writes X percentage of my code. I wouldn't be surprised if GitHub is using some "misleading, but technically true" metric for this.
+1.
>Nearly a third of new code on GitHub is written with AI help
May be true if you count VSCode's IntelliCode, JetBrains IntelliSense, CoPilot, and other similar tools together. But no way copilot has that much marketshare yet.
>Nearly a third of new code on GitHub is written with AI help
May be true if you count VSCode's IntelliCode, JetBrains IntelliSense, CoPilot, and other similar tools together. But no way copilot has that much marketshare yet.
[deleted]
Martin from GitHub here. While Copilot is good it's not quite up to those numbers yet I'm afraid. It’s early days, but we’re already seeing that for some languages when people have it installed, we see about 30% of newly written code being suggested by Copilot. Vary’s between languages and the numbers of users in the technical preview is a LOT less than the 73m folks we have on GitHub, so a long way to go before it makes up a third of everything..
For more information, the interview with one of the creators of Copilot has been posted here https://www.githubuniverse.com/2021/session/707714/inside-gi...
Like any algorithm, it is dependent on the quality of its training data. In a study, a group of academics from New York University found 40% of the code produced by Copilot had cybersecurity flaws.
Yeah let me use this tool where 40% of its output is garbage. Toss this in the ever-growing AI Winter bin.
Yeah let me use this tool where 40% of its output is garbage. Toss this in the ever-growing AI Winter bin.
Given the fact that Copilot is trained on a corpus of real code, is it not reasonable to assume that the number of security flaws just reflects how insecure most public code is? If human-written code has about the same 40% suffering from security issues then then using Copilot is certainly no worse. It definitely doesn't seem fair to invalidate the idea of AI-assisted code generation without testing that.
> is it not reasonable to assume that the number of security flaws just reflects how insecure most public code is?
It sounds to me like that's not an inference that can easily be drawn. Copilot was trained on predicting code, it doesn't understand the code it produces syntactically. Security issues can be highly context dependent. For example, in most cases it's fine to log a variable, but when it happens to contain a password, it's a security issue. This is a flawed example as the algorithm may be able to learn that variables with names or contexts suggesting that they're secrets should not be logged, but I can imagine much more subtle issues can crop up.
It sounds to me like that's not an inference that can easily be drawn. Copilot was trained on predicting code, it doesn't understand the code it produces syntactically. Security issues can be highly context dependent. For example, in most cases it's fine to log a variable, but when it happens to contain a password, it's a security issue. This is a flawed example as the algorithm may be able to learn that variables with names or contexts suggesting that they're secrets should not be logged, but I can imagine much more subtle issues can crop up.
Think of it this way, how many security-hardened open-source monolithic kernels are there to learn from?
This number just doesn't feel remotely plausible. Hasn't Copilot supported just one editor and five languages until now? And it's advertised as being available to a limited number of testers, and gated behind a waitlist. Oh, and according to the article half the people who were granted access to Copilot have stopped using it.
There is no way to make these numbers work unless we assume that the people using Copilot are committing orders of magnitude more code. And if they are, how likely is it that they're really legit users?
There is no way to make these numbers work unless we assume that the people using Copilot are committing orders of magnitude more code. And if they are, how likely is it that they're really legit users?
The only plausible way to interpret that statistic that I can think up with is if it is saying:
a third of all code committed by active copilot users has been accepted copilot suggestions…
a third of all code committed by active copilot users has been accepted copilot suggestions…
Most of AI tools like TabNine, Copilot or Kite are quite resource intensive.
For example I have TabNine, but it is active ONLY when my laptop is plugged in, otherwise it drains batteries pretty quickly.
That's why I don't believe that more than 30% of developers are OK to heat up their laptops for getting AI suggestions. It more looks like a marketing title
Copilot is done remotely via an API so no extra resource drain on the computer.
That could be correct if by AI they mean Googling to get to Stack Overflow. Search used to be considered a core AI problem!
But then it would be a bit low wouldn't it ? :)
I find it weird that people keep slamming copilot. IDEA (and other IDEs) have for years auto complete for for loops, functions and a few other things. The most this tool does is put in a couple of extra like 'user.someInterestingProperty' based on your context inside those same for loop shells. It's just that the for-loop shell is now, for better or worse, similar to what people write.
I've seen some of the suggestions in my workflow at this point and it's much slower than IDE assist stuff, and generally barely helpful. So I don't see any reason to slam the crap out of it.
I wonder if it's a mix of people feeling like they A) will lose their job, B) make the software around the world worse (it's already horrendously bad) or C) hate Microsoft?
I can't really tell why anyone would honestly hate this tool other than a lot of pent up whateverism.
I've seen some of the suggestions in my workflow at this point and it's much slower than IDE assist stuff, and generally barely helpful. So I don't see any reason to slam the crap out of it.
I wonder if it's a mix of people feeling like they A) will lose their job, B) make the software around the world worse (it's already horrendously bad) or C) hate Microsoft?
I can't really tell why anyone would honestly hate this tool other than a lot of pent up whateverism.
I've always been somewhat suspicious of this AI generated code stuff. I get what's happening, but I'm not convinced it's all that fundamentally different than what compilers have been doing for decades.
Isn't a compiler a code generator? And isn't the stuff we humans write simply instructions to a compiler on what code to generate?
Compilers have tons of intelligence in them. At least when I was a CS grad (an ancient time ago) complex sets of heuristics were considered AI. And compilers use complex sets of heuristics to determine things like optimizations.
So I guess what I'm saying is that this news feels over hyped to me.
Isn't a compiler a code generator? And isn't the stuff we humans write simply instructions to a compiler on what code to generate?
Compilers have tons of intelligence in them. At least when I was a CS grad (an ancient time ago) complex sets of heuristics were considered AI. And compilers use complex sets of heuristics to determine things like optimizations.
So I guess what I'm saying is that this news feels over hyped to me.
> The open-source software developer GitHub says as much as 30% of newly written code on its network is being done with the help of the company's AI programming tool Copilot.
reply
(Emphasis mine)
I wonder if what this really means is ~30% of code written in codespaces or the web/cloud version of vscode you can use within a repo is written with co-pilot. Which I find much easier to believe than 30% of all code uploaded to github was written with AI assistance (how would they ever be able to measure that).
(Emphasis mine)
I wonder if what this really means is ~30% of code written in codespaces or the web/cloud version of vscode you can use within a repo is written with co-pilot. Which I find much easier to believe than 30% of all code uploaded to github was written with AI assistance (how would they ever be able to measure that).
> The open-source software developer GitHub says as much as 30% of newly written code on its network is being done with the help of the company's AI programming tool Copilot.
I wonder how they distinguish between Copilot code and code manually copied from Stack Overflow
VS Code enable mandatory spyware if you try to use the Copilot extension. I guess they track it by that and the git extension somehow?
Or maybe they just count what share of commiters have the Coopilot extension installed ...
Or maybe they just count what share of commiters have the Coopilot extension installed ...
And code that just is kinda vanilla and so would be something that co-pilot would actually generate...
[deleted]
[deleted]
> 40% of the code produced by Copilot had cybersecurity flaws.
Off-topic, but it's quite ironic to me that "cyber" had such a resurgence. In ~2010 to 2015, most people in my circle considered cyber to be a washed out joke, and yet, in the last few years, it gained massive popularity again. It's still hard for me to take anything "cyer-" seriously, though.
Off-topic, but it's quite ironic to me that "cyber" had such a resurgence. In ~2010 to 2015, most people in my circle considered cyber to be a washed out joke, and yet, in the last few years, it gained massive popularity again. It's still hard for me to take anything "cyer-" seriously, though.
cyberx is still a strong signal that something is corpo bullshit.
Maybe, all in all, we spend 33% of our time writing very dumb code that can easily written by some AI ? (I must recognize that a good chunk of the code I was writing as a professipnal dev was boiler plate stuff...)
> The open-source software developer GitHub says as much as 30% of newly written code on its network is being done with the help of the company's AI programming tool Copilot.
where is the reference link?
where is the reference link?
If that number is true, it's probably being thrown off by a research group somewhere studying Copilot by having it generate gigabytes of code.
The actual quote is:
> For some languages, we already see about 30% of newly written code being suggested by Copilot.
It writes about 30% of Python and 20-25% for other languages.
> For some languages, we already see about 30% of newly written code being suggested by Copilot.
It writes about 30% of Python and 20-25% for other languages.
Interesting, i am since 2008 on Github and still haven’t been offered to join the Product nor do i know of many professionals who have been.
Well, Google is an AI.