Sourcegraph Master Plan(about.sourcegraph.com)
about.sourcegraph.com
Sourcegraph Master Plan
https://about.sourcegraph.com/plan
36 comments
The best way to define the problem is iterating on it. This (and other tools) allows you to iterate faster. That's why tools like this are useful.
I'm not saying its not useful. My daily work currently is VR development in Unity and so I'm using C# in Visual Studio with ReSharper which together give one of the best and most complete "basic code intelligence" experiences I know of. It's great and useful but while it makes me more efficient it just doesn't have much impact on the actual hard parts of programming. It improves the workflow for the routine parts but they are merely time consuming, not difficult/hard problems.
Sourcegraph CEO here. Thanks for posting this. Happy to answer any questions!
What is the relationship between Sourcegraph and the LSP? The Master Plan reads like you were part of creating it?
Sourcegraph CEO here. Microsoft designed and released LSP. We are eternally grateful for that! We played no part in its creation. The first LSP mention links to the repo at https://github.com/Microsoft/language-server-protocol with more info.
We have, however, built and/or sponsored many language servers. (Those, in turn, build on compiler APIs that other people tirelessly created, to which we also owe thanks.)
We have, however, built and/or sponsored many language servers. (Those, in turn, build on compiler APIs that other people tirelessly created, to which we also owe thanks.)
The LSP is a really wonderful, simple and well executed idea. The simplicity of the protocol, cross platform (not bound to a COM interface or similar Windows only technology), ease of creation of an implementation in any language, all well done.
This is where the deep experience of Microsoft in building developer tools together with the desire to be cross platform are shining.
As a user of Emacs under Linux for the past 20 years to code in Fortran, they succeed in me using Visual Studio Code[0] and hack into a Fortran Language Server!
[0]: https://www.ceondo.com/ecte/2018/09/vscode-visual-studio-cod...
This is where the deep experience of Microsoft in building developer tools together with the desire to be cross platform are shining.
As a user of Emacs under Linux for the past 20 years to code in Fortran, they succeed in me using Visual Studio Code[0] and hack into a Fortran Language Server!
[0]: https://www.ceondo.com/ecte/2018/09/vscode-visual-studio-cod...
Yes, and LSP is one of those ideas that I respect even more because it was laughed at initially! Unlike most other approaches at solving this problem, it did NOT try to model concepts or structures in programming languages. It is presentational, not semantic; it is solely concerned with "what does the user want to do/see". It was radically simpler than other approaches at solving the same problem, and it has achieved tremendous adoption, so I think it's fair to say it's a success.
BTW, very cool that you built the Fortran language server!
BTW, very cool that you built the Fortran language server!
Simplicity is hard to achieve!
I am not the author of the Fortran Language Server, just fixing bugs and trying to improve it. But as the protocol is simple, this makes the work to improve it also way simpler. This is the part where I am now sure that the LSP will take over the world for the benefits of all the code editors, closed or open source, cross platform or not. And this is good for us as programmers :)
I am not the author of the Fortran Language Server, just fixing bugs and trying to improve it. But as the protocol is simple, this makes the work to improve it also way simpler. This is the part where I am now sure that the LSP will take over the world for the benefits of all the code editors, closed or open source, cross platform or not. And this is good for us as programmers :)
[deleted]
How much of this plan do you envision will be provided by Sourcegraph and how much comes from the many other people working on similar things? (For example, many people are working on language servers these days.)
Hey, Brian! Our (Sourcegraph's) unique role is that we have a product used by a ton of developers to search and browse code (in public and inside their company), so we have an audience for distribution of these tools. We also have some expertise. So, I envision most of our work being on that, and perhaps a few reference implementations of language servers and other APIs, and then letting the true language experts outside of Sourcegraph own the language-specific parts.
This is because the best people to build language servers are the language maintainers/experts themselves. A great language server should be a core part of every programming language, just as most languages have packages for parsing their own AST, typechecking, etc. (E.g., the javac API, go/types and related packages, and the TypeScript API.) This is incentive-aligned because great tooling boosts language adoption.
Rust has done a great job here, with an official language server (https://github.com/rust-lang-nursery/rls). The Go team is investigating building an official language server, which we would love (we built an unofficial one and are happy to deprecate it in favor of an official one, or help make it official--whatever is easiest for the Go team/community).
For languages that don't yet have a good language server, we would dearly love to sponsor the creation of one, but it's quite time consuming to do so. It takes a while to find the right person to build it for each language (esp. for languages we aren't familiar with), and the majority of these projects have not yielded a working language server, so that hasn't been as easy as we had hoped. This is the main bottleneck.
And in all cases, we want to make it so there's a huge reward to building a great language server. When you build a language server, you can use Sourcegraph to run it for all code you view on GitHub, Bitbucket Server, and (soon) other code hosts. That means you get hovers, go-to-definition, find-references, etc., in code files and PRs in your existing tools. (See https://chrome.google.com/webstore/detail/sourcegraph-for-gi... or the Firefox extension to start using that.) The process for taking a language server and deploying it to Sourcegraph currently requires some intervention from us, but we are changing that with https://github.com/sourcegraph/sourcegraph-extension-api. This will also make it much easier to take a language server and make it work in all editors, without writing per-editor shim extensions.
This is because the best people to build language servers are the language maintainers/experts themselves. A great language server should be a core part of every programming language, just as most languages have packages for parsing their own AST, typechecking, etc. (E.g., the javac API, go/types and related packages, and the TypeScript API.) This is incentive-aligned because great tooling boosts language adoption.
Rust has done a great job here, with an official language server (https://github.com/rust-lang-nursery/rls). The Go team is investigating building an official language server, which we would love (we built an unofficial one and are happy to deprecate it in favor of an official one, or help make it official--whatever is easiest for the Go team/community).
For languages that don't yet have a good language server, we would dearly love to sponsor the creation of one, but it's quite time consuming to do so. It takes a while to find the right person to build it for each language (esp. for languages we aren't familiar with), and the majority of these projects have not yielded a working language server, so that hasn't been as easy as we had hoped. This is the main bottleneck.
And in all cases, we want to make it so there's a huge reward to building a great language server. When you build a language server, you can use Sourcegraph to run it for all code you view on GitHub, Bitbucket Server, and (soon) other code hosts. That means you get hovers, go-to-definition, find-references, etc., in code files and PRs in your existing tools. (See https://chrome.google.com/webstore/detail/sourcegraph-for-gi... or the Firefox extension to start using that.) The process for taking a language server and deploying it to Sourcegraph currently requires some intervention from us, but we are changing that with https://github.com/sourcegraph/sourcegraph-extension-api. This will also make it much easier to take a language server and make it work in all editors, without writing per-editor shim extensions.
Looks awesome and I hope to hear from succeed!
The blue box half way down is really bad on mobile. Too much margin.
edit: https://imgur.com/a/8vCwBjh
The blue box half way down is really bad on mobile. Too much margin.
edit: https://imgur.com/a/8vCwBjh
Thanks for pointing that out. Fix deployed, should be online in a few min.
So, I've written a language server for a very niche programming language (Paradox Interactive games' scripting language).
I started it as a VSCode extension (via LSP), and am halfway to getting a Sublime Text extension to work, and gave up on intellij. Emacs/Vim shouldn't be hard, but I have no idea how I would release updates. While I agree LSP is technically making it an "n-plus-m" problem, implementing the plugins that wire things up to the IDE is hard.
My main pain-points have been:
- Initial integration: Each IDE has its own extension framework, and it takes a lot of work to get my LSP server to work at all
- Deploying updates: Once I have it working, I then need to spend days automating updates for each IDE.
- Distributing LSP updates: For VSCode, I bundle the executable, but that means extension updates for every LSP update. I could have system level installation, but I don't want to ask users to take that risk (nor manage installers myself)
- Syntax highlighting: There is no good standard for semantic highlighting (technically textmate, in practice not). LSP doesn't include highlighting.
- Settings management: Each IDE has a different way of managing settings, and gluing that to the LSP isn't easy.
Basically, I have to spend a week or two of my free time to support an extra IDE at all (not well), which is frustrating. Probably a month or two for full support, and that would be for maybe a dozen users.
I started it as a VSCode extension (via LSP), and am halfway to getting a Sublime Text extension to work, and gave up on intellij. Emacs/Vim shouldn't be hard, but I have no idea how I would release updates. While I agree LSP is technically making it an "n-plus-m" problem, implementing the plugins that wire things up to the IDE is hard.
My main pain-points have been:
- Initial integration: Each IDE has its own extension framework, and it takes a lot of work to get my LSP server to work at all
- Deploying updates: Once I have it working, I then need to spend days automating updates for each IDE.
- Distributing LSP updates: For VSCode, I bundle the executable, but that means extension updates for every LSP update. I could have system level installation, but I don't want to ask users to take that risk (nor manage installers myself)
- Syntax highlighting: There is no good standard for semantic highlighting (technically textmate, in practice not). LSP doesn't include highlighting.
- Settings management: Each IDE has a different way of managing settings, and gluing that to the LSP isn't easy.
Basically, I have to spend a week or two of my free time to support an extra IDE at all (not well), which is frustrating. Probably a month or two for full support, and that would be for maybe a dozen users.
Sourcegraph CEO here. We have a project underway that lets you write one editor extension and use it on multiple editors (and other tools with code views, such as GitHub, GitLab, Sourcegraph, etc.). We intend it to solve your problem.
https://github.com/sourcegraph/sourcegraph-extension-api
https://github.com/sourcegraph/sourcegraph-extension-api
Awesome mission. But god I wish JS weren't eating the world like this.
> At Sourcegraph, we want to fix this and eventually enable everyone to build software. For now, we're revealing our master plan for phase 1: how we're going to make it easier and faster for today's developers to build software.
These are lofty goals, and sorely needed.
We've come a long way but there is a lot of "suckage" both in developing software and in using it still.
I wish Sourcegraph all the best :)
These are lofty goals, and sorely needed.
We've come a long way but there is a lot of "suckage" both in developing software and in using it still.
I wish Sourcegraph all the best :)
One thing in particular that grinds my gears about a lot of software is how limited the actions I can perform on data are.
Take Spotify for example, a fantastic product by many measures, but which even so does suck in some ways. Being able to create playlists and to enqueue songs is a start, but think about a list as a special case, limited kind of graph, and think about what kinds of operations we are missing out on by restricting ourselves to just lists.
And that's not all. Spotify is also annoying in terms of looking at the currently playing list of songs. If you go someplace else and look at another list and then go back to the current playlist you'll usually be brought back to the top and if the list is long then you have to scroll a lot in order to get back to where you were.
And I would like to make arbitrary queries like "show me all playlists in which song x occurs".
And I would like to have access to the full history of songs I've listened to, including information about leaving the song before it was done, and skipping forward and back. And I would like to have the point in time in a song where I starred it or added it to a playlist.
Then I could query for example "show me all songs that I've listened to in the past six months, that I listened to at least 80% of the full length of, but which I have not listened to within the last two weeks" and play the resulting list.
Of course, the playlist being a simple concept makes it easy for users to understand. I guess the amount of people that want what I want is probably not many...
Take Spotify for example, a fantastic product by many measures, but which even so does suck in some ways. Being able to create playlists and to enqueue songs is a start, but think about a list as a special case, limited kind of graph, and think about what kinds of operations we are missing out on by restricting ourselves to just lists.
And that's not all. Spotify is also annoying in terms of looking at the currently playing list of songs. If you go someplace else and look at another list and then go back to the current playlist you'll usually be brought back to the top and if the list is long then you have to scroll a lot in order to get back to where you were.
And I would like to make arbitrary queries like "show me all playlists in which song x occurs".
And I would like to have access to the full history of songs I've listened to, including information about leaving the song before it was done, and skipping forward and back. And I would like to have the point in time in a song where I starred it or added it to a playlist.
Then I could query for example "show me all songs that I've listened to in the past six months, that I listened to at least 80% of the full length of, but which I have not listened to within the last two weeks" and play the resulting list.
Of course, the playlist being a simple concept makes it easy for users to understand. I guess the amount of people that want what I want is probably not many...
Arbitrary data querying are literally the problem databases exist to solve. It's a monumentally hard problem.
And there's another issue on top of it, the company 'crown jewels' problem. If Spotify makes it too easy to query Spotify, then it becomes ridiculously easy to compete with Spotify. Which isn't a bad thing necessarily, but not something we should expect Spotify to expend millions of dollars to enable.
And there's another issue on top of it, the company 'crown jewels' problem. If Spotify makes it too easy to query Spotify, then it becomes ridiculously easy to compete with Spotify. Which isn't a bad thing necessarily, but not something we should expect Spotify to expend millions of dollars to enable.
> And there's another issue on top of it, the company 'crown jewels' problem. If Spotify makes it too easy to query Spotify, then it becomes ridiculously easy to compete with Spotify. Which isn't a bad thing necessarily, but not something we should expect Spotify to expend millions of dollars to enable.
Yeah that's true. But mainly the data I want to be able to query is the data that is directly connected to my own activity. If Spotify made it possible to get access to metadata about my activity and the songs that were involved in my activity then it's useful for users but still limits somewhat what a would-be competitor could do. Combine that with a ToS that forbids redistribution of the data and I think Spotify could provide this without helping would-be competitors too much.
Even if a would-be competitor had access to various data from Spotify they would still need to both negotiate agreements about streaming rights with the IPR owners of the music itself, and they would need a lot of marketing in order to get users. In other words they would need a lot of capital. Where would a would-be competitor even get that kind of money? I don't see why any VC or startup accelerator would be willing to fund someone to build this given how big Spotify is and how limited the group of people probably is that would decide to leave Spotify for this feature.
Whereas on the other hand, this kind of access to data could be a premium feature, and then it could be another reason for users that cared about these things to pay for Spotify Premium. Open source third-party clients could implement query engines, building on exactly what you mentioned in the first paragraph;
> Arbitrary data querying are literally the problem databases exist to solve.
Open source clients exists today, and using them with Spotify requires premium. If Spotify provided the data, someone would follow suit in making use of the data in open source clients. Which could further boost sales of premium membership, while at the same time like I said would still not make it too easy for would-be competitors.
Actually, the client wouldn't even need to be open source, it could be a commercial product, and by integrating with Spotify Premium they would still not be a competitor, and they would not need to go through licensing, and marketing might be easier as well. "Enjoy advanced interaction with your Spotify Premium listening experince by using our client."
Yeah that's true. But mainly the data I want to be able to query is the data that is directly connected to my own activity. If Spotify made it possible to get access to metadata about my activity and the songs that were involved in my activity then it's useful for users but still limits somewhat what a would-be competitor could do. Combine that with a ToS that forbids redistribution of the data and I think Spotify could provide this without helping would-be competitors too much.
Even if a would-be competitor had access to various data from Spotify they would still need to both negotiate agreements about streaming rights with the IPR owners of the music itself, and they would need a lot of marketing in order to get users. In other words they would need a lot of capital. Where would a would-be competitor even get that kind of money? I don't see why any VC or startup accelerator would be willing to fund someone to build this given how big Spotify is and how limited the group of people probably is that would decide to leave Spotify for this feature.
Whereas on the other hand, this kind of access to data could be a premium feature, and then it could be another reason for users that cared about these things to pay for Spotify Premium. Open source third-party clients could implement query engines, building on exactly what you mentioned in the first paragraph;
> Arbitrary data querying are literally the problem databases exist to solve.
Open source clients exists today, and using them with Spotify requires premium. If Spotify provided the data, someone would follow suit in making use of the data in open source clients. Which could further boost sales of premium membership, while at the same time like I said would still not make it too easy for would-be competitors.
Actually, the client wouldn't even need to be open source, it could be a commercial product, and by integrating with Spotify Premium they would still not be a competitor, and they would not need to go through licensing, and marketing might be easier as well. "Enjoy advanced interaction with your Spotify Premium listening experince by using our client."
Wouldn’t an example of another ‘type’ be something akin to a graph for Spotify Radio? Particularly Artist or Song Radio.
Though, I do still see the point of your example.
Though, I do still see the point of your example.
Phase 2 sounds particularly interesting to me as it's a problem I also want to solve. Are you guys willing to talk about the direction you're moving in, or is this still to come?
I think making development as simple and accessible as possible is essential for our software-driven economy to scale efficiently.
I think making development as simple and accessible as possible is essential for our software-driven economy to scale efficiently.
Isn't Kythe[0] trying to achieve this already, with a graph based, multi-language framework?
Of course, there are no integration with current dev tools, but the core is there to make it work, maybe?
[0] - https://kythe.io
Of course, there are no integration with current dev tools, but the core is there to make it work, maybe?
[0] - https://kythe.io
Sourcegraph CEO here. Yes, Kythe is a very ambitious project and we have a lot of respect for the team building it inside Google. source{d} (https://sourced.tech/) is another company doing awesome things in the code analysis space.
From our POV, the more people trying to accomplish these things, the better, because:
1. As developers and humans, we want these things to exist!
2. Our business and product is a code search and browsing tool, which benefits from better code intelligence/analysis. We don't sell language servers directly; we sell a tool that makes use of them.
(For interactive tools (editors, code search, and code browsers), which is what we're primarily targeting, LSP (https://microsoft.github.io/language-server-protocol/) is a better fit than Kythe.)
From our POV, the more people trying to accomplish these things, the better, because:
1. As developers and humans, we want these things to exist!
2. Our business and product is a code search and browsing tool, which benefits from better code intelligence/analysis. We don't sell language servers directly; we sell a tool that makes use of them.
(For interactive tools (editors, code search, and code browsers), which is what we're primarily targeting, LSP (https://microsoft.github.io/language-server-protocol/) is a better fit than Kythe.)
> From our POV, the more people trying to accomplish these things, the better
Sure, im totally ok with this vision, given this is how people create new things, even if it looks 'solved' in a given context.
> As developers and humans, we want these things to exist!
The more iteration, the better.. we definitely need more polished tools in this space.
> LSP (https://microsoft.github.io/language-server-protocol/) is a better fit
Oh thanks, thats kind of the context i was looking for.. in what problem domain you guys were pointing to.
It makes perfect sense giving the current goals of Sourcegraph, to create a more complete solution to the dev community it targets.
(I didnt wanted to sound the "Dont reinvent the wheel" kind of guy here, just curious about what you were trying to acomplish, and used Kythe as a reference to the problem domain)
Sure, im totally ok with this vision, given this is how people create new things, even if it looks 'solved' in a given context.
> As developers and humans, we want these things to exist!
The more iteration, the better.. we definitely need more polished tools in this space.
> LSP (https://microsoft.github.io/language-server-protocol/) is a better fit
Oh thanks, thats kind of the context i was looking for.. in what problem domain you guys were pointing to.
It makes perfect sense giving the current goals of Sourcegraph, to create a more complete solution to the dev community it targets.
(I didnt wanted to sound the "Dont reinvent the wheel" kind of guy here, just curious about what you were trying to acomplish, and used Kythe as a reference to the problem domain)
[deleted]
Someone has read their Thiel. +1
Impressive that they’ve sold a free IDE for 20 million to investors...
Sourcegraph CEO here. We sell a code search and browsing product for developers to companies (https://about.sourcegraph.com/).
I don't get what problem this solves that Emacs with appropriate language modes doesn't already solve.
Sourcegraph CEO here. I have been a near-daily Emacs user for 15+ years. The unsolved problem for you and me as Emacs users is this: I wish I had the same high-quality extensions that VS Code users get. I use VS Code 1/3 of the time because the editing experience is so much better in many languages (such as TypeScript).
Have you tried VS Code? I would highly recommend you try it if you haven't yet. I think you'll see what you are missing.
(Of course, Emacs has many awesome features/modes that other editors lack, too. The same principle applies there.)
Have you tried VS Code? I would highly recommend you try it if you haven't yet. I think you'll see what you are missing.
(Of course, Emacs has many awesome features/modes that other editors lack, too. The same principle applies there.)
What the point in using VS Code compared to emacs when we have LSP servers and an emacs lsp-mode ?
VS Code extensions for language support are (usually) a combo of a language server plus VS Code extension code. Other editors using just the language server won't get the features implemented in the VS Code extension. So Emacs lsp-mode and a language server won't get you all the features that VS Code users enjoy.
For example, the Go language server (if enabled) in vscode-go is only responsible for hovers, definitions, references, and a few other things. Auto-imports, code lens for tests, helpful commands, coverage, etc., are all implemented in vscode-go's TypeScript extension code.
I recommend just trying VS Code for the languages you use in Emacs to see what you're missing. I bet a blog post about your experience would get to the front page of HN. :)
(One solution, of course, is to make it possible to write all of those things in a way that can be consumed by all editors. Check out https://github.com/sourcegraph/sourcegraph-extension-api to see our work to let you write any kind of editor extension in an editor-independent way (not just the language/code intelligence features). When this gets into beta, we'll feel comfortable updating this master plan to denote the progress toward the goals.)
For example, the Go language server (if enabled) in vscode-go is only responsible for hovers, definitions, references, and a few other things. Auto-imports, code lens for tests, helpful commands, coverage, etc., are all implemented in vscode-go's TypeScript extension code.
I recommend just trying VS Code for the languages you use in Emacs to see what you're missing. I bet a blog post about your experience would get to the front page of HN. :)
(One solution, of course, is to make it possible to write all of those things in a way that can be consumed by all editors. Check out https://github.com/sourcegraph/sourcegraph-extension-api to see our work to let you write any kind of editor extension in an editor-independent way (not just the language/code intelligence features). When this gets into beta, we'll feel comfortable updating this master plan to denote the progress toward the goals.)
People use a lot of different editors. The solution isn't "everyone switch to emacs".
Not suggesting that. I was asking why I as an Emacs user should switch to Sourcegraph. And now I have a pretty good answer.
Programming is hard because it's hard to define the problems that need to be solved, hard to design good solutions and often hard to implement them efficiently. A smart IDE is a useful tool but it does little to nothing to address the fundamental difficulty of programming. The real challenge to making programming more accessible is how to teach/train people to be better thinkers and problem solvers (and I include myself in needing that).