Golang dependency hell in Haunts The Manse game(kickstarter.com)
kickstarter.com
Golang dependency hell in Haunts The Manse game
http://www.kickstarter.com/projects/2066438441/haunts-the-manse-macabre/posts/373466
8 comments
Golang should be the name of the language. Ironic that you can't Google for "go" and get relevant results, but "golang" works well.
I've got 3 minutes. I've spent more than 3 minutes searching, and I can't find a way to specify a version in an import (though you can tag the go version, you still can't tie the import to a specific tag or version or commit). Am I missing something? It seems the only way around this is to check out a point in time release and import that (which would solve the problem the article mentions). However, that's a workaround for a problem that package managers like NPM and RubyGems have trivially solved.
You can't. The right way to spend the 3 minutes is to read the docs. [1] [2]
Also, just because it doesn't work the way NPM does doesn't mean it doesn't work.
[1] http://golang.org/doc/code.html#remote
[2] http://golang.org/cmd/go/#hdr-Remote_import_path_syntax (linked from [1])
Also, just because it doesn't work the way NPM does doesn't mean it doesn't work.
[1] http://golang.org/doc/code.html#remote
[2] http://golang.org/cmd/go/#hdr-Remote_import_path_syntax (linked from [1])
Neither of those links address the issue the author mentioned, nor do they describe the need to use your own local copy of a package in case the package authors introduce non-backwards compatible changes. Yes, it needn't behave like NPM, or RubyGems, or Aptitude, or Homebrew, or any of the dozen or so systems out there for install remote packages that let you indicate version.
There is no version control for dependencies. You pretty much have to download the version you want manually, and then reference your local copy. Many people are aware of this and I imagine there will be work done to support versioning in some way.
This has nothing to do with Go. It sounds like a case of poorly managed dependencies and build rules and likely weak engineering. The fact that they wanted to support the iPad and chose Go also shows this wasn't well thought out.
I would say bad engineering:
* Go 1.0 had a terrible GC, especially on 32-bit platforms (their target platform)
* No manual memoray management (and unlikely to ever exist)
* Can't bind to C++, can't cross-compile cgo (gccgo can though)
Go is really cool, and I use it every day, but it doesn't make any sense for games, especially not for their target platform. Maybe in a few years the GC will be more reliable, but I wouldn't try to build a game in Go if I thought I might want to try to sell it.
* Go 1.0 had a terrible GC, especially on 32-bit platforms (their target platform)
* No manual memoray management (and unlikely to ever exist)
* Can't bind to C++, can't cross-compile cgo (gccgo can though)
Go is really cool, and I use it every day, but it doesn't make any sense for games, especially not for their target platform. Maybe in a few years the GC will be more reliable, but I wouldn't try to build a game in Go if I thought I might want to try to sell it.
"There is nothing special in Go that makes it particularly well suited to video games, but on the other hand there is nothing inherent in the language that makes it a bad choice, except for the fact that it's new and not well supported."
I would say that "new and not well supported" makes this a potentially bad choice for shipping code. Not to say that it can't be used but the problem with new technologies is that not all warts have been properly exposed and you can open yourself up to major headaches.
I would say that "new and not well supported" makes this a potentially bad choice for shipping code. Not to say that it can't be used but the problem with new technologies is that not all warts have been properly exposed and you can open yourself up to major headaches.
Previous discussion: https://news.ycombinator.com/item?id=5707019
And the March 5 update on the Kickstarter page seems to imply the project has been abandoned: http://www.kickstarter.com/projects/2066438441/haunts-the-ma...
My summary from the last time it was posted.
"Original author didn't localize dependencies, obviously caused problems."
"Original author didn't localize dependencies, obviously caused problems."
"it makes a local copy of that external library. And if you have a local copy, you can edit it. And meanwhile, that person you copied the OpenGL code from? They are also changing their version (fixing bugs, adding features, adding bugs). Therefore there is no guarantee that your version and their version remain compatible: it takes work to maintain compatibility."
Remote packages in Go are like DLLs for other languages. Yes, if you edit a DLL locally, and other people download the public version of that DLL, they won't be the same. DON'T DO THAT.
Make your own copy of the public Go repository, so you can merge in changes from the main branch as you see fit. This is akin to making a copy of the DLL and using that to distribute with your code, rather than just telling people "go download the newest version of the DLL from github".
It's also good practice for go authors to keep any one repository backwards compatible, make it clear when the code is in flux and may have breaking changes... and to make new branches if they have to make significant changes to their code.
Remote packages in Go are like DLLs for other languages. Yes, if you edit a DLL locally, and other people download the public version of that DLL, they won't be the same. DON'T DO THAT.
Make your own copy of the public Go repository, so you can merge in changes from the main branch as you see fit. This is akin to making a copy of the DLL and using that to distribute with your code, rather than just telling people "go download the newest version of the DLL from github".
It's also good practice for go authors to keep any one repository backwards compatible, make it clear when the code is in flux and may have breaking changes... and to make new branches if they have to make significant changes to their code.
> and put import ("github.com/go-gl/opengl/gl") directly into your code. Viola! Thus hilarity ensues.
Of course it does! All they had to do is fork the project (which they have) and reference that it instead. Viola, problem solved!
Of course it does! All they had to do is fork the project (which they have) and reference that it instead. Viola, problem solved!
No need to fork, it puts a copy of those libraries under ./src, so you always have a sealed (hermetic) referentially complete copy.
You don't bet your production software on things randomly downloaded from the internet at compile time. This is why go never auto-updates when you do "go get", you have to explicitly use "go get -u".
Always localize what you depend on, you never know when a github repo could be pulled, go offline, be renamed, a lawsuit could break out making the library vanish. There are TONS of reasons you should ALWAYS localize your dependencies for building software.
EDIT: This has the amazingly cool side-effect of meaning even on complex projects, you can just do a git clone and then go build FOO and it just works, no nonsense, no fuss, and it is EXACTLY in the state it was when it was checked in -- no surprises.
You don't bet your production software on things randomly downloaded from the internet at compile time. This is why go never auto-updates when you do "go get", you have to explicitly use "go get -u".
Always localize what you depend on, you never know when a github repo could be pulled, go offline, be renamed, a lawsuit could break out making the library vanish. There are TONS of reasons you should ALWAYS localize your dependencies for building software.
EDIT: This has the amazingly cool side-effect of meaning even on complex projects, you can just do a git clone and then go build FOO and it just works, no nonsense, no fuss, and it is EXACTLY in the state it was when it was checked in -- no surprises.
> No need to fork, it puts a copy of those libraries under ./src, so you always have a sealed (hermetic) referentially complete copy.
While that is true, it doesn't solve the problem with distributed teams. The library path MUST be pointing to a 'localized' repository to solve the problem the Haunt guys are having.
While that is true, it doesn't solve the problem with distributed teams. The library path MUST be pointing to a 'localized' repository to solve the problem the Haunt guys are having.
What? It ONLY looks for localized copies.
When you have import "github.com/gorilla/mux"
it will look in
$GOROOT/src/pkg/github.com/gorilla/mux and $GOPATH/src/github.com/gorilla/mux
if it is missing, you can fulfill it with
go get github.com/gorilla/mux which will place it in the first place in $GOPATH
What are you talking about breaking with distributed teams? Since $GOPATH/src/github.com/gorilla/mux is in source control you NEED NOTHING and it will be a known working git clone.
EDIT: Additionally, it will never auto-update those dependencies (and break your build) unless you explicitly tell it to with "go get -u".
When you have import "github.com/gorilla/mux"
it will look in
$GOROOT/src/pkg/github.com/gorilla/mux and $GOPATH/src/github.com/gorilla/mux
if it is missing, you can fulfill it with
go get github.com/gorilla/mux which will place it in the first place in $GOPATH
What are you talking about breaking with distributed teams? Since $GOPATH/src/github.com/gorilla/mux is in source control you NEED NOTHING and it will be a known working git clone.
EDIT: Additionally, it will never auto-update those dependencies (and break your build) unless you explicitly tell it to with "go get -u".
I know how it works :) I'm not new to Go.
BUT, what you are describing is exactly how the Haunt guys used Go and consequently the issues they have run into. Please re-read the update from Rick and read my comments again, you should then understand the issue and why changing the import path to a fork/clone/localized copy is beneficial (though not required).
BUT, what you are describing is exactly how the Haunt guys used Go and consequently the issues they have run into. Please re-read the update from Rick and read my comments again, you should then understand the issue and why changing the import path to a fork/clone/localized copy is beneficial (though not required).
I don't understand, so explain it to me like I am a child, I am certain I am missing something basic and the developers on the project aren't as dumb as it seems.
Seriously, it seems so goddamn basic. I am going to walk you through step by step how I would add an external library to my Go project.
#1. Switch to the go project directory #2. Fork for the new feature (adding_mux) #3. Run go get github.com/gorilla/mux #4. Add import "github.com/gorilla/mux" to my code, use mux for great goodness #5. Commit my code (this is commit of ./src which includes two things, the addition of the new github.com/gorilla/mux/* and my code using that new library) #6. Merge #7. Push
At this point, the commit that was just checked in doesn't need a change to its import path, will never break and will work as it was committed FOREVER. git clone FOO; go build BAR -- victory!
If at some point I want to update that library (new cool feature!)
#1. Switch to go project directory #2. Fork for updating library (update_mux) #3. Run go get -u github.com/gorilla/mux #4. Write code using new awesome features #5. Commit my code (again, "my code" in this case is both the update to the mux library, and my new usage of it, again committed together for great goodness!) #6. Merge #7. Push!
I feel like one of us is bat shit insane, and at this point, I am just hoping it isn't me.
EDIT: Another cool side effect of this is you end up with localized hermetic copies of your code with references for posterity to where you got it, bonus points!
EDIT 2: For reference, there are various forks of the source he "released" (abandoned? Sold for 28k? whatever) ... I think you can start looking at the github network from here: https://github.com/runningwild/haunts
Seriously, it seems so goddamn basic. I am going to walk you through step by step how I would add an external library to my Go project.
#1. Switch to the go project directory #2. Fork for the new feature (adding_mux) #3. Run go get github.com/gorilla/mux #4. Add import "github.com/gorilla/mux" to my code, use mux for great goodness #5. Commit my code (this is commit of ./src which includes two things, the addition of the new github.com/gorilla/mux/* and my code using that new library) #6. Merge #7. Push
At this point, the commit that was just checked in doesn't need a change to its import path, will never break and will work as it was committed FOREVER. git clone FOO; go build BAR -- victory!
If at some point I want to update that library (new cool feature!)
#1. Switch to go project directory #2. Fork for updating library (update_mux) #3. Run go get -u github.com/gorilla/mux #4. Write code using new awesome features #5. Commit my code (again, "my code" in this case is both the update to the mux library, and my new usage of it, again committed together for great goodness!) #6. Merge #7. Push!
I feel like one of us is bat shit insane, and at this point, I am just hoping it isn't me.
EDIT: Another cool side effect of this is you end up with localized hermetic copies of your code with references for posterity to where you got it, bonus points!
EDIT 2: For reference, there are various forks of the source he "released" (abandoned? Sold for 28k? whatever) ... I think you can start looking at the github network from here: https://github.com/runningwild/haunts
So the key part of the update I was referring to was the following:
Fast forward to when the community takes over, they clone the Haunts project and attempt to build it. Now they are using the HEAD of the 'github.com/go-gl/opengl/gl' repository, as specified by the imports path and they find that it no longer compiles.
This is where the problem exists.
The simplest way around this problem is to simply fork the OpenGL Go library (like they have), and reference the fork in the import path. That way no matter what happens in the upstream repository, the Haunts project will continue to compile. The original developer could of then just pushed the changes he was making in his local 'src' directory back to the fork and everything would be fine. When they are ready to take updates from upstream, they simply update their fork and then 'go get -u' on all developer machines.
By forking, or maintaining your own copy of library, you get a level of isolation from the upstream developers.
What you are saying is 100% correct, the Go tools automatically give you this when running 'go get', it clones the library into your 'src' directory. What I am saying is that when working in a team/community environment, you need an additional level of isolation (the fork), so that the entire team can be using the exact same version of the library.
Because of course that's not how it really works. Instead, it makes
a local copy of that external library. And if you have a local copy,
you can edit it. And meanwhile, that person you copied the OpenGL code
from? They are also changing their version (fixing bugs, adding features,
adding bugs).
Fast forward to when the community takes over, they clone the Haunts project and attempt to build it. Now they are using the HEAD of the 'github.com/go-gl/opengl/gl' repository, as specified by the imports path and they find that it no longer compiles.
This is where the problem exists.
The simplest way around this problem is to simply fork the OpenGL Go library (like they have), and reference the fork in the import path. That way no matter what happens in the upstream repository, the Haunts project will continue to compile. The original developer could of then just pushed the changes he was making in his local 'src' directory back to the fork and everything would be fine. When they are ready to take updates from upstream, they simply update their fork and then 'go get -u' on all developer machines.
By forking, or maintaining your own copy of library, you get a level of isolation from the upstream developers.
What you are saying is 100% correct, the Go tools automatically give you this when running 'go get', it clones the library into your 'src' directory. What I am saying is that when working in a team/community environment, you need an additional level of isolation (the fork), so that the entire team can be using the exact same version of the library.
> I feel like one of us is bat shit insane, and at this point, I am just hoping it isn't me.
> In short, original developer has the clone of the OpenGL library in their 'src' directory. While working on Haunts his making changes to it and not pulling updates from upstream. All is well, things will always build on his machine because as you say, it won't update unless he does 'go get -u'.
The important addition is -- he does not check this clone into his OWN source control, so his source control doesn't have a complete hermetic version of the application, it has a "partial" application... or really "application bits".
> Fast forward to when the community takes over, they clone the Haunts project and attempt to build it. Now they are using the HEAD of the 'github.com/go-gl/opengl/gl' repository, as specified by the imports path and they find that it no longer compiles.
If he had checked it into his version control system (you might call it forking, I call it localizing), they would have had the EXACT same version as him and they would not have had to fulfill that dependency on their own. Also, the same import path would continue to work off this local checked in version.
> By forking, or maintaining your own copy of library, you get a level of isolation from the upstream developers.
I thought that maintaining your own local copy of libraries under /src was the norm for go developers, as the value of hermetic checkouts goes hand in hand with some other Go'isms like static built binaries.
The important addition is -- he does not check this clone into his OWN source control, so his source control doesn't have a complete hermetic version of the application, it has a "partial" application... or really "application bits".
> Fast forward to when the community takes over, they clone the Haunts project and attempt to build it. Now they are using the HEAD of the 'github.com/go-gl/opengl/gl' repository, as specified by the imports path and they find that it no longer compiles.
If he had checked it into his version control system (you might call it forking, I call it localizing), they would have had the EXACT same version as him and they would not have had to fulfill that dependency on their own. Also, the same import path would continue to work off this local checked in version.
> By forking, or maintaining your own copy of library, you get a level of isolation from the upstream developers.
I thought that maintaining your own local copy of libraries under /src was the norm for go developers, as the value of hermetic checkouts goes hand in hand with some other Go'isms like static built binaries.
Sorry, but problem not in Go. Code of external libraries shouldn't be edited (or should be forked before changing).
Syntax errors:
Performance problems:
Bad version control:
Those sorts of mistakes seem consistent with the type of coder who wouldn't set aside the 3 minutes it takes to understand Go's dependency system and either use it correctly or build a super-simple makefile to pull the correct versions of code from elsewhere.
Also, pedantic but worth noting: Golang is not the name of the language.