Buffer: Composable Buffers for Go(github.com)

50 pointsby tombenner11 anni fa14 comments

https://github.com/djherbis/buffer

4 comments

stinos11 anni fa

   // Buffer 32KB to Memory, after that buffer to 100MB chunked files
  buf := buffer.NewUnboundedBuffer(32*1024, 100*1024*1024)

  pool := NewFilePool(100*1024*1024, "") // "" -- use temp dir

I seem to have a twofold feeling about this code: on one hand I really like the brevity of these statements, but on the other hand it does come at the cost of being rather cryptic: without the comments one can only guess what exactly is going on. To the point that should I want to use this in my own code and make it understandable I'd almost would be forced to either copy the comments as well or else wrap it in a method with another name or. In such cases it probably would have been better if the original API already had done this.

gouggoug11 anni fa

I feel like we could say that about any code. It doesn't sound cryptic to me. It does require to read the doc (http://godoc.org/github.com/djherbis/buffer#NewUnboundedBuff...) but I'd argue that's the case for everything.

djherbis11 anni fa

Just to add to the other comments made here, the NewUnboundedBuffer is meant to be a convenience method. It's actually just a (hopefully useful) example of how to compose the other buffer types to create a more complicated buffer. Thanks for the comment, hope it helps clear that up.

nkatsaros11 anni fa

If anything this is just a nomenclature problem. It's common for Go packages to have New* functions for common use cases and this is one example. After looking at the source it is very clear what it does.

TillE11 anni fa

That's one problem that named arguments solve pretty nicely. Unfortunately, I don't think Go supports them.

MetaCosm11 anni fa

There are a lot of ways to make arguments nicer in Go. The simple on is to just use a configuration struct (or anon struct) at that has nice names. But, for complex stuff, functional options as laid out by Dave Cheney: http://dave.cheney.net/2014/10/17/functional-options-for-fri... (video: https://www.youtube.com/watch?v=24lFtGHWxAQ)

lobster_johnson11 anni fa

You have anonymous strict literals, though, which are nearly as succinct and can serve the same role.

elithrar11 anni fa

Nice work. buffer.NewPool is a great convenience over writing your own sync.Pool.

I was previously using https://github.com/oxtoacart/bpool as a 64K buffer pool for rendering (concurrently) template/html contents—so I can check for the errors from template.Render—before then using io.Copy to copy the "known good" contents to the http.ResponseWriter. I may have to look into using this.

djherbis11 anni fa

Hey author here, happy to answer your questions.

heavenlyhash11 anni fa

Have you / would you consider extending this concept to buffers that are still single writer but support many readers -- something like a `FreshReader() io.Reader` function that returns a reader that starts from the beginning again? (I have an application that needs this semantic and ended up just rolling it quick-and-dirty; this sounds like very similar concepts done up with better gift wrapping.)

djherbis11 anni fa

I'm currently working on something that does that sort of thing. https://github.com/djherbis/fscache

It doesn't have the composability features yet, but it already handles 1 writer with many concurrent readers. Let me know if that helps!

kenferry11 anni fa

As a relative go newb, I do have a question about this: I thought channels/go routines were already essentially composable buffers. No?

fishnchips11 anni fa

Yes and no. Goroutines are lightweight threads - what other languages might call fibers. Channels are synchronisation primitives designed to work with goroutines but they do indeed provide some buffering. What OP is doing though is creating customisable buffers that allow you to control for memory usage and that are likely designed to replace things like bytes.Buffer rather then buffered channels. Of course I may be totally wrong ;)

djherbis11 anni fa

You've got the right idea. bytes.Buffer is great (I even use it under the hood) but this repo is intended to give you more control over how your buffer behaves as the amount of data stored in it changes.