37ef_ced3 · HackerTrans

37ef_ced3

no profile record

Submissions

AVX-512 code generator NN-512 vs. TensorFlow

jonatron.github.io

2 pointsby 37ef_ced35 yıl önce1 comments

Fabrice Bellard: Fast, Standalone CPU Inference for Transformer Language Models

bellard.org

7 pointsby 37ef_ced35 yıl önce1 comments

Show HN: NN-512 – Generate standalone C code for neural nets

nn-512.com

111 pointsby 37ef_ced36 yıl önce28 comments

comments

4 yıl öncediscuss

Linus Torvalds:

  And I claim that that is the real problem with AVX-512 (and pretty much any vectorization). I personally cannot find a single benchmark that does anything I would ever do - not even remotely close. So if you aren't into some chess engine, if you aren't into parsing (but not using) JSON, if you aren't into software raytracing (as opposed to raytracing in games, which is clearly starting to take off thanks to GPU support), what else is there?

Answer? Neural net inference, e.g., https://NN-512.com

If you need a little bit of inference (say, 20 ReNet50s per second per CPU core) as part of a larger system, there's nothing cheaper. If you're doing a small amount of inference, perhaps limited by other parts of the system, you can't keep a GPU fed and the GPU is a huge waste of money.

AVX-512, with its masked operations and dual-input permutations, is an expressive and powerful SIMD instruction set. It's a pleasure to write code for, but we need good hardware support (which is literally years overdue).

4 yıl öncediscuss

There are completely stand-alone systems like:

https://NN-512.com

The stand-alone code generator (a statically linked executable written in Go with no dependencies outside the Go standard library) generates stand-alone POSIX C code for the neural net, requiring only gcc to compile.

Also see Fabrice Bellard's LibNC:

https://bellard.org/libnc/

C API. Small library, no external dependencies, available for Linux and Windows.

4 yıl öncediscuss

And https://NN-512.com

4 yıl öncediscuss

You're in for a big surprise. Try using the language.

Spend some time using Go, and you will be impressed by its performance.

You'll wonder, "Were all those haters on Hacker News misinformed?"

4 yıl öncediscuss

Convolutions are fused into convolutions, elementwise operations are fused into convolutions, everything is inlined except where function calls are needed for pthread work units (and those work units are all custom/arbitrary).

4 yıl öncediscuss

So...

...you were unable to decipher this Hacker News comment thread...

...unable find some C code and build it with GCC and make an executable for yourself...

...but you think you can reverse engineer the executable?

4 yıl öncediscuss

Here is the C code for a DenseNet-121 generated by NN-512:

https://nn-512.com/browse/DenseNet121

Even if you had the C code available to you, you would have a hard time producing the input graph.

Good luck reverse engineering it after GCC has compiled it!

NN-512 has an incredibly flexible code generator. It can easily be tweaked to produce completely different code for the same convolution, so everyone can apply their own twist to defeat the reverse engineers ("the intellectual property thieves").

4 yıl öncediscuss

This will not be able to reverse engineer fully-customized, fully-fused neural networks generated by NN-512:

https://NN-512.com

NN-512 generates custom code for all the operations, custom units of work for the threads, custom code around tensor edges, everything is fused and unrolled and customized. If they can deduce the network graph specification from the AVX-512 code, I will be astonished.

If you can do it, show me. But I know you can't.

Anyone who cares about model privacy will use their own variant of a tool like NN-512. It's security through obscurity, but that's the best you can hope for if you are distributing an executable.

4 yıl öncediscuss

Are you seriously suggesting that the language should not have notation for allocating zeroed primitive types and receiving the address of the allocation?

  var (
      p1 = new(int)
      p2 = new(*int)
      p3 = new(**int)
      p4 = new(complex64)
      p5 = new(complex128)
  )

I feel like you must be joking. Should we say

  var (
      c complex128
      p = &c
  )

every time?

4 yıl öncediscuss

You can think of Go as the modern representative the Plan 9 and Oberon schools of thought -- their answer to (and replacement for) C and C++.

The fact that it was developed at Google is incidental. That's where Robert Griesemer (Oberon) and Rob Pike (Plan 9) and Ken Thompson (Unix, Plan 9) and Russ Cox (Plan 9) and Ian Lance Taylor, etc., happened to be employed.

4 yıl öncediscuss

It's used like this

  p := new(int)

since you can't say

  p := &int{}

4 yıl öncediscuss

For small-scale transformer CPU inference you can use, e.g., Fabrice Bellard's https://bellard.org/libnc/

Similarly, for small-scale convolutional CPU inference, where you only need to do maybe 20 ResNet-50 (batch size 1) per second per CPU (cloud CPUs cost $0.015 per hour) you can use inference engines designed for this purpose, e.g., https://NN-512.com

You can expect about 2x the performance of TensorFlow or PyTorch.

4 yıl öncediscuss

https://NN-512.com

4 yıl öncediscuss

Go. The obvious right choice.

But Meta is in the advertising business and Google is a competitor.

4 yıl öncediscuss

As an engineer who uses Go and appreciates it as a replacement for most purposes where you might use C or C++, the complaints just seem bizarre.

You've got people who argue against explicit error checking and people who don't understand the important role of nil pointers and people who don't see the massive net win of garbage collection in concurrent programs, and so on.

Endlessly.

It feels like a waste of time defending the language when the people criticizing it seem to hold such a vastly different point of view. It's like trying to convince people that Natural Born Killers is a good movie or that Primus makes good music. There's an unbridgeable chasm between you and the people you're trying to convince.

4 yıl öncediscuss

From a customer's perspective, it isn't clear that a hardware/software taxi driver is in any way better than a human taxi driver, particularly with (human) driver assist preventing collisions and all the other warnings provided by a modern car.

In both cases someone else is driving for the customer. With a human driver (plus driver assist braking and collision warnings) you have the most flexible, sophisticated intelligence on Earth driving. With a robotaxi you have something inferior. But maybe it's a lot cheaper, right?

The robotaxi can only compete on price because that's its only advantage. If you own a car that drives itself, that's a different story. Everybody can see the value proposition.

But is the robotaxi actually cheaper at all? We would have to look at the cost of the hardware (how often do lidars fail and how much do they cost to replace?) and the cost of the software development and the cost of the fallback human remote operators (fleet monitoring and teleoperation) and the years of huge R&D investment (billions of dollars) to evaluate whether a robotaxi fleet is indeed cheaper. So how much cheaper is it, exactly? 5%? 10%? 15%?

As a customer, would you pay a little more to have the most flexible, sophisticated intelligence on Earth (human brain + driver assist) or would you want to save a few dollars and risk having some dumb piece of software strand you in the middle of the road somewhere?

We all use Google Maps or Apple Maps when driving and most of us have seen these systems do boneheaded things. Just imagine the dumb things a robotaxi could do. It's hard for a normal person to be excited about this. I don't know a single person who is excited by robotaxis.

4 yıl öncediscuss

Yes, but did he buy her a horse?

4 yıl öncediscuss

From a customer's perspective, it isn't clear that a software taxi driver is in any way better than a human taxi driver, particularly with (human) driver assist preventing collisions and all the other warnings provided by a modern car.

In both cases someone else is driving for the customer. With a human driver (plus driver assist braking and collision warnings) you have the most flexible, sophisticated intelligence on Earth driving. With a robotaxi you have something inferior. But maybe it's a lot cheaper, right?

The robotaxi can only compete on price because that's its only advantage.

If you own a car that drives itself, that's a different story. Everybody can see the value proposition.

We would have to look at the cost of the hardware and maintenance and fallback remote operators and the R&D investment to evaluate whether a robotaxi fleet is indeed cheaper. How much cheaper is it, exactly? 5%? 10%? 15%?

Would you pay a little more to have the most flexible, sophisticated intelligence on Earth (human brain + driver assist) or would you want to save a few dollars and risk having some dumb piece of software strand you in the middle of the road somewhere?

We all use Google Maps or Apple Maps when driving and most of us have seen these systems do boneheaded things. Just imagine the dumb things a robotaxi could do.

It's hard for a normal person to be excited about this. I don't know a single person who is excited by robotaxis.

4 yıl öncediscuss

Colin Percival's Tarsnap: https://www.tarsnap.com/

Colin is "Security Officer Emeritus, FreeBSD".

4 yıl öncediscuss

Animats, I was a fan of your Downside site: http://downside.com/

Downside's goal is to remind people that we're experiencing a financial bubble, and that financial bubbles always burst. (We originally wrote that in April, 2000. Any questions?)

And here we are, living through another huge bubble.