There is no real way to prevent this, but there are ways to increase the cost of doing so. For example, one level of obfuscation is, OAI could internally run synthesis and adopt a “netlist-in” model in which Broadcom gets a netlist - a description of a huge amount of gates and wires and how they connect - instead of the plain Verilog (or other language). It is possible to reverse engineer the netlist, but it’s a certain level of indirection and effort.
A big part of the semiconductor industry also operates on a reputation basis. Broadcom (like TSMC) is a neutral party as a design house, but if they did something like this, it might ruin that reputation.
The typical way a chip effort in a non-chip company works is that the "design" is the RTL (e.g. SystemVerilog that defines the behavior of the chip) and then this is handed off to a third-party "design house" (such as Broadcom) that turns that code into a real image of a chip, which is called a GDS (basically you can think of this as a very big layer by layer photoshop file) that can actually be sent to a fab. This is called "backend design", in contrast to the "frontend design" (the RTL itself).
As another commenter said, Broadcom is very experienced with backend design (as well as the supply chain management, testing, etc. that comes after the chip is taped out) and so this can't be regarded as a "first chip". Richard Ho (the head of hardware at OpenAI) is also extremely experienced and used to be the head of the Google TPU effort -- where he actually worked with Broadcom in a similar tapeout already. So yes, this is not a "first design"!
Chip CEO here. It really depends on what "design" or "production" means. Does "design" mean that the design was complete? Does "production" mean the beginning of production, i.e. tapeout? If measuring from RTL-freeze to tapeout, this is a fairly typical (even somewhat unimpressive) timeline (accounting for some unexpected issues) for a large, complex 3nm chip. If measuring from concept (no RTL at all, block diagram of architecture) to tapeout, this is an amazing timeline. The truth is probably somewhere in between. A more concrete statement would use actual technical milestones and gates.
Hey, tangentially- I am CEO of Fabric, a company building orders of magnitude faster hardware accelerators for next-gen cryptography on the latest fab technologies.
Founder of AlphaSheets here -- we built this back in 2015 and developed it for 3 years. We built Python, R, SQL and full excel formula/hotkey/format/conditional formatting/ribbon compatibility. It was a long slog!
I wish you good luck and all the best. It's a tough field but a big market. And I still think the potential is there.
I always wondered what special genius people saw in Girard.
The observation that people want what each other want is not new and doesn't require philosophical genius to observe -- "keeping up with the Joneses" is what it's called by normal people.
What about "avoiding competition is good so chase blue oceans?" That's certainly a decent fund thesis, sure. Is it a genius one? Certainly seems like the returns come from the application of the maxim and not the maxim itself.
The real insight would've been to propose a fun way out of this "mimetic hell" for society. Girard's observation is that this usually takes violence against a scapegoat -- certainly not a fun way out.
The title is wrong about it being an integrated design. Here's an excerpt from the paper abstract:
"This paper presents a new type of photonic accelerator based on coherent detection that is scalable to large (N≳106) networks and can be operated at high (gigahertz) speeds and very low (subattojoule) energies per multiply and accumulate (MAC), using the massive spatial multiplexing enabled by standard free-space optical components"
Caveat: the paper mainly focuses on the "standard quantum limit" which is the fundamental photon energy needed for the operations. If other things are taken into account (for example, modulation energy for the weights in this homodyne scheme, which scales with N² and not N, or the limits of the ADC) then the energy they are proposing is nowhere near possible. Furthermore, substantial alignment and packaging problems exist for free space optical systems, which prevents them from beating integrated approaches in the near term. In fact, it seems that Fathom Computing has potentially pivoted away from free space, based on the latest verbiage on their website, and they've been trying to get it to work for 3 years now.
However, it still presents an interesting case for the fact that the fundamental floor on optical scaling is absolutely tiny. It'll be interesting to see who wins in this space :)
You can go to Groq.com -- that startup claims to have 125fj per flop (and each mac is two flops thanks to marketing logic). Started by 8 out of 10 founding TPU team members.
I don't mean to be too negative here, but this is hardly a new development, so can someone clarify the novelty in this paper? Neural nets have been extensively demonstrated in memristor-based architectures [1] and several memristor-based training architectures have previously been proposed and tested [2]. The abstract's claim that "no model for such a network has been proposed so far" is prima facie blatantly false.
In any case, I have yet to see a conclusive, publicly explained solution to the significant system-level problems with memristor-based neural architectures, or indeed any analog neural architecture. The best claimed digital architectures are around ~250 fJ per multiply-and-accumulate (MAC) [Groq], and these generally involve 8-bit multiplication, which is extremely expensive in the analog domain thanks to the exponential scaling of power with precision levels. Even if you set aside the monstrous fabrication and device-level variance issues with memristors, DAC and ADC consume tens of pJ per sample in the realistic IP blocks that are commercially available. Although only one pair of DAC and ADC operations is required per dot product, this is still 40 fJ per MAC from DAC and ADC alone, assuming a 256x256 matrix multiplication and not taking other system-level issues into account. This limits memristors to a 5x over current digital architectures, and as nodes shrink, by the time memristors come out, this will be around a 3x. While a 3x is considerable, I don't think it justifies the moonshot-level deep tech risk that memristors will continue to represent. Many hardware companies [Tabula...] have failed attempting to reach something like a 3x in the main figure-of-merit, only to find that system-level issues get them a 1x instead. Besides, I'm sure digital architectures have more than 3x room for improvement- plenty of tricks left for digital!
I'm hoping for a breakthrough, because I am fundamentally an optimist, but memristors have been failing to deliver since 2008.
Although probably not sufficient for AGI, network architecture is essentially guaranteed to be important, because of both ample empirical evidence of the importance of architectures and ample reason, from facts about numerics, to believe that it is important.
In the first category (empirical evidence),
- The discrete leap from non-LSTM RNN to LSTM network performance on NLP was essentially due to a "better factoring of the problem": breaking out the primitive operations that equate to an RNN having "memory" had a substantial effect on how well it "remembered."
- The leap in NMT from LSTM seq2seq to attention-based methods (the Transformer by Google) is another example. Long-distance correlations made yet another leap because they are simply modeled more directly by the architecture than in the LSTM.
- The relation network by DeepMind is another excellent example of a drop-in, "pure" architectural intuition-motivated replacement that increased accuracy from the 66% range to the 90% range on various tasks. Again, this was through directly modeling and weight-tying relation vectors through the architecture of the network.
- The capsule network for image recognition is yet another example. By shifting the focus of the architecture from arbitrarily guaranteeing only positional invariance to guaranteeing other sorts, the network was able to do much better at overlapping MNIST. Again, a better factoring of the problem.
These developments all illustrate that picking the architecture and the numerical guarantees baked into the "factoring" of the architecture (for example, weight tying, orthogonality, invariance, etc.) can have and has had a profound effect on performance. There is no reason to believe this trend won't continue.
In fact, there are some very interesting ways to think about the principles behind network structure -- I can't say for sure that it has any predictive power yet, but types are one intuitively appealing way to look at it: http://colah.github.io/posts/2015-09-NN-Types-FP/
Shameless plug: I am a founder of AlphaSheets, a company working on solving all of these issues. It's quite scary (building a spreadsheet is like boiling an ocean) but our mission feels very meaningful, we're well-funded, and we are now stable and serving real users.
A big problem in finance workflows is that there is a tradeoff between several factors: correctness, adoption / ease-of-use, rapid prototyping, and power. We aim to solve several of these major problems. We've built a real-time collaborative, browser-based spreadsheet from the ground up that supports Python, R, and SQL in addition to Excel expressions.
Correctness is substantially addressed, because you don't need to use VLOOKUP or mutative VBA macros anymore. Your data comes in live, and you can reference tables in Python as opposed to individual cells. A lot of operational risk goes away as well, because the AlphaSheets server is a single source of truth.
We help with adoption of Python and adoption of correct systems as well. You can gradually move to Python in AlphaSheets -- many firms are trying to make a "Python push" and haven't succeeded yet because the only option is to move to Jupyter and that's too much of a disruption. It's less brittle than Excel. The important keyboard shortcuts are there.
And finally, the entire Python ecosystem of tools (pandas, numpy, etc.) and all of R is available, meaning that many pieces of functionality that had to be painstakingly built in-house in VBA and pasted around are simply available out of the box in well-maintained, battle-tested packages.
Our long term plan is to broaden our focus into other situations in which organizations are outgrowing their spreadsheets. We think there's a lot of potential with the spreadsheet interface but the Excel monopoly has prevented meaningful innovation from happening. For example, every BI solution tries to be "self-serve" and "intuitive" these days, but encounters resistance from users who end up sticking with spreadsheets due to their infinite flexibility and immediate familiar appeal.
We hope to bring the spreadsheet in line with the realities of the requirements of the modern data world -- big data, tabular data, the necessity of data cleaning, data prep / ETL, the availability of advanced tooling (stats, ML), better charting -- because we think there's a giant market of people waiting to move to a modernized but familiar spreadsheet.
If there's anyone interested, contact me, because I'd be very interested in chatting! I'm michael at alphasheets dot com :)
Shameless plug: I'm a founder of Alphasheets, a company seeking to solve problems like these! I couldn't resist replying after seeing these comments.
We make a collaborative (Google Sheets style) spreadsheet with Python and R running in the sheet. You can define functions, plot using ggplot, embed numpy dataframes, matrices and all that good stuff. We don't let people use macros, all the code runs in cells because we think macros are too brittle. You can check out the website at http://alphasheets.com .
We're seeing that many enterprises (for example, in finance) that have Excel power users are moving to Python because of limitations like these, and are running into adoption issues because people like spreadsheets so much. That's generally where we come in and provide a bridge from the Excel world to Python through a more friendly frontend.
We're also seeing that Alphasheets can help a lot with shortening feedback cycles on more sophisticated data analyses- Excel is the most popular self-serve analytics tool out there, but doesn't cover cases where you need Python/R/fresh data.
Frontend Developer (React) | AlphaSheets | up to $150k (depends on level of hire and equity tradeoff) + equity | Contractors welcome; remote or onsite | Bay Area
You can check more examples out at alphasheets.com.
AlphaSheets marries the capabilities of spreadsheets (simple WYSIWYG calculation interface) with the full power of programming. We've gotten excitement from wall street quants, marketing analysts, pharmaceutical scientists, insurance analysts. Our broader audience is the burgeoning population of people who can write small bits of code but aren't full-on software engineers. We envision a future where tens of millions of people with these skills see AlphaSheets as their tool of choice for data analysis.
Experience is a plus, but not a must as long as you're smart. We have a React+ES6+Flow / Haskell stack. We love seeking leverage through good architecture, languages (Haskell!), frameworks, and tools. (Doesn't matter at all for this position if you don't know Haskell.) We're well funded (big seed round) and have 4 years' runway so we're not going away overnight.
Our culture is one of efficient, open communication and rational decision making. You'll be joining a founding team of 4 guys out of MIT.
Email our CTO (Anand Srinivasan) at anand (at) alphasheets (dot) com
Senior Backend Developer (Haskell) | AlphaSheets | up to $150k (depends on level of hire and equity tradeoff) + equity | Bay Area
We're building the future of spreadsheets: collaborative, programmable in multiple languages, and highly extensible. We think spreadsheets haven't reached their full potential as a general computing platform. We envision a world where many more people are able to code thanks to the intuitive interface of a spreadsheet, where non-technical and technical analysts can share the same interface to data, and where everything from fully-featured data analysis and visualization apps to new spreadsheet functions can be shared, Google-Sheets style, on the AlphaSheets platform. We're an ambitious company with plenty of runway.
If you're an opinionated functional programming proponent who hacks in Scala, Clojure, or Haskell in their free time, that's a great sign of a fit. Experience is a plus, but not a must as long as you're really smart. Doesn't matter if you don't know Haskell as long as you can learn. We have a React/Flow/Haskell stack. We love seeking leverage through good architecture, languages (Haskell!), frameworks, and tools.
You'll be joining a team of 4 MIT dropouts (among them, one owned a multimillion-dollar Bitcoin mine in high school, two were USA Math Olympiad winners, and one made ~$300k on stat-arb trading in high school).
Email me (Michael Gao, CEO) at michael (at) alphasheets (dot) com
A big part of the semiconductor industry also operates on a reputation basis. Broadcom (like TSMC) is a neutral party as a design house, but if they did something like this, it might ruin that reputation.