xgk · HackerTrans

xgk

no profile record

comments

letzten Monatdiscuss

Mixture-of-Expert (MoE) was introduced in the 1990s [1, 2], see also [3, 4]. The idea was that MoE scales up model capacity and only introduces small computation overhead. MoEs did not become viable for high-performance applications until sparse routing was integrated with modern deep networks, made possible by large-scale distributed computation. The breakthrough came with the development of sparsely gated networks [5], which showed that it is possible to maintain model accuracy while activating only a small fraction of a large parameter network during both training and inference.

[1] R. A. Jacobs, M. I. Jordan, S. J. Nowlan, G. E. Hinton, Adaptive mixtures of local experts. (1991)

[2] M. I. Jordan, R. A. Jacobs, Hierarchical mixtures of experts and the EM algorithm. (1993)

[3] L. Xu, M. Jordan, G. E. Hinton, An alternative model for mixtures of experts. (1994)

[4] S. Waterhouse, D. MacKay, A. Robinson, Bayesian methods for mixtures of experts. (1995)

[5] N. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q. Le, G. Hinton, J. Dean, Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. (2017)

vor 4 Monatendiscuss

Indeed, that is the standard approach. It is also how some of the aforementioned languages desugar opaque type synonyms during compilation. It has the slight disadvantage that we can no longer use variables like

in some situations, but need to use

    x._polynomials_gf_2

or whatever is the structure's field name. It is nice to avoid this boilerplate, which can become annoying quickly. Let the type-checker not the human do this work ...

> You do not need another language for this.

By the Church-Turing thesis you never need another language, but empirical practise has shown that the software engineering properties we see with real-world code and real-world programmers differ significantly between languages.

vor 4 Monatendiscuss

There are even more algebras on the same bits, when you take signed integers into account, such as saturating arithmetic.

One interesting programming language construct that might be useful in this context are Opaque Type Synonyms, a refined form of C's typedef, which modern languages like Rust, Haskell, Go or Scala offer. This allows the programmer to use the same underlying types (e.g. int), give it different names, and define different algebras with the alias. The typing system prevents the different aliases accidentally to flow into each other. Of course that alone does not help to manage the profusion of algebras over the same bits. I think a better approach for a high-level programming language is to follow assembly and really use different names for different operations, e.g. not have + build in. Instead use explicit names like add_uint32, add_polynomials_gf_2, add_satur_arith, etc etc. The user can then explicitly define (scoped) aliases for them, including +, as long as the typing system can disambiguate the uses. The Sail DSL for ISA specification (https://github.com/rems-project/sail) does this, and it is nice.

vor 4 Monatendiscuss

> CPUs actually implements 5 distinct data types

Yes, that's true, but the registers themselves are untyped, what modern CPUs really implement is multiple instruction semantics over the same bit-patterns. In short: same bits, five algebras! The algebras are given by different instructions (on the same bit patterns).

Here is an example, the bit pattern 1011:

• as a non-negative integer: 11. ISA operations: Arm UDIV, RISC-V DIVU, x86 DIV

• as an integer residue mod 16: the class [11] in Z/16Z. ISA operations: Arm ADD, RISC-V ADD/ADDI, x86 ADD

• as a bit string: bits 3, 1, and 0 are set. ISA operations: Arm EOR, RISC-V ANDI/ORI/XORI, x86 AND.

• as a binary polynomial: x^3 + x + 1. ISA operations: Arm PMULL, RISC-V clmul/clmulh/clmulr, x86 PCLMULQDQ

• as a binary polynomial residue modulo, say, x^4 + x + 1: the residue class of x^3 + x + 1 in GF(2)[x] / (x^4 + x + 1). ISA operations: Arm CRC32* / CRC32C*, x86 CRC32, RISC-V clmulr

And actually ... the floating point numbers also have the same bit patters, and could, in principle reside in the same registers. On modern ISAs, floats are usually implemented in a distinct register file.

You can use different functions in C on the bit patterns we call unsigned.

vor 2 Jahrendiscuss

> China’s terrible demographics

Are China's demographics appreciatively different from other industrialised countries? Questionable. Not to mention that it's unclear why an aging society is a problem at least for the next 100 years or so (e.g. most violent crime is perpetrated by the under-30s). It's also easy, at least for a dictatorship, to increase the birth rate (e.g. restrict access to all birth control, give massive preferences to families with children, like housing, salary, "bachelor tax" etc).

> military containment of China.

That is expensive. One frequently cited explanation for the collapse of the Soviet Union was that its poor economy could not support its oversized army, which it needed to keep the Soviet Bloc at heel (not to mention all the revolutions and secession movements it fostered elsewhere).

Also, does anyone actually believe that China is expansionist? Compare:

https://en.wikipedia.org/wiki/List_of_wars_involving_the_Uni...

https://en.wikipedia.org/wiki/List_of_wars_involving_the_Peo...

The strategic problem of China's rise is more that other countries will eventually switch allegiance away from the West.

vor 2 Jahrendiscuss

> In terms of design, China has world class companies.

And not just companies. This is currently the world's top open-source RISC-V development https://github.com/OpenXiangShan coming from the Chinese Academy of Science.

(Whether China will remain investing in RISC-V, given that the US government has started to pressure US RISC-V development to limit their involvement with China is another question.)

vor 2 Jahrendiscuss

An alternative explanation might involve both of:

1. Stopping technology transfer worked exceedingly well for the west weakening the Soviet Union during the Cold War.

2. A distinct lack of (non-violent) alternatives for the West preventing China becoming the world's leading technological superpower (and hence also strongest military).

I doubt this will be successful in the long run, because China is not burdened by the Soviet Union's extremely inefficient way of organising its economy. Not to mention that China is the worlds biggest market.

vor 2 Jahrendiscuss

Sorry, I should have said: ... (row)hammer hard enough, every sufficiently dense/modern DRAM ...

vor 2 Jahrendiscuss

I think if you (row)hammer hard enough, every DRAM will eventually flip a bit.

vor 2 Jahrendiscuss

This is misleading.

I recommend [1] as an introduction to the semiconductor physics behind the Rowhammer problem. Rowhammer is an instance of the "weird machine" problem behind many security problems, i.e. a mismatch between two abstractions: the abstraction we pretend describes the system, vs the reality of the system. In the case of Rowhammer, that is the abstraction of memory as a digital device, against the reality of storing bits with capacitors and wires, ie. analog devices. Clearly a leaky abstraction. The denser you pack those capacitors and wires, the more leaky.

[1] A. J. Walker, S. Lee, D. Beery, On DRAM Rowhammer and the Physics of Insecurity. https://ieeexplore.ieee.org/document/9366976

vor 2 Jahrendiscuss

Agree. The extreme secrecy of DRAM manufacturers about the innards of their chips puts an additional obstacles in the way of memory controllers (MCs) implementing efficient Rowhammer defences. In particular, if the MC doesn't know which addresses are corresponding to neighbouring rows, how can an MC know with certainty that any concrete row is being attacked? (And, to the best of my knowledge, DRAM manufacturers don't give away this information.)

vor 2 Jahrendiscuss

The possibility of flipping bits in DRAM in a Rowhammer like fashion, was known in the DRAM industry since at least the 1990s (sorry, no reference handy), and Rowhammer-like access was used in DRAM quality testing.

As silicon density increased, the issue became more urgent.

vor 2 Jahrendiscuss

It has been possible to re-purpose such additional refresh cycles as an additional Rowhammer attack vector, see https://www.usenix.org/conference/usenixsecurity22/presentat...

vor 2 Jahrendiscuss

Have you ever seen any even moderately detailed specification of what the DRAM manufacturers do in this regard? I have not, and I looked. I am deeply sceptical ....

I don't believe that Rowhammer mitigations happen inside the DRAM chips themselves, I think that they are being put into the memory controller that talks to DRAM. Since DRAMs with built-in Rowhammer defences would have to spend transistors on this defence, those transistors would be 'wasted' in situations where Rowhammer is not part of the attacker model.

vor 6 Jahrendiscuss

   On one side ...
   On the other side ...

Only one of those sides has children in substantial numbers, so this polarity will sort itself out over longer time scales. As somebody living next to a primary school in a very progressive city I already see this in a big way!

vor 6 Jahrendiscuss

   questionably from the beginning

Agreed. If you look at what's the majority of compute loads (e.g. Instagram, Snap, Netflix, HPC) then that's (a) not particularly security critical, and (b) so big that the vendors can split their workload in security critical / not security critical, and rent fast machines for the former, and secure machines for the latter.

I wonder which cloud provider is the first to offer this in a coherent way.

vor 6 Jahrendiscuss

Other networking protocols coming from a telco background, in particular ATM and ISDN, were all circuit switched, and had suitable resource reservation for QoS. Acceptance of telephony degradation was probably driven by cost: VoIP was free and that made a difference, especially for international calls. In my experience, in 2020 the VoIP calls I make are really high quality, even better than 1980s-style ISDN calls, and the main cause of audio quality degradation are people using "hands-free" setups with their laptops.

vor 7 Jahrendiscuss

   studies are bs because

We should not let the perfect be the enemy of the good. The problem is that we do not currently know how to carry out studies on the efficacy of programming language paradigms under "controlled laboratory conditions". It's known to be hard. Feel free to change this and become famous! The studies discussed in the article are but first steps towards:

- a better empirical grounding of programming language & software engineering research;

- more emphasis on reproducibility in science.

I'm glad that Vitek, Berger et al are starting serious empirical PL/SE research, and care about reproducibility! Bravo!

vor 7 Jahrendiscuss

   somehow messed with Cs 
   absorption line

In a simple case yes, but if all other time keeping mechanisms would also be messed with by dark matter, there might be no way of saying which one is the right own.

vor 7 Jahrendiscuss

The clock drift of A1 and A2 could be correlated for some reason that we don't currently understand.