But in the process of finding every security vulnerability in the software you write, would you be ok with your model hacking AWS to start mining bitcoin? Would that still be aligned with you? (I'm guessing not)
That's the alignment problem I'm referring to (which is one of the many aspects of alignment), for which we do not have robust recipes, and not only that but for which research suggests it is becoming harder to create guardrails for as base models get smarter.
I guess we could debate what counts as alignment, but I think my initial point remains that if the underlying base model needs these classifier guardrails so badly then the way we train the base models is creating fundamentally misaligned models that are happy to pursue illegal behavior.
I'm sure OpenAI would argue that base model + guardrail is aligned, but considering the "relative intelligence" of these two pieces, the fact that guardrails can just be turned off, and these kind of incidents, I am not reassured. We may well get another "oopsie" moment with much more catastrophic consequences even from otherwise well intentioned actors.
Echoing other responses to you but this isn't a capability problem. You're totally right that this is so last year in terms of LLMs being capable in infosec. The issue here is an alignment one, i.e. the model seemingly isn't "aware" (especially with its guardrails turned off it would seem) that it is doing something immoral/illegal by hacking HF for the answer to its (vague) query of "solve this problem". Or if it is "aware", it's not trained to care, i.e. it's not an aligned model (alignment is considered hard).
In Canada, all three major AI research centers use clusters created with public money. These clusters receive regular additional hardware as new generations of GPUs become available. Considering how these institutions work, I'm pretty confident they've considered the alternatives (renting, AWS, etc). So that's one data point.
The intent behind media matters but isn't all that matters. How people might interpret something is important (albeit often unpredictable).
I think the symbolism of "let's crush all remaining vestiges of creative culture" is a pretty obvious _potential_ interpretation from a _non-trivial amount_ of people. In that sense it is an interpretation that matters for our present discourse, even if it isn't the interpretation that the creator of the ad intended.
I know, the CAQ is full of xenophobes, and it can suck not looking white in Quebec (although much less so in Montreal). I'm skeptical that it drives rent prices in any detectable way.
Not really no, there's been a bump which studies seem to agree is an aftereffect of COVID, but the media is also paying more attention to it. There's always been some amount of shootings and stabbings in Montreal, and 99% of Montreal is safe during night time.
Wow, what a take. I seriously doubt that people in Montreal/Quebec are more or less xenophobic than the average North American; if you have data supporting this I'd be happy to change my mind.
Also please be more skeptical of thoughts of the form "X is Y because of hot take Z". Reality is complex and things have many causes. Montreal has a long history of doing city planning differently than other cities in Canada.
This is definitely the case in DL (and I'm assuming elsewhere too but I wouldn't know).
I've lost count honestly, running 1-2 year old paper github repos with some detail missing (like the Python version!) that make it non-trivial to run as is. Libraries make undocumented breaking changes, wrong pickle format, authors used a nightly version which didn't make it to a tagged version, and so on.
This perhaps says also something about the CS (versus software eng) background that most people engaging in DL publishing have.
Yeah! This kind of reasoning is why it's exciting to think that we could use the GFlowNet machinery to construct latent representations--ones that map more directly to the notion of "beliefs" that we're used to think about as humans, something "discrete" and relational.
Concretely what this could mean is using these tools to generate causal hypotheses, like what's been done here: https://arxiv.org/abs/2202.13903
What the permutation invariance gets you is that the model doesn't arbitrarily prefer one (graph) configuration over another, but this seems tangential. The difference between this and RL is in what we do with the reward:
- RL says, give me a reward and I'll give you its max.
- GFlowNet says, give me a reward and I'll give you all its modes (via p(x) \propto R(x)).
Yes you would ideally have a loss (well, a reward/energy) that is invariant and operates e.g. directly on the molecule rather than on some arbitrary ordering of the nodes.
I think the inspiration came to me from looking at SumTrees and from having worked on Temporal Difference learning for a long time. The idea of flows came to Yoshua and I from the realization that we wanted some kind of energy conservation/preservation mechanism from having multiple paths lead to the same state.
But, yes in the moment it felt like some very serendipitous insight!
Yes, although they may not be ideal unless you're able to define a "distance" between a proof that the agent is proposing and whether that proof is correct (i.e. unless you're able to define a reward or energy function).
It may be possible to infer/learn a score from existing proofs though. We have a paper that manages to both learn a flow and an energy function (the score) from data: https://arxiv.org/abs/2202.01361
I don't know much about theorem proving though. Can some value be attributed to partial proofs?
Hi, first author of the original NeurIPS paper here! Someone just shared this HN post with me. I'll go through the comments but happy to try to answer questions as well.
Yeah the "net" in GFlowNet refers to how the underlying state is interpreted, not to an architecture. It is a way to train generative models, on any kind of discrete data (like graphs).
Source: am first author of original GFlowNet paper.
It's quite likely some people will come out of reading this thinking we should stop sending people to grad school. Shouldn't we instead take this as a signal that it needs improvement?
Grad students are one of the very few subpopulations of humanity allowed to take on extraordinary epistemological risks; a kind of immune system of our civilization. I'm not even talking about some elusive notion of "progress", just [intellectual] societal health. It would feel to me like a tremendous loss if we let go of such a component of society.
> Getting China to stop coal would be more effective than getting the USA to stop beef.
Fighting climate change is not a game of "what would be more effective" and ranking solutions (especially in between countries), it's a game of "what are ALL the things we can realistically do". Both _must_ be done.
I fail to see how this is the case, but perhaps I'm missing the point you're trying to make? What you're referring to is about colonialism, whereas here the USA is very much incentivized to work within its own borders, on its own food industry. It just so happens that climate change is a global problem as well, and the USA being a large rich country it has more causal influence on the outcome of climate change.