llmzero

vor 2 Jahrendiscuss

What is a little contradictory is that designing a system to use less resources can increases the number of people fine tuning models so that the final result can be a net global increase in the total energy use. A hypothetical goal could be to reuse fine tuning, that is designing a knowledge graph in which you fine tuning from a previously fine tuned model (like dynamic programming, save the result of previous computations). Lora allow us to store the small matrices with low cost.

llmzero

vor 2 Jahrendiscuss

I liked that you link to renting a dual 24GPU for 0.60cents/hour, but how long could it takes to fine tune a 70b model using your system (4 bits for weights)?

If I were a consumer I would be interested in the final price of fine tuning, for example a table with model size, training size, cost of training, and expected loss of quality with this technology.

One obvious question: Can you apply your technology with the recent (-1,0,1) encoding?, I think you will answers that the (-1,0,1) model is not available and you can't try it, but my question is whether once/if that model is available answer.ai will be able to use the same technology that this post to fine tune a big model in two very small GPUs, and then I should ask for a new table with cost/benefits analysis.

Edited: I should add that I find this kind of work very useful for enhancing individual users like me to be able to compete in the applications of LLM market, this is great work and along the lines of the book "from zero to one" (not that I like or dislike the author) to solve the kind of problem that nobody is trying to solve.

Edited: Now that I have a total of 23 points in HN, I will change my password to some random one, just to cure my desire to look for votes, and try to make some work, and again some tomorrow create a new presence in HN.

llmzero

vor 2 Jahrendiscuss

Think about the following scenario: I write a calculus book and the agents of this model just modify every example and every definition and change a little the ordering of the material to teach students. Now they are using my book but it seems they are not using my book. Are they trying to copy without copying?

llmzero

vor 2 Jahrendiscuss

The superformula depends of four parameters and is able to model many different curves. I wonder if that superformula would be useful to learn to generalize the form of a curve given few points. It could be that, in same way, the four parameters of that curve are a orthogonal bases in the hypothesis space, in the sense that each parameters add a lot the information. If this intuition has any meaning, it could be the start of a new theory for constructing bases of the hypothesis space, that is models with few parameters but great expressive power.

Edited: (1) The following link explains expressivity and generalization power in machine learning: https://blog.evjang.com/2017/11/exp-train-gen.html

So my question is whether the superformula constitute an example of great expressivity and powerful generalization for curve fitting by using machine learning models.

Edited: (2) In the following link they use the superformula, Automatic Generation of Smooth Curves from Interpretable Low-Dimensional Parameters.

So the intuition seems fruitful. https://arxiv.org/pdf/1808.08871.pdf

llmzero

vor 2 Jahrendiscuss

My take away of a long post.

<< It’s a general feature of machine learning—and AI—techniques that they can be very useful if an approximate (“80%”) answer is good enough. But they tend to fail when one needs something more “precise” and “perfect”.

llmzero

vor 2 Jahrendiscuss

What's new in this edition?

llmzero

vor 2 Jahrendiscuss

In page 2, there is the theorem: Theorem ∀n in Nat.f n (fib p) (fib (p+1)) = fib (p+n), I think it should be for all p in Nat (fib p) + (fib (p+1)) = fib(p+2), otherwise there is something mysterious here.

llmzero

vor 2 Jahrendiscuss

Shorter ruby version: phrase.split.select{|x| x.count("aeiou")<=2}.max_by(&:length)

The enumerable module of Ruby provides many methods that can be easily implemented in J. Just to show one of them

   max_by =: 1 : '{~ (i. >./)@:(u&>)'

For example the list.max_by(&:length) is (# max_by)

llmzero

vor 2 Jahrendiscuss

I agree that in a team you should not use code golf for others to review your work. But for your own explorations or because your team can read easily your code golf then use it when is appropriate.

llmzero

vor 2 Jahrendiscuss

Just for fun, a solution in language j to the problem of selecting the longest word with less than 3 vowels.

solution =: >@:{.@:(\: #&>)@:(((+/@:(e.&'aeiou') <: 2:) # ])&.>)@:;:

Example

solution 'yes, today you are reading something that is not so easy to grasp'

the result is: today

Ruby : frase.split.select{|x| x.count("aeiou")<3}.sort_by(&:length).last => "today"

Edited: Added a comparison with Ruby. It seems Ruby here is easier to read and to compose.

llmzero

vor 2 Jahrendiscuss

Since: (i) the father and the mother of Sally may be married with other people, and (ii) the sister or brother relationship only requires to share one parent, we deduce that there is no a definitive answer to this question.

  Example:  Sally has three brothers, Sally and their brothers have the same mother but a different father, and those brothers have two sisters Sally and Mary, but Mary and Sally are  not sisters because they are from different fathers and mothers, hence Sally has no sister.

For those mathematically inclined: Supposing the three brothers are called Bob (to simplify) and the parents are designed by numbers.

FS = father of Sally = 7

MS = mother of Sally = 10

FB = father of Bob = 12

MB = mother of Bod = 10

FM = father of Mary = 12

MM = mother of Mary = 24

Now MS=MB=10 (S and B are brothers), FB=FM=12 (Bob and Mary are brothers), (FS=7)#(FB=12), and (MB=10)#(MM=24). Now S and M are not sisters because their parents {7,10} and {12,24} are disjoint sets.

Edited several times to make the example trivial and fix grammar.

llmzero

vor 2 Jahrendiscuss

Just thinking about some intuition for the result of the theorem: If the off diagonal elements are zero then the diagonal element is an eigenvalue, by continuity of the determinant, if the off diagonal element are small then $det(A-a_{ii}\lambda)$ is almost zero, that is the new eigenvalue is near aii. So it suggests that the off diagonal elements measure how far is aii from being an eigenvalue.

Submissions

A Recursive Definition of AI

comments