nothing0001

vor 3 Jahrendiscuss

I find a little sad that many people could change books and sharing experiences with others with using tools such as chatgpt. Those tools could be used to not discuss things like education and guide people to follow the agenda of the fine-tunnings authorities.

nothing0001

vor 3 Jahrendiscuss

How can they avoid that someone perform a post-training from this model to obtain a different model that perhaps could not be protected by copyright?

nothing0001

vor 3 Jahrendiscuss

I think you are absolutely right, but those sames problems apply when the paper claim that the average of two models gives a good model. So in that case the weight space could have additional properties that could make the proposed approach a little more plausible with some modifications. As you suggest, features are encoded in many different ways and in many neurons, so the suggested approach could only be applied for features that are encoded using only one neuron. To reduce a little the ways the features can be encoded, the proposal could be applied to an encoding of both models. Looking for matching neurons using as distance the L1-norm of the difference of outputs.

nothing0001

vor 3 Jahrendiscuss

A negative value cannot be produced but in the hidden layers the output of one neuron is multiplied by weights so the sign can be encoded in those weights.

nothing0001

vor 3 Jahrendiscuss

Reading the paper, I was thinking about the following: Given the weights of two models w1 and w2, then at each neuron k compute some average of the absolute difference of the outputs of neuron k over the training set. Then perhaps the neurons with low differences are those that capture general knowledge shared by w1 and w2, just an idea.

nothing0001

vor 3 Jahrendiscuss

I wonder what happens when one change the activation function, is there some related results in that direction?

nothing0001

vor 3 Jahrendiscuss

I asked chatgpt to generate the different orders of 5 words, it failed at it. Then I ask it to generalize the problem and then it refers to permutations (giving the formula). Then I ask it to use that information to generate the differents orders and then the chatgpt answer the original problem correctly. So it can use some of the information that it can't understand.

nothing0001

vor 3 Jahrendiscuss

Now that I am interested in large language model and as I begin reading Street Fighting Mathematics, the first thought is that large language model can learn a lot (statistical approximate inference) from dimensional analysis, that is using words like meters and seconds they can predict the formula for the velocity using only words related to physical dimensions

nothing0001

vor 3 Jahrendiscuss

It seems I used a lot of prompts: Some of them follow. Define the projection of word1 with respect to word2 as a word that is has a similar meaning to word1 and word2 and that is usually used with word1 -- The projection of word1 with respect to word2 is word1 when both words don't have anything in common in other case it should be a synonym -- The projection never increase the level of the word, so the projection of big can't be huge, the projection of two words look for the common thing between the words but always smaller than those words -- the projection should never add meaning to add1 and be more specific that word2 and should be something commonly found in situation that involves both words -- in the case that word2 is more general that word1 the proyection should be word1, so the projection of dog and animal should be dog -- What is the projection of hard respect to clear? -- If you project word1 with respect to word2 and it gives word3 then the projection of word3 with respect to word2 should be word3 -- what is the projection of dog with respect to animal? -- Now we define the reflection of word1 with respect to word2 so that first you compute the projection of word1 with respect to word2 given word3 and now you compute a word that is the poryection of word1 with respect the opposite of word3 -- the projection of dog with respect to "not-dog" should be nothing because there is no common thing for those words -- what is the projection of hard with respect to clear? -- what is the reflection of hard with respect to clear -- The projection p(w1,22) should never mean the contrary of w1 -- the reflection r(hard,clear) should be thing that are hard but not clear -- r(hard,clear) = something difficult to learn -- r(rich,honest) -- give me an insightful example of reflection of two words -- you should put more emphasis in the first word, so the reflection should clearly imply intelligence is that is possible -- I don't think that timid implies intelligence in general -- you are getting better at grasping the concept of reflection -- Give me another insightful example of reflection -- you are still not putting enough emphasis in the first word, the solution should mean creative without any doubt -- you use the word may be, but it should be necessary, so spontaneous doesn't imply creative -- that is better -- give another example of reflection -- Reflection and projection could be well used when word1 could be defined as having a small number of characteristics and projection and reflections help us to understand better the meaning of that words and how it interact with other words -- Give me an example of reflection in which the first word is composed of a small number of useful characteristics and word2 is related to one of those characteristic in a positive or negative way -- That is not correct when you project efficient over inefficient there is nothing in common, so the result is nothing. then the negation of nothing is all and the projection of efficient with respect to all is efficieent -- the p(w1, all) is w1 since you look for commonalities but also inside of the meaning of w1 that is more specific or smaller in grade or cuantity that ww1 -- Explain to me the concept of projection used in our conversation -- some axioms p(w1,w2) = w3 then p(w3,w2) = w3 -- You should be able to recover the meaning of w1 if you know the projection of w1 with respect to some other words. Give me an example in which you project w1 over two words w2 and w3 and then you try to recover the meaning of w1 using the computed projections -- To be useful you should try to project w1 with respect to words that are not very similar to w1 so you capture a little of information in each projection and then from those tips you can recover later the meaning of w1 -- you should look for words such that the projections over such words capture the complete meaning of w1 widout ambiguity -- give me an example using w1 and other three words for the projection using the indication of above -- give me the words to project rich and recover the meaning of rich using 5 words to project rich -- You don't know the projection of rich over opulent because some rich people are not opulent -- use 5 words to project the word love -- r(love, passion) -- the reflection r(w1,w2) does not have any of w2 characteristics -- r(love,passion) -- you say not as intense but it should mean necessary not having -- it should me love without passion -- that's correct ~ ~ ~

nothing0001

vor 3 Jahrendiscuss

Just make a loss function that gives +1 for correct answer, -1 for incorrect answers and 0 for unknown. Since this idea took me like 10 seconds, I suppose something like this must/should have been used before.

Unrelated, what happen when we train a model using a discontinuous function? Could the trained model be used to detect some pattern in the data?, for example if the input vector to such a model is a direct sum of two independent variables could such model be used to detect that the problem can be decomposed in two independent problems. Sorry of being off-topic and thinking aloud.

Edited: The following result is from (1): In summary, while softmax classifier probabilities are not directly useful as confidence estimates, simple statistics derived from softmax distributions provide a surprisingly effective way to determine whether an example is misclassified or from a different distribution from the training data, as demonstrated by our experimental results

(1) https://arxiv.org/abs/1610.02136

nothing0001

Submissions

Teaching GPT Word Reflections

[untitled]

comments