Brief Notes on Ilya30u30 Papers 1-3/30

Summary of notes

My Notes:

• attn is all you need
  • k q v is super super cool, my understanding is: q is "what is the ctx of token a", k is "given the context, how relevant is token b to token a", v is "how does b change the meaning of a, proportional to how relevant it is to a in this context" so k*q*v => change to meaning of token
• unreasonable effectiveness of RNNs
  • RNNs are apparently turing complete, based
• complextrophy
  • sophistication seems eerily like compression
  • oversimplified sophistication def according to my understanding: sophistication of str s = smallest size of program that always computes a random elem of S (s parent set) given a criteria for S / size of S
  • random string: small program to compute / large set size = low sophistication. Cleo integral solution : giant program to compute / small set of possibilities : high sophistication