finally understand relative self attention

gonna try to implement that correctly this time

so far though, generation is sounding pretty great

fine tuning rn, will see if that helps at all

July 28, 2023

model loss is still not plateauing

a little scared that is is going to overfit

paused training to listen to generation, output was not great 😔

not really sure why though, since loss is lowest its ever been, and the don't remember changing loss function before i started training

i wonder how many pg essays i can read in one workday

gonna start grinding leetcode

want to do ~2 problems a day

fun side project could be tik tok generator

could replace pinkydoll

friday in the 4HL, can really feel the pull of the weekend

July 27, 2023

> woke up this morning

> loss has not even come close to plateauing

> LETS GOOOOO

hopefully it doesn't take too long though

need to learn what a superconductor actually is

also need to set up SDXL

July 26, 2023

tried using relative attention last night, but didn't seem to work

going to re implement by myself (still in torch)

current implementation has too much abstraction

first embedding layer might be bottleneck

weight initialization is probably wrong

how come no one told me about gradient checkpointing

now i have infinite GPU memory

250M PARAMS😲

450M PARAMS

GIVE ME MORE

July 25, 2023

trained overnight for ~8 hours

definitely getting somewhere though

making block size larger might have an effect on some of these

also gonna look into relative attention like in original paper

> generally pretty solid

I am dumb as hell

all I had to do was sample from the distribution, instead of taking most likely event at inference

sounds 1000x better

July 24, 2023

stuff to fix:

July 23, 2023

Oppenheimer was so good

Barbie was a big letdown

finished the model

basically is just Karpathy’s minGPT with a custom loss function

and inference is different obviously

rn I pretty much have the biggest model I can run without out of memory error on gpu

im sure I could figure a solution out though if needed (to get a bigger model)

July 21, 2023

todo:

today's goal is actually just to train one iteration

also want to read a bunch of Exit, Voice, and Loyalty – didn't read at all this morning

maybe after piano music, try to recreate sounds of "MoogTube" playlist on spotify

July 20, 2023

bro why did no one ever tell me about coffee

played with some tts tools yesterday

gonna focus on music generation first though, want to try to finish *something* before school

i should build this

but actually make it seem like texting a girl

gpt4 is way too good

had no idea how to write script to take midi file and convert it to a vector to use as training data

and it instantly got it completely perfect

i thought preparing the data was gonna be such a pain

lets go

July 19, 2023

got llama-2 running last night, gonna try out 70B today since 13b was really fast

trying to find best tts model, none of the good ones are open source though

need to clone ScarJo's voice to be like Her

Got two new books:

July 18, 2023

main problem with music generation is you have to encode a bunch of stuff for a single note

with text, its just character after character

idk, will finish those papers today

llama-2 released, gonna get that running later

July 17, 2023

Why hasn’t generative music caught on

I’ve seen people try it with GANs before, but never with transformers

going to try to automate as much work for internship as possible

wondering if someone has made a tool to generate requirement docs (word docs)

>be me

>meet with client about changes to program

>everything looks good, client makes small adjustments

>come in to work today, boss wants to meet with client

>now i have to wait, have nothing to work on

is that how greentext works, im not really sure

sike naw it got approved

lets gooooo, i can work on sum

Wondering if tiktok has an upload api

Could create infinite stim videos

forgot to bring airpods to work😭

ideas of stuff to build:

also would be cool to do something in decentralized protocols

there is probably a lot of money to be made making AI girlfriends

some interesting papers:

July 16, 2023

Got a sick domain for essays site

July 15, 2023

testing upload to notes from site

Would be cool to have macro keyboard

Uses:

need to start drinking coffee, so much untapped alpha

Need to be working way harder

July 14, 2023

cool papers is shipped

visit here:

lmk what you think

July 13, 2023

new monitor

hell yeah

gonna try to finish and ship cool papers today

and maybe this too

lgtm

ok not gonna ship but basically done

will ship this tho

shipped

July 12, 2023

welcome

these are my notes, straight from my brain to yours

wrote a cool script to upload these from my terminal

just set up linux the other day on new pc

Mishima is such a good movie (philip glass did the music)

current projects:

need to learn more about quant models before trading bot tho

assume I could just use a plain ANN or transformer to make predictions on whether price is higher or low for the next day?

would be fun project, def won't work though

I need to do something cool with this gpu

project I forgot: