bought the caffiene, taurine, and l-theanine last night

apparently l-theanine has noticeable effects even when taken alone

time will tell

for supplements that "increase brain function" a lot of the literature just says it increase oxygenation

implying that oxygenation is way upstream of everything

being outside is the best supplement

January 31, 2024

finally got mnist diffusion up on website

that too way too long

it is still really slow

for the actual music app, i will have to actually learn how to host models

no way that took me 10 days to actually ship

i am not working nearly enough on this

January 29, 2024

never heard about these before

going to go vegetarian this week

January 28, 2024

saw a tweet about how you can compile cpp code into web asm

January 27, 2024

recognizing complacency in yourself might be the first step, but not the most important

January 25, 2024

i hate aws

January 24, 2024

got anki on my pc

goal is to be able to watch a French movie before summer w/o subtitles

or read le petite prince (this should be easier)

January 23, 2024

that is essentially the good outcome

bad outcome:

most orgs devolve into massive bureaucracies

standard of living slightly increases, but jobs become very mundane

most people are addicted to phones/entertainment a la Infinite Jest

honestly the main difference between the two is centralization

most decentralized = more people can use it how they want = free market = better for the masses

January 22, 2024

if agi actually really close, this is what I think

short term: white collar job market gets bad

wealth gap increases massively

basic standard of living also gets way better

long term: more artists, creators

some sort of UBI

January 21, 2024

out on the other side of aws hell, lambda is too slow (probably my fault)

gonna try something new

got a jank setup running flask on ec2

way faster tho

might grind out the whole post tonight

realized my youtube intake has drastically plummeted

consumption is still good if high quality (books, some movies, some podcasts)

you can buy caffeine extract, taurine, and glucuronolactone on amazon (stimulants used in redbull)

might cook up a home brew

writing with left hand is becoming easier

got the mnist post up, model is still kinda slow

nevermind, http means it doesnt work on prod

January 20, 2024

since model is so small, it actually runs on cpu relatively fast

so i don't need expensive gpu servers :)

time to break out the good ol' lambda function image that has pytorch installed

totally forgot about the pytorch game, that was a pretty cool project i should really finish

gonna write it in a flask server before i get bogged down in aws hell

January 19, 2024

need to be working way harder on music gen

this weekend will have demo of MNIST diffusion on website

i need to get some more posts on there

i haven't shipped in months

MNIST model trained

lets goooooo

results are pretty good, gonna scale it up a lil though

wondering the best way to host this

easiest would probably be something like replicate

recap on fast:

seems like i have a case of "singularity stress" (coined by yacine, i think)

January 18, 2024

agi is near, better prepare

although idk how to do that

purpose of this generation is to take us from where we are to limitless abundance once we have agi

all white collar work is completely automated in ~10 years

and that is conservative

anything that happens solely online will be automated within 5

next big step is robotics

after that, if implemented correctly(!), abundance is achieved

it’s time to build

for a couple years though, there is going to be mass unemployment

people will flock to trades, then that will fall

building wealth now is probably the most important thing you can do

as nice as libertarianism sounds, universal basic income is probably necessary in some form

open source ai is the most important thing to be working on

massive leverage in the hands of a few companies is not going to turn out well

January 17, 2024

day 4 of fasting

feeling pretty great

yesterday was definitely worse, I felt way more tired and weak

probably am going to do one more day

January 15, 2024

isn't college where you go to become radicalized

why is this not happening

feels like i'm missing out

day 2 of fasting

tired and fairly hungry, nothing too bad yet though

January 14, 2024

day 1 of the fast

feeling good so far

best way to understand math in ml paper is just derive everything yourself

gives you way better understanding when looking at the code

January 12, 2024

before i do diffusion model for my audio images, i'll start with mnist

seriously doubt i'll be able to train model on my local gpu, since images will be order of magnitude larger than mnist

time will tell

January 11, 2024

wonder if you could apply VAEs to text models

the latent vector would then not contain information about an image, but about some text

it would be the pure distilled information, like a thought

not sure whether you could actually do this, but having language model do the "thinking" in some latent space, and then translating that into english seems interesting

this latent information would be passed to the encoder block of the transformer

so the analog is first it will think up a solution in vector space, and then articulate it into words

really cool book i just found:

gonna take all notes this semester with my left hand

pretty sure by the end I’ll be totally ambidextrous

January 10, 2024

ai "devices"(humane,rabbit,etc.) are cool toy projects

if they cannot completely replace your phone, they are useless, and will be completely replaced by siri-like features on smartphones

i think the tipping point is when they start to prompt you (al la Her)

good video on diffusion models

January 8, 2024

demucs is so fast on gpu 🤑

should be able to have all train/test data ready by tonight

definitely need to look into which kinds of architecture to use (some kind of diffusion, but the actual specifics)

may have small problem in that the beginning and the end of a song usually wont have drums

i guess i could just delete the first and last n images tho

cbtm

January 3, 2024

goal for today is to write script that takes single audio file, and turns in into N spectrograms that are 10 seconds long

seems like a useful dataset to start with/train baby model on

done

on cpu, demucs runs at about 2x song duration

January 2, 2024

transcribing to midi is harder than I thought, especially for percussion

generating spectrograms with diffusion may work better

idk cbtm

once loop is generated, could then just transcribe that audio clip

so pipeline looks like this:

> get audio files

> separate into layers

> convert audio to spectrogram

> use img gen models to create new spectrograms

results from SD sound pretty good here

yeah training diffusion model on spectrogram is definitely the move

January 1, 2024

first step is getting the data

datasets below are okay, but i'll probably need to get some myself

will likely need model that turns audio into midi (which has already been solved)

these models work really well for audio recording of single piano, but more complex songs w/ multiple instruments may be difficult

end goal of data collection is to have discrete groups of midi files that just contain single ~instruments (drums, lead, rhythm)

midi approach should work perfectly for drums/percussion, lead/melody may need different strategy

seems promising

nevermind it breaks down with multiple instruments

there are ways to separate instruments though, just need to find open source model

pipeline now looks like this:

> get large number of audio files(mp3, wav)

> split them into track layers (voice, drums, melody)

> turn these into midi files

> train model on single type of track layer

seems to be sota oss model

demucs works but is very slow (might change when running on gpu)

problem is now that audio -> midi does not work for percussion, need to find new model

December 31, 2023

seems like my toy classical music generator's architecture is actually fairly similar to meta MusicGen

both are basically pure transformers (just the decoder)

maybe scale/tokenization method was limiting factor

if I want to make abelton/garageband type tool, will need to be able to generate single layer of a song in midi

should be fairly simple, since midi file has different layers for each instrument

December 30, 2023

gonna start doing more music gen stuff

feels like music is way behind image generation for no good reason, since the models are probably quite similar

main difference is probably the temporal aspect of music, but that has been solved in text generation

so there must be some intersection between both kinds of models that would fare really well for audio

how fun would GarageBand be if you could generate certain layers of a track, without having to know any theory

where is the midjourney/ChatGPT (consumer facing, high level tool) for music?

new strategy:

always have some big project to work on

changing projects is totally fine, but trying to think of new project by not working is not

current project: tinygrad contribution (may change to music gen once I have my pc again)

re: tinygrad

working on moe model to get experience with high level api

December 28, 2023

didn't finish moe model

got distracted and watched great movie: the moment of truth

i need to go to a bull fight

updated reading log page, looks way better now

moe model is like 90% done

its just buggin a lil bit

December 27, 2023

today: write moe model in tinygrad from scratch

going to use same tiny shakespeare dataset for simplicity

December 26, 2023

made first contribution to tinygrad!

although it was just a comment in a pr conversation

hopefully someone finds it useful, I wish it had been there a week ago when I first tried running it

December 24, 2023

always choose the option that requires the most agency

December 21, 2023

you can literally just become smarter by reading more

90% of the time raw intelligence is not as useful as deep understanding

December 20, 2023

wishing I had my gpu right now😢

December 18, 2023

December 17, 2023

need to be more low level

learn to write GPU shaders (for metal)

December 13, 2023

new goal: reasonable contribution to tinygrad

December 12, 2023

fine tuning model on text message data would be fun

not too sure where to get that tho

December 11, 2023

probably wont be able to fine tune 7B tho

that would take ages