seems like my toy classical music generator's architecture is actually fairly similar to meta MusicGen

both are basically pure transformers (just the decoder)

maybe scale/tokenization method was limiting factor

if I want to make abelton/garageband type tool, will need to be able to generate single layer of a song in midi

should be fairly simple, since midi file has different layers for each instrument

December 30, 2023

gonna start doing more music gen stuff

feels like music is way behind image generation for no good reason, since the models are probably quite similar

main difference is probably the temporal aspect of music, but that has been solved in text generation

so there must be some intersection between both kinds of models that would fare really well for audio

how fun would GarageBand be if you could generate certain layers of a track, without having to know any theory

where is the midjourney/ChatGPT (consumer facing, high level tool) for music?

new strategy:

always have some big project to work on

changing projects is totally fine, but trying to think of new project by not working is not

current project: tinygrad contribution (may change to music gen once I have my pc again)

re: tinygrad

working on moe model to get experience with high level api

December 28, 2023

didn't finish moe model

got distracted and watched great movie: the moment of truth

i need to go to a bull fight

updated reading log page, looks way better now

moe model is like 90% done

its just buggin a lil bit

December 27, 2023

today: write moe model in tinygrad from scratch

going to use same tiny shakespeare dataset for simplicity

December 26, 2023

made first contribution to tinygrad!

although it was just a comment in a pr conversation

hopefully someone finds it useful, I wish it had been there a week ago when I first tried running it

December 24, 2023

always choose the option that requires the most agency

December 21, 2023

you can literally just become smarter by reading more

90% of the time raw intelligence is not as useful as deep understanding

December 20, 2023

wishing I had my gpu right now😢

December 18, 2023

December 17, 2023

need to be more low level

learn to write GPU shaders (for metal)

December 13, 2023

new goal: reasonable contribution to tinygrad

December 12, 2023

fine tuning model on text message data would be fun

not too sure where to get that tho

December 11, 2023

never mind, m1 is good enough

I just had some goofy architecture

training on tiny Shakespeare, seems to work pretty well

might try to implement moe later this week

probably wont be able to fine tune 7B tho

that would take ages

December 9, 2023

mlx.core.random.categorical is not the same as torch.multinomial, despite what docs may want you to believe

😭😭😭😭

took too long to figure that out

i know mlx just came out, but the docs are horrendous

# of freshman CS majors in ~4 years is probably going to plummet

maybe 2 years until no more trad swe entry level roles

math, physics, EE, nuclear will be where they go

this is because CS major is effectively just trade school

that trade will be 90% automated very soon

if trad software is solved, there are two options: (1)AI research/dev, (2)hardware

if nuclear makes a comeback, energy is third option

other than that, will be hard to find entry level job in tech

there will always be room for pure software startups, but they will rarely need large engineering teams like they do now

mlx on m1 chip is still not fast enough to train any kind of transformer

crypto-first venmo

who's building this

could be really useful overseas

venmo is almost digital cash, but not quite

December 8, 2023

people are overestimating how much search wi be replaced by AI

At least 1/2 of my searches are to a specific site, where I use some product

Search is not the same as information querying

first winter break project is training baby llm with mlx

would be fun to fine-tune some 7B model on tweets(farcaster casts)

December 4, 2023

text2movie

who is building this

realtime sd could have a lot of use cases

could make a sick game

dingboard for videos

would be professional software though

might be fun to work on

AI-first editing software

December 3, 2023

Would be fun to build something with this when code is released

December 2, 2023

should buy one of these

December 1, 2023

want to have >100 LC problems done before end of winter break

perhaps a few hards???

time will tell

November 29, 2023

should probably make a post about Mootr

the goal is to be as upstream as possible

current twitter following is doing a fairly good job of this

there are definitely still sources further upstream though

(who is being read by the smartest/most contrarian/most upstream people)

doesn't necessarily have to be people I agree with

haven't uploaded a picture in a while

November 25, 2023

putnam exam

November 17, 2023

stripe integration for scooter app should only take a weekend

November 16, 2023

PG says that best startup ideas come from being on the edge of innovation and looking for holes

contrarian ideas are almost certainly the same

every contrarian thinker(yarvin, thiel, etc.) had huge influences

so having innovative ideas out of the blue is not the correct mode to think about this

instead of sitting in a room and trying to think of new things, the idea is to just select the right influences

and the right influences are the people who are on that edge

(the edge is just the societal idea equivalent of scientific advances for technology)

although, they aren’t necessarily new like the technology side is

November 15, 2023

need to be more ambitious

scooter ride share app going fairly well

might not be a good business, but a light gpt wrapper that is for tutoring would be useful

a marketplace to find other students that will do your homework

November 6, 2023

built a prediction market for a hackathon, but it barely works since I wrote most the smart contracts at 4am after consuming large amounts of redbull

would be nice to clean it up and put it on website

there are surely still some cool undiscovered use cases for crypto

October 31, 2023

will have a lot more time now

ideally:

might be fun to build diffusion model from scratch

also could be fun to do some kernel level stuff, maybe jump back into C

maybe for pytorch game, could use webGPU to train tiny models w/ some visualization

October 25, 2023

brain seems to have fog only lifted by caffeine in the past few weeks

October 24, 2023

i have become complacent

is AI going to actually change everything?

seems like most of work people do is bureaucracy/process/bs

maybe in a hyper efficient market(one that we do not live in)

will definitely increase variance

income gap will increase massively(not necessarily bad)

AI has yet to produce novel ideas

seems to mostly be compression of all information

if intelligence is ability to be contrarian(scroll down), gpt models will probably not achieve AGI

appreciation of aesthetic values( in the harold bloom sense) will becoming increasingly rare but useless(professionally)

but in a world of abundance, liberal arts should be becoming more desirable, not less

there should be less need for "blue collar" type education(hard sciences, computer science)

"blue collar" in the sense that you are learning practical skills

there is so much free information available

the easy things to learn(practical skills[computer science, hard science, math]) would ideally be left for people to do on themselves

while harder things to learn(aesthetic values, love of pursuit of knowledge) should be taught by institutions who are specifically designed to teach

i think main problem is the mental model of "college = educated"

college should ideally be starting point for education, it should be where you learn how to teach yourself

once you graduate, you should not be classified as done(educated)

this used to be english university system(I think)

the smartest kids should be in liberal arts degrees, not engineering

because they are able to teach themselves the engineering

where the value of liberal arts comes out from discussion that is difficult to achieve alone

basically I am saying the value of college lies in the peer group

having a very smart peer group is much more useful for liberal arts-type learning than engineering is

therefore to maximize value of higher education, pursuing aesthetic values would ideally be prioritized

*but since its not, and the smartest kids are not in liberal arts, to have the smartest peer group you should study engineering

i need to think about this more

It is definitely possible that gpt4 and similar models can produce independent and contrarian thought, and that they have just been sanitized to make sure nothing too crazy is said

libertarianism is so interesting because a huge portion of people prescribe to it theoretically but not practically

I wonder what is that delta

i should read more classic american literature

hemingway, steinbeck, emerson

pynchon too

October 22, 2023

Would be fun if Snapchat had inpainting

maybe a project idea

need to add text editing for canvas app, then it would be pretty much done

chrome extension that changes page styling to early internet era vibe

pictionary that makes photorealistic image from inpaint sketch

October 17, 2023

babe wake up new pg essay just dropped

yet another banger

October 15, 2023

canvas solver is working pretty well

need to add image cropping

getting to a pretty usable point

might share w/ a few ppl

October 13, 2023

finished Hackers & Painters (first book ive finished since august☹️)

like halfway done with Confederacy of Dunces

built v1 of canvas quiz app, but not sure how useful it will be once GPT-V api is out

need new project, pytorch game is lacking in inspiration

October 9, 2023

I wonder what kinds of formats of text have stayed in the past

October 7, 2023

i should definitely be self studying more ML math in case

if a startup condenses working life into a couple of years, what does condensing 4 years of school look like

maybe 1.5 semesters?

ideal way to spend rest of semester:

need to cut the fat

I am becoming complacent

maybe I should posting on twitter more

seems clear that the most contrarian/independent thinkers are also incredibly well-read

October 6, 2023

is the most agentic version of myself in school right now

October 4, 2023

farcaster visualization is shipped and done

that took way too long

September 29, 2023

visualization is live at:

just have to add description and write post for website

also want to post on twitter/farcaster

September 27, 2023

got search by name/zooming working

farcaster vis probably 90% done

would like to ship this week

September 26, 2023

app that takes any website and turns it into podcast would be so useful

September 22, 2023

radius of a circle inscribed in unit circle and sqrt x