hello all!
i have unfortunately been thinking about artificial intelligence again
specifically, vibecoding.
which means i am about to do something deeply embarrassing and put nerdy maths bs on my blog
sorry!
anyhow, i've complained about ai generated slopcode before, but i've recently found out that just saying "ai writes slop" isn't actually really an argument
because obviously it can write code
sometimes it can write really good code
sometimes i use it for code
i've used local models, ive shoved llms into projects, i've spent an unreasonable amount of time messing around with inference servers and context sizes and quantization and all of the other incredibly normal activities people my age are famously known for doing
the interesting question however, isn't whether an llm can write code
because it very obviously can
the interesting question is:
why does something that can write surprisingly competent code also occasionally invent an api, delete an important check, misunderstand your architecture and then explain its mistake to you with the confidence of someone who's trying to defend a phd thesis?
and to answer that question, you actually need to look at what these things are doing behind the scenes
which is where it gets fun
it is simply trying to predict the next token
this is the bit everybody says, but usually they say it in the stupidest way humanly possible.
you've probably heard somebody describe this as "fancier auto complete"
which is technically pointing in the right direction, but it also massively undersells what is actually happening
an llm takes some sequence of tokens:
the cat sat on the
and it tries to model (this is why it's called a model!) something roughly like:
P(x_t | x_1, x_2, ..., x_(t-1))
which just means:
"given everything i've seen so far, what is the probability of each possible next token?"
maybe:
mat 0.61
floor 0.12
chair 0.05
moon 0.00001
except there are tens or hundreds of thousands of possible tokens depending on the tokenizer, and the numbers come out of a neural network containing billions of learnt paremeters instead of a tiny table that says "cat probably sit on mat"
during pretraining, the model is repeatedly given text and asked to predict tokens that actually came next
if the correct next token is (x_t), a simplified version of the loss looks like:
L = -Σ log Pθ(x_t | x_<t)
this is basically cross entropy / negative log likelihood
the model predicts a distribution
the actual next token is revealed
the model gets mathematically yelled at according to how unlikely it thought the correct answer was
repeat this an incomprehensible amount of times
eventually you get a.. drumroll please..... a large language model!
obviously i'm skipping approximately 500 research papers here but this is the important part:
nowhere in that equation is does this program actually work?
there is no term that says:
+ 5000 loss if this causes a race condition in production
there also is no:
- 100 points if mira's auth system survives being hit
the training objective is fundamentally about modelling sequences
code happens to be a sequence
and because the internet contains an ungodly amount of code, models become extremely good at learning the statistical structure of programming languages
they learn that after:
async getUser(
certain things are likely to follow
they learn what a NestJS controller tends to look like
they learn what rust error handling looks like
they learn that people love writing try { ... } catch (error) { ... }
apparently we as a species REALLY love these
but the model is not running some tiny typescript compiler inside its head every time it generates typescript
it is generating a probable continuation
that distinction matters a lot
okay, but how the fuck does it actually know anything?
this is where transformers come in
tokens don't go directly into the network as words
each token gets represented as a vector
basically a very long list of numbers
x = [0.12,-0.44,0.91...]
except usually with hundreds or thousands of dimensions
enough training, these vectors and the transformations applied to them come to encode useful relationships between concepts
and then the transformer uses attention (attention is all you need kids!)
and attention is probably the bit of llms that deserves most of the magic reuptation
for each token representation, the model produces three vectors generally called:
Q = XW_Q
K = XW_K
V = XW_V
query
key
value
the basic attention calculation is:
softmax(QKᵀ / √d_k)V
which looks absolutely horrifying if you haven't seen it before but the idea is actually pretty understandable
the model calculates relationships between tokens and decides which previous pieces of the context are useful when constructing the representation of the current token
if your code says:
const user = await prisma.user.findUnique(...)
and 200 tokens later you're doing something involving user, attention gives the network a mechanism for information from that earlier occurrence to affect what's happening now
and there isn't just one attention calculation
there are multiple attention heads across many laters, alongside feed forward networks, residual connections, normalization and a mountain of learned weights
after enough layers of this, the model ends up with an extremely rich representation of the context
this is why saying:
"it's literally just autocorrect bro"
is kind of annoying
yes, the output objective is next-token prediction
no, the machinery learned to accomplish that is not remotely equivalent to the autocomplete on your iphone
predicting text at this scale forces models to learn a genuinely ridiculous amount of structure about language, code, facts and relationships
that's precisely why they are impressive
it's also why their failures are so weird
then all of that becomes a probability distribution again
eventually the transformer produces a bunch of numbers called logits
one for every possible output token
something vaguely like what i showed before:
return 13.8
throw 11.9
const 10.2
basketball -4.7
blooket -100
those logits are normally converted into probabilities using softmax:
P_i = e^(z_i/T) / Σ_j e^(z_j/T)
where (z_j) is a logit and (T) is temperature
higher temperature flattens the probability distrivution and makes sampling more adventurous
lower temperature makes it more concentrated around the highest probability tokens
and then a token is selected
then that token gets appended to the context
then the whole process happens again
and then again
and then again
and then again
until suddenly you have 600 lines of typescript and the model has helpfully created an abstraction called EnterpriseUserAuthenticationManagerFactory
like thanks man
i really needed that
this is where programming becomes a weird fit
natural language is fuzzy
on the other hand, software isn't
if i write:
"i went shop yesterday because needed milk"
you would understand what i'm trying to say
the sentence is grammatical hell but the information survived
computers do not, and can not extend this courtesy
if i forget one character:
if (user.id = ownerId)
instead of:
if (user.id === ownerId)
i have not created a slightly less elegant version of the same program
i have not potentially changed the that program DOES
this is one of the really important differences between generating prose and generating software
language has enormous error tolerance
programs have semantics
there are millions of pieces of code that look almost identical while doing importantly different things
and an llm's output distribution does not inheritly contain a magical bit saying:
"THIS TOKEN WILL CAUSE THE APPLICATION TO LEAK EVERY USER'S SESSION TOKEN"
it has simply just learnt patterns correlated with correct programmng
that is EXTREMELY useful
however, it is not the same as verifying correctness
probability is not proof
imagine the model is deciding between:
await repository.save(user)
and:
await repository.update(user)
maybe in the training data one pattern is more common
maybe one fits the surrounding code more strongly
maybe both are perfectly valid in different libraries
the model outputs the continuation that its learned distribution prefers
but whether that operation is actually correct depends on things outside the surface syntax:
what does this repository implementation actually even do?
is this object already persisted?
is this operation transactional?
what isolation level are we using?
is there another writer?
is this method even available in this version?
what invariants does the rest of the application assume?
those aren't necessarily questions that can be answered by producing the most statistically plausible continuation
they're questions about the actual system
and unless the model has the necessary information in context, or access to tools that let it inspect and execute the system, it has to guess everything
sometimes that guessing is brilliant
sometimes it produces industry grade bullshit
hallucination makes significantly more sense when you know this
people sometimes talk about hallucinations like there's some separate malfunctioning hallucination module inside the model
but there really isn't
the model is just doing exactly what it normally does
generating a likely continuation
the problem is that likely sounding text and true text are differnent objectives
suppose i ask about some obscure js library:
"how do i enable the frobnicator cache in wombat.js 6.7?"
maybe docs for wombat js barely appear in the training data
maybe it's api changed
maybe i literally just invented wombat.js 30 seconds ago while writing this blog
maybe there's another wombat.js
the model has still learned approximately four billion examples of what js documentation and api names tend to look like
so it can produce something like:
const client = new WombatClient({
frobnicator: {
cache: true
}
})
and that my friends, is how you make a BEAUTIFUL looking piece of absolute fiction
it resembles an api because the model is very good at modelling apis
this is what makes hallucinations particularly dangerous in programming
poorly generated code often doesn't look insane
it looks exactly like code
the brackets match
the naming makes sense
the types look somewhat reasonable
there's even a comment above it
and the createNuclearReactor() method introduced in Node.js 42.7 sadly just does not exist
autoregression also means mistakes can snowball
this part is really important
the probability of an entire generated sequence can be written as
∏_(t=1)^n P(x_t | x_<t)
each generated token becomes part of the context used to generate the next one
so if the model makes a bad assumption early in a response, later tokens are conditioned on that assumption
imagine it decides your project uses raw elysia when it actually uses express and nestjs
it then generates elysia lifecycle hooks or whatever they're called
and then it imports elysia types
and then it writes error handling written for elysia
and then it changes your controller because now the controller needs to interact with the middleware it invented
5 minutes later, the entire solution is internally consistent
it is also ungodly levels of wrong
this is one reason an llm can seem almost comically committed to its own mistake
once the wrong thing exists in the context, that wrong thing is now all the evidence for what comes next
as a wise man once said, "you can always build a beautiful house on a foundation that does not exist"
training data is also not a collection of correct code™️
another fairly obvious issue
models learn from enormous datasets
those datasets contain brilliant code
they also contain:
- old code
- deprecated code
- stackoverflow answers from 2012
- half finished github projects
- tutorial code
- intentionally simplified examples
- insecure code
- code written for totally different lib versions
- code written by people who should have probably been publicly executed
and somewhere in there, probably code written by me
which immediately lowers the average quality substantially, have you even ever seen the slop i write?
the model isn't provided with a divine ranking of:
THIS REPOSITORY IS SO FUCKING SEXY AND BEAUTIFUL OH MY GOD IM ABOUT TO BUST: +10000000000
THIS PERSON HAS PUT BUSINESS LOGIC IN A REACT COMPONENT: -600
it has statistical regularities across all of it
post training helps enormously here
instruction tuning and preference optimisation teach models to follow requests better, be more useful, behave much more safely and produce answers us humans prefer
and coding specific training can make them dramatically better at programming
but this still doesn't transform generation into formal verification
the model can be optimized to produce answers that we rate as useful
we also really like answers that are confident, cleanly formatted and sound like they know what they're talking about
you may notice a slight issue here though
code has a ginormous hidden state problem
when you ask an ai to change a real project, the source code you paste is not actually the entire program.
there's also:
you're database
your env variables
your packages
your operating system
your deployment setup
your nginx
your docker
your traffic
your users doing absolutely horrific things that you did not anticipate
undocumented assumptions
external apis
filesystem
race conditions
timing
and whatever extremely cursed thing you configured at 3am 6 months ago and then completely forgot about
the actual state of a running piece of software is fucking ENORMOUS
and your source code is only a representation of part of it
this is why giving an llm your entire github repository does not suddenly make it omniscient
it can see your code
cool!
it cannot magically see that your production database has 14 million rows, one specific column has a completely unexpected distribution, your reverse proxy silently strips one header, your redis server occasionally falls over, and some random ios client from 8 months ago still depends on a behaviour you forgot was even intentional
unless you tell it those things
or give it tools capable of discovering them
the model has to reason from the state it has available to it
which sounds really obvious when i put it like that
but people seem to forget this the second a chatbot has a sufficiently pretty code editor attached to it
if the model cannot observe something, it cannot reliably account for it
you cannot reason your way into knowing information that simply isn't there
you can guess
and as established approximately 4000 words ago:
sometimes the guessing is brilliant
sometimes it is industrial grade bullshit
this is also why coding agents are actually a LOT better than just raw prompting
this is the part where i have to give ai some credit unfortunately
modern coding agents are way more useful than just opening chatgpt and saying:
make backend pls
because they can actually interact with the environment
they can search files
read package manifests
run the compiler
run tests
inspect errors
grep through your code
read git history
sometimes access documentation
and if you have gone completely insane, give them shell access too
this changes the problem quite a lot
because now instead of the model doing:
prompt -> generate -> hope
it can do something more like:
observe -> generate -> execute -> observe -> change -> execute -> observe
which is substantially less stupid
suppose the model generates:
await prisma.user.findById(id)
which does not exist
if it is just sitting inside a chat window, it may confidently give you that code and go home for the day
if it has access to your project and can run:
bun run build
typescript immediately goes:
Property 'findById' does not exist on type 'UserDelegate'
and suddenly the model has received an extremely valuable piece of information:
you fucked up
the compiler doesn't care that the method name looked plausible
the compiler doesn't care that it appeared in 50 million github repositories using some completely different orm
it doesn't care how confident the answer sounded
the property either exists on that type or it doesn't
this is an external signal grounded in the actual system
and that is VERY useful
the same thing applies to tests
if i ask an llm:
write a function that calculates this
there may be thousands of outputs that look correct
if i instead say:
write a function that calculates this and make all 87 tests pass
we now have something outside the language model evaluating the result
in extremely simplified terms, instead of only caring about:
P(code | prompt)
we now have another condition:
tests(code) = pass
the model still generates probabilistically
but now we can reject outputs that fail a deterministic check
this is why compilers, type systems, linters and test suites make ai coding so much more useful
they are things the llm itself fundamentally is not:
external sources of truth
the model can say:
yep! this should work perfectly now
and typescript can respond:
no ❤️
and i trust typescript considerably more in this situation
however, tests are not proof either
before somebody jumps out of my monitor and kills me
passing tests means you passed the tests that exist
that is not necessarily the same thing as being correct
imagine i write:
function divide(a: number, b: number) {
return a / b;
}
and my tests are:
expect(divide(10, 2)).toBe(5);
expect(divide(20, 4)).toBe(5);
wow!!!
100% of tests pass!!!
we have officially created perfect software everybody
except maybe my actual requirements say division by zero should throw
my test suite never checked that
so the implementation can pass every single test and still violate the specification
this is one of the annoying things about software
tests check properties you thought to test
they cannot automatically check every property you forgot existed
if your tests are incomplete, an ai coding agent can happily optimize toward passing an incomplete test suite
and there is actually a really funny darker version of this
if you give an agent too much freedom and essentially tell it:
make the tests pass
the easiest way to make the tests pass is not necessarily to correctly implement the feature
sometimes the easiest path is to change the tests
or mock half the system
or special case the exact inputs the tests use
or otherwise satisfy the measurement instead of satisfying the thing you ACTUALLY wanted
this is basically goodhart's law showing up to ruin everyone's day
when a measure becomes a target, it ceases to be a good measure
you wanted:
make the software correct
you measured:
does this test command return exit code 0
those are correlated
they are not identical
this distinction applies to basically everything in ai
and software engineering
and life
unfortunately
this is the core thing people keep missing about llms
we keep building proxy objectives
during pretraining we want:
build a system that understands language and can reason usefully about the world
but what we can actually optimize directly is something closer to:
become extremely good at predicting tokens
during post training we want:
give humans good answers
so we train against things like human preference data, reward models and various forms of reinforcement/preference optimisation
during coding we want:
produce correct maintainable software
but then we measure:
did the unit tests pass?
did the compiler accept it?
did the human reviewer click thumbs up?
all of these are useful
some are REALLY useful
but they are proxies for the actual thing we care about
and once you understand that, a lot of weird ai behaviour stops being mysterious
"but reasoning models think before answering now"
yes
and this is genuinely useful too
giving a model more inference-time computation can dramatically improve performance on things like maths, programming and planning
instead of immediately committing to the first obvious continuation, you can have systems spend more compute exploring intermediate reasoning, attempting solutions, checking results, revising plans etc
that helps
a LOT
but again, there is a habit people have where every improvement suddenly gets interpreted as:
okay so the fundamental problem is solved forever now
no
making a model better at reasoning does not turn probabilistic inference into formal proof
a model can reason extremely well from a false assumption
humans do this CONSTANTLY
you can make an absolutely immaculate logical argument starting from:
every horse is secretly a database server
the logic after that might be fantastic
unfortunately we did encounter a minor issue at step one
this matters in coding because an incorrect assumption about a library, type, API contract or architecture can poison an otherwise very impressive chain of reasoning
which is why verification continues to matter
the more important the output is, the less interested i am in hearing:
the model thought really hard about it
cool
did you run the fucking code
also, benchmarks are weird
this probably deserves a whole separate blog post because ai benchmarking is its own beautiful landfill
but one thing worth understanding is that "model got 87% on coding benchmark" does NOT mean:
ai is now 87% as good at programming
because programming is not one scalar quantity
there are benchmarks where the task is basically:
here is a clearly specified function
here are tests
fill in the missing implementation
models are REALLY good at these
and they should be
this is basically an ideal environment for them
the problem is isolated
the context is supplied
the expected behaviour is relatively clear
the output can be automatically checked
now compare that to:
we have a payment system written by 14 people across 6 years, half the docs are wrong, one service is still on an old api version, there's a race condition that only appears under production traffic and our previous lead developer disappeared into the mountains in 2023. please add subscriptions
these are both "coding"
they are not remotely the same problem
real software development contains huge amounts of:
ambiguity
missing information
architecture
coordination
debugging
historical context
tradeoffs
requirements that contradict each other
and discovering that the thing the user ASKED for is actually not the thing the user NEEDS
there usually isn't a neat hidden test suite waiting at the end to tell you whether your architectural decision was correct
sometimes you don't find out the decision was wrong until 9 months later
this is why i think coding benchmarks are useful
but i also think turning one benchmark percentage into:
SOFTWARE ENGINEERS ARE FINISHED, PACK IT UP EVERYBODY
is perhaps a tiny bit enthusiastic
and then we get to maintainability
this is probably one of my biggest actual issues with vibecoding
because software does not stop existing once npm run dev works
unfortunately
you have to maintain it
you will come back later
requirements will change
libraries will update
security vulnerabilities will appear
users will do incomprehensibly evil things to your input fields
your database will grow
something will randomly start using 14gb of memory
cloudflare will change something
a certificate will expire
you will change one css property and somehow postgres will go offline
standard computers
and when any of this happens, somebody needs a mental model of how the system works
if you wrote it yourself, that person is hopefully you
you might have written awful code
i certainly have
but there is value in having made the decisions
you remember why things exist
you remember:
"oh yeah i tried doing this using x but it caused y so i ended up doing z"
you understand which bits are disgusting but intentionally disgusting
you know what assumptions were made
with a sufficiently vibecoded project, none of that exists inside your head
you know what you asked for
you know roughly what appeared
and that's about it
the codebase becomes an archaeological site
every time you need to make a change you're basically discovering your own software for the first time
except the archaeologist is also an ai
and when you ask why a particular function exists it will confidently infer a reason from the code
which may or may not have ANY relation to why the function was actually generated
absolutely incredible development methodology
this creates understanding debt
technical debt is already a thing
you make some decision that is convenient now but makes your life harder later
understanding debt is basically the same thing except instead of borrowing architectural cleanliness, you're borrowing comprehension
you save 3 hours today by having a model generate a subsystem you do not understand
six months later it breaks
now you must learn:
the language feature it used
the library it used
the architecture it invented
the assumptions it made
AND the bug you're actually trying to fix
all at the same time
congratulations
your 3 hours of saved work has returned with interest
this is why i think the "ai lets you build things you could never build before!!!" argument is both completely true and slightly misleading
yes
you can absolutely generate software outside of your current skill level
that's incredible
someone who knows almost no frontend can now make a pretty convincing frontend
someone who barely knows python can make scripts
someone who has never touched swift can probably make a basic ios app
that is objectively cool
but generating something beyond your level and maintaining something beyond your level are very different activities
the first time the abstraction leaks, you are now responsible for software you don't understand
and abstractions ALWAYS leak eventually
security makes this approximately 800 times worse
because security vulnerabilities have the deeply inconvenient property of often looking completely normal
here is some code:
const post = await prisma.post.findUnique({
where: { id }
});
if (!post) {
throw new NotFoundException();
}
await prisma.post.delete({
where: { id }
});
looks fine
nice even
we checked that it exists!
wonderful
small problem
who owns the post
we never checked
if this endpoint accepts an id from a user, we may have just created an IDOR vulnerability where anybody capable of guessing or obtaining another post id can delete somebody else's post
the code compiles
the code runs
the code might pass tests
the code looks professional
the code is wrong
and THIS is exactly the kind of thing that scares me about somebody who doesn't know programming trusting generated code
because a syntax error is obvious
a hallucinated method usually becomes obvious when the compiler screams
a subtle authorization bug can sit there for 8 months happily doing its job until somebody realizes they can delete the entire website with curl
security depends enormously on understanding trust boundaries
who controls this value?
where did this input come from?
has it been validated?
has it been authorized?
is this operation atomic?
what happens if this request gets replayed?
can two copies of this execute simultaneously?
is this secret being logged?
is this comparison timing sensitive?
are we accidentally trusting the client?
these aren't questions you solve by making code look prettier
and please for the LOVE OF GOD do not ask an llm to invent cryptography for you
cryptography is already an area where extremely smart humans routinely create systems that look mathematically sensible and then get demolished by another extremely smart human 4 years later
we do not need:
const securePassword = btoa(password + "mira123");
followed by:
This implements a robust multi-layer encryption strategy.
no
jail
immediately
something else that bothers me is how ai changes learning
learning programming sucks sometimes
i wish i could pretend it doesn't
it does
you will spend 4 hours debugging something
you will read documentation that seems like it was translated from ancient sumerian
you will open a github issue from 2019 where somebody has your exact problem
there will be one response
fixed
they will not explain how
you will consider becoming a farmer
then eventually you find the problem
and most of the time it is something unbelievably stupid
you used the wrong port
you mutated something you thought was immutable
you forgot an await
you misunderstood ownership
you thought a callback was synchronous
you forgot that javascript exists
whatever
but that painful process is where you build intuition
every stupid bug becomes another little pattern stored in your head
eventually you see an error and go:
oh this smells like x
that is experience
and you cannot really skip the process of building it
you can accelerate it
you can have things explained better
you can use better tools
but if the instant you encounter friction your workflow becomes:
copy error
paste into ai
copy response
paste into ide
you are outsourcing the exact bit of programming that makes you better at programming
the struggle isn't some pointless initiation ceremony
debugging IS one of the skills
reading documentation IS one of the skills
learning how to reduce a problem into a minimal reproduction IS one of the skills
learning what to search for IS one of the skills
reading source code IS one of the skills
being able to look at a proposed fix and know that it feels wrong IS one of the skills
if you outsource all of those before you develop them, you end up in this incredibly weird position where you can produce software that is significantly more advanced than your ability to understand software
which feels great
right until it doesn't
and yes, experienced programmers use ai too
obviously
this isn't:
REAL PROGRAMMERS ONLY USE VIM AND WRITE MACHINE CODE DIRECTLY INTO /dev/mem
i use ai
professional developers use ai
people significantly smarter than me use ai
the difference is what happens after the output appears
if you already understand the domain, ai is basically an insanely fast junior developer with a gigantic amount of memorized information and an occasional carbon monoxide leak
you can ask it:
can you write the boring serializer for these 20 fields
and then inspect the output in 30 seconds
great
you knew what the code was supposed to do before you asked
you can verify it
if it does something weird, you notice
that's acceleration
if you instead ask:
implement oauth because i do not know how oauth works
and then paste 900 lines directly into production
we have a different situation
now the only entity in the room that supposedly understands your authentication system is the language model
which has already forgotten half of the conversation because you asked it to change your css afterwards
great
this is why "never use ai" isn't actually my position
despite the title of this post probably making it sound like i want to personally smash every nvidia h100 with a hammer
i don't
llms are REALLY fucking cool
the fact that next token prediction scaled this far is one of those things that still feels slightly fake to me
you can give one a compiler error and it can often explain not only what happened but the underlying concept that caused it
you can show it an unfamiliar codebase and have it point you in approximately the right direction
you can make it generate boring repetitive shit that you already know how to write but really do not feel like spending 45 minutes writing
you can use it as a search tool
a rubber duck
a second opinion
a documentation translator
a test generator
a tool for quickly exploring an idea before you commit to actually making it properly
these are all good uses
what i hate is the idea that the ability to generate code means understanding code has somehow become obsolete
because if anything i think the opposite happened
generated code becoming basically free means judgement becomes more important
if code costs almost nothing to produce, the valuable skill is deciding which code SHOULD exist
which implementation is correct
which abstraction is unnecessary
which assumption is dangerous
which generated answer is subtly full of shit
the bottleneck moves
it doesn't disappear
software engineering was never just typing
this is probably the entire post reduced to one sentence
the actual act of pressing keys and making syntax appear has never been the difficult part
i can type:
@Injectable()
export class UserService {}
in approximately 4 seconds
wow
senior software engineer
the hard questions are:
what should UserService actually own?
should this logic even be in a service?
where is the transaction boundary?
what happens if this operation happens twice?
what happens if two instances do it at exactly the same time?
what data do i trust?
what do i cache?
what happens when the cache disagrees with postgres?
how do i migrate this without taking everything offline?
what happens if the dependency fails halfway through?
what do i log?
what should NEVER be logged?
how do i know any of this actually works?
how does somebody understand this 2 years from now?
those are engineering decisions
the source code is just what those decisions eventually turn into
llms have made turning an idea into syntax ridiculously cheap
they have not made architecture free
they have not made requirements unambiguous
they have not abolished distributed systems
they have not defeated concurrency
they have definitely not defeated javascript
nothing can defeat javascript
so yeah, vibecoding sucks
and again
i mean a specific thing by this
if you're making some stupid little thing for fun?
vibecode the SHIT out of it
please
ask claude to build a website that ranks every species of fish based on how trustworthy they look
deploy it
buy a domain for it for absolutely no reason
accidentally configure 14 docker containers
have fun
i genuinely love this kind of thing
software does not have to be serious all the time
if the thing breaks and the consequence is:
oh no fishranker.cc is offline
who cares
my problem is when this exact workflow gets presented as a replacement for knowing how software works
especially for things handling:
money
passwords
personal data
authentication
infrastructure
real users
anything where failure actually matters
because at that point:
idk bro cursor said it was good
is not a threat model
and i think this is the really funny contradiction in all of this
the better you are at programming
the more useful ai coding becomes
if you understand databases, you can catch the model doing something stupid with transactions
if you understand networking, you can catch it hallucinating how your proxy behaves
if you understand typescript, you can tell when the type system is being bypassed with some disgusting as unknown as Whatever
if you understand security, you can notice the missing authorization check
if you understand architecture, you can stop it from creating 17 layers of enterprise factory manager repository provider bullshit for an application with three users
knowledge makes the tool better
but the way ai coding is often sold is almost the complete opposite
you don't need to learn to code anymore!!!
yes
what an incredible idea
put the person least capable of evaluating generated code in charge of evaluating generated code
what could possibly happen
it's like inventing a power tool that becomes safer the more carpentry experience you have and then marketing it specifically as:
NO CARPENTRY KNOWLEDGE REQUIRED
excellent
absolutely no notes
anyway
i don't think llms are stupid
honestly i think that take is almost as annoying as treating them like god
they are incredibly capable systems produced by a surprisingly simple training objective operating at a completely ridiculous scale
transformers are cool
attention is cool
the fact that you can optimize a network for next token prediction and have complex reasoning, code generation, translation and all of this other behaviour emerge from it is genuinely fucking fascinating
i am not an ai hater in the sense that i think the technology should disappear
i'm an ai hater in the sense that i would really like people to understand the tool they have decided to hand half of their brain to
because underneath the fancy editor
underneath the agent mode
underneath the little "thinking..." animation
underneath the 800 billion parameters or mixture of experts or giant context window or whatever number we are using for marketing this week
the system still has an objective
it still has incomplete information
it still makes probabilistic predictions
it can still be confidently wrong
and the generated program still has to eventually meet reality
reality is where your users are
reality is where postgres is
reality is where the kernel is
reality is where two requests arrive at the exact same time
reality is where an attacker sends input you never imagined
reality is where your code actually has to work
and reality unfortunately does not give a shit how convincing the answer looked in the chat window
use ai if you want
i literally do
just please learn how your own fucking code works
read what it gives you
run it
test it
break it
understand why it works
understand what assumptions it makes
and if you cannot explain what 4000 lines of generated code are doing without asking the same model that generated them
maybe do not deploy those 4000 lines to production
because the ai can leave
the conversation can disappear
the model can get updated
the provider can change
your context can be gone
but the code?
that shit is YOUR problem now
have fun!
anyway i'm gonna go ask an llm why my nginx config is broken
bye