Computers process audio in blocks, not sample by sample. The buffer is that block, and its size is the knob every producer eventually learns: 64 samples for playing instruments live, 1024 for mixing heavy sessions without crackles.
The tradeoff is mechanical. Small buffers mean the machine must finish its work in tighter deadlines, risking dropouts; large ones add delay, since audio waits for its block to fill. At 48 kHz, a 512-sample buffer is about 10.7 milliseconds per trip.
Reactive visuals inherit the same physics: audio arrives for analysis one buffer at a time, so buffer size sets the floor on how quickly any visual can know a kick happened.