No, you stop time at the start. You slurp up those 1024 samples and then start to process and output, hence the latency.hmmm...
Ok, so a tune has been going through the filter, and then we stop Time for moment...
so 1024 samples are loaded, which one of those 1024 am I recalculating with all the multiplying and summing?
You can start right away but then your starting processed samples would behave oddly/be wrong since you don't have the past samples to do it right. This is sometimes done in video at the line boundaries but not in audio.