2nd try topping (Do-Si-Dos)

You are correct, those do not usually amount to anything. Typically they will always stay small and never ripen up to the point of being anything worthwhile to smoke. You can remove them at any point, earlier the better though as your plant has already wasted energy on them and not your top buds. If you are looking to wash them for hash or grind them up to put into edibles, you can leave them on if you wish. I typically get so many B grade buds higher up my plant anyways that I use for those purposes that these tiny ones are usually just a waste. Hope this helps.