Ignore the aphids, they won't do much damage at this late stage and since you are growing outdoors, some predators should show up soon enough and eat them, so no need to panic chop before the plants are properly ripened.
The aphids may damage some of the leaves a bit, but your buds will be fine, trust me, I have been growing outdoors for close to 40 years and have seen this a number of times before and your plant will be ok.
I would not be spraying the buds with anything this close to harvest, no matter how safe it is supposed to be, smoking stuff is much different to eating things.
In other words, do nothing and proceed as normal.
