As long as you don't get rain toward the end of flowering, everything should be fine. The odd bug won't eat much and I believe it is far better to not spray any plant you intend smoking, with anything, no matter how "safe" any product claims to be.
Spraying kills good bugs as well as bad ones and if you don't spray, the good bugs will be there and ready to eat any bad bugs if they show up.
I have been growing outdoors, in the open air for 35+ years and have never had any real issues with bugs or mildew, just bud rot/botrytis once in a while and only ever if there are early Fall rains and the plants are advanced in flowering. Up until about two thirds way through flowering, rain is not that damaging, it is only right towards the end of maturity that rains can cause rot, but that is the gamble of growing outdoors. Sheltering your plants with beach umbrellas or a tarpaulin if early rains are predicted is about the only thing you can do to prevent wet buds. Since you are growing in pots, it is pretty simple to move them under cover and keep them dry. I have never even once sprayed my plants with anything and have relied on healthy plants being able to take care of themselves, which they can do quite easily and without the need to spray them with anything "just in case".
Hope this helps, Organoman.