AsNoriuanswered grow question 5 months ago if there is no aphids or spider mites, you should not. my grandmother had theory, that ants are cowboys of aphids, kill some and milk others, but keeps the herd alive.
if nest is inside of pot, you can have some minimal root damage, but generally ants are never a thread to weed growers
have a look at this article for more info
https://www.royalqueenseeds.com/blog-how-to-remove-ants-from-your-marijuana-crop-n341