Yeah, you want to get rid of the ants as they can do damage to the root system. Easiest way to do this is by sprinkling some diatomaceous earth on top of your soil... as the ants crawl across it, it actually cuts them and they die pretty quickly (I use this to kill fleas or ticks or any other crawly, jumpy thing in my house - it's organic although some people are affected if they breathe it in).
Good luck!