They are not harmful and live on decaying organic matter, turning it into a source of humus the plants can access.
If they bother you, just scrape them around with a stick to dislodge them and leave them to rot in the pot as a source of food for your plant.
And yes, generally a sign that the soil is being kept too wet.........let it dry out more between irrigations.