I would be scraping away the surface layer and removing the chunky bits where they are hiding and no doubt, breeding. Also letting the surface dry out a bit might discourage them. I do not know if they are a real problem, but it would probably be safer to reduce their numbers down to a minimum. If an infestation were to occur, there is no guarantee that they may induce a secondary effect whereby other insects are also attracted or a fungal problem arising. There is nothing wrong with organic soil, they have been growing like that for Millenia and has been my preferred option for 35+ years. Maybe catching some and taking them to your local garden centre to see if they can be positively identified, and this may solve the problem of if they are indeed a pest and ways of treating them if they are, It would, for garden hygiene reasons, probably be better to not have them, unless they are actually beneficial. A positive ID will solve this problem for you.
Hope this helps,.... Organoman.