lol i don't even use soap. i just take a slightly coarse rag and a garden hose and wipe them down. (plastic nursery pots)
fabric pots have additional concerns. Might want to give them more attention to strip away at least most of the buildup of minerals that solidified as the water evaporated. I quit using fabric solely for this reason. I wasn't very steadfast at cleaning them but had no problems after 4 or 5 uses with plenty of buildup on the sides. I did try to soak them inbetween uses but it never got them clean.
heck, if it doesn't easily go back into solution it can't do much to the roots either, lol. Maybe my concern was for naught. Soaking at least gets the stuff that is soluble off the fabric or reduced enough. Anything insoluble is less of a concern.