Is it doing any major harm moving my pots from 1 spot to another a couple of times a day. I keep them facing the same way but i read they don't like being moved so thought I'd ask.
Yes that's right they don't like to be moved around, everytime you move a plant it has to reposition and adjust it self to the sun/light which takes time and energy so everytime it's moved it will slightly stunt it's growth. You can move it but keep it facing the same direction as the sun/light.
The plants need light, so if you move them for this reason, you're doing well. Until they're in flowering, I wouldn't move them anymore. Yes, they're very delicate because the root hairs are very small and fragile.