Yeah - you should be worried. You've got a nasty, bad infestation of some kind of leaf-sucking insect (which one doesn't matter at all)... get yourself either some Dr. Zymes Eliminator or some Spinosad and use as directed... both are organic, both can be used up to the day of harvest if necessary without affecting the taste of the smoke.... but whatever the insect is, it's sucking the life out of your plant and will kill it if not taken care of...
Good luck!