I would assume so. While you might likely have worms in the surrounding soil where you're planting, I can't imagine they're large in number or provide an abundant amount of resources, especially if you reuse the same outdoor soil patch. Adding additional worm castings shouldn't harm anything. I've seen some people plant in soil mixes containing up to 40-50% castings, so I imagine adding additional outdoors would only be beneficial.
If you are reusing the same soil patch, I would recommend planting some perennial dutch white or crimson clovers on them during off seasons and when you're not growing. They're nitrogen fixers and will charge the soil with nutrients over time and are perfect for outdoor use.