Swarm is probably not the right word here, as a true swarm wouldn't care about near-by people. They're looking for a new home if they're swarming. And it's actually more likely that their colony is in the area and they're defending it. It could be a longterm issue....