You can try and knock it down as long as the birds aren't present. If they return and start building in the same nest, you're a bit stuffed I'm afraid. You could net over the old nest site (but use very fine mesh, otherwise the swallows might get tangled in the netting trying to get through) and put up a des res like
this one in a location that suits you better. They might not take to it, but it's worth a shot.