Permanently relocating and basically re-arranging the seating arrangements is rude, as is abandoning a dining companion completely).Ideally at these events there is a manager or hostess who will come over and politely move them back (perhaps under the pretext of the servers e.g. Oh it is time for entr???e, you will need to sit back at your own table now!) but obviously this is not always the case.