Forum:Soul Exchange Ruling

I was dueling this person and he used Soul Exchange on my monster. He didnt have any cards in his hand so he used Soul Exchange to release my monster for no reason, without card effects or a tribute summon. Is that legal? Our judge even said it was illegal but we fought about this for like 10 mins. We had to end the duel because he wouldn't cooperate.