Gemara:


Ein Ayah: Birkat Hamazon, which is a set of berachot that comes after the meal, leaves us with a strong message. Even at the time that one is satiated and has indulged his body, nothing is able to take hold of him and remove his heart from his Master. Rather, he clings to His great Name even when his heart may tend toward feeling overly free in his satiation. This is why "you shall eat and be satiated" calls for blessing Hashem even when one is drowsy after satiating himself with wine (Yerushalmi, Terumot 1:4).
The above is a very important idea because a person sometimes is forced to be in a very physical state. Therefore, he requires a reliable guarantee that even when his physical side is dominating him, he should not give up his position but should remain firmly vigilant to defend his service of Hashem and the completeness of his ways. This is accomplished by thanking Hashem and realizing that everything comes from His bountiful Hand, as He is the one who gives us the strength to succeed. A place can be connected to a status and to abstract concepts such as these. Therefore, it is proper to bentch specifically where one ate. This demonstrates that even in the state of ostensible physical dominance that he was in at the time of eating, his heart is cognizant of the need to thank Hashem.
Let us deal with one who forgot to bentch until he arrived at a new place, where he did not have the materialistic experience of eating. Since naturally he is beyond the point of the dominance of physicality, part of the message of Birkat Hamazon is missing. That is why Beit Shammai instruct him to return to his place so that bentching can be done in a place where there is an imprint of the physical state that he was in at the time. When he bentches there, part of the damage that was caused by his initial forgetting to bentch is remedied.
Beit Hillel posit that the best situation is to be at a distance from the state of materialism, which should be avoided when not necessary. When he is at a distance from that potentially dangerous state, we do not want to send him back to give him the opportunity to overcome his physical tendencies. Such a challenge is worthwhile only for those who are above the spiritual level of the average person.
We should know that at the same time that one is eating and his physical side has power, he also has the inspiration to be thankful for the enjoyment he is receiving. This feeling strengthens the righteous tendency in man to not deny the good he has received. However, after he has left, he is in a situation where the experience of enjoyment has already passed. If he puts himself in a situation of heightened physicality, it might not be accompanied by the desired feeling of thankfulness, in which case the problematic element will be present without the good. This is a bad situation according to Beit Hillel who build their approach based on the path that is good for the average person, even if a particularly wise person could do better.