Gemara: There are six things that are negative for a talmid chacham to do: ...He should not enter the beit midrash (study hall) at the end [on a regular basis], for people will call him a posheia (usually translated as negligent).

Ein Ayah: If someone who is not involved on a regular basis in Torah study regularly comes late to the beit midrash, people will assume that he was busy with his business affairs. However, a talmid chacham, who usually has no regular occupation besides learning and can be assumed to be studying Torah in his home as well, does not have a standard excuse for being late to the beit midrash. Thus, people will conclude that he does not care about the advantages of being part of a group of scholars and G-d-fearers.
This would make him a posheia, which is a term for one who sins without the prospect of benefit. The word is etymologically related to the word poseia, which means taking a step, in this case, a step out of the group.
It is a matter related to great completeness for a person to realize the value of being part of a positive group, both for his own benefit and for that of Hashem’s nation. The idea is as follows. Usually one’s motivation to act improperly comes from stupidity or the strength of his desires. We call resulting wrongdoings chet and avon. There is also, though, an evil inclination to see oneself as outside the order one is required to have based on the justice of the Torah. One who wants to do the aveira for the purpose of not feeling contained in the circle that surrounds the nation is called a posheia.
A moral person should see how the spiritual welfare of the whole world depends on the extent to which there is a love of the correct divine order that is the foundation of the holy Torah. Love of being part of the group of scholars is something that particularly leads one to this proper order. Even though being by oneself can bring one to grasp higher ideas than he can as part of a group, the benefit to the world in regard to the order of being part of a group of complete people is so great that a scholar should be diligent in joining the group at the first opportunity.

The Danger of Eating With Lowly People
(condensed from Ein Ayah, Berachot 6:57)

Gemara: There are six things that are negative for a talmid chacham to do: ... He should not take part in a meal of amei ha’aretz (lowly people), for he will come to follow them.

Ein Ayah: We saw in the previous piece that even though one may be able to reach a higher level if he keeps to himself, it is still preferable to give that up in order to join up with other scholars. However, now we are told to realize that the benefit of the group is true only in regard to scholars, but he should not join the group of lowly people. He should not think that if he lowers himself a little bit, it will still help in that he can raise the others. To the contrary, it is they who will lower him rather than his raising them up. As opposed to the aforementioned problem of coming late to the beit midrash where people only say that he is a posheia, where objectively he is sometimes justified in doing it to increase completeness, in eating with lowly people he actually does lower himself.