You wrote in Bemareh Habazak (IX:94) that an avel may enter shul once sheki’a (sunset) has passed, even before the end of Kabbalat Shabbat. Should we say that, similarly after sheki’a, the shul should not "welcome" an avel by saying Hamakom yenachem …"? Answer:
You may be assuming that one may not be menachem avel on Shabbat. The gemara (Shabbat 12a-b) (reluctantly) permits being menachem avel on Shabbat, as does the Shulchan Aruch (Orach Chayim 287:1). Since this is not the optimal time to do so (Shabbat 12b; Magen Avraham 287:1), we should not be surprised by the minhag to not be menachem on Shabbat, at least for Ashkenazim (see Gesher Hachayim 20:5:2).
The timing of an avel’s entrance is based primarily on his ability to enter shul (not before Shabbat – see Tur, Yoreh Deah 393). Note a historical fact. Until relatively recently, Ma’ariv of Shabbat in shul was done during daytime, so that the community accepted Shabbat early with the saying of Barchu (see Rama, Orach Chayim 253:2, as one of many sources). The custom developed to recite Mizmor Shir L’yom HaShabbat, which became the acceptance of Shabbat (Shulchan Aruch, OC 261:4). Between Lecha Dodi and Mizmor Shir became the perfect time for the avel to come in. People could be menachem freely because it was still Friday, and he could enter as it would immediately be Shabbat, when his presence in shul became appropriate. As we pointed out in Bemareh Habazak, if Shabbat began before the community accepted Shabbat, it is also permitted for him to go into shul. Your question is a good one. Does it become forbidden to say Hamakom yenachem …?
307 - Selling Food Supplements Online
308 - Ushering in an Avel after Sunset of Shabbat
309 - Chasing after a Child during Kedusha
One could argue that it is still permitted, as we saw that it is permitted to be menachem on Shabbat, despite the minhag not to do so. Since the custom is to welcome the avel at that point in davening, we would follow the halacha that it is permitted. However, the Pri Megadim (Mishbetzot Zahav 287:1) says that once the community has said Mizmor Shir, they may no longer announce "to go out to welcome the mourner" (it was apparently more elaborate than today) because it is a public display of mourning (this is not obvious assertion), which is forbidden (see Shulchan Aruch, YD 400:1). He allows individuals to go over and express consolations, but not the shul and not with the standard weekday formula. The Mishna Berura (287:3) states that some authorities permit regular language.
Thus, the purist will logically agree with you that after sheki’a, the shul should not say Hamakon yenachem. We could compromise and say that during bein hashemashot (doubt whether it is day or night), one can be lenient and allow the marginally problematic group consolation. (In Bemareh Habazak we allowed Shabbat leniency from sheki’a because we are lenient on matters of aveilut (Moed Katan 18a).) However, the minhag seems to allow welcoming the mourner even after tzeit hakochavim (nightfall). In theory, we can say the minhag is a mistake, by not updating the practice after the timing changed.
However, it is possible (and preferable) to uphold the minhag for two reasons. On a matter that is not overly serious halachically, we uphold minhagim even when they appear to be "wrong." Sometimes we also do not know the wisdom behind the minhag. Let us suggest a possible fundamental justification, while not being sure that it is a correct explanation. The Pri Megadim is talking about a case where the community consciously accepted Shabbat with Mizmor Shir. As such, public aveilut behavior is inappropriate. However, when the community has not yet accepted Shabbat with their behavior, it is not forbidden to welcome the avel. While each individual is not allowed to violate Shabbat because the time of Shabbat has come, consoling the mourner is not a violation (see above). If just the public nature of the practice is the problem, sensitivity to that may begin only with Mizmor Shir even the time of Shabbat came earlier.
In any case, we will uphold the minhag of our shuls, many or most of which still welcome mourners before Mizmor Shir despite your good question.