I thought that at Ma’ariv of Rosh Chodesh (or other times there is something new to say), the gabbai calls out "Ya’aleh V’Yavo" (=YVY) before Shemoneh Esrei. But in many shuls, someone just bangs. Which way is correct? Answer:
While all agree that semichat geula l’tefilla (connecting the beracha of "Ga’al Yisrael" to Shemoneh Esrei) is important at Shacharit, not all agree regarding Ma’ariv (see Berachot 9b). Since the conclusion is that it does apply at Ma’ariv, one may not talk before Shemoneh Esrei of Ma’ariv (Shulchan Aruch, Orach Chayim 236:2).
342 - Announcements before Shemoneh Esrei of Ma’ariv
343 - Do the Chatan and Kalla Need to Eat at Sheva Berachot?
Nevertheless, the Rashba (Shut I, 293) justified a minhag to call out "Rosh Chodesh" before Shemoneh Esrei at night. He reasons that talking for the needs of tefilla is not considered a hefsek and that the fact that Ma’ariv is an optional prayer reduces the severity of such a break. Indeed we rule that pertinent announcements are permitted at Ma’ariv (Shulchan Aruch ibid.), but not at Shacharit (Taz, OC 114:2).
The Maharashal (see Bach, OC 236) disagrees with the Rashba. He argues that the only speech permitted between geula and tefilla is reciting things instituted by the Rabbis (such as Hashkiveinu and Baruch Hashem L’Olam). He posits that Ma’ariv is no longer optional because Klal Yisrael accepted it as binding, and that in any case, in the midst of tefilla, even if it were optional, one may not make a break. The Mishna Berura is among those who bring no dissenters on the Shulchan Aruch’s permission to announce YVY at Ma’ariv, and this is the standard approach presented by contemporary Ashkenazi tefilla compendiums (see Ishei Yisrael 28:24; Tefilla K’hilchata 19:20).
Some poskim, though, cite minhagim which do not permit calling out "YVY." The K’tzot Hashulchan (27:5) cites the Ba’al Hatanya’s siddur as forbidding it; the Kaf Hachayim (OC 236:17) says that the minhag in Yerushalayim was against it, and the Yalkut Yosef (OC 422:2) rules this way. One explanation (see Kaf Hachayim ibid.) of these counter minhagim is that they are concerned that the Maharashal, not the Rashba, is right. It is perhaps more likely that it is a shame to allow speaking when there are effective, preferable alternatives.
As you mentioned, many suffice with simple banging, as in many shuls everyone understands what they are hinting at. Producing sounds, like other forms of non-speech hints, is not a hefsek in davening except for in Shemoneh Esrei and the first parasha of Kri’at Shema (Shulchan Aruch, OC 653:6; Mishna Berura 104:1). It is likely that the minhag of banging developed not as a rejection of the possibility of announcing, but out of a realization that, in some shuls, it is unnecessary.
Another alternative (see Magen Avraham 114:2, in a related context; Kaf Hachayim ibid.) is for one who gets up to YVY in Shemoneh Esrei to remind others by saying those words out loud. While one generally should not daven Shemoneh Esrei out loud, it is permitted for one davening at home when there is a reason for it (Shulchan Aruch, OC 101:2). In shul we are concerned that this will disturb others (ibid.). However, it is hard to have such an objection when one person is saying two words to help the tzibbur. An advantage of this system is that the reminder comes closer to the time people recite YVY, and is in that way more effective. Do note that some consider saying words of Shemoneh Esrei out loud to be disrespectful (see opinions in Dirshu 422:2), at least if not done by someone appropriate like a gabbai or the chazan (Halichot Shlomo, Mo’adim p. 1). There is often a technical problem – if the one saying out loud does not start early or daven faster than others, many will get to YVY before him.
In summary, there are three legitimate ways to remind people to recite YVY, each with advantages and disadvantages, some of which depend on the shul (e.g., if people understand the bang). Since people have seen each system, many shuls develop a hodgepodge of practices, which is neither great nor terrible. If the rav has not set a policy, any alternative is fine.