[We return to the review of classical responsa on topics related to beit din.]
You asked about a city’s set beit din to whom a document was presented and beit din knows that it is forged by an expert and that the witnesses who signed it are liars, although this has not been formally substantiated in court. Based on their personal knowledge [based on experience/ intuition, …], beit din ripped up the document. Does the document’s owner have claims against beit din? Does a respected beit din that is careful to not extract money from people without basis, have the power to take such steps as a hora’at sha’ah (exceptional remedy to a situation)?
The answer is that if the beit din has an established reputation as a reliable, G-d fearing, distinguished beit din, they have the power to take the type of steps you mentioned. This is apparently Chazal’s intention when they spoke of "judgment of truth to its truth" (Sanhedrin 7a).
There is a similar case that the gemara describes in Bava Metzia (39b). Mari bar Isak had a brother who came from Choza’ah, asking to receive a portion in their father’s inheritance. Mari bar Isak said he did not recognize his brother, which was reasonable since the latter was young when they had last seen each other. When asked to produce witnesses of his identity, the brother said that his witnesses were unwilling to testify because they feared Mari, who was an intimidating person. Rav Chisda ruled that Mari was responsible to produce witnesses that the person was not his brother. When Mari asked how Rav Chisda could go against the rule that the one who wants to extract money has to produce evidence, Rav Chisda answered: "This is how I judge you and other intimidating people like you."
We see that Rav Chisda altered the normal law based on a specific need and a desire that the judgment should come out to be true in earnest. However, beit din has to investigate and deliberate carefully before it rips up people’s documents. We see this from another Talmudic story (Ketubot 85a). Rav Pappa said: "Now that we know that a person’s intuited knowledge has halachic status, I would rip up a document based on the assertion of my son, whom I know to be fully honest." The gemara said he could not have possibly meant he would rip up a document based on his son, but just weaken its reliability.
I have heard that a case came before our teacher, the Ramban, and he ripped up a document with his own hands. However, everything depends on how one checks and the intent for the sake of Heaven; one who fears Hashem finds a way to do the most correct thing.