Question: I want to use an otherwise kosher oven that was used for cooking food in a manner of clear chillul Shabbat. Has it become "treif"?
Answer: Food that is cooked on Shabbat is one of many examples of ma’aseh Shabbat (the result of chillul Shabbat), and as such is forbidden to be eaten. Your question is a good one: does such food treif up utensils?
The answer seems dependent on whether ma’aseh Shabbat regarding food is a prohibition against benefit (which, for food, is usually eating) or whether the food is considered ma’achalot assurot (what we call nonkosher). If the former, any residue in the oven will not bring you real benefit. If the latter, then the food is like any other that treif up an oven (we will not discuss how an oven becomes treif or how it is kashered).
One reason to not consider this food ma’achalot assurot is that it is prohibited for an external reason – not because of an intrinsic problem with the food per se, but due to its connection to a bad situation. The Ktav Sofer (Orach Chayim 50) compares ma’aseh Shabbat food to bishul akum, as that food is also not intrinsically problematic but tainted by a situation. There is a machloket Rishonim whether bishul akum treifs up a pot (see Tur, Yoreh Deah 113 – the Rashba is strict; the Rosh is lenient). The Shulchan Aruch (YD 113: 16) cites both positions, but prefers the stringent one (he is slightly lenient on how to kasher it).
Indeed, the Magen Avraham (318:1) cites the Rashba as saying that ma’aseh Shabbat food treifs the utensil in which it was cooked, and he and the Mishna Berura (318:4) accept this position. Regarding the above fundamental chakira, Rav Orbach (Minchat Shlomo I:5) sees this Magen Avraham as a proof that ma’aseh Shabbat food is ma’achalot assurot.
On the other hand, many disagree. Besides significant opinions that are lenient regarding a pot used for bishul akum, this case includes additional reasons for leniency. The Mateh Yehuda (cited by Livyat Chen 42) says that the Rashba only implies that according to R. Yochanan Hasandler (Ketubot 34a) who views ma’aseh Shabbat as an intrinsic Torah law, a utensil would become treif. However, according to the Tannaim that ma’aseh Shabbat is a penalty, only the actual food, which gives real benefit, is forbidden. Some (see Teshuvot V’hanhagot II:196) point out that the Gra rules like R. Meir (Ketubot ibid.) that even the food itself becomes permitted after Shabbat.
Finally, there are strong indications that ma’aseh Shabbat does not create ma’achalot assurot. According to the opinion of R. Yehuda, which the Shulchan Aruch (OC 318:1) accepts, the food is forbidden forever only for the person who was mechallel Shabbat. This distinction is difficult if ma’aseh Shabbat is ma’achalot assurot, which are generally objective prohibitions (Ktav Sofer, ibid.). I would add that the fact that ma’aseh Shabbat applies to many nonfood melachot works more cleanly if they all share the categorization of prohibitions of benefit.
It is hard for an Ashkenazi posek to argue with the opinions of the Magen Avraham and the Mishna Berura, at least without other grounds for leniency (see Orchot Shabbat 25:53). Rav Ovadia Yosef (Livyat Chen 42), on the other hand, concludes that the basic halacha is to be lenient and views kashering utensils in this case as only laudable.
In your case, there is little room for concern. We forbid ma’aseh Shabbat after Shabbat only when the chillul Shabbat was intentional, and then only for the one who was mechallel Shabbat. According to most, it is not even forbidden for a person for whom it was done (see Magen Avraham 318:4); it is certainly permitted for others (see Orchot Shabbat ibid.). Therefore, since you had nothing to do with the chillul Shabbat, even the food and certainly its residue in the wall are permitted. (You did not ask and we will not discuss the topic of classic kashrut questions regarding an oven of one who is not Torah observant.)