If one has to undertake a "conversion of doubt" (because the mother’s family were persecuted by antisemites and thus concealed any evidence of their Jewishness; and now the mother has no living relatives to substantiate this), can they marry a Kohen? Would the fact that the Kohen’s not so observant have an impact?

There is no clear answer, and all will depend on the specific circumstances of the doubt. The Kohen's observance would generally not be a factor.