If I ask my friend to forgive me, that itself will hurt him (he doesn't even know that I slandered him)! What can I do?

In order not to hurt that person, R. Yisrael Salanter says that it's wise to utilize the common custom before Yom Kippur to go around and ask forgiveness from anyone whom you may have wronged. This way he won't be hurt by your asking his forgiveness, and probably won't ask: "what did you do to me?". This way, the world will be a much nicer place for all! Gmar Chatima Tova, Rav Ari Shvat (Chwat)