Avoda Zara and Ayin Hara

As a Spharadi, I was brought up with the concept of protecting myself from ayin hara (the evil eye). We wear chamsa, fish, and an eye figure. I had a few people tell me that this is a form of avoda zara and I should not wear them. I also heard that it is mentioned in the torah about protecting oneself from ayin hara. To this I am confused. I do not want to engage myself in any avoda zara. Can you please clarify the confusion?

Wearing these amulets does according to some opinions create a problem of Avoda Zara (Rabbi Chalfon Moshe HaCohen of Jerba, Tunisia in his responsa Shoel V'Nishal Yoreh Deah V 140 prohibited engraving a sign of a fish and a chamsa on the doors of a synagogue). There are verses that can be said against Ayin Hara (they usually appear in the siddur after maariv on motzei shabbat). The best advice is, however, to keep mitzvot, do chessed, pray at the required times and have faith in G-d.