Being a Cohen

My father told me he is a Cohen. His father told him. They are not religious people. Never seen them in shul blessing like Cohanim do. My grandpa is not alive. Can I enter a cemetery?

This would not be sufficient to establish identity as a Cohen. Staying out of a cemetery would be a stringency you are not obligated to accept, but you might be able to check you grandfather's gravestone to see if he is recognized as a Cohen. If it's not there, then it's a good bet he was not.