Is it a sin if you do not pray?

There are worse things, but yes... I think you should look at praying as a privilege we have to speak to Hashem more than an obligation we have to fulfill and that might help change the attitude towards Tfilah.