So how can I stop him?

It is simple. Firstly, be clear in your mind what you want. Secondly, clearly and briefly tell him so. Thirdly, when he asks you questions like "What is your belief on this matter" or "Why don't you wish to come to the meeting with me", clearly, politely and persistently repeat your first statement. "Thank you for your invitation but I would rather not come".
"Why not?"
"That is really my business. I would rather not come."
"But there will be many interesting people there."
"I am sure there will be but I would rather not come."
"I am inviting you because I care about you."
"I am glad you care about me but I would rather not come."
If you clearly, patiently and persistently repeat yourself and refuse to allow him to get you involved in a discussion he will eventually give up. It is a shame that you have to do this, but it is very important for people to learn that they cannot impose their beliefs or wishes upon others.