How to break up with someone? Nice ways..?

Okay, so I first started dating this guy and his friends didn't really like me because I was all positive and they were all negative and thought I was going to take their friend away. So they made a bunch of rumours up and slandered my name. He didn't really stand up for me, until I told him he had to in which it was too late. I ended up breaking up with him. However, he apologized over and over and acted really sweet and somehow managed to squeeze his way in again. Anyway so we ended up getting along and fixing things and it was good for a few months. But for some reason recently I can't really get over the way I wad treated before. Maybe because he keeps slipping it in nonchalantly. He smokes weed which I don't enjoy at all. He has cut back for me and has done lots of things I have asked for but I just don't feel like I could love him. I feel like he tries so hard to get me to love him, it's like I just don't want to and if I do decide to love him, is he going to go back to smoking weed and not sticking up for me. Plus he always tells me we are going to do this or that, and when I get to his house we never actually end up doing anything. Sometimes we do but most of the time the plans fall through. Really it is just too boring for me. I am 21 years old and need to get out there and figure myself out more. Plus he is sometimes negative and grumpy which I feel brings me down. But don't get me wrong I'm just focusing on all the bad right now to justify my decision to break up with me. He is a really nice guy and really would do anything for me. Help!