Don't want my BF Brother near my child?

My Boyfriends older brother has been a to me since day 1 of our Relationship. First it was my age i'm 30 & he is 22. Then we got engaged after 2 months which was after a big car crash which we both escaped unhurt thankfully. We both realised we never want to be apart. We are together 3 years now and his brother always has something to say. I'm also 4 months pregnant with my first child. This is our 4th pregnancy (1 ectopic pregnancy & 2 Miscarriages). This guy is so childish he's even resorted to threatening me because my boyfriend took his GF off facebook as a friend apparantly thinking i made him do so. We have had confrontations which haven't been too nice so i have stopped visiting my future in-laws because of this, every time i visit he has a tantrum and starts to abuse me. It's got to the stage were i have had panic attacks and i am worried about how things are going to be when i have my baby. I know he's my BF brother but I hate this guy so much he's made my life a living hell and don't want him near my child. He threatens me then texts my boyfriend after & is all nice. I think it's a bit f****** up.This is a guy who smokes , drinks & never works. I have considered giving up my relationship because of this but why should i ruin my happiness i love my boyfriend so much. Somebody please give me advice i told my mother i don't want my child subjected to that kind of behaviour and have said to my boyfriend that his parent's can come see the child at our house. My job as a mother is to protect my child is it not?? I have considered going to a solicitor also to get a letter sent to him for slander