Sorry you've had to deal with all that.

I'll defer to Jo as she has provided some excellent advice. What would be the downsides if you were to let go of your brother? You don't have to answer that here if you don't want to but maybe that's something you can ask yourself. It doesn't sound like the rest of your family is supportive either way. As Jo mentioned, you and your wife should focus on yourselves, at least for the time being.