It won't be a big problem to me, because, well, it's me and my point of view. I cannot give a good advice, though.
Here's what I suggest: don't listen to criticism and try to balance both, care and fun. I know how difficult it might be. But you love your brother, don't you? Most people feel content when they help others, maybe you do too. Think of it as a challenge that no one else has, the one that makes you grow inside. I'm sure it is possible to arrange time in a way that you are able to go to some parties, cinema, concerts, etc. You are not locked, right? So just do it.
Your friends might not know the things you do, they know their wants and needs and apply them to your life. The thing is, it's not the same as theirs. So, tell them you are strong and mature enough to understand what responsibility is and to be happy at the same time. It's their problem if they can't. You shouldn't push them away, just let them know you feel good. Maybe they're just jealous they can't be as good and devoted as you? So, try to balance your duties, your free time and show everyone that you don't live in misery. Who can criticize you for feeling fine?