Life isn't about partying 24-7, but it also isn't all about responsibilities. You are doing the best thing you can, helping your family and taking care of your brother, but you should be able to take some time off and do something you enjoy, otherwise you'll end up resenting them.
Try to explain that to your friends, and if they really are friends they'll understand you. It seems to me that they are quite young, and they don't have the responsibilities you already have, but eventually we all find ourselves balancing what we have to do with what we want to.
Good luck.