No matter when you work and no matter what job it is you're working. There will always be something you will miss or have to sacrifice because you're working. I worked retail, so I had to miss some holidays and evening events, I shoot weddings so I miss Saturday stuff. I do commercial shoots during the week so I miss stuff when I do those.
There will always be something.. What you need to do is put family first and keep them first. If you can make family always be at the top of your list then you will never have an issue.
Plus there are 52 weekends in a years. You're obviously not working every weekend. If you are, then you're doing it wrong...