Grandparents not able to see their Grandkids

I have two beautiful grandchildren ages 2 and 4. I have missed most of their lives. I know that they see their grandmother on the other side of the family. It breaks my heart that I have missed out on everything. I have tried to be patient so I don't lose my relationshp with my son but it is so hard.