Maternal mother obtaining rights from grandparents

This is a complicated one. I have always been a part of my 9 year old daughters life to begin with. When she was born I was a teenage mother and my parents helped watch her while I worked. At some point the lines were crossed between grandparents and parent. We had tried moving out when she was very young but myself being young as well knew this was not the best decision I could not care for her the way they could. So I moved out (my parents and me fought non stop day and night) and left her with them. This was to be temporary. I would visit while not at work be there for all mild stones just did not sleep there. My father would tell me , once I was at a place he felt that he was comfortable living with me than he would return her to me. I worked all the time , rented a two bedroom apartment and tried everything in my power to be right by his eyes. But every time it would not be good enough. A year or two later I became very depressed and moved back in, I was still mommy to my daughter and everything was going back to normal. I met a man (now my fiance) we purchased a house in a wonderful kid friendly neighborhood in the best school in our district. I became very successful in my career and make a pretty decent income as well does my fiance. When we purchased the home my dad was moving out of state to help a friend with a business endenvoure . This was okay because we worked out an arrangement that she would gradually move into the home (weekends than a week here and there) to allow her to adjust . At the time she was 5. Everything was going smoothly until my mother asked if they could visit my dad. With no reason to believe that this would not be okay I agreed to a 2 week visitation out of state. After the two weeks they did not come back, I threatened calling the police and was calmly told that I should read over paperwork when I signed it. See before I moved out the first time I thought maybe the military would be the best option for us, but chickened out, my fathers attorney worked up paperwork that was suppost to state in the event of my death they would have custody. Little did I know at that time being 19 and trusting I signed off rights to my parents. So here is were we stand. My daughter is now 9, we have gone through some court custody battles (I decided I did not want to do that to her) and we have gone over agreements over the past couple of years ( holidays and summers I get her) We have gone round and round in arguements that she is now calling them mom and dad and other things I don't agree with but I am always the one depressed at the end of the conversation because they manipulate me into thinking it's whats best for her. Now here we are she is 9, an arguement came about her not coming up for Christmas. My father calls me the next day and states she wants to come home and she misses her family and this is what she wants. He now wants me to take on all custody. I am completely ready, my fiance and family in the area are on board and we have everything ready to go as far as school child care and everything. My father was very gun ho on this and said it's what she really wants , now 2 weeks later he is not so sure about the idea and is calling me all kinds of names out of anger and saying that I'll never be the mother my mom was and just putting me down so bad. I want to do what's right for my daughter but I grew up in a very emotionaly abusing household from both my parents and fear the same for her, not to mention I feel it's right for her to be around her family and with me. I just do not know how to look at this and what the long run risks will be for her once she is older. I do think she is going to change her mind on wanting to be with me because of seeing them upset and that is not the right thing for her.