Why Do You Back Off From things You Want

Today I decided to end a new relationship with someone I was increasingly found myself drawn to. My major concern was an age difference that brother me more than I was totally ready admit to myself.Not only did it hurt me but I'm sure that it came as a surprise to this person that the depth of my feelings where at that point.
 A little back ground maybe of some help. At my age 66 I have a full live and have recently have major health issues,do mostly to a party type of life so it is all self infected. Coming out of a divorce that was the shock of my life and would have utterly destroyed me if I had let it. I have been extremely careful of letting my heart dictate to my head of what best for both.Recently I have become attractive to someone I have known from my past.With common interest in sports as well as in animals safety it was easy to let my guard down, let her into my protective area where heart and mind agree more on things.There is an age difference of 20 or so years,another concern, but having gotten somewhat past it I began to see possibilities for a relationship.
 When I say relationship I don't mean wham, bam  thank you madam type. I am looking for a companion.best friend,partner type and if sex is there all the better but medical problems I don't know..maybe. So I am at a point where I know it was best,but there is a little sadness on my part cause maybe I finally let someone get close and pushed them a way.