I would say he spooked because he either heard something, or smelled something he didnt like. did you have some estrous scent out?
Did he smell well you had walked in? Some people would disagree with me, but I have tried a lot of the estrous stuff over the years and many of them have spooked more deer than brought them in. Especially if a doe is with him. Just my observances over the years. I still use them occasionally with a drag rag, but only when I know the area I am hunting is on. As far as coming back I am sure he will. I have shot at deer, miss and come back a few minutes later.