My dog seems to like sitters much more than me?

So I work from home but from time to time (roughly twice a week) have to go somewhere and leave my dog to a sitter for daycare. But every single time, for several different dog sitters, when I bring him back he whines, barks and with his full force tries to go back to the sitter. The degree varies a bit, but once I let him go to where he wanted to go after walking away from the sitter's house, and he went back to the sitter's house and was sitting there staring at the door. So.. I mean at this point it hurts my feelings to be honest haha. He's a puppy less than a half year old, and it's been a month and half since I brought him. I take him out for walks 5 times a day and at least once among them I take him to a dog park to let him play with other dogs for 30-60 minutes. I work during the day so I can't play with him as much, but try to train him and play with him at least once a day.

Is this common? I'm wondering if I'm doing something wrong or not doing something that sitters are doing. Thanks folks!