How do I politely decline a potential client after a meet and greet?

I got a walk request for two dogs, I set up a meet and greet and met with the owner. During the meet and greet I felt unsafe and uncomfortable, the owners dogs didn't even like him (he stated this many times). I want to be polite, but i am not sure how to decline this service. I was afraid to do it in person because it is not commonplace for me to do so. He took down my personal number too which I shouldn't have given him in the first place but I panicked and just acted like everything was ok. What can I do?