Dealing with a client that is difficult to communicate with?

How do you deal with unresponsive clients or ones that are difficult to communicate with?

I've been trying to set up a meet and greet with a client for the past 3 weeks for several drop-ins in about 2 weeks. So far I've attempted 3 meet and the greets: for the first a situation popped up and they were unavailable (they told me the hour before the meet), the second they never confirmed the meet so I offered an alternative time (they said they forgot to hit send to confirm the second), and the third time they just never responded to my text requesting second confirmation (which I requested the day of, trying to be safe).

I went anyways (they live in an apartment complex I was passing by anyways, which is why I scheduled for that time that day), only to find essentially every door to the complex requires a keycard---they did not notify me about this. Maybe I missed an obvious entrance, but I searched for about 15 minutes and couldn't find any way in.

I'm getting a bit fed up with it---to me it shows a lack of respect for me and my time. I'm tempted, at this point, to refuse their request---I accepted to take it in the first place (only verbally, no actual confirmation) because I love helping people and want to make their search easy. However, if I were to actually take on this client it would be extremely inconvenient for me (for someone with bad motion sickness, I would have to take public transportation for 2 hours every day to reach their dog for 30 minutes, it adds up very quickly). They aren't aware, of course, but it makes the hassle that I'm going through with just trying to get a meet and greet have me lean towards refusing this client.

Thoughts on how to deal with the situation?