When do you give the house key back--at the end of the day or at the end of the job/final visit?

I just had a new client/owner whose dog I cared for today and am supposed to care for tomorrow as well called me in the middle of the night (10 pm) and asked she's not finding the house key that she gave me. I was surprised and said, oh I thought I was supposed to put the key back in the mailbox at the end of the job/final visit which is supposed to be tomorrow. During our meet and greet, she never clarified that she wanted me to put the key back in the mailbox at the end of each visit, as opposed to at the end of the job. She then told me to bring the key back in the middle of the night when I was on a call with my family. When I explained, she said she would call Rover support and then texted me she would call the police if I don't bring the key back right away. I was seriously annoyed and offended. So, I went ahead and dropped the key off and also cancelled the rest of her booking tomorrow. When I talked with Rover support, they also told me that it's the norm that sitters give the key back at the end of the job.

How do you handle any such lack of or mis-communication like this? I just don't think it was right for her to demand that I go out of my way in the middle of the night to drive to her house to return the key and also did not admit that she failed to clarify this during our M& G.