I know this is an older question but I just thought I would add my experience. I do drop in visits mostly. I have a two year old and a 6 year old. They have both been on pet sits with me before. The two year old will sit in the car with a tablet while I video chat her. This is if I am only doing one drop in and the weather is nice. Also I sit in the Mountains, so most homes will have long driveways into their property. So she isn't sitting alone on some busy traffic street. There was a pet sit where I was at the Meet and Greet and realized I knew their child! He was in my daughters class the year before. They had super friendly sweet dogs and cat. When we realized the connection she said I could feel free to bring my whole family with me. So I brought both kids on one visit and just my older one on the other visits.

I think it is possible but you have to be clear and upfront with your expectations and careful with the circumstances you agree to.