How to tell an owner you can't dog sit again for them?

Just tell the client there isn't a good fit for you and you are allergic to smoke. Honesty is the best policy

If the dog didn't smell like smoke, you would take him again? If that's the case, this is what I, personally, would do. This may not fit with your personality, but I also hate confrontation.
First, I wouldn't say anything unless they tried to book me again. No point rocking the boat for nothing, and if you tell them now, they may leave you a bad review.
If they wanted to book with me again, here's what I would consider, based on their personalities, how I feel about the dog, and my life circumstances at the time:

1. If I really didn't want to take him again and I'm busy, I'd tell them I'm not available during that time (I only really say this when it's true because I hate lying and my Rover calendar will betray me anyways, lol).
2. If I wouldn't mind taking the dog again, but want to fix the issue and get paid for all the trouble, I'd say something like "Thanks so much for reaching out to me again. It's great to hear from you! I would absolutely love to watch Rex again, but I think I should be honest with you. Unfortunately I have a very high sensitivity to cigarette smoke, to the point where it can actually make me sick, so if it's alright with you, I would be happy to bathe him and wash his stuff when he arrives [optional, add "at my bath rate of $XX]." I really hope you understand that I don't mean any offense to you. This is just something I would need to do for myself so I could give Rex all the energy and attention he deserves." You can also offer for THEM to do these steps before bringing him over, but they probably won't, or they'll do it too long before his stay and he'll get all smokey smelling. Either way, if you choose this option, they probably will not get back to you or they'll just say no thanks. Or they may be super apologetic, who knows?
3. If I really want/need the money or just don't have the energy to deal with any potential confrontation, I'd accept the dog, give him a bath (maybe outside if he's real big and you have a spot to do so and it's not too cold) and wash all his stuff without telling them. I'd probably ask in advance if his skin is sensitive to any products or if they mind if I use stuff on him or give him a bath.