Law 3 states that refs should wait till the ball is out of play before cautioning players that make a keeper switch without permission. Why?

What about the moment after the keeper has the shirt off but before the new keeper has it on? Right then the defense is playing WITHOUT a keeper. That’s forbidden.

What if the other team attacks while the keeper jersey is laying on the ground? Certainly this is to be avoided.

I’m pretty sure this is just angels dancing on the head of a pin, because I have never seen it, but the instant I saw the keeper take his shirt off, I would be sorely tempted to stop play. Is my position defensible?

USSF answer (March 30, 2011):
No, your position is not defensible. How can we say that? Read on.

As appears to be the case in your question, if the goalkeeper and the field player haven’t actually exchanged jerseys yet, it can’t be an illegal goalkeeper change because—guess what?—no shift in positions has occurred. Were they ABOUT to? Sure (at least a reasonable inference), but it is not illegal to attempt to change places or to have the thought in one’s head that you want to change places. About the only thing you could get them on is for removing their shirts, and that would be a mighty long stretch.