I wipe a lot when I play, whether I'm wiping off dust/sand, mud, moisture, snow. I don't like the feel of anything between my hand and my disc (I have no use for birdie bags).
About once a year (when they start to look pretty grungy), I let them soak in warm water & mild dish soap then scrub with dish sponge. The soak loosens some of the heavier grunge, resulting in less elbow grease expended.
To be honest, any difference is probably all in my head... by that doesn't mean it's not real. I feel better when my putter's clean, so it's no surprise that I putt better when it's clean.