I keep really good records and when questioned I always offer to share the cost/budget of events I run. No one has ever taking me up on that. I think those that complain just do it because that's who they are.
Last event I ran someone got up in my face and i did what i always do...offered to show them the budget for the event....includes copies of receipts and all. They didn't want to see just complain. After they started at me again, i said offered to refund their entry fee so they could walk away...and that didn't work.
I decided before that event i was ready to be done running events....and that just confirmed my desire to be done.