I recall this being a problem raised on a post some time back in the dim and distant past but can't remember if it was fixed or not. Quick glance through the forums and I couldn't find the original post - maybe I'm having a bad brain day.
Anyway, I just placed an item up for auction and the fee took my character into negative (~$2.7k) cash. I had cash on another char so was able to transfer but as with forum posting there should be a little message "uh huh, you don't have the cash for this auction" kinda thing...