I agree with you, the belt isn't necessarily always an indication of skill. But in our system if you're not skilled enough we can and will hold you back. I've been training for 8 years and certainly am no slouch in a fight.
I got a black belt in karate a few years ago, but I had to quit because they pretty much converted it to little kid karate a few months later. Then I went on to krav maga, but again quit because of school. So in other words, you could probably kick my ass, but I'm a black belt too. =P