A downvote isn’t “I don’t think this should be here”. It’s just “I don’t like this”, don’t agree, thing it’s bad/wrong, whatever. Up/downvote systems work under the process of the wisdom of the crowd, and as a result, works under the assumption that while the ways in which people use these tools and the reasons may be different, having a simply up/downvote system means they’ll average out into something more useful. It doesn’t work all the time of course, but it does work generally.