My advice is simply to ignore the aspect of EL&U's game mechanics that you dislike. A number of users with high reputation scores at this site have observed that the scoring system favors short, trivial, clever answers over thoughtful, carefully researched answers—but unless you are focused on piling up reputation points, you're under no pressure to devote yourself to short, clever answers. In the long haul, your reputation score will accumulate regardless of how you approach asking and answering questions.
I agree that the prominent display of an answerer's reputation score interferes with a reader's effort to appraise answers purely on their own merits; but countering that drawback in objective fairness is the fact that some answerers have demonstrated sound judgment and substantial authority over a large number of previous answers. A person's overall reputation score must be taken with a grain of salt—there are a number of contributors to this site who currently have lower reputation scores than I do but who are vastly superior to me in knowledge and insight on the subject matter of EL&U. In some instances, the disparity reflects their relative newness to the site; in others, their less frequent visits. But a voter would be extremely foolish to prefer my answer to one of theirs simply on the basis of accrued reputation points.
Over an extended period, the point system does help distinguish useful questions and answers and useful contributors from untried or unhelpful ones. For that reason, I don't find its shortcomings too objectionable. And the features that I don't care to participate in (such as downvoting), I am free to abstain from.