To borrow a term from Robert De Niro’s character in “Meet the Parents,” the 200 or so people I follow on Twitter are my “circle of trust.” For me, they provide the perfect amount of quality of content.

Of course, my circle isn’t set in stone. Sometimes I add to it, but I like to keep it at a low number. It keeps things personal and under control.

The number of people who follow me (around 370 as of this writing) grows fairly consistently, and I appreciate those who have deemed me worth following.

Yet without fail, I see that number grow and shrink on a weekly basis. I think I’ve zeroed in on the reason people and brands unfollow me; unfortunately, it has nothing to do with my tweets.

It’s because I don’t follow them back.

We all know them. They have 28,000 followers and are following 27,999. Their goal is to build a mountain taller and taller, and then—taller and taller and taller. Often, an app helps them follow en masse and unfollow the ones that don’t return the favor.

There’s no bigger pet peeve of mine in the social media world than people who are in it for the illusion of influence. If you have great insights, people will find you—I can almost guarantee you will build a following. But if your No. 1 goal is followers (as has seemingly been the case for many self-proclaimed “social media experts,” ironically enough) you’re missing the point.

Michael Jordan wasn’t great because of the number of people who bought his jerseys. He earned greatness through his play. Everything else was a reward.

Building a following on Twitter is no different. Prove the value of your presence first, and you should have no need for an automated system, especially one that neglects value and takes a shortcut to attracting followers.

People sometimes forget that you can still engage with those who aren’t following you. Use that to your advantage. Show you’re more than a picture. Engage. Answer a question or offer an interesting and relevant article. Proving your value will demonstrate your credibility and establish connections. It will make you worth following, but more importantly, build your circle of trust.

As I’m sure goes for plenty of other people, it’s highly unlikely I’ll follow you back immediately, if at all. We’re on Twitter for the value, not the meaningless number. So instead of giving up and unfollowing, the question you have to answer is, “Why should they follow me?”

Come up with a good answer for that question, and you’re well on your way to making Twitter a better place.

Aaron Gottlieb handles social media and marketing at IF Management, an on-air media talent representation company in Midtown Manhattan. You can follow him @aarongottlieb, but don’t unfollow because he didn’t follow back.