In an ideal world, your broker would help you get the best deal possible whenever you made a transaction. But unfortunately, you have to watch your broker closely to protect your own best interest. From high-commission products to nickel-and-dime fees, brokers are in business to make money — and if you’re not careful, you’ll be left footing more than your fair share of the bill.

Many customers have never heard of one source of potential revenue for brokers, but it’s an increasingly important part of the business. As competition among exchanges for stocks and other securities has heated up, exchanges have decided that a great way to entice your broker to place your trades with them is to offer cold hard cash for the privilege.

Making money that you’ll never see
It wasn’t that long ago that investors didn’t have many choices in where to buy and sell stocks. If a stock was listed on a given exchange, that’s where you’d typically go to trade shares.

The boom in new exchanges, however, has given investors — and, more importantly, the brokers who typically serve them — alternatives on where to place trades. In theory, that should be a good thing for investors. With multiple exchanges, brokers can route trades through whichever one will give their customers the best execution price.

To attract traffic, upstart exchanges started offering brokers a payment to place orders there. In some cases, exchanges then charged an extra fee to anyone who filled one of those offers. So in other words, brokers were rewarded for adding liquidity to the exchange but had to pay a premium if they reduced liquidity by taking out one of the orders.

Over time, the practice became commonplace enough that even NYSE Euronext (NYSE: NYX  ) and Nasdaq OMX (Nasdaq: NDAQ  ) started doing it. According to a recent article in The New York Times, during the first three months of 2012, the Nasdaq paid more than $300 million in rebates, while the New York Stock Exchange paid a somewhat smaller proportion of its overall revenue. A recent Woodbine Associates study suggests that rebates may cost investors as much as $5 billion annually.

The practice isn’t limited to stocks, either. For instance, CBOE Holdings (Nasdaq: CBOE  ) offers rebates for certain types of options trades on its C2 platform.

What it means for you
In an ideal world, none of this would make any difference to you. In deciding where to place your trades, your broker would aim to get you the best deal possible after factoring in any rebates. If the broker earned a rebate, it would pass that money along to you in the form of lower commissions. Interactive Brokers (Nasdaq: IBKR  ) , for instance, gives customers who use its cost-plus pricing structure the benefit of any rebates it gets from exchanges.

But some brokers keep any rebates to offset general expenses. For instance, TD AMERITRADE (Nasdaq: AMTD  ) says in its order-execution FAQ: “The majority of exchanges and market makers provide incentives for brokers to route orders to them. Typically, this involves a rebate or payment to the broker for routing orders to that exchange or market maker. … This payment is used to offset the costs of doing business and ultimately helps to reduce the overall cost to our clients.”

If you’re not a frequent trader, then these small boosts in execution price — which often amount to less than a penny per share — probably won’t make a huge difference to your investing results. But even investors who never trade stocks at all can be affected. The Woodbine study pointed to mutual fund companies facing the same situation when they make trades for their accounts, and with much larger trades involved, those pennies add up.

Watch out
Trade rebates are just another way that brokers can end up having a conflict of interest with their clients. Making sure that you learn as much as you can about how your broker gets compensated will put you in a better position to understand your broker relationship. You may not be able to avoid paying slightly more, but it will at least give you a more accurate picture of what your true costs are.

Finding the best broker you can is important to your long-term financial success. So is making the best investments you can. The Motley Fool’s special report on long-term investing can help you figure out where to find the best stock prospects for the long haul, with three names submitted for your consideration. Let me invite you to get your free copy right now.

Do you need more help finding a broker that will do a better job of meeting your needs? Go to the Fool’s Broker Center and learn more about the brokers that cater to investors like you.

It’s a bad idea to buy a stock without looking at its quarterly earnings statement or annual report, but many customers fail to do their due diligence when signing up with financial advisors or brokers, opting for little more than gut instinct or a recommendation from a friend.

That was how Steve Singer, an attorney and father of two from New Jersey, found his advisor. He says that while his current broker doesn’t typically manage portfolios under $1 million, she made an exception for him: “She was handling my mother’s cousin’s portfolio, then my mother’s, and at my mother’s request is now handling mine.”

But such an approach can yield mixed results. A new study by J.D. Power & Associates showed that while overall customer satisfaction with brokers is slowly increasing to pre-2008 levels, customers still aren’t crazy about their brokers’ fees, performance, and service.

When bad brokers and advisors happen to good people
When Maggie Welch’s broker told her to sell some of her stock holdings, she assumed he knew best. Yet he’s sold off several well-performing stocks, causing her to miss out on huge gains, while holding on to underperforming ones. He’s even bought the wrong stock with a similar name to the one Welch had requested. “It is very difficult for me to get him to listen to me,” she says, “and I’m very frustrated.”

Welch echoes a more pressing fear many customers have. “Things have gotten so bad that I’ve tried to take a more active role in my financial planning, but I only know enough to be a potential hazard to my financial well-being.” She’d like to find another financial advisor, but is finding it difficult. “No one can recommend an advisor and tax expert who will accept my paltry $197,000 portfolio.”

Debbie Klug was working with an advisor with whom she felt comfortable. “The first advisor asked tons and tons of questions, sent us — must’ve been eight or nine pages worth of questions and paperwork, wanted to know our goals, near and far-term expenses, that sort of thing. He asked the right questions.”

But when her advisor moved to another firm, her portfolio was too small to take with him. Klug followed his recommendation to go to another advisor, but her experience was totally different. “This guy’s strategy was ‘invest it all.’ He had us sell all the bonds that I had as my safe money and didn’t advise us that we’d get hit like a ton of bricks at the end of the year by taxes.”

Because she didn’t feel as though she was making any more money with him than with she would managing it on her own, she opened a Vanguard account, where she says she feels very comfortable. “If you’re not comfortable with somebody, don’t go there. You’re going to second-guess everything they do because you don’t trust them.”

Advising done right
Before you think all financial professionals are unhelpful, several customers shared stories of exceptional, long-term relationships with their financial advisors, spanning decades, states, and generations.

When Nan Langen Steketee married her husband Scott, she also married into his extended family’s relationship with their financial advisor, who had worked first with his parents a generation before. The couple, who live in Philadelphia, continue to work with the Toledo, Ohio-based advisor, who also manages accounts for Scott’s three siblings, the couple’s grown children, and Nan’s sister. The extended family is scattered along the East Coast from Boston to Atlanta, but Nan says all work well with the Toledo-based advisor on a variety of issues.

“She has advised us on loaning money to our children, helped us with risk decisions, and invested with social and environmental criteria in mind,” which Steketee says is a key criterion for her and her husband. “She has made wise decisions, although she didn’t prevent the 2008 market collapse!”

Don Canale had been less than inspired by his previous broker, a college friend of an employee, when he switched to a new broker. “There’s a day-and-night difference between the level of service,” Canale says. “From what I know of friends of mine who have had similar experiences, I got to be a big fan of this guy because he does a little bit more than everybody else.”

That little bit more included changing firms when Canale mentioned he thought the fees were a little high. His broker agreed and during their next quarterly meeting, he had switched to another firm with lower customer fees.

That high level of service has also been backed by some notable returns; during the 2008 crash, Canale recalls receiving a call. “He said, ‘Well, buddy, we’re up for the year.’ I said, ‘How much are we up?’ and he said, ’1%.’”

“It’s not a big gain,” Canale says, “but 1% is still better than a 30% loss.”

Stuck in the middle with you
Most broker/customer relationships fall somewhere between those extremes. Dan Abbas, an investor from Webster, N.Y., is working with a broker who meets his basic needs — with a little encouragement. “I am quite confident that he is doing a good job of keeping abreast of changes that might affect our finances, that he thinks a lot about alternatives, and is conscientious about doing what he feels is best for his clients. He is not always as attentive to details as I’d like, but when I point out something that does not make sense to me, he either explains it or corrects it.

“He is the first advisor I’ve worked with whom I feel really understands finances better than I do. He is not as sharp with the mathematical details as our previous advisor, but I keep him on track there as I am quite comfortable with the math.”

Singer, whose recommendation came from his family, says that while he’s relatively confident in his broker’s recommendations, he’s not necessarily loyal to her. “If she left tomorrow, I’m not certain I would follow her. I like her as a person, but I don’t know that I would go through the rigmarole of her new brokerage.” Singer says instead, he would most likely look for a recommendation of another broker within the same firm.

Advice from brokers
It would be easy to blame advisors or brokers alone for lackluster service, poor returns, or not meeting customer expectations. But, advisors see their customer relationships as a two-way street.

“Some clients will put you to the test,” says Frank Masselli, president of the Masselli Group and an industry speaker and trainer. “They’ll come in and say, ‘What do you have? What should I do?’ The best advisor will say, ‘I don’t know yet. I don’t know you well enough yet. I want to take the time to get to know you.’ It can be painful for the advisor and for the client.”

Carl Richards, a financial advisor, New York Times columnist, and author of The Behavior Gap, encourages investors and advisors alike to take in the bigger picture. “Nobody can make investment decisions in a vacuum; it all depends on the situation. … Good investment decisions can only be made in the perspective of something larger. An objective third party can help you determine your goals, make a plan, and at end of that process make sure the investments you choose reflect what you said you wanted.”

While it’s up to the financial advisor to give the best advice to his or her customer based on their needs, it’s ultimately up to the customer to determine what those needs are, and if a particular advisor is right for them. Research is critical, and to help with that research, we’ve prepared a checklist of questions to ask yourself and your broker before entering into any relationship — and for determining when it’s time to leave.

Click here to read three customers telling us in their own words how their brokers and advisors have come through or failed them.