One of the benefits of working on an Eclipse project is that you get to see firsthand how open source is produced and consumed. In a series of "Antipattern" posts, I will talk about some things you don't want to do with open source: sort of like a set of "worst practices."
For the first installment, let's talk about how to consume open source components in a commercial product. The lack of a viable consumption process is one of the biggest problems I've seen, and is most severe when the company consuming the components is also working on that open source project. That last bit is counter-intuitive: if your company is working on open source components that you also consume, shouldn't it be easier to manage the quality, features, timing, etc.? In some circumstances, yes, but crucially it all depends on your consumption process. This process has an impact on how you participate in the open source project and how attractive it will be for others to join your efforts.
The heart of the problem is binding commercial products too tightly to open source components. When you do so, you expect that the open source components will work in a way that you alone have determined necessary for your product, and that they will evolve in a controlled path aligned with product aims. Yet sharing in open source goes beyond simply making source code available. It goes beyond the other associated transparency areas (bug lists, development mailing lists, etc.). That is, success open source is about sharing control of the components as well. If you really want an engaged community to rally around your open source efforts, you have to share control, otherwise you limit how much people can actually participate, in ways meaningful them, in your project.
Now, here's the rub: when you share control, sometimes the open source components will move in a direction not aligned with your product goals. If your products are very tightly bound to the open source components, then you'll either have to accept the loss of control or stop using the components to regain full control.
With a better consumption plan, however, you can enable the best of both sides: control over your product while allowing the open source to move with the community. What you need is a process that is flexible and that can adjust the open/closed source mixture as necessary. You should also consider the wisdom of crowds: if the community is moving the open source components in a certain direction, perhaps that's better than what you had previously considered for your product? Perhaps you're getting hyper-early beta feedback at a time when you can actually react to it?
Whatever you do, don't stifle an open source project with commercial product goals and then wonder why a strong community hasn't developed around either.