Spoon, fork or knife?

Administrative note: thanks for all the comments on my previous post! I haven't found a solution, but have moved on to the “latest and greatest,” which has circumvented the issue. For those interested in the “question overhead” discussion take a look at BZ241335, comments 7 and 8.

OK, so I'll try one of these metaphorical postings. Be kind and work with me here, I'm no Ed Merks for writing these, after all!

How do you consume your open source? Spoon, fork or knife? It's nice if you can use a spoon; take it as is, slurp it up whole. Even nicer if someone spoon feeds you. What if you don't exactly like the taste? Maybe add some seasoning, or prevail on the chef to alter the recipe. Perhaps suggest variations to the chef, or even help the chef cook (assuming the chef wants you mucking around in the kitchen the insurance policy allows it).

You could also consume it with a fork. This gives you more control to pick and choose the next bite, but also brings the substantial responsibility of not sticking your fork into more than you can swallow. Poor use of the fork is a primary cause of choking, and it's not pretty to watch or hear. Of course, if you are skillful with the fork and can take just the right amount, you get to eat as you wish.

Then there's the knife. You're given a meal, but clearly can't consume it whole (unless you're a creature with one of those magical expanding jaws and a big enough mouth). Also, there might be parts of the meal you don't like, and want to avoid through judicious use of the knife. So you carve the meal and take it as you want, when you want. It's extra work – wouldn't it be nice if the chef carved it for you, so you could just use the spoon – but worthwhile to get the bites you want.

But probably you're like most people: you use the spoon, fork and knife in some combination that seems to make sense for the meal. The spoon is fastest, and the others require more work. Sometimes you might carve with the knife and use it in a stab to consume the separated piece (which is dangerous, by the way. I hear that people long ago were more skillful with the knife and actually ate like this). Or maybe you are like me: you tend to use the fork to both carve and stab (a habit, which I am told, betrays my lack of a proper upbringing).

In the past I've mostly consumed Eclipse meals using a spoon, with (rare) occasions for the knife to make bites suitable for the spoon. Over time, I find myself using the knife more and more, and the fork seems to have a renewed purpose. How about you?

Walking away from bugs

One of the most frustrating things that can happen to an open source committer is to meet someone who tried your software, ran into some problem, and just walked away. Did you check the error log? Did you ask on the newsgroup/mailing list/forum? Did you file a bug? As a committer, these questions roll off your tongue. After all, open source is a collaborative effort, and success depends in large part to the collective effort of the community. Walking away like that is just not good citizenship!

Or is it?

Recently I needed to evaluate some plug-in code that was written on a Ganymede-based target platform. For reasons not important here, my development environment is based on Europa at the moment. Eclipse has great support for these cases, however, and setting up various target platforms is routine for me. So, I go and set up the necessary target platform, load the evaluation code into my workspace, and set up a launch configuration. Right – so let's run these plug-ins and see how they work. The splash screen appears and shortly after – bang! It dies with an error* Strange. I've been developing open and closed source plug-ins for Eclipse since V1, and have dealt with all sorts of platform configurations and error conditions: never saw this one before... Mess around with the launch configuration, read config.ini (like the error message suggests); nothing helps. Ok – off to an internet and Bugzilla search for the issue: clearly I must be doing something stupid, because Europa should be able to launch Ganymede-based target platforms (right?). Hmmm... some discussion about this type of error, but nothing seems to fit my case. Try launching the target platform directly, and it works fine. A few more minutes of searching, a few more minutes of messing around with settings. Finally, I walked away.

And then I thought about it. Why did I walk away, and what does that tell me about open source users who do the same? Well, in my case, and I suspect this is common, the reason was pretty simple: my task at hand was to evaluate the plug-in code, not to debug an Eclipse launch failure. I was trying to use Eclipse to work on another problem, not on Eclipse itself. I could get the information I needed in another fashion, and that's what I did. Now, being an open source committer, I felt a little guilty: surely I'm not walking away, but rather I'll get back to that problem. I'll do The Right Thing and check all possible avenues: I'll read everything I can about it on the net, I'll ask on the newsgroup, I'll file a bug and follow up. Yes, yes I will! I will get back to it, right after I get these other things (that I'm paid for) done... Then again, maybe not (to be realistic).

I even felt guilty about writing this blog entry. Who I am to complain when I haven't done my homework? But then I realized that something larger was interesting: my attitude toward the whole situation. Maybe a kind reader will point out an answer for my problem, one which will probably seem entirely obvious, if I had looked in the right place. If so, thanks! But fixing this specific issue is really not the point. Maybe you'll flame me: You don't understand how open source works, you've got to engage the community! To this I'll answer: You don't understand your users! How many of them have walked away silently?

When I was in graduate school, a friend of mine was doing his Ph.D. In the physics department at MIT. Maybe it's a measure of what it takes to operate at that elite level, but he once told me (without pride or arrogance, but simply matter of fact) that he was never shy about asking a question in class, because if he didn't understand something, it was the professor's fault for not being clear.

Just to be clear (because I don't mind being flamed, but prefer to be flamed for what I actually meant): I'm not blaming the Eclipse platform or any other Eclipse project/committer for the error I encountered or my inability to fix it quickly. Rather I am just pointing out the observation that I feel compelled to do a lot of work before raising an issue, and I wonder if this “barrier to the question” is the reason why people walk away. If so, what can be done?

*java.lang.IllegalStateException: Unable to acquire application service. Ensure that the org.eclipse.core.runtime bundle is resolved and started (see config.ini).

Returning to the planet

It's been a while since I last blogged. In blog years, that's like a lifetime. I guess it's because I've been reincarnated in a new job and in a new role. It's taken me a while to get settled in my new environment and to figure out how, if at all, I should continue to blog.

My new position involves creating Eclipse tools for JBoss ESB, and there's no shortage of things to do. :) And, yes, it's all open source. I'll continue to blog about the same sorts of things that previously appeared in this blog, but also include more about SOA and plug-in development in Eclipse.

For now, I'll just make a general observation, which has been guiding my use, selection and development of tools recently: the Linux Hater says: "The best code in the world won't change the world if people can't use it."

Toxic Requirements

Regardless of the process (or lack thereof) you use for creating software, there has to be some idea of requirements. These requirements tell us, in whatever form they are expressed, what it is that we’re trying to do. Most people who have worked on software projects for a while have faced impossible/highly unlikely requirements foisted on the project by people who do not understand technology and are prone to wishful thinking. (Simply saying that something will happen doesn’t make it occur.) Clearly unreasonable requirements based on wishful thinking are not good. But that’s not what I want to talk about here. Rather, I’d like to concentrate on requirements which are actually reasonable in some local context, but in the overall project context are just as bad, if not worse, than wishful requirements. These are toxic requirements.

Fifteen years ago Gerald Weinberg discussed how considerations around software quality are ultimately political: someone has to make a decision about what “quality” means in a certain case, and doing so can result in more alignment to one person/group’s interests over another. Implicit in Weinberg’s work is that requirements are political as well: statements about software quality are expressed as requirements, and by extension all decisions about what to do (and hence what not to do) are made by designated people/groups balancing options.

How does this apply to open source, and how is it an antipattern? As I discussed in an earlier post, tight binding between open source and product work can cause problems. Toxic requirements emerge in this context when product requirements are foisted on the open source community. Of course many times product requirements are also community requirements, so there is no imposition. But there are also times when the decision makers (committers) use their position to impose requirements on the community that have not been demonstrated as having broad applicability beyond specific product needs. In some situations, the case for broad applicability (crucially usually including some modifications of the requirement) can be made, but it is more expedient to use committer authority to push the process. In such cases, committers will often use the “I do, therefore I decide” principle of open source to justify the decision. But these rationalizations avoid the real issue: if your idea of open source includes principles like “openness,” “transparency,” and “community” then making decisions in the narrow scope of product requirements is poison. Instead you must be willing to either build community consensus (or at least the lack of objections), or simply not implement the requirement in open source. This is very hard to do, especially when your sponsors (who “pay the bills”) are demanding fast action in a direction that was decided by a closed process.

Open Source Antipatterns: How to hurt yourself with open source (2)

OK, so it's been a while since the first post in this series... Maybe there's an open source antipattern lurking? :-)

The second antipattern I'd like to describe deals with the costs of developing in open source. Since open source software typically can be redistributed freely (within the terms of use in particular cases, YMMV), somehow otherwise savvy tech and business people jump to the conclusion that open source is free. Free to develop; a sort of perpetual motion machine for creating software! Money for nothing!

But, of course, it does cost something to create open source software, even if the end result is made freely available. That's probably obvious (once the hype fog mentioned above clears). But, here's a secret: developing open source software makes you incur costs that you would have otherwise avoided if you decided to create the software some other way (say, for example, the standard closed-source model). That's not to say closed-source development doesn't have costs (potentially) avoided by open source-development, but rather the costs associated with open and closed-source development not always the same. To further complicate the picture, many open source initiatives are taken by employees on behalf of their companies, and so -- if you're not careful -- you can end up incurring both open source and closed source development costs for the same component! (Though, unless you try really hard, you probably will not incur all the possible costs associated with each.)

So now the antipattern: Not understanding the cost structure discussed above before embarking on open source development. Next, you (or worse, your boss!) discovers the open source-specific costs, and things get unpleasant. One way of helping the situation (aside from understanding the cost structure in advance) is to isolate the additional, open source specific costs and seek to share these specific costs with the community.

Open Source Antipatterns: How to hurt yourself with open source (1)

One of the benefits of working on an Eclipse project is that you get to see firsthand how open source is produced and consumed. In a series of "Antipattern" posts, I will talk about some things you don't want to do with open source: sort of like a set of "worst practices."

For the first installment, let's talk about how to consume open source components in a commercial product. The lack of a viable consumption process is one of the biggest problems I've seen, and is most severe when the company consuming the components is also working on that open source project. That last bit is counter-intuitive: if your company is working on open source components that you also consume, shouldn't it be easier to manage the quality, features, timing, etc.? In some circumstances, yes, but crucially it all depends on your consumption process. This process has an impact on how you participate in the open source project and how attractive it will be for others to join your efforts.

The heart of the problem is binding commercial products too tightly to open source components. When you do so, you expect that the open source components will work in a way that you alone have determined necessary for your product, and that they will evolve in a controlled path aligned with product aims. Yet sharing in open source goes beyond simply making source code available. It goes beyond the other associated transparency areas (bug lists, development mailing lists, etc.). That is, success open source is about sharing control of the components as well. If you really want an engaged community to rally around your open source efforts, you have to share control, otherwise you limit how much people can actually participate, in ways meaningful them, in your project.

Now, here's the rub: when you share control, sometimes the open source components will move in a direction not aligned with your product goals. If your products are very tightly bound to the open source components, then you'll either have to accept the loss of control or stop using the components to regain full control.

With a better consumption plan, however, you can enable the best of both sides: control over your product while allowing the open source to move with the community. What you need is a process that is flexible and that can adjust the open/closed source mixture as necessary. You should also consider the wisdom of crowds: if the community is moving the open source components in a certain direction, perhaps that's better than what you had previously considered for your product? Perhaps you're getting hyper-early beta feedback at a time when you can actually react to it?

Whatever you do, don't stifle an open source project with commercial product goals and then wonder why a strong community hasn't developed around either.

Code wants to be miscellaneous

I’ve been reading the fascinating Everything is Miscellaneous by David Weinberger, and have a few preliminary ideas about how this relates to open source. The final two sentences of his prologue: “…information doesn’t just want to be free. It wants to be miscellaneous.” I’d like draw out an implication of this statement:

Code doesn’t just want to be free. It wants to be miscellaneous.

So, what’s that supposed to mean? I’m only going to make a few suggestions here, as I best understand it at the moment, and leave the implications to more study and discussion of Weinberger’s book.

Open source code is free, as we all know, like in “free speech” not “free beer." Well, it is free in this sense modulo the open source licenses morass, but that’s another post… But it is not "miscellaneous" in the sense that Weinberger uses it. For him, an important characteristic of information being miscellaneous is the ability to recombine it in ways desired by the consumer of that information. Information is allowed be an unstructured, unordered, teaming mass with rich associations. Consumers of this information take slices configured according to their wishes, and in the process both create additional information and associations.

I wonder if open source can get to this miscellaneous state? A state in which components, source files, and even parts of source code itself can be recombined as desired by consumers, and in which rich meta-data enables this usage. An example from Eclipse: there are feature definitions which bundle plug-ins into groups based on usage patterns or functional areas. While the feature mechanism is a great tool, anyone who has provided Eclipse plug-ins to enough adopters can tell you that no set of feature definitions will ever satisfy everyone. There’s just too many ways to slice and dice plug-in sets, even in smaller projects, and by the time you get to the size of most Eclipse Foundation top-level projects, the possibilities are huge. In practice most adopters just take the best approximation of their needs based on existing features, though some apparently define their own custom feature definitions. In this system, the plug-in set is structured into features at the wrong end (following Weinberger’s theory): the providers are imposing what they feel to be the appropriate feature set on adopters, whereas it should be the adopters who are defining features as desired. And I don’t mean the adopters suggest a new feature set to the Eclipse project: I mean that the adopters dynamically create their feature sets based on their needs as the moment.

Another example: what if there were an open source project where anyone could modify the code? I don’t mean “submit a patch” or “modify their local copy,” but rather an open source project where anyone could commit changes. You say that this would quickly become chaos? The code would never run? Security bugs could be injected on purpose? How can the quality/intellectual property be managed? Yes, these are real concerns today. But, to me, this points to the need for a new set of tools and a new way of working with source code. One which lets me understand the lineage, modifications, and intent of various “layers” (versions) within the miscellaneous code base. One which lets me have the stability demanded by (especially commercial) adopters, with the free-flowing experimentation of an ideal(istic?) open source community.

I can see two objections here: this whole “miscellaneous” business is a sham, and arguments that we can/are doing so already. Maybe the first objection is correct, but I’d reserve judgment until the ideas and implications of Weinberger’s book are better understood. Personally, I doubt this is a sham: on the contrary, it is pointing to something very important. For the second, I’d agree in theory, if not practice. In other words, in is possible, but not at all practical, to do these sorts of things with existing technology. But that’s not the point: the reason why we’re not still writing everything in assembly language, for example, is that higher-level languages bring abstractions and enable thinking in different ways. The same is true here: we need tools working at the right “level” so miscellaneous information can be harnessed on a routine basis, without requiring large effort, knowledge or time.