Tuesday, November 29, 2011

Confusing intent and outcome

Mikeal Rogers recently published a blog entry entitled "Apache considered harmful". He writes:
The open source section of my brain was seeded and curated by Ted Leung, long time ASF member, and it is this ethos, Community > Code, that I've dedicated a significant portion of my life to. It is this ethos that has led me to the hard conclusion that as the world has changed Apache has become a net negative for its projects.

The Mikeal raises a fundamental question. Do the costs of developing software at the Apache Software Foundation, or any other foundation for that matter, outweigh the benefits? Apropos, what exactly are the costs and the benefits? Lets start with the costs.

The costs

The principal cost of developing software at Apache is loss of autonomy and freedom. The foundation owns your software. As far as your software development efforts are concerned, the foundation becomes your pointy-haired and clueless boss. In his very informative blog entry, Ben Collins-Sussman writes:
The ASF has a great set of cultural norms that it pushes on its communities via political means and lightweight processes
The reader should not be fooled by the seemingly innocuous reference to cultural norms, political means and lightweight processes. After all, the official name for North Korea is "The Democratic People’s Republic of Korea" and not the more accurate "The Totalitarian Stalinist Dictatorship of North Korea". In the same vein, DDT was qualified as "harmless" in the 1950's, as are its considerably more toxic equivalents of today. In the world of IT, everybody describes their software product as lightweight and simple. No body advertises their product as an over-engineered piece of crap.

Alright, coming back to Apache, once you join, you will be expected to obey rules disseminated throughout the organization many of which are open to interpretation. Rules are not necessarily bad. Any cohesive group must a have a set rules guiding its actions. It just happens that many Apache rules are forged by discussion and are not the result of careful experimentation. The conquest of false opinion by free discussion is a philosophical question. For a much deeper discussion of this topic, do yourself a favor and read Democracy is not a truth machine. Again, forging rules by discussion is not bad per se. It becomes problematic only when the group is formed by electing similar thinking individuals (co-opting). As time goes by, less and less opposing voices are heard since dissenters leave quietly and those who remain tend to agree with the prevailing dogma. As a corollary, co-opting favors convergence of shared vision and stricter adherence to group's rules over time. Whether such convergence is "desirable" depends on the "desirability" of the rules. I should emphasize that of group convergence is not all black-and-white but a matter of degree, from loose counter-cultural groups on one end, to cults on the other the extreme. It would be a great exaggeration to qualify Apache as a cult.

More concretely, Apache projects cannot designate a formal project leader. Every committer has strictly equal rights independent of past or future contributions. This rule enjoys very strong support within the ASF. It is intended to foster consensus building and collaboration ensuring projects' long term sustainability. Worthy goals indeed! However, one should not confuse intent with outcome. Note that strict committer equality contradicts the notion of meritocracy, one of the core tenets of the ASF.

As I have argued in the past, rejection of project leadership in addition to lack of fair conflict resolution mechanisms constitute favorable terrain for endless arguments. In case of conflicts arising during the lifetime of an Apache project, participants often resort to filibustering. In an all-volunteer project, where most of the participants are only lightly invested in the project, project disruption is almost cost-free to the disruptor. In nature, one of the opponents in a struggle eventually backs off because he or she has something to lose, typically the risk of bodily harm. At Apache, the only cost incurred by the disruptor is the time taken by the disruption. In a highly-competitive industry such as IT, wasteful expenditure of time may imply software stagnation and eventually obsolescence.

When all else fails, a project can ask for the Apache Board's intervention to settle an argument. In the examples I've witnessed, lacking the proper context, the board will tend to intervene with the finesse of an elephant in a china-shop.

As another example, consider the Apache Board position on author tags. The board minutes from February 2004 state:
  - author tags are officially discouraged. these create difficulties 
   in establishing the proper ownership and the protection of our 
   committers. There are other social issues dealing with collaborative 
   development, but the Board is concerned about the legal ramifications
   around the use of author tags
I fail to see the legal ramifications the Board has in mind. However, given copyright law in continental Europe, and in particular in France, removing author tags infringes upon the author's moral rights which are perpetual and inalienable. In short, I would argue that removing author tags is illegal in continental Europe. The board statement also mentions social issues in relation with collaborative development. The idea here is that developers should not mark code that they contribute as their own to avoid territorial conflicts. Thus, the board indirectly discourages developers to take pride in their code. Moreover, it removes one of the currencies in which open source developers are compensated: increased prestige.

The benefits

As far as I can tell, Eclipse, Apache and the FSF all give unconvincing explanations about the benefits they provide. The main benefit of joining any organization is being part of said organization. Seriously, tautology aside, once you join a FOSS foundation, you automatically assume part of the aura of that foundation and have better access to its other members. Mike Milinkovich of Eclipse-fame lists the following benefits of FOSS foundations:
  1. IP management
  2. Predictability
  3. Branding and community
  4. Industry collaboration

Regarding IP management, it is not too difficult to determine whether the dependencies of a software project are compatible with your chosen licensing policy. Eclipse takes the additional step of contacting the authors of projects to ascertain ownership. Who cares?

Regarding branding, it is by now apparent that Apache, Eclipse and the FSF all have very good as well as utterly useless projects. The same is true for software projects outside any FOSS foundation. While software projects in a foundation are perceived to be on average of better quality, many of the top 1% of FOSS projects are outside the control of any foundation. I suspect that given the overhead involved, being part of a foundation ultimately hurts quality. In any case, FOSS foundations do not oversee product quality, they only ensure that projects are alive and follow IP-related guidelines. (Apache projects in addition have to prove diversity before graduating from the incubator.) Good software will stand on its own and does not need any additional branding. If your software responds to an unmet need, users will come in their numbers.

As for industry collaboration, depending on project size, belonging to a foundation may truly help collaboration among corporations. I also think that collaboration can be achieved outside any foundation for large as well as small organizations.

In the early days, joining Apache felt like joining a friendly and informal group of capable geeks. This aspect of Apache is not mentioned often enough.


Thus far, FOSS foundations have been quite successful in attracting software projects. In fact, the Apache Software Foundation has been so successful that it can now brag about not having to advertise to attract newcomers. The prevailing circumstances in 1999 allowed Apache to attract key Java software projects such as Ant, log4j, Struts and Tomcat. From then on, the immense aura of these early projects allowed Apache to be selective. However, by instituting rules largely tangential to software quality, the ASF is squandering the good-will capital it had accrued a decade ago.

Bertrand Delacretaz describes Apache as the turbocharged Diesel of Open Source and Matt Asay compares Apache to IBM. My comparison is less flattering. I consider Apache analogous to a carnivourous plant trapping and slowly digesting insects. As unsuspecting insects find out, it is easy to step foot inside the rolled leaves of a pitcher plant but escaping it unscathed is a different matter. Donating software to Apache is a one-way street, you can enter but you can't leave without being subjected to the digestive juices of Apache dogma. I've been told that Eclipse is better managed and less dogmatic. I don't buy that. I believe that the disadvantages described here apply to software donations made to Eclipse as well as other FOSS foundations. You can enter a FOSS foundation but you cannot leave, much like an insect on the surface of a carnivorous plant.

For the reasons presented here, I urge you to not fall into the trap of complacency. Setting up and maintaining the infrastructure required for a software project takes time, but is becoming cheaper and easier by the year. If your software meets a need, then you don't need the brand-value associated with joining a FOSS foundation. It takes time and effort but by being consistently open, friendly and respectful you can attract long-term contributors to your open source project.

In short, if you are thinking of donating software to and joining a FOSS foundation but have not actually done so, don't, joining is not worth the trouble. If you are forced to join a FOSS foundation, well, by definition you don't have a choice.

If you have already joined a foundation but would like to get out, try convincing the other project members to leave. If they share your point of view, then get out and liberate your software. If a sizable majority of the project members oppose departure, then there are two cases to consider. The software project you are trying to liberate is 1) marginally to moderately popular or 2) highly popular. In the first case, you can leave and start over to create a better product. Your users, few in this case, will follow. In the second case, unless you are willing to put all your energy in the forked project and continue to do so for a very long time, I am afraid you are stuck with the foundation. It might come as a surprise but your users have a radically different agenda than yours. They will consistently value stability over your purportedly innovative but unproven fork. Thus, you need to swallow your pride and learn to live with the foundation's rules regardless of how inane you find them. Patience is your best friend. With enough patience, most arguments will eventually reach a reasonable settlement.

Now, if you have joined a FOSS foundation and are thrilled with the outcome, then this post does not apply to you - until it does. The risk of the management/board raining down its wrath upon your project always exists and is non-negligible. After all, it is the management/board's prerogative to shutdown projects it disapproves of.

Or course, I am making a number of assumptions which may be totally incorrect in some relevant context. Studies indicate that former cult members tend to shade the truth and blow out of proportion minor incidents, turning them into major incidents. My main point however, that there is an important distinction between intent and outcome, is hopefully as valid as it is generally applicable. Organizational ethos of FOSS foundations do not correspond to reality. You don't want to obey rules imposed left and right, writing FOSS software should be first and foremost fun! Why would you want to put up with masters if you do not have to?


Lincoln Baxter, III said...

These organizations are not for anyone, but I'd hardly go so far as to discourage people from becoming involved in some of the most influential and professionally rewarding experiences that they might otherwise miss out on if they take your advise blindly. I don't think this applies to those who are joining, not starting a project.

In that same vein: doing FOSS at a well-known foundation frequently means getting awesome, highly-paid jobs.

If you want to host and take on the maintenance burden yourself, like we've done at http://ocpsoft.com/ by effectively creating an entity to manage IP and support, that's certainly something that can be done. But I would also posit that if you start to run into IP problems, you might wish you'd joined Apache, JBoss, or another OSS company/foundation. It took a LOT of effort to get OCPsoft set up, and I don't blindly recommend doing that for anyone. Know your passion. I agree with you that it's not for every project, but it is completely up to the participants to weigh their options properly.

Even getting a decent single-project website set up can be difficult, expensive, and incredibly time-consuming.

If you don't care and just want to write OSS, there's nothing to stop you from doing that, so I don't really see why the overly general statements on FOSS foundations is relevant in that context.

You say that the Apache guidelines were created through discussion, not experimentation; can you prove that? Guidelines are required for any kind of large-scale interaction, and I'd be surprised if they didn't go through numerous changes over their time.

It seems like something happened that personally angered or disappointed you. Is that the case?

Unknown said...

"belonging to a foundation may truly help collaboration among corporations" looks like a key point to me.

The four benefits that you mention might not be very relevant to an individual developer, but for a corporation they often make all the difference.

Ceki said...

Yes, doing FOSS at a well-known foundation frequently means getting awesome, highly-paid jobs but doing FOSS development on a well-known project *also* means getting awesome, highly-paid jobs.

You are right. Setting up and maintaining a project site takes time, but is becoming cheaper and easier by the year.

You see, those just wanting to write FOSS are the ones most tempted to join a foundation when they would probabaly be better served on the long term by not joining a foundation.

I can't prove that the Apache guidelines are created through discussion and not experimentation, but such is my observation.

Yes, something has happened that has disapponted me several years ago. More importantly, I see it happening to others. Let this blog entry serve as a warning to the innocent.

G. said...

@Unknown: Actually "collaboration between companies" is easier with Foundations. This might be fine for Companies. The question is, how do the Individuals inside the communities feel? Are the projects really targeting at including new committers from outside? Sometimes there is discussion on diversity, but no, there is not a complete oversight. Isn't a Foundation, helping Companies to collaborate, nothing else then some kind of Meta-Company? And does this Meta-Company get some kind of dynamics and build up a ruleset it self to act as a company? And finally, is it really fun for Individuals to work there, in a Companies playground? Good that there are some projects which do NOT have so strong Companies in back. This is not the outcome of good oversight, it is simply luck.

Now lets assume you are working at such a project. Is it really "Community over Code" or is it "project success over Community" (as can been seen on some projects in action). One cannot say. There is no measurement on project healthiness. There is no measurement on community healthiness.

If one joins a foundation, the guy will surely see amazing software and met great developers. But he must also know that in some cases Community < Code. Because in an "company environment" there is heat from time to time.

Interesting post.

Unknown said...

Interesting ... I've occasionally been so frustrated with the ASF that I've thought about pulling out; of course, it's not clear we could take the "Tapestry" name with us.

The infrastructure is clunky, such as the use of exported confluence to manage site documentation combined with multiple hours between making a change and seeing it go live.

The promise of a switch to Git is keeping me going, for the meantime.

mikeal said...

@Lincoln Baxter, III

The intended audience for my article "Apache Considered Harmful" is people who are considering bringing their project to Apache and it is, in fact, an effort to discourage them.

New projects need to gain momentum to be successful, Apache's policy insure that will no longer happen without big companies behind the project pouring in resources.

Existing projects with a community that consider joining would have a tough time convincing their community to come with them and live under the Apache policies.

The only project that is a good fit for coming to Apache nowadays are corporate open source projects with some resources that want contributors from other companies involved. This is sad, Apache was intended to be an organization for the community, not corporations.

Ceki said...

> This is sad, Apache was intended to be an organization for the community, not corporations.

and this state of affairs seems irreversible because the rules are drawn up as a result of exhausting discussions and with good intentions

Ian Skerrett said...

Full Disclosure: I work at the Eclipse Foundation so obviously my comment have a certain perspective. :-)

I would certainly never claim OSS Foundations are right for everyone and certainly Eclipse or Apache is not right for everyone or every project. Apache is obviously not right for you. :-)

I would like to comment on some of your statements about the benefits Mike Milinkovich stated for the Eclipse Foundation.

- Lots of people care about IP management. You might not but if you are large company or a small growing company, intellectual property is important. If you aspire to be a widely adopted open source project, IP cleanliness matters.
- I would dispute your claim many of the top 1% FOSS projects are outside the control of a foundation. If you start with LAMP, with the exception of MySQL, foundations support LAMP. This research would certainly back up my position http://openlife.cc/blogs/2010/november/how-grow-your-open-source-project-10x-and-revenues-5x

It is not to say all foundation projects are of high quality or good technology can't exist outside foundations. I would however claim successful oss projects have some type of legal entity, for-profit or not-for-profit, to support it.

fwiw, the additional branding from a Foundation is the opportunity for more exposure. We have over 1 million visitors to eclipse.org a month. Eclipse projects get a lot more visibility than projects hosted other places.

I do agree with your final recommendation: writing FOSS should be fun. I know at Eclipse the sense of community between the individuals makes it fun.

Anonymous said...

Fyi: iBatis did the move from apache away to mybatis

Bradley M. Kuhn said...

Ceki, I notice you mention in the original post you don't mention Conservancy at all. I would challenge you to find any member of Conservancy who isn't generally satisfied with Conservancy's rather extensive service plan. In fact, Conservancy is currently funding a number of developers, some of whom are putting in full time consulting hours, to write and improve their projects.

Honestly, it seems to me that you didn't do any even anecdotal research to find out what projects like or don't like about being members of a FLOSS Foundation and if the benefits have value to their project.

BTW, I also wrote a brief response to the original blog post on this issue.