Know-how costs

Looking at Sun man pages versus Linux man pages is like looking at a Van Gogh or Monet after studying the work of the high school football player taking art as an "easy" elective.

Linux has substantial advantage over Solaris in know-how costs and availability. That's mainly to better published books availability and larger community of vendors who are supporting the OS including such heavyweights and IBM and HP. IBM actually is doing more in this area that Red Hat and Novell combined just by having several high Web quality sites like linux section of Developerworks and publishing a stream of Red Books and RedPapers. Of course, IBM tries to promote its mainframe and Power based linux installation, but still some of those free Redbooks are of very high quality and are often useful in a wider context.

But this advantage is not even and in some areas Solaris is superior. For example Sun Blueprints series is an extremely well done and has very high percentage of really useful, relevant to day to day administration materials. We will split know how space into four major directions: manpages, books and magazines, open source applications support and training. Also the most interesting are not OSes but open source applications and linux-based books about open source applications can be reused by Solaris administrators with minimum efforts and inconveniences.

Man pages for linux are just a scam

They are mostly useless. Especially annoying are the man pages that send you to some long defunct GNU documentation format (texinfo). Content of some man pages is partially obsolete and does not correspond to the version of software installed. Solaris has a huge edge here. Inability of Red Hat and Novell to rewrite man pages while charging for annual support is really deplorable.

Historically, documentation is a poor relative in many (if not all) open source projects. This is not by design, but simply because the resources are limited and the main emphasis tends to be on the product and not the supporting functions. If you have a very limited resources you need to chose and the product is the right choice. But as a result the documentation suffers the most and this is quite visible in linux man pages.

Published books and electronic documentation from the vendor

Books are probably the most important component of "know-how" space. Availability of books is better for linux and Red Hat books outnumber Solaris books more then 10 to 1. If you add IBM Red Books (and among them there are several really interesting like Linux Performance and Tuning Guidelines) here Solaris loses by a wide margin. Paradoxically IBM performs much better in documentation area then either Red Hat (pretty dismal record) or Novell (slightly better level). Still quantity is not equal quality and substantial portion of linux books are either outdated or junk or both. Also neither Red Hat nor Novell are really interested in providing high quality documentation outside of their expensive training programs. with opening of Solaris Sun logically gets to the same category but inertia can last for some time.

All-in-all the acquisition of critical "know-how" for both Red Hat and Suse might be cheaper then for Solaris as some problems that can be resolved by using published books in Red Hat might requires expensive consultants-related costs in Solaris. Please note that books which are devoted to the usage of open source applications on linux (specifically the books devoted to Apache, Perl, PHP, MySQL and others) can often be reused by Solaris administrators.

I would like to stress that IBM Red Books provide an important free stream of ebooks for Linux community. Of course they are of various quality but IBM are a powerful brand name that increases chances that they will be read as well as chances that they will be updated. A couple of them are extremely well written and provide state of the art information about linux. Also many tools developed for AIX can be ported with some efforts to linux and accompanying them high quality documentation reused ftp://ftp.software.ibm.com/aix/tools/perftools/perfpmr and Red book about optimizing AIX performance is one such example). Some tools for AIX have linux ports (for example nmon ). Still the differences between linux and AIX are such that it is Solaris not AIX that can boast "linux affinity" (whatever this term means ;-).

I would like to stress it again that quantity of books does not mean quality. That is especially applicable to most published by mainstream press linux-related books. They are by-and-large of "pulp fiction" variety and the available Solaris 10 books while far fewer in numbers essentially match those available for linux. All those regularly reprinted books with each new version Fedora books are usually too entry level to be useful for professional administrators.

Another problem with Red Hat books (especially Fedora-related) is a very short life span: useful shelf-life of a book only slightly exceed useful shelf life of packed food like serials. Red Hat as a distribution is still in flux and each version adds/remove tools and change emphasis for existing tool. For example Yum became the standard way of working with RPMs in versions 5.x, new version of syslogd daemon was introduced in version 5./2 and so on and so forth. It is also true that there were several talented guys who wrote excellent free books for linux (especially during "romantic" period of its development, let's say before 2000) but those case are more of exception them a rule and most of this books are now outdated.

Recently the number of Red Hat books diminished substantially as Ubuntu became the most sexy Linux distribution, but still a dozen of Red Hat and Fedora books of various quality was published in 2006. But it looks like only a couple of books are above "pulp fiction" quality (Sobel's introductory book and may be RHCE Red Hat Certified Engineer Linux Study Guide (Exam RH302) by Michael Jang ). This number declined in 2007 and again in 2008. Currently I see just four Red Hat books printed of scheduled to be printed in 2008 (Unleashed, Bible, Sobel's Practical guide, . All of them are updated versions of older books. And like in 2006 just two ( Sobel's book and Jang's book) are above "pulp fiction" variety.

In a way Suse position is even worse then Solaris as there are very few books published about this distribution. Quality of existing published Suse books is questionable. But Novell had the most democratic training options available with their more or less OK (quality is mixed and generally does not match Sun's quality) and reasonably priced Certified Linux Engineer 10 Self-Study Bundle ($652.00.) That makes Suse the premier enterprise distribution from the point of view of the cost of training for the certification. Also Novell provides lifecycle guarantee for its distributions (7 years). The latter means that Novell certification might eventually get some traction on the marketplace (not yet). Quality of OpenSuse is better then quality of Fedora so here Novell also has a some edge over Red Hat. Patching for OpenSuse 10.2 works more or less OK and is based on yum which is a good solution (Suse 10 uses red carpet)

While O'Reilly is the leading publisher for linux books, Sun Press which should be its competitor is definitely a semi-dead organization. I know less then a dozen usable and current Solaris books (with old Sobel's book as the leading introduction to Solaris). They have tremendous success with Java but as for Solaris there is not much interesting from Sun on the shelves.

The compensating factor for Solaris is that Sun provided a very good documentation with OS and several free electronic books. Only IBM exceeds Sun in the amount and quality of electronic books provided to administrators (including mentioned above famous eBooks series called IBM Red Books). Neither Red Hat nor Novell come close. Sun also shines in its extensive "blueprint" series that are somewhat similar to long dead linux How-To project and IBM "Redpapers" concept but I would say is more successful. More then two dozens out of approximately hundred published blueprints are of very high technical quality and address important areas of Solaris deployment. Like the best IBM Red books they worth real money despite the fact that they are provided for free.

Infrastructure for compiling and packaging of open source application

As open source applications are more important the the underlying OS, the next important part of the "know-how" space is the level of support of the infrastructure for compiling and packaging of open source application.

Currently Ubuntu (or more correctly Debian) and Gentoo are undisputable leaders in this space with Red Hat as a distant second. Almost any package can be found for Debian and it usually compiles out of the box. Gentoo is build on the BSD-derived idea of compiling packages during installation ("real open source" as opposite to "fake open source" practiced by other distributions like Red Hat and Suse). Solaris infrastructure for compiling open source application is still very rudimentary although all major open source application can be found in precompiled and packaged versions.

Situation with the availability of binary packages on Solaris is not bad but also needs improvement. Installer for precompiled open source packages provided on Software Supplement CD is too primitive and unfortunately supplement CD that contains Solaris precompiled and packaged applications is not upgraded on a quarterly basis like the OS. The most bright spot for Solaris is http://www.blastwave.org, the site that has a very impressive list of up-to date packages versions of major open source applications and advanced package technology that automatically resolved dependencies). There is also older, more rudimentary but still very useful www.sunfreeware.com that contains large number of prepackaged applications but dependencies should be resolved manually and some packages are pretty old. The quality of packaging is really outstanding, though.

The importance of open source applications for Solaris (and especially popular in enterprise space OSS applications including but not limited to scripting languages, apache, Postgress, bind, postfix, Sendmail Jboss and Tomcat) can not be overestimated due to the multicore architecture of latest Sun CPUs (T1 line). On multicore CPUs commercial applications usually cost premium. In might make sense for Sun to increase the number of OSS applications that are included into the standard Solaris distribution. Quality of installation of existing applications (especially Perl) also needs to be improved. Postgress, TCL (and Expect), and ksh93 are natural candidates for an inclusion into standard Solaris distribution so that users never need to install them additionally. Actually I cannot understand the logic of sticking to old POSIX shell (ksh88 variant). Also it is unclear what is driving Sun to supply obsolete and extremely user-unfriendly version of Borne shell as a default shell. Who on the Earth need to use Borne shell in XXI century ? It is outdated even for palmtops to say nothing about servers with four or more gigabytes of RAM.

Training and certifications infrastructure

Is the third important component of "know how" space. Large enterprises have money to allow formal training of personnel via vendor run courses, despite high cost of such training. Some large enterprises are actually pretty generous and budget one course per year per person as a "skill refreshment" or "career development" perk. Level of training of system administrators also becomes more important as we move to higher end servers. Solaris used to have an edge due to high quality of training but right now Red Hat looks like a favorite. Solaris administration courses still have higher quality training materials but training itself became outsourced and lose the edge. But this situation might further deteriorate in the future. Red Hat recently considerably improved its printed manuals and moved closer in quality to Sun. Several Red Hat courses are now rival or superior to Sun's offerings. for example RH333 is a more valuable course then SC-300.

But where Red Hat really excels and have or used to have talented people is the testing area. You can instantly feel the level by trying free placement tests available from Red Hat site. Curriculum for mainstream courses from Red Hat is broader then Sun Solaris curriculum and pays more attention to mainstream open source software. The same is true for Novell. Red Hat certification is a higher quality certification then Solaris certification as it is based on the person's ability to fix the system, not so much on the ability to answer multiple choice questions (which always has the danger of degrading into a memory test).

At the same time the evaluation of results of "fix the machine" tests is pretty complex and without really good evaluation software such an exam degrades into a "do as we tell" torture, when the answers not only need to be right, they need to be close to those that Red Hat test writers expected and Red Hat course teaches. That smells as bad as multiple choice questions based test. Still the idea is promising even if currently execution is far from being perfect.

Red Hat certification is a higher quality certification then Solaris certification as it is based on the person's ability to fix the system, not so much on the ability to answer multiple choice questions (which always has the danger of degrading into a memory test).

Even networking course in Solaris were Solaris used to shine is no longer superior to the Red Hat networking course in several topics: for example Red Hat course has better, more up-to-date DNS lab. Still as of 2006 the quality of student book for Solaris is slightly higher.

Still Solaris certifications, especially Certified System Administrator for Solaris are viable not due to their inherent quality (Red Hat certification tests quality is vastly superior) but due to existence of a single version of the OS. Red Hat certification is very well thought out and involves solving of a carefully selected predefined set of problems (which requires real understanding of issues involved), while Solaris certification is based on set of questions and as such can be more easily cheated. Still Red Hat certification suffers from the fact the linux market is fragmented and Red Hat itself represents just over half of installations. Solaris certifications covers all installed instances of OS (or to be more correct particular version of Solaris).

Generally any linux certification is a very fuzzy notion indeed as Linux is the kernel and there are multiple competing distributions based on this kernel each different from the other to the extent that it requires its own certification.