Softpanorama (slightly skeptical) Open Source Software Educational Society

May the source be with you, but remember the KISS principle ;-)

Solaris vs. Linux: Framework for the Comparison

by Dr Nikolai Bezroukov

9. Conclusions

Previous version of conclusions begin to resemble an separate article and was converted into Summing up section ;-).  Business decisions are always compromise and much depends on the goals. The key in OS area is the total cost of ownership.

Here we just reiterate the major points which might help to determine right compromise. Please note that I am talking not about cash-strapped universities and/or start-ups and not about firms located in developing countries. I am talking about making decisions in the environment of more or less well to do (although now far from flush with money) large enterprise IT:

1. Enterprise OS mix behaves like an ecosystem and side effects as well as complexity of the task of adding yet another flavor on Unix in a large enterprise environment (be it linux or Solaris) should not be underestimated. Those side effects tend to eat into savings. Proliferation of unix flavors increase sysadmin overload and lessen the quality of each individual environment as workforce became spread too thin and lack critical mass necessary for acquiring and improving knowledge. With large overload the situation is more about survival then about quality. Even with huge cost-effectiveness of Intel Duo CPUs killing one of existing flavors of Unix usually can save more than adopting a new one.  Cost and tradeoffs typical for excessive diversity of Unix environment are often ignored in simplistic calculations of benefits of linux adoption (or any other new flavor if Unix adoption). 
    
2. Excessive complexity of modern OSes leads to "lowest common denominator" style of deployment in large enterprize environment with multiple flavors of Unix used.  The mere fact of presence of several flavors of Unix almost guarantees that some of the most interesting and unique capabilities of a particular flavor of Unix will not be used.  Some can be poorly understood and thus underutilized (RBAC and zones in Solaris, ACLs, bash 3.2 debugger, expect, Jumpstart/kickstart, flash archive in Solaris). Many features are typically disabled or misconfigured (built-in firewall,  RHEL SELinux, SUSE AppArmore,  Xen,  snapshot capability of linux LVM,  etc).  Typically the most advanced usage of OS can be observed in mono-culture and dual-flavors environments. Dull, absent of any ingenuity, basic deployment style is a rule in any environment with more then two flavors of Unix used.  That significantly increase TCO both directly (lower productivity of equipment and people) and indirectly (acquisition of packages and hardware with the explicit goal to tame the excessively diverse environments)
    
3. Enterprise linux flavors are in all dimensions very similar to proprietary software and generally suffer from the same weaknesses (and first of all overcomplexity) with some (instability) even more pronounced.   Promises have been made. Assurances have been given. Commitments have been published. But far less has been delivered. Linux became just another operating system choice, a clear case of be-careful-what-you-wish-for. It is clear that in some more important in my view aspects linux in technically inferior to Solaris 10, while in others less important aspects Solaris 10 is inferior to enterprise linux flavors.  But in no way linux can claim technical leadership. It's more or less successfully is following tailgates of proprietary Unix flavors (and not only Solaris).  Stability of enterprise linux distributions is definitely less then stability of Solaris 10 and other major commercial Unix flavors (AIX 5.3 and HP-UX 11i).  Linux distributions are rather bloated, complexity of the kernel is high, regression testing is limited and sometimes errors including kernel errors can be introduced during regular patching process due to the update of the kernel or particular important subsystem.  "Blue screen of death" due to driver problems are not uncommon with the only differences from Windows that at the time of the crash linux kernel often is unable to display even the  panic message.  Server simply froze.  Patching in linux is more dangerous process then in Solaris 10 and for critical servers, whenever possible, should probably be limited to the minimum subset of security patches.  Due to stability problems linux should preferably be used in applications were redundancy is built-in in the design.  Integration with directories (including Active Directory) is important to the extent that it can be said "It's the Directory, Stupid" and here the favorite is unclear but it might be Suse.  Novell looks slightly better then others in this area and LDAP authentication with eDirectory synchronized with the Active Directory is used in many enterprises.
    
4. Qualification of sysadmins is the key to stable and reliable work of any Unix or linux server farm. There is no substitutes and neither Solaris not linux can fly well and do not crash without experienced pilots.  High qualification can be achieved only if sysadmin is responsible for no more then two flavors of Unix. The diversity of Unix flavors in a large enterprise environment should be tightly controlled and "counter proliferation" efforts should be an important part of any sound enterprise datacenter policy. If introduction of linux increases the diversity it generally makes infrastructure less cost efficient, not more cost efficient.
   
   This is connected with the fact that the complexity of modern OSes had risen to the level when it is almost beyond the capability of single, even very intelligent, person to understand them. Also OS themselves represent a moving target (linux to more extent then Solaris or, AIX, or HP-UX) with new versions arriving at regular intervals.  Due to this top level admin skills can be acquired only after many years of hard work (forget about people claiming to be "experienced system administrators" with just one or two years of administrator work under the belt, unless they are former programmers on the same OS).  Due to the level of variety between different Unix flavors sysadmin skills are to considerable extent Unix flavor specific and that's why usually people tend "naturally" concentrate on a single ("loved") Unix flavor and dislike others (in addition one "minor" flavor can be learned reasonably well too).   Administrators with deep knowledge and passion for the particular Unix flavor currently used in the datacenter represent important part of the company intellectual capital, the capital that can be easily wasted in case of transition.  That actually might can help to explain such a persistent phenomenon as "OS nationalism" often demonstrated in discussions like Solaris vs. Linux as they usually pretty well resemble the style of USA culture vs. Great Britain culture (Canada or Australia can be substituted for Great Britain) discussions (you know both countries share the same language, don't they ;-).  Unix sysadmins who moved to a different flavor of Unix feel much like expatriates for several years as considerable part of their skills is Unix flavor specific and the higher qualification they have is heavily based on deep knowledge of this "specialized", flavor-specific  part.   For administrators with almost a decade of experience in a particular Unix flavor under the belt, to quote  Linux Torvalds, switching from administering one OS to another is not unlike “performing brain surgery on yourself”. This is one of the major reasons why adding any new Unix flavor to the large enterprise Unix mix usually does not provide for expected savings. 
   
   From the point of view of sysadmin training Solaris and linux are the most compatible with each other and least toxic pair of enterprise Unix flavours available.
    
5. We need to distinguish benefits of Intel/AMD hardware from benefits of the new OS adoption.   Advantages of linux are too often uncritically mixed with the advantages of switching to Intel/AMD hardware and first of all its dramatically better (often twice or more lower)  price/performance ratio of Intel Duo CPUs in comparison with Itanium and RISK CPUs. In reality, for large enterprise environment with multiple Unix flavors installed, linux offers very limited advantages and some noticeable disadvantages if compared with using existing commercial Unixes on competitive hardware (for example with Solaris 10 on Intel/AMD hardware).  Fashion-based replacements of Solaris SPARC servers with linux servers without considering possibility of moving to Solaris 10 on  Intel/AMD hardware increase costs instead of providing savings: the key advantage is faster hardware.  That suggests that days of Sparc on low and mid-range servers might be numbered despite T1/T2 CPUs.
   
   While Linux does has important advantages over Solaris in entry level and mid-range servers deployment areas, Solaris holds its own on midrange and large servers and in light weight virtualization space.  Until recently an most important advantage of linux was non-technical: it was the  availability of cheaper hardware from vendors like Dell (although Sun Opteron-based servers were more or less price competitive). But since late 2007 Solaris 10 is officially supported by Dell on its most popular and very good PE1950 and PE2950 servers. Still linux has wider hardware spectrum supported, as well as due to "home field" advantage (as a platform of development for many open source applications). Those advantages partially fade as we move to middle range servers. They are completely absent for high end servers were linux simply is not a real player (if we discount the role of DOS it is playing in IBM's VM/linux mainframes). 
    
6. Linux plays a tremendously progressive role at large enterprise environment as a counterbalance to strangulating IT bureaucracy.  Born as an alternative OS it still can live to its promise in this particular environment. It permits running small, "guerilla" projects and experiment with new technologies like scripting languages.  It also can lessen the negative effects of "pseudo-security" efforts of  "overzealous know-nothings" at desktop area.  Actually, in security area large enterprises IT should fear more from the bungling of the incompetent than from the machinations of the wicked.  In principle, Solaris 10 can play the same role, but it requires more efforts both in installation on corporate desktops and configuring the necessary software.
   
   Bureaucratization of IT has very positive influence on linux/Unix adoption and significantly diminishes attractiveness of  "pure" Windows on desktop and stimulates adoption of "mixed" model with linux and Solaris virtual instances. 
   
   The litmus test of the level of bureaucratization is prevalence of form over substance and as a side effect fashion rules and logic does not necessarily prevails in discussions about the relative benefits of introduction of a new OS. Aging IT bureaucracy like any other bureaucracy develops goals strictly related to self-preservation.  The more dominant are those self-preservation tendencies the more bizarre and damaging (from the point of view of common sense )enterprise IT moves can be expected,  the more politically motivated major technological decisions become( misdirected SOX compliance efforts are a good example here ) and the less they care about you, the Unix administrator.  On the other hand the same bureaucracy in Windows space push the most technically astute users to the "Unixland" as bizarre and arbitrary limitations make it difficult to use Windows productively.  With Active Directory group policies available, Windows world more and more reminds mainframe world.  In this sense linux (and to lesser extent Solaris) serve a very positive and extremely important role in modern IT: the role of "freedom fighters weapon of last resort". 
    
7. Due to the complexity (should I say overcomplexity ?)  of modern Unixes the value of OS certification cannot be overestimated. Having certified in particular flavor of Unix administration for at least some administrators on the floor is probably the most reliable way to  avoid rather painful errors and horror stories at the initial stages of introduction of any new flavor of Unix. For example, it is clear from the content of the paper, that the expertise in Solaris or AIX administration is not directly translatable into linux domain and attending one intro or "transitional" course is not enough -- such a bootstrapping approach and the idea of "growing sysadmin expertise with the system" might backfire discrediting the OS in question more then actual or perceived shortcomings.  At the same time linux certifications suffer from the same "multiple-personalities disorder" that linux itself is suffering from.  Among vendor certifications Red Hat certification looks like more objective measure of skills and IQ then Sun's certification  (although I noticed several bad apples here too, it is more difficult to fake by memorizing the material without understanding it).
   
   Still Sun has an extremely good and largely deserved reputation in terms of quality of support, training and certification. In those areas it is superior to offerings from Novell or Red Hat although Red Hat has an advantage of keeping training "in-house" while Sun outsourced it and that negatively affects quality.  Novell currently is more democratic vendor as for training and certification in linux enterprise space (Red Hat has the most expensive training and certification options, expensive even if we are talking about large enterprise financial capabilities).  
   
8. In no way linux can completely replace enterprise Unixes; after more then fifteen long years, the zeal to build a brave new OS is cooling. The leadership, from Linus Torvalds down to the lowliest kernel driver coder, seems more tired than inspired. The ruling "Linux elite" seem reluctant to make way for younger men. Cynicism due to "make money fast zeal" among the Linux elite during dotcom boom and maladministration of the kernel development further dulled the efforts. Moreover Linux kernel development efforts are spread too thin trying to encompass all the hardware spectrum from laptops to high end servers with just a fraction of resources at hand in comparison with Microsoft. The decision by Linus Torvalds to abandon stable branch of the kernel (previously with great success maintained by Alan Cox) and essentially delegate debugging of the kernel to distributors in version 2.6 does not help too.  That means that outside its major deployment area (low end servers, especially front end web servers) you should expect raw spots. Solaris on X86 is more suitable for midrange servers if and when corresponding applications are available. Solaris is more focused on servers OS, although recently Sun brass also tried to position Solaris 10 to be "all things to all people" and repeat linux mistakes. In case you order Sun X86-based hardware you get an important additional advantage of using a single hardware and software vendor (which eliminates finger pointing), the advantage that is absent for any enterprise linux distribution.
    
9. Contrary to hype, linux does not have advantages over Solaris in the development model. With opening of the code Solaris adopted the same model of distributed collaboration. And less democratic nature of Solaris development with the core concentrated at a single place might be more an advantage then liability. Large scale open source software development projects actually stimulates hierarchical power redistribution with the Great Chairman at the helm and less powerful but no less autocratic "members of Politburo" as the second level of hierarchy, the process of consolidation of power and emergence of elite that long before linux kernel development saga was masterfully depicted in Animal Farm (as reflected in popular quote "all pigs are equal but some pigs are more equal then others").  Like in fluids with certain concentration of salts this process of "crystallization of the elite" is an objective process that occurs independently of the will of the participants and their goals. Moreover reliance on faceless Internet-based communication might amplify some of problems typical for corporate environment and stimulate power struggle at the expense of real work.  Financed by consortium of hardware and commercial software vendors cooperative model used by linux (with the support of enthusiasts from many countries) demonstrated weakness of architectural vision which in turn leads to dominance of imitation at the expense of innovation.  Wrong choice of direction or changes that badly effect stability can propagate all way to the top pretty easily as seeing the whole picture is a difficult task even for the most devoted and talented developers.  With the current level of complexity of the kernel developers including Linus Torvalds looks more like proverbial blind men and an elephant. 
   
   The traditional corporate model with more clear cut lines of responsibility (when a person can be hired for a particular important task or fired for a particular blunder), more concentrated presence of developers at one place, partial suppression of "vanity fair" motives by copywriting the work by faceless corporation and monetary stimulus like stock options might be not as bad as some open source enthusiasts try to depict it.  One of undeniable advantages is that communication between key developers can still be face-to-face.
   
   Due to the age of linux there is an inevitable problem connected with the forthcoming change of the leader, the change that is more problematic and painful that similar change in proprietary unix teams.
    

10. Historically Linux has far from being impressive compatibility record.  Recently it became better (Suse is the leader in this area), but still abrupt changes are the way of life despite the senior age of the OS (sixteen years is an advanced age for the OS; many successful OSes died before reaching it).  As Steve Ballmer aptly answered the question how one OS beats another, the availability of source is just one factor in the battle:

    "The way you beat any other competitors: You offer good value, which in this case means good total cost of ownership, right? Because total cost is really, at the end of the day, the issue. And the fact that, quote, Linux is open source, therefore it appears to have a zero price -- that actually made it easier to shine a spotlight on the thing that always mattered anyway, which is total cost.

    In a way both Linux and Solaris are niche players in the data center stuffed with Microsoft servers and applications and as such should more cooperate then compete.  In X86 space both are definitely riding on coattails of Microsoft as both the cost of X86 hardware and average specifications (including typical amount of RAM) on low and midrange are determined by Microsoft's share of the market.  From the point of view of X86 desktops and servers technical specification neither linux not Solaris really matter. Large companies now decide about Solaris or Linux, not because they hate one and love another; but because of perceived risks, TCO and how well it will play with their Microsoft part of infrastructure.  That means that a good interoperability is vital and more cooperation between teams is essential.  After all old saying states that the enemy of my enemy is my friend ;-)

    The second aspect of compatibility is the danger of proliferation of flavours. It should be stresses that  Solaris does not have the danger of proliferation of flavors.  This issue cannot be swept under the carpet as there is a real danger to bet on a wrong horse and later face the necessity to support two enterprise flavors of linux in one organization. The leading linux vendor (currently Red Hat) does not occupy very stable position (Oracle alternative support model really cuts into the profits) and can be eventually displaced by Novell Suse  or (less likely) Ubuntu which is currently a rising star among linux distributions.  Red Hat already lost to Ubuntu a lion share of the market in linux books. Suse has been tuning kernel for AMD for a few years (they wrote the GCC x86-64 back-end) and now enjoys support of IBM.  All-in-all internal linux fragmentation is the replay of old Unix wars and as such is  a serious threat.  I doubt that enterprise system administrators can benefit from remembering 3 ways of doing things, for example, changing resolution of the screen (one for Suse, one for Red Hat and one for Ubuntu). Just a threat of competing distribution winning at the marketplace over adopted in the particular company (say, Suse vs. Red Hat) somewhat cools enthusiasm for linux.  No amount of hype can hide the fact that the cost of switching from one flavor of enterprise linux to another is comparable with the cost of switching from one proprietary Unix to another: the same level of vendor lock-in and associated problems with re-certification of applications, partial retraining of administrators, etc.  No amount of Linus Torvalds interviews can hide the fact the linux is fragmented into two major enterprise flavors which can be viewed as competing OSes with common kernel.  If you do not understand the value of single version of OS please browse Windows evangelism documents starting from page 9.  While it is highly Microsoft-centric it's pretty instructive as for the role of single standard for the prosperity of ISVs. Note the knockdown of competitors with .NET recently achieved by Microsoft. 
    

11. Solaris currently has more technically advanced kernel with much better instrumentation capabilities (due to DTrace and solid OS dump infrastructure) while linux has superior "external" personality.   Due to those advantages Solaris 10 is more suitable for deploying complex applications like databases and ERP systems (for example Oracle and SAP/R3).  Dump infrastructure in linux is primitive and buggy. Tracing also leaves much to be desired and far behind the capabilities of DTrace.  Due to better instrumentation with proper tuning Solaris 10 can achieve performance comparative or better then linux on X86 architecture.  This is especially true for complex applications like databases-driven Web applications. While linux is definitely fast, rumors about linux being significantly faster then Solaris 10 on Intel/AMD architecture are greatly exaggerated. Actually Solaris has performance edge over linux for applications that heavily use threads.  Availability of  enterprise applications might still be a problem for the adoption of Solaris 10 for X86.
    
    As for "personality" of OS linux beats Solaris: linux looks like more modern OS for administrators and provide them with a lot on non-trivial and important capabilities (better package management, YAST (which is now available on Red Hat due to Oracle porting efforts), loopback interface,  etc).
    
    As for networking Solaris beat linux: better implementation of NFS and other complex networking protocols, more flexible TCP-stack.
    
12. On server level security side Solaris 10 has a substantial lead over any linux distribution and its security mechanisms are less disruptive for applications. Both RBAC implementation and zones are superior to mechanisms used in current linux distributions (with the possible exception of Suse AppArmor, which is a very elegant technology indeed).
    
    Solaris RBAC has tremendous value as a security mechanism perfectly suited for the large enterprise environment. In combination with zones RBAC represents a unique method of preventing "root sharing hell" by ensuring real separation of duties. Solaris zones essentially allows application owner to control it own lightweight virtual machine and as such greatly reduce conflicts in access control in Unix environment.
    
    Recently Solaris RBAC also became one of few ways to channel large part of SOX compliance efforts in a constructive way and limit the negative influence of "SOX socialism" on large enterprise IT environment. For RHEL 4 no amount of hardening can match the security of Solaris with applications running in zones and well structured RBAC "separation of duties". Suse AppAmor is an elegant technology and does has promise, while RHEL security infrastructure suffers from overcomplexity and due to overcomplexity actually is the weak spot of distribution (cases of systematic switching it off in production servers are not rumors, they are fact of life).  This fact, combined with the necessity (and dangers) of more frequent patching for linux means that maintaining the same level security of servers on linux servers will always be more expensive for large enterprises then maintaining the same level of security on Solaris 10 servers be it X86 servers of UltraSparc servers.  Of course here like in most other areas the qualification of staff  is more important factor then differences between those two OSes. 

Created Jan 2, 2005.  Last modified: August 22, 2008