Softpanorama (slightly skeptical) Open Source Software Educational Society

May the source be with you, but remember the KISS principle ;-)

Solaris Competitiveness on Midrange Servers

People who typically post comments about Linux vs. Solaris usually don't run anything mission critical and have probably never seen anything more powerful than your average 2U box. They concentrate mainly on initial costs. Things like reliability, scalability and manageability never even enter the equation for them.

But for mid-range servers beyond the lure of low cost of hardware an organization needs to consider the TCO which includes operating system and hardware support costs (1-2K a year), database or ERP licensing costs and maintenance fees (often per CPU/core), cost of  system administrators and network infrastructure. Stability is one of the primary concern in this range. Actually stability that is pretty adequate for development desktop/laptop is completely inadequate for midrange server.

The fact, that the initial costs of the midrange servers is probably the least expensive part in the total cost of ownership (TCO) of such a server (let's assume five year lifescan). During those five years lecencing and maintance costs of software intalled as well as the cost of administration and maintance contracts usually exceed the initial cost of hardware. Sometimes exceed many times. 

Midrange and high end server are not very competitive with low end servers on the base of hardware costs only (CPU are the same or just slightly better, I/O is better but not equal to the sum of I/O channels on equal number of low-end servers), but they are more reliable and more subsystems duplicated in hardware and they are very competitive on the base of TCO costs as administration costs are lower that the cost of administration of several low end servers.  The amount of supported CPUs and, especially,  max memory that can be installed are really important too. You cannot generally call a mid-range server any machine that does not support more then 8G of RAM because this defeats scalability. Often desirable upper limit for midrange servers is 64G and some midrange servers like SAP/R3 application servers have this amount of memory actually installed from the day one.

Saving on administration costs is a very important consideration. If you can save one Unix administrator position (for example, by not adding a new one due to increased workload after a large acquisition) that means that in the USA you saved approximately $60K-$100K  per year.

There are two major areas of deployment of midrange servers in large enterprises:

• Consolidation game. This is pure attempts to lessen administration costs by running applications that previously used to run on many smaller servers on a single physical server. Now consolidation game is played mainly in virtualization space and we will address this issue later.
• Database servers, ERP and financial systems. Those will be discussed in this section.

Midrange servers deployments in large enterprises are dominated by database servers. Databases are big business -- just look at what some companies spend on licensing Oracle. And in most companies database servers are used very inefficiently: the level of a database group in a typical large company is usually rather basic and they rely more on "brute force" then sophistication in designing and managing the database.  Also the structure of many databases are simply beyond their control.  For example,  often the database is a part of an ERP system or enterprise application developed outside the company and as such should be taken "as is".

Enterprise Resource Planning (ERP) systems have become popular  in large corporations and they support critical business functions such as supply chain, inventory, and customer relationship management. The choice of the right platform for ERP systems is extremely important for large enterprises as it greatly affects the total cost, efficiency and future extendibility. Any unplanned downtime can lead to reduced productivity and lost revenue. The ERP platform must also be able to easily handle growing amounts of data and large numbers of concurrent users with reasonable response times. Hardware should have some space for growth and allow for, say, at least 30% growth of the user population with just memory and CPU  upgrades, without total replacement of the current hardware.  Therefore if system currently requires 32G of memory the server should be able to carry 64G (16 * 4G chips).  That means that ERP servers represent upper zone of midrange servers and in many respects are close to high end servers.

In case the database server is used in ERP system like SAP/R3 the structure of database is dictated by the ERP vendor and a typical benchmark often means more then for low-end servers and price of the server means less.  The number of CPUs or cores that are necessary to achieve given performance also matter as database licensing is per CPU and per core.  Generally, the less CPUs and cores the server has the better, and that's why IBM has reasonable success with its Power 5 line in ERP systems area.  But with Intel Duo 5160 available no more. Here are  Fourth Quarter 2006 SPEC CPU2000 Results.

It looks like hardware-wise old lines of CPU including P5 and Sparc started moving into "mainframe road", the road to a niche player.

Please note that the level of loyalty to linux and the level of support from the major vendors (IBM and HP) is marginal for this server class: for midrange servers both usually try to sell their own proprietary versions of Unix (AIX 5.3 and HP-UX 11i, respectively).  IBM openly claims that AIX is the strategic platform for all mission critical, core business applications and until Intel Duo 5160 was available there was some truth in it. Only Dell can be called a true linux midrange server vendor, the vendor that does not have "dual loyalty" problem. 

Another important factor that hunts linux is that loyalty shifts among major supporters for the flavors on enterprise linux. It looks like IBM started downplaying Red Hat in flavor of Suse for some time. So there is a distinct danger that any large enterprise which adopts a particular enterprise flavor of linux; events beyond their control might eventually force them to support both main enterprise distributions (Red Hat and Suse).

Large variety of hardware supported matters much less on mid-range.  For example hard drives are often used via NAS (sometimes without clear cut advantages at at considerable additional cost; see below). 

Sun curse

UltraSparc CPUs seriously lag in benchmarks latest Intel 5160 CPU offerings (more then P5 in the table above) and in this metric Sun servers lost competitive advantage. Only newer line of midrange Sun servers with Fijutsu UltraSparc CPUs are competitive with Opterons (but not 5160) in SPECINT, and SpecFP and newer and more accurate  SpecCPU benchmarks.  At the same time Sun Opteron based servers get high marks on all benchmarks although currently they lag Intel 5160 CPU. 

But we should not mix the problem with Sun hardware and slow speeds of UltraSparc CPUs (or IBM P5 CPUs) with the advantages or disadvantages of Solaris and linux although it is difficult to completely separate them .

Solaris on midrange servers is closely associated with the UltraSparc in the minds of many enterprise customers. This is a tremendous marketing problem that Sun currently solves unsatisfactory and some customers who already have substantial Solaris presence and trained personnel migrate to linux instead of migrating to Solaris 10 X86 and saving headache of introducing a new OS in the enterprise mix.  But there not that much logic in large enterprise IT behavior anyway, it is  a kind of Alice in Wonderland environment in its best  ;-). Still a commercial where Sun brass sign with blood the commitment to support X86 line of Solaris on equal basis with UltraSparc might be very beneficial  :-)

The main problem for Sun in this space is that Sun's CPU design team is chronically late. Like with low end V210 and V240 boxes that are hampered by using slow UltraSparc IIIi CPUs, hardware-wise Opteron is squeezing UltraSparc on midrange servers: you could buy a large X4600 for less money than a V490. A X4600 with 4 x Dual Core 885 Opteron CPUs and 32 GB of RAM and a 4 way V490 with 32GB of RAM both cost approximately $40K. With higher speed of memory the X4600 gives you significantly better performance on typical database benchmarks and it is more expandable.  The only advantage of V490 in the advantage of  "tried and true" solution and nobody was fired for buying it.  And here you definitely can be fired if things go wrong.  Also you can get Sun SPARC Enterprise M4000 Server for more money and it is faster then V490.

Also the new Xeon 51xx (Woodcrest) made harder for Sun to claim performance advantage on midrange: four Woodcrest cores are about twice as fast as the 8 core T1 so the marketing offence that Sun unleashed wit the launch of T1 line had come to its natural end. Energy efficiency of Intel-based offerings was also improved. 

Here is the relevant summary from AnandTech Intel Woodcrest, AMD's Opteron and Sun's UltraSparc T1 Server CPU Shoot-out

Sun's T2000 server and it's 32 thread T1 CPU turned out very variable results. It is not the best choice for open source databases. PostGreSQL and MySQL scale better on Solaris than they do on Linux, but both RDBMS have trouble scaling over multiple cores. It is likely that the DB2 and Sybase results will be much better on the T2000. The SAMP web performance of the T2000 was good when we cached the PHP pages and we had few accesses to the MySQL database. When PHP pages had to regenerated with every access and the query cache of MySQL was used, performance was pretty bad compared to the x86 competition. The best purpose for the T2000 is a JSP server with SSL authentication.

The Intel Xeon 5160, a.k.a. Woodcrest, will simply be the most powerful server CPU this year (though it's not yet available for purchase of course). As our extrapolated calculations show, even a 2.6 GHz Woodcrest will outperform the current Opteron 285 with a 5 to 55% margin, nothing short of impressive. The new Xeon is however not invincible: the Opteron can still give some serious resistance when running some instruction mixes with lots of rotates, add-carry or load effective address instructions. RSA, AES and other benchmarks clearly show this. Intel will still have to convince some software vendors to port to SSE if it wants Woodcrest to be the completely superior CPU. The advantage in MySQL is also rather small, a result of the relatively high latency of the FB-DIMMs. But we are nitpicking: Intel's newest Xeon has taken back the performance/Watt crown. In one word: Woodcrest rocks!

And what about AMD? The Opteron remains a powerful architecture with a flexible platform. It is quickly becoming the most popular platform for 4 sockets and the upcoming Tulsa CPU is most likely not going to change that. However part of AMD's success has been Intel's Prescott/Nocona failure. In the K6 and Athlon (K7) years, AMD managed to improve the architecture every two years. In 1999 we had the original Athlon, in 2000 we got Thunderbird (integrated L2 cache) and in 2002 we got the Athlon XP. For the few past years, the Opteron architecture has made the move to dual-core and received a better memory controller, but the necessary IPC improvements and cache enlargements have not materialized. "Only the Paranoid survive", remember?

The Intel P-M architecture went from 1.7 GHz Single Core (Banias) in 2003 to 3 GHz (Conroe, Woodcrest) in 2006, while it quadrupled the L2 cache and significantly improved the IPC. At the same time, AMD's K8 series went from 1.8 GHz to 2.8 GHz dual-core, with the same amount of cache, and almost equal IPC. The result is that AMD will not be able to regain the performance crown in the dual and quad-core market until the K8L arrives. The future looks bright in the quad socket market however.

In summary:

Intel Xeon 5160 (Woodcrest)

Advantages:

• Best server performance across all applications
• Best Performance/Watt in the high end
• Absolutely stunning web server performance
• FB-DIMM enables high RAM capacity and bandwidth (quad channel)

Disadvantages:

• Needs SSE optimized code for some special case code (RSA, AES)
• FB-DIMM adds extra latency, cost (small) and power

UltraSparc T1 / Sun T2000

Advantages:

• Superb SSL performance
• Excellent Performance/Watt with SSL and Java code
• Solaris, a robust and well scaling OS
• Quad channel enables high RAM capacity

Disadvantages:

• Heavy optimizing is necessary; out of box software performance is low
• Low single threaded performance; also results in low performance in server software that scales badly
• Price/Performance compared to Woodcrest

AMD Opteron

Advantages:

• Well rounded CPU: performs well even with non optimized code; still excellent MySQL server results
• Excellent Quad socket platform
• Does not need FB-DIMM for high capacity thanks to NUMA

(DDR2 (socket-F) offers lower latency, less power and less cost )

Disadvantages:

• Web server performance compared to Woodcrest
• Power at higher clockspeeds (110 W vs. 80 W)

The importance of memory size and speed

Both size and speed of memory are of paramount importance as database applications are very unfriendly toward caching. Memory in modern computer is a very complex subsystem. See an excellent Ulrich Drepper's article What every programmer should know about memory  for some important details the need to be understood. Figure 2.13 in the article is reproduced below and shows the names of the DDR2 modules in use today.

Array Freq.	Bus Freq.	Data Rate	Name (Rate)	Name (FSB)
133MHz	266MHz	4,256MB/s	PC2-4200	DDR2-533
166MHz	333MHz	5,312MB/s	PC2-5300	DDR2-667
200MHz	400MHz	6,400MB/s	PC2-6400	DDR2-800
250MHz	500MHz	8,000MB/s	PC2-8000	DDR2-1000
266MHz	533MHz	8,512MB/s	PC2-8500	DDR2-1066

Figure 2.13: DDR2 Module Names

Array Freq.	Bus Freq.	Data Rate	Name (Rate)	Name (FSB)
100MHz	400MHz	6,400MB/s	PC3-6400	DDR3-800
133MHz	533MHz	8,512MB/s	PC3-8500	DDR3-1066
166MHz	667MHz	10,667MB/s	PC3-10667	DDR3-1333
200MHz	800MHz	12,800MB/s	PC3-12800	DDR3-1600
233MHz	933MHz	14,933MB/s	PC3-14900	DDR3-1866

Most computer textbooks teach that a memory access is roughly equivalent to a time of execution of one CPU instruction (especially for RISK architecture).  But with new technologies the reality is that a memory operations, for example in case of a cache miss, may cost you 100 CPU instructions. Also currently the top speed of CPU is approximately three times higher than the top speed of memory (5Ghz and 1.33 GHz respectively). The gap between memory speed and disk latency is even worse.  So other things equal if you can install enough memory to keep 80% of the database in RAM you can achieve spectacular improvements in performance.  And keep in mind that 32G database is a pretty large database, but it can easily stored in memory on any modern mid-range server amount of RAM (a Dell 2950 server with 32G of RAM, two 2.66GHz quad CPUs and four 80G 15K RPM drives in April 2007 can be bought for less then $20K -- small amount of money by large enterprise standards).

Simplifying we can provide the following definition of a database server: midrange server is a database server if it  has the most expensive disk subsystem available for  particular server (that's why each SCSI controller on a database server often costs more then a low-end server), the fastest disks money can buy (now this is less important due to SAN and NAS but for the OS itself 15K RPM disks are a must), the fastest RAM the money can buy with possible mirroring (which probably means DDR2 667 MHz in current environment) and the ability to scale to 64G or more of RAM.

That's why unlike low-end server database servers really can benefit from 64-bit addressing. This is essentially is a requirement for any modern database server and that's why until recently Intel and AMD were essentially shut out of this market. Still even now the level of maturity of 64-bit version of OS and key applications are important considerations for database servers. UltraSparc has longer history of 64-bit application development. Solaris on UltraSparc has more then ten year record of using 64-bit address space.

Because on those memory requirements on midrange servers Red Hat uses more expensive distribution with a different kernel (AS kernel) and this version of kernel implements stack protection feature of modern AMD and Intel CPUs. So this security advantage of Solaris disappears while RBAC-based advantage remains and even increases especially for security conscious companies with non-Dilbert style IT management.

Larger amount of memory (64G)  that are often required on such mid-range servers as SAP/R3 applications servers are generally better handled by Solaris. Also important is the fact that Solaris 10 now shares with Linux the distinction of being the Unix that runs best on Intel-compatible hardware and level of its tuning to Opteron quickly increases with each release of OS.

Among Opteron servers the most attractive midrange offering is Sun's x4600 server which can scale to 128G of memory making it essentially scalable above midrange systems. Sun has a blueprint paper for x4600-based virtualization ([PDF] Application and Database Server Consolidation on the Sun Fire X4600 Server Using Solaris Containers ).Paul Venezia in his Computerworkd review published on Dec 6, 2006 noted:

 "the Sun Fire x4600 is a superb server with an obvious focus on virtualization, high-performance computing and database applications. If the local RAID controller supported RAID 5 and the local disk I/O had more headroom, this server would be nearly perfect."

 He also praised the motherboard design of the server:

Looking at Sun Microsystems Inc.'s brand-new Sun Fire x4600 M2, most would figure it for a quad-socket system. After all, at 4U (7 in. high), it matches the profile of the four-way HP ProLiant DL585 and Dell PowerEdge 6850. A quick peek under the hood tells a different tale, however: The Sun Fire x4600 M2 holds eight easily swappable sockets.

Armed with current dual-core Opteron processors, this equals 16 cores per server, so a full rack would bring the total to 160 cores. The Sun Fire x4600 M2 will also be able to run with the next-generation quad-core AMD Opteron chips, bringing the total core count per rack to an amazing 320. Packed with 128GB of RAM per server, that's a full terabyte of RAM in the same rack. At $51,995 for a single x4600 with eight dual-core Opteron 8128 CPUs, 32GB of RAM and two 73GB SAS drives, that power doesn't come cheap, but the x4600 offers quite a bit of bang for the buck.

While unpacking the x4600, the first thing I noticed was the enormous fan arrays. Two sets of four large fans sit right in the front of the case. The only cooling fans in the entire chassis, they push air directly over the vertically mounted CPU modules, and they are surprisingly quiet during normal operation. Behind these modules are six half-height PCI-E and two PCI-X expansion slots with plenty of elbow room. The almost complete absence of ribbon cables was also surprising.

Hardware error conditions in the x4600 are handled with aplomb. Should a dual in-line memory module (DIMM) fail on any of the processor modules, the release handles light up. Further, pulling a CPU module out of the server and tapping a button will cause the socket containing the bad DIMM to light up using power from an on-board capacitor. All these features lead me to the conclusion that this is one of the best-designed server chassis I've ever seen.

Powering the x4600 are four 850-watt power supplies operating in a two-by-two redundant configuration. Although the system wouldn't power up with only two functional power supplies present, it would continue to run following the loss of the same two power supplies. This adds up to a lot of power consumption, but not as much as I expected given the server's overall performance and abundance of processors.

I suspect that 64-bit applications maturity advantage that Sun was enjoying in its UltraSparc line and partially in Opteron line disappeared in 2007, when dominant part of Intel servers shipments as well as mainstream desktops and laptops has become 64-bit. Also the fact that UltraSparc CPUs are not very competitive speed wise with AMD and Intel CPUs does not help.  If a database engine or an application is licensed per core or per CPU, then using slower CPUs can translate into significant additional cost for software maintenance. For example T2000 with its 1.4GHz eight core CPU is nice server but running Webshere will on it will cost you twice as much then running Webshere on Dell Power Edge 2950 with one quad core CPU. Pleas note that  counting core we are not counting threads which T2000 can use more efficiently due to its architecture.

Moreover 64-bit advantage, while temporary, does still exists and slightly favors Solaris over linux.  Even in 2007 a lot of applications on linux are deployed in 32 bit mode as the dominant part of linux developers (especially outside of the mainland USA) still works on 32-bit machines. While both Red Hat and Suse provide 64-bit versions of OS and all major applications, nobody will deny that a large enterprise which goes this path takes additional risks due to relative immaturity of the platform in comparison with classic 64-bit RISK platforms (be it Sun's, IBM's or HPs). 

I need to stress it again that for database applications the speed of memory and the speed of I/O is often the bottleneck and other things equal the server that has fastest memory, the most expensive SCSI controllers (or NAS storage) and fastest drives (15K RPM) wins. With modern speeds of CPU, the raw CPU speed is bottleneck less often although it does influence the total throughput (generally less and less after speed of CPU approximately doubles the speed of memory (with approximately corresponds to the CPU speed 2.66GHz with current memory offerings). With higher CPU speeds cache misses probably eat most of gains due to specifics of database type loads (but sometimes you also can see this effect on different loads too, even as cache friendly as Webspec benchmark which reproduced in the previous section of the paper; please note that slower CPU si dual-core CPU):

It is logical to assume that the return on investment in CPUs faster then 2.66GHz with current memory subsystems is marginal, especially if you have several CPUs: on database loads memory subsystem becomes a bottleneck.

Filesystems comparison

Solaris 10 also can claim some advantage in filesystem area due to presence of ZFS filesystem. Basically linux needs a new filesystem. Ext2/ext3 filesystem is showing its age. While ZFS is new (and probably slightly overhyped ) it is a very good scalable filesystem with a lot to offer and it definitely makes Solaris 10 more attractive on midrange.

ZFS does represent state of the art filesystem that satisfies the most stringent requirements of large businesses. At the same time you face problems with all major linux filesystems: ext3 is slow and ReiserFS v3 (still used in Suse 10 including Suse 10 SP2  as default) has scalability problems and its author is in jail and faces an uncertain future. As Jeff Maloney from Suse Labs noted, due to those problems ReiserFS v3 was replaced with Ext3 in OpenSuse 10.2 (the original post is available from many SUSE-related blogs, see for example here):

We’ve been using ReiserFS as our default installation file system for the last 6-7 years now, and it’s served us well in that time. Unfortunately, there are a number of problems with it. ... ReiserFS has serious scalability problems. ... While I realize that XFS-style scalability isn’t a real goal ... for ReiserFS, the scalability problems are real. ... ReiserFS has serious performance problems with extended attributes and ACLs.
... ReiserFS v3 is a dead end. Hans has been pushing reiser4 for years now and declared Reiser3 in maintenance mode. ... Reiser3 lacks a number of features ... such as extents and growth beyond current limits. Since it’s in maintenance mode, that’s unlikely to change. I view reiser4 as an interesting research file system, but that’s about as far as it goes. I’ve been unimpressed with its stability so far.
... The solution for replacing an aging file system isn’t to switch to a brand new unproven file system, but rather a proven one with a clear upgrade path. That file system is ext3. Ext3’s performance in some situations may not be on par with Reiser3, but it scales better....

It should be noted that XFS, a high quality file system donated by SGI  is available on linux (it is an installation option), but for some reason is rarely used. It probably should be used more, but the quality of integration and support issues turn it into Catch 22.

I would like to repeat it again: quality of the filesystem and the speed of the disk subsystem has tremendous importance on midrange servers, especially database servers,  and there is significant difference running the same database on server with the most expensive SCSI controller and 15K RPM drives and, say, integrated with motherboard-SCSI controller and 10K RPM, or worse 7,200 RPM drives.  NAS storage can get even better (closer to mainframes) results due to large memory caches that it uses but network latency can be a problem.   Other things equal for typical database applications money should be spend on the fastest (and most expensive) I/O subsystem the money can buy, then on as much memory as money can buy and only after that on CPUs...

Fighting SANs Fashion on Midrange

One important but often overlooked advantage of using Intel/AMD hardware and linux or Solaris is the ability to cut expenses of SANs. While SAN has legitimate usage in large enterprise environment for large databases and often is indispensable, this is a very expensive solution and one should not try to shoot birds with a cannon.   Small, especially catalog type databases (mostly read operations) can perform very well with the local storage. SAS RAID cards can scale up to 4GB/s.  That' s a very high I/O speed and you can use most of it by using solid state drives (for example Intel X25-E;  they have almost zero read latency, sustained sequential read speed of up to 250MB/s and sustained sequential write speed of up to 170 MB/s)

At the same time when dealing with really large databases accessing data is not the single issue. Other operations such as importing/exporting data into database, backups and restore operations are equally important.  A SAN with dedicated ports for data and backups separates those operatios.  A 10GbE, when properly configured can get 1Gb/sec from a single port. That's important for backups (see  Joe Chang Recommended Server Systems, 2008 Q3 - Dunnington six-core)

One need to understand that transmit data via fiber or copper with speeds that current hard drives generate is not that easy. Network latency and possibility of oversubsctiption of availbale netwrk channels are issues that should not be overlooked.  Additional complexity is another factor: you add more expensive components like cards and switches each of which can fail. You can duplicate them by suffering from adding costs but sometimes a simple question arise: "Can local storage be just adequate for the task in hand?" And the answer in many cases is definite yes.

Also the cost of  SAN connection is substantial and for equal comparison the money spend should be used for improving I/O (by increasing the number of controllers and switching to solid state drives) and increasing memory size and possibly its speed on the server.

The rule "use SAN as the weapon of last resort" is often violated.  Flexibility achievable with SANs is often overrated and using drive images and appropriate software can provide 80% of flexibility with lesser costs.  In many cases it is used with RISK-based servers just because of fashion considerations, aggressive marketing or due to some kind of  limitation on the size of individual hard drives (which probably is long the thing of the past).  In any case Intel/AMD servers has no such limitation and can use large size drives (for SAS that means 300G and, amazingly, for SATA that means up to 1T per drive) without problems.  That means that other factors that scalability should be considered before making a decision.

Internal filesystem get similar of sometimes better performance (on continued transmission of large amount of data where caching does not work local subsystems have an edge; after all this is a local filesystem and no fiber connection with expensive switches is needed -- saving on two fiber optic cards and switch alone can lead more flexible, faster I/O subsystem (controller and drives) and large amount of RAM on the server at less cost). Using such trivial improvements as high-end I/O controller and 15K RPM drives, bigger RAM and larger amount of physical drives (one pair per critical filesystem) can be adequate and save money that otherwise might land in EMC coffers. It is not uncommon for a large enterprise paying EMC almost the same sum annually as for two major Unix vendors server maintenance. This amount can (and probably should) be cut using other vendors but the same question  "Can local storage be just adequate for the task in hand?" should still be asked before making final decision.

In this area ZFS leads the pack due to its unique advantages but linux is not far behind and with proper tuning is very competitive.  Among other filesystems it supports IBM JFS.

We should not believe in marketing hype and in this case losses due to overinvestment in SANs can be substantial. Benchmarks for the equal cost solutions are a must (usually you can increase the number of cores and/or amount of memory on the server in solution that use local storage instead of SANs: just two cards required cost over $2K or, for a $20K server, 10% of the total cost ).

Proof of the pudding is in the eating

But all this theoretical considerations does not worth the paper they are printed on. The configuration issues tend to be complex and depend on the application which will be running of the server. Without real benchmarks on actual hardware it is difficult to tell what improvement each platform can bring. The money involved are considerable and without a pilot and laboratory testing of actual applications and actual OSes on real hardware enterprises are bound to make an expensive mistakes in selection of midrange servers either because of vendor hype or their own incompetence or both.  I just want to suggest that Solaris 10 on X86 should be a part of such pilot. 

For example, in late 2006 Tweakers.net performed an interesting  test using X4200 (4 cores). while in no way this the results can be taken for granted and need to be reproduced in your particular environment (tuning is important issues here and the more your know about particular OS the better it will perform in your environment) they have found that "concurrencies of 15 and above made Solaris between 6% and 14% quicker than Linux, running PostgreSQL and MySQL, respectively. Quite remarkably, the latter does show a higher performance peak running Linux." As margin of error in such experiments is probably around 10% this suggests that there is no any performance advantage of Linux over Solaris on Intel platform. 

Reliability issues

Reliability is huge issues for midrange servers. On the latest Intel hardware one linux server with two quad-core CPUs (or one Solaris 10 X86 server) can probably replace two similarly priced UltraSparc-based midrange servers with the same amount of RAM but if it crashes you lose money. Possibly quote a lot and pretty fast.  The feeling when you are losing $5K each hour because you saved $20-$30K on the hardware and now you can do nothing about particular problem, the problem which with more expensive server can be solved (for example by hot-swapping of the component ) or would never occur in the first place (for example driver problem, like described below), is very difficult to forget. 

The availability of certified (or in general more or less qualified) Suse and RHEL sysadmins is very low. This made it difficult to provide internal support for key applications.  This situation is made worse by the instability of the system and the challenge of finding who is responsible for a particular crash: hardware vendor or linux distribution provider.  For example Suse 10 before SP1 was known to rather unstable and only with SP1 reached the "post-beta" level. Some of the errors that hunted Suse were present also in RHEL 4 despite multiple fix packs so the number of fix packs does not tell you the whole truth. It is more about the level of maturity of particular version of kernel, which in case of 2.6 was far from satisfactory partially due to Linus Torvalds decision to sideline Alan Cox and treat "stable" 2.6 tree as the development tree. As a result all major Linux vendor fork the kernel at certain level and backport patches.

No matter how much you pay for support in case of hangs, unexplainable reboots and like you usually find usual mutual finger pointing and very slow progress.  It was not uncommon that finding the reason for crash takes more then one day. In this case servers from recovery center can be used but still the situation is tricky.

My resent exposure to the commercial Linux problem is rather limited to Suse 10 on Dell hardware  but I did encounter several problem with Suse on Dell 6850, and Dell 2590 (the latter can be considered to be a mid-range server only in "fully loaded" configuration -- 2 dual-core or Quattro CPU and full stack of 4G RAM chips.).

The first serious problem with Suse 10 we encountered almost immediately after the installation but, paradoxically, only on some servers. So we lost quite a bit of time suspecting Dell hardware.  For some reason among a dozen of absolutely identical Dell 6850 servers several  were immune to this problem while some experienced it frequently.  But there was a couple of servers which experienced it almost each weak.  The problem surfaced approximately in Feb 2007. It proved to be quite critical for us and at one point (just before getting SP1 from Novell) we started thinking about "alternative variants". 

Here is how the problem in question is described in Novell support note 3605538 Ext3 filesystem goes read-only without the underlying storage reporting errors  The frequency of experiencing of this bug on production servers diminished after installation of SP1 but it did not went away. As of  Feb 2008 it still exists as we were never informed about workaround proposed (may be the workaround was created only in Feb 2008; we discovered the document only in May 2008):

Status(Last updated: 2008-02-25)

This issue is not yet fixed in a maintenance update of the kernel. Root cause analysis has been performed and it is expected that a fix for this issue will be included in the next maintenance update of the kernel.

Workaround

Explicitly disable barrier support for the affected filesystems, e.g. by specifying
 

barrier=0
 

in /etc/fstab's mount options field for the affected filesystems.

Another instructive problem at the end of the day proved to be not linux error but a grub configuration error (wrong parameter was passed to the kernel as a result of patch installation of some application installation). Still we found that it was not that easy for the vendor to find and fix the problem. It took quite a while and involved usual problem of "falling between two chairs" when the OS vendor and hardware vendor are different and have different agendas.

This error demonstrates itself on 64-bit Suse with the message:

"hdb: cdrom_pc_intr: The drive appears confused (reason = 0x01)"

where hdb is the device corresponding to DVD/CD-ROM drive. 

Server hangs at random time after booting (and sometime during booting)  with an average uptime of three hours.  Deletion of DD-ROM drive cured the problem completely. The same effect was disabling ide-scsi and ide-cd modules (for some reason both were present).

The problem falls in-between Dell and Novell and it took them several weeks to future it out.  Drak card was suspect, firmware was suspect, an so on and so forth. But the problem was resolved almost accidentally when Novell tried to get core dump using  kdump and discovered that even after the update to latest and greatest version it still was not working. While trying to troubleshoot this problem Suse engineer discovered the long grub line. Comparison with other servers has shown a lot of extra parameters some of them were introduced during troubleshooting but a couple was not. One of those two proved to the source of the problem. During those days we suspected everything including the kernel and while this version proved to be wrong we did discover that we are not alone and there are many interesting discussions of the error on the Web (you can try to Google it yourself) and very little understanding of what actually happing in this particular case.

The most confusing part of the problem was that removing DVD/CD-ROM drive from the server cured the problem and after that server worked just fine.  The other confusing thing was that there were many description of this error on the Web (it looks like it occurs in various circumstances and not only on servers but X86 laptops and desktops too) which pointed to the kernel as the course of the problem. Below is one of the earlier posts related to the problem . This is how Alexander Fieroch describes it in the LKML:

I've tested different versions of linux as follows. None works fine but
all have different error messages:

... ... ...

Kernel 2.6.11.11
----------------
Kernel 2.6.11.xx is the first kernel where the message "drive appears confused" is thrown.
I also recognize the following lines in syslog where the kernel hangs
for some seconds (nearly a minute) while booting:

May 28 13:47:10 orclex kernel: ACPI: PCI interrupt 0000:01:0a.0[A] ->
GSI 21 (level, lo
w) -> IRQ 21
May 28 13:47:10 orclex kernel: hda: dma_timer_expiry: dma status == 0x64
May 28 13:47:10 orclex kernel: hda: DMA interrupt recovery
May 28 13:47:10 orclex kernel: hda: lost interrupt

I have enabled "ACPI APIC" in my bios. Disabling this causes the kernel
to hang on the last message above and repeating "hda: lost interrupt"
continually.

The "nobody cared" message is still here.

...
hdb: cdrom_pc_intr: The drive appears confused (ireason = 0x01)
hdb: cdrom_pc_intr: The drive appears confused (ireason = 0x01)
hdb: cdrom_pc_intr: The drive appears confused (ireason = 0x01)
hdb: cdrom_pc_intr: The drive appears confused (ireason = 0x01)
hdb: cdrom_pc_intr: The drive appears confused (ireason = 0x01)
hdb: cdrom_pc_intr: The drive appears confused (ireason = 0x01)
hdb: cdrom_pc_intr: The drive appears confused (ireason = 0x01)
irq 18: nobody cared!

Call Trace:<IRQ> <ffffffff80150064>{__report_bad_irq+48}
<ffffffff80150128>{note_interrupt+91}
       <ffffffff8014fae3>{__do_IRQ+257} <ffffffff80110105>{do_IRQ+71}
       <ffffffff8010d83d>{ret_from_intr+0}  <EOI>
<ffffffff8015e0a1>{clear_page_range+845}
       <ffffffff8015e10a>{clear_page_range+950}
<ffffffff801641bb>{exit_mmap+222}
       <ffffffff8012faa8>{mmput+50} <ffffffff80134adc>{do_exit+306}
       <ffffffff80134e78>{sys_exit_group+0}
<ffffffff8010d296>{system_call+126}
handlers:
[<ffffffff803021af>] (ide_intr+0x0/0x17a)
[<ffffffff8034e53a>] (usb_hcd_irq+0x0/0x68)
Disabling IRQ #18
hdb: lost interrupt
hdb: cdrom_pc_intr: The drive appears confused (ireason = 0x01)
hdb: cdrom_pc_intr: The drive appears confused (ireason = 0x01)
hdb: cdrom_pc_intr: The drive appears confused (ireason = 0x01)
hdb: cdrom_pc_intr: The drive appears confused (ireason = 0x01)
hdb: cdrom_pc_intr: The drive appears confused (ireason = 0x01)
hdb: cdrom_pc_intr: The drive appears confused (ireason = 0x01)
...

After booting this is repeating continually and I get every minute a
message on the console:

Message from syslogd@orclex at Sat May 28 13:33:02 2005 ...
orclex kernel: Disabling IRQ #18

Here is Alan Cox's letter that might be related to the problem:

From: Alan Cox [email blocked]
To:  linux-ide, [email blocked]
Subject: Multiple functionality breakages in 2.6.12rc3 IDE layer
Date: Thu, 28 Apr 2005 16:48:05 +0100

Ages ago we added an ide_default driver to clean up all the corner cases like spurious IRQs for a device with no matching driver (eg ide-cd and no CD driver) as well as ioctls and file access.

2.6.12rc removes it. Unfortunately it also means that if your only IDE interface is one you hand configure you can no longer run Linux. It also changes other aspects of behavior although they don't look problematic for most users. You can no longer
- Control the bus state of an interface
- Reset an interface
- Add an interface if none exist
- Issue raw commands
- Get an objects bios geometry
- Read the identify data by ioctl (its still in proc but may be stale)

without having a device specific driver loaded matching the media - and that only works if its already detected the device correctly.

I don't have the tools at the moment to generate spurious IRQ's for devices with no driver loaded but it does look like the code may well then crash. From the way the changes were done it appears the current IDE maintainers never appreciated that ide_default existed for far more than just cleaning up ide-proc but also to handle IRQ's, opening of empty slots, ioctls and power management ?

The ability to specify the IDE ports on the command line as needed for some Sony laptop installs have also become "obsolete" over time. They still appear to work but spew a warning that the user will soon be screwed.

Alan

Greg Dickie in his post to Dell PowerEdge maillist explicitly suggested that the problems might be related to a bug in the kernels versions 2.6.13 and higher (Suse Enterprise uses 6.13 kernel):

dsm_om_connsvc scewing up the CDROM

Greg Dickie greg at max-t.com
Fri Sep 22 16:33:42 CDT 2006

• Previous message: netstat output
• Next message: pxeboot installation of Win2003 on 2950
• Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]

---

Hi,

  We've been having problems with some CDROM drives throwing errors like
"hda: lost interrupt" and then hda: cdrom_pc_intr: The drive appears
confused (ireason = 0x02)". The root cause of this appears to be a bug
in kernels > 2.6.13 in scsi_ioctl.c where they are specifying a timeout
in jiffies and not milliseconds (or vice versa) but we have found that
OMSA actually provokes the problem by sending the errant ioctl to the
CD. 

Specifically stracing dsm_om_connsvc shows this:
open("/dev/hda", O_RDONLY|O_NONBLOCK) = 23
ioctl(23, 0x30d, 0xa7dde6e0)            = 0
ioctl(23, CDROM_SEND_PACKET, 0xa7dde490) = -1 EIO (Input/output error)

Any idea why?

Thanks,
Greg

Debugging of this error proved to be more difficult then it should because the OS dump facility (kdump) in particular version of  Suse 10 Enterprise was initially broken and that linux core dump requires as much storage as there is memory (this was a server with 64G of RAM). Actually making kdump work (the version supplied with the original Suse 10 distribution was buggy) turned into a separate parallel task and this task required several days to be resolved and several upgrades. We also learned that linux patching is a very tricky art: for the application we used only vendor approved non-security related patches can be installed without breaking the support contract and application was certified only for a particular version of the kernel. 

Such errors instantly improve the level of understanding of modern hardware and to certain extent helps to get superficial knowledge about linux kernel development and its major players,  but they are real pain to fix as the linux distribution vendor and hardware vendor disagreed on the causes of error and both vendors instead of digging the code and finding and fixing it for several weeks proposed all variants of "waiving a dead chicken" type of fixes ;-).

As I mentioned before, the  removal of DVD/CD-ROM drive from the server was the only clean solution of the problem that helped to solve the problem in a reasonable period of time (we we need IDE drive in the $40K server, SCSII drive probably is as good :-). The beauty of this "radical" solution is that it saves a lot of  time otherwise uselessly spend on hardware and software vendor support with typical fingerprinting at each other but unfortunately this solution was not appreciated and troubleshooting continued.  

It looks like in linux you not only can shoot yourself in the foot, you can blow out your head instead. And after this incident the minimization of linux installation on midrange servers no longer looks for me to be an option, now it looks more like a survival strategy. As I mentioned before, the  removal of DVD/CD-ROM drive from the server was the only clean solution of the problem that was found within reasonable time. Actually it was the best solution (which  unfortunately was not fully appreciated at the time it was found ;-) :  it is unclear why we need IDE DVD/CD-ROM drive (and IDE subsystem in general in view of posts reproduced above) in the $40K server, SCSI drive probably would be as good for the same price and provides an opportunity to get rid of potentially fragile subsystem .  It looks like you need to be very careful with the set of drivers you deploy on the linux box.

Solaris on UltraSparc has slower CPUs, hardware is more expensive but it is a real workhorse and both OS and hardware is provided by the same vendor so there is no finger-pointing between hardware vendor and OS vendor. And Solaris 10 works well on X64 servers and Sun produces at least one X86 server that can compete with Dell 6850.   

Sun has worked hard to create a system that will maximize application uptime and ensure the performance and scalability to meet the business needs of  ERP and database users.  It has very good hardware fault detection capabilities, and that alone can compensate some problems with slower, if any,  performance of servers as you can use a server instead of cluster and get approximately the same reliability without neck breaking complexity.

Most faults are detected automatically and for almost everything except the motherboard the faulty hardware can be simply switched off after reaching certain threshold.  

Compatibility issues

Solaris has unmatched compatibility record. Linux compatibility record is tainted to say the least. Linux kernel  is actually a generic term for "not very compatible" line of kernels that now reached version 2.6.  Complex applications certified for version 2.4 are not guaranteed to run on 2.6.  Moreover they often are certified for a  particular release of the kernel, for example just 2.6.13. That's actually pretty funny situation which is unique to linux.

At the same time the recent agreement between SAP and Novell about joint support of SAP/R3 application stack promises great improvements both for stability and compatibility for users who run SAP/R3 on SAP-approved version of Suse 10. And Oracle definite can improve the linux compatibility issues with its move into distribution support area.  Therefore there is still a promise in this area for linux.

Open Source applications important on midrange servers

Range of open source application suitable for mid-range is extremely narrow (probably less then a hundred) and here Red Hat loses some of the advantages it enjoyed on low end servers in comparison with Sun.  Moreover Sun recently started providing customers with Studio 11 compiled packages which provide additional advantages both in speed and security in comparison with gcc-compiled packages.

Home field advantage and availability of prepackaged open source applications still matter but the total price of the system is such that you can (and probably should) hire a consultant to create set of scripts that compile and optimize application for your particular configuration of kernel and hardware. Unless you have bad luck with a particular consultant you will get more efficient application and more optimal architecture that using precompiled package. Those un-enlightened customers who use precompiled packages of critical open source applications on midrange servers lose important part of the advantages of using open source by excluding tuning, and first of all by using more appropriate for the situation compilation options. And if the application is business critical you might benefit from using every bit of such an advantage.

Multicore/multi-CPU licensing issues

Enterprise application that typically run on midrange servers often are licensed per CPU and per core. Other things equal it is better to have minimum amount of faster CPUs and minimum amount of cores. Here advantage is with Opteron and Intel servers but Fijutsu CPU-based UltraSparc servers are not bad in this metric too.

Both Solaris and Red Hat favorably treat multiple core CPUs: A CPU chip containing multiple hyper-threaded or multi-core processing elements is counted (from licensing perspective) as a single CPU.  That provides pricing advantage for using multicore CPUs like T1 and dual and four core CPUs from Intel and AMD. But here the analogy ends.  Red Hat charges three times more for linux version that is capable to scale to mid-range configurations: you need to use so called  AS server, so in comparison with Solaris you are instantly $2K (or 4G of memory) down and you probably can be better off by spending  those money on additional memory then on lining Red Hat pockets. Anyway here are current (as of end of 2006) prices. Please note that Solaris minimal level of annual support is around $300 per CPU socket (not that you should used such minimal level on midrange servers):

Conspicuous Consumption Problem

It is interesting to note that large enterprise IT department which in comparison with small startups are flush with money for hardware acquisitions and are often slipping into what Thorsten Veblen called "Conspicuous Consumption" mode: they clamor for the latest and greatest CPUs and often "under-purchase" other subsystems and first of all memory. Most of us definitely know about cases when IT staff of large enterprises bought  servers with a several 3.6GHz (or whatever is the latest and greatest Intel GHz offering) CPUs and with just 2G  of RAM per CPU as if this is a sign of a mature enterprise IT.  Considering the heat and throttling issues of the old Intel 3.6 GHz chip, this was a really bad move :-).  The only way to rightsize hardware for a complex application like ERP or database application is by piloting it on several real servers (most vendors provide free 60 days try period for such servers to selected customers) and deciding based on real data what matters and what does not.

Otherwise the purchasing strategy often looks like buying the same server with, say,  2.8GHz CPUs when 3.6GHz CPUs are available is just for suckers, even if due to differences in CPU prices the first server can have twice as much memory then the second (that actually happens not without help form hardware sales reps :-). But we can do nothing for the natural human desire of "keeping with Jones" ;-). 

In this stupid "megahertz game" Sun does not looks too attractive as it neither T1 nor Opteron scale to top megahertz. Still Sun CPUs run above memory speed and it is memory speed that at the end determine the real throughput of most midrange server that are running ERP applications or databases (unlike computationally intensive applications all those game with L2 and L3 caching do not change the big picture for this class of applications).

Although outdated  (released before dual core CPUs became available) January, 2006 Anandtech's   benchmarks "  correctly states: "Single core, single threaded CPUs do not have a chance in this market anymore". But comparison of  T1 and Power5 is still relevant and might be interesting for enterprise IT staff , so I decided not to delete it:

SAP 2 Tier

This benchmark is based on the number one ERP software, according to De Gelas, with the database back-end and application running on the same machine. He points out that some of the queries are more complex, and therefore, the T1 cannot outperform the fatter cores. However, he emphasizes, "the performance per watt is unbeatable."

• Sun Fire T2000: Processors: 1x 1.2GHz UltraSPARC T1; Power Dissipation CPUs: 72-79 W; # of Cores: 8; # of Active Threads: 32; Score: 4780 % Score: 97%

• IBM p5 550: Processors: 2x 1.9GHz POWER5+; Power Dissipation CPUs: 320-360 W; # of Cores: 4; # of Active Threads: 8; Score: 5020; % Score: 102%

• HP DL580: Processors: 4x 3.33GHz Xeon MP; Power Dissipation CPUs: 440-520 W; # of Cores: 4; # of Active Threads: 8; Score: 4700; % Score: 96%

• HP DL385: Processors: 2x 2.2GHz DC Opteron; Power Dissipation CPUs: 140-180 W; # of Cores: 4; # of Active Threads: 4; Score: 4920; % Score: 100%

Reflecting on this UltraSPARC T1, De Gelas states that it is the beginning of a new generation of server CPUs. "Single core, single threaded CPUs do not have a chance in this market anymore," he writes. "The Sun UltraSPARC T1 simply wipes the floor with the competition when it comes to performance per watt. According to this metric, the UltraSPARC T1 is 4 to 12 times better."

Still it is clear from this benchmark that servers with new Intel dual-core 1560 CPUs beat both T2000 and IBM p5 550 on the same benchmark.  Still it is also clear that before Intel dual-core CPUs P5 used to be a very good midrange platform -- you need only a few CPUs to get very decent benchmark results.  T2000 is good but only for open source applications: you need to pay for 8 relatively slow cores if you run Oracle on SAP/R3 on T2000, you are penalized unless you have enterprise-wide unlimited license for Oracle (actually many large enterprises have such licenses). 

One passionate tale about Solaris superiority

Here is one tale that is somewhat relevant to the topic but should be treated with a great of skepticism because of level of passion involved [Solaris Deployment Embarasses Linux... (8189) ]. Despite the nature of the site and the content of the tale there are some considerations  (especially in comments on site) that are worth reading. They provide some interesting view on the problem, the view that usually escapes major trade publications. As one reader aptly commented "When we're talking Linux and Dell, we're talking about a company slapping together parts manufactured by other companies and slapping an OS made by another entity on result. This is NOT the same process at all. ".

Again I would like to stress that I was first hesitant to include it but still I found it to be an interesting illustration to the topic. It is dated by mid 2004, so please keep in mind that things definitely changed since (please note that you can buy refurbished Sun E4500 mentioned below can be bought for approximately $3500 in mid 2006 [Sun Enterprise E4500]).  This is a nice illustration how more expensive SCSI controller and tested OS beats much faster CPU with less tested OS :-). In some areas the more things change, the more they stay the same. Anyway here is the story that you definitely need to take with a grain of salt. But I think there is some truth in the experience described: 

This is a summary of real world technical project. I have not embellished nor fabricated anything.

Please try to ignore whatever the cloud of denials or rationalizations which the Cult Of Linux Loonies who haunt this site will try to bury this in. There is I believe important information in this summary for any IT manager wondering about all the hype of Linux in the datacenter.

This is a summary of a project I recently completed to upgrade the core production database from a thoroughly 'modern' Dell Intel server running Linux 2.4.23 to an obsolete (i.e. last generation) Sun Enterprise 4500 server running Solaris 9.

A little trivia to occupy the Cult Of Linux: How much RAM can a Linux 2.4.26 kernel (with full PAE enabled) allocate to a given process, for example a database process?

[If you said 64GB -- WRONG. It's actually less than 4GB.]

First The Specs:

Linux "server"
Dell Poweredge 2650
2 x 2.4Ghz Xeon Proc
4GB RAM
Linux 2.4.23 Kernel
MySQL 4.0.18 Standard

Sun Server
E4500
8 x 400 Mhz UltraSparc II (yep that old)
8GB RAM
Solaris 9
MySQL 4.0.18 (SPARC)

Please note, this is a fairly up to date and hot Intel based 'server' vs. older generation Sun technology with much slower CPU's.

----

Immediately following deployment:

- Server paging reduced to 1/5th(!!) previous
- Load during peak periods reduced from 30-40(!) to 2-4
- Processor usage (aggregate) reduced by 20%

The Service was visibly improved, and Traffic growth curve steepened immediately.

Note the exact same databases, tables, indexes, etc was migrated. There was no reduction in data size or anything which could conceivably be used by any Linux apologist to account for such significant differences.

Anyone who thinks the Linux kernel manages memory horribly, well here's confirmation. The kernel's sloppy management on the Linux server consumed left the 4GB server with about 20MB(!) free. If the database was stopped, free RAM would jump up to 3.7 or so GB. Once the database was started, free RAM would steadily drop and in a few hours would be back to 20MB.

By contrast, the Solaris server manages somehow to contain itself and the exact same database, same data, same indexes, same customer load, in 1.9GB of RAM. The Solaris server exceeds 6GB of free RAM and graphs of it show a slow, steadily decline, much as one would expect to see. It does not plunge down to practically zero in a few short hours.

With the Linux database, adding a small number of new database connections (say 25) dramatically affected the load. The Sun server doesn't even notice this small an increase, it's not even a blip on the monitors.

So on the technical side, we see vastly better memory management, and a kernel performing better under load. Gee which do you really want to put in your datacenter? The amount of lost revenue incurred by having such an underperforming database at the core of the business can only be imagined, but the idea the Linux will save you money is very questionable. Once Linux is taken out of the basements of its enthusiasts and put into production in anything but a rank start up, it's usefulness disappears and its severe limitations as a production grade NOS become more and more apparent.

This is not news to anyone who has used both in production, but may come as a surprise to some. Linux is a workstation and needs to stay there for quite a while. 2.4 was supposed to be the 'fixes everything' (Now they say 2.6 is!) release but it has not. You could scour the world and I doubt you will find any NOS which manages memory so poorly. The SMP code which is supposed to basically fall apart performance wise past 2 processors is a topic for another day.

Oracle and Solaris

Until recently Oracle developers used to develop the database on Solaris. And the temporary anomaly that for some time Larry Ellison forgot that he became rich as Croesus selling proprietary database and was promoting Red Hat, including switching Oracle developers to Red Hat  platform is probably now over :-). After Red Hat JBOSS acquisition he probably discovered his mistake and took appropriate steps (and the number of MySQL 5.x and Postgress deployments in large enterprises might help him to see a bigger picture outside the narrow area of direct competition of Oracle and Microsoft SQL Server on Intel platform :-). Recently he became more favorable to Solaris and stated:

"A certain number of customers will choose Linux, and we want to make sure our products run on Linux and perform best on Linux. But we also feel that, on an ongoing basis, customers are evaluating performance, technology enhancements and want choice when it comes to architecture. We feel that Solaris 10 provides that, and, for us, it is the one operating environment that allows you to design for scale-out and scale-up environments".

So, all of a sudden Linux went from strategic to a niche (32-bit only) platform for Oracle [Is Oracle moving away from Linux] Still as some developers were moved and will probably stay with Red Hat, Red Hat position probably will still continue to improve for some time, may be for a couple of years. Usage of a particular OS as the development platform creates a very important "home field advantage" and other things equal it is always extremely wise for large enterprises to select as the deployment platform exactly the same platform on which a particular enterprise application is developed. 

But in October 25, 2006 Oracle announced support for a clone of Red Hat with significantly lower support costs and that decision moved the tables in favor on linux.  That means a rather severe blow to Red Hat but it is detrimental to any other OS vendor including Solaris too.  Some people think that this is Oracle's revenge for Jboss, but big business is not about retribution.  Still details are pretty interesting and somewhat damaging for Solaris at mid-range:

Currently, Red Hat only provides bug fixes for the latest version of its software. This often requires customers to upgrade to a new version of Linux software to get a bug fixed. Oracle's new Unbreakable Linux program will provide bug fixes to future, current, and back releases of Linux. In other words, Oracle will provide the same level of enterprise support for Linux as is available for other operating systems.

Oracle is offering its Unbreakable Linux program for substantially less than Red Hat currently charges for its best support. "We believe that better support and lower support prices will speed the adoption of Linux, and we are working closely with our partners to make that happen," said Oracle CEO Larry Ellison. "Intel is a development partner. Dell and HP are resellers and support partners. Many others are signed up to help us move Linux up to mission critical status in the data center."

"Oracle's Unbreakable Linux program is available to all Linux users for as low as $99 per system per year," said Oracle President Charles Phillips. "You do not have to be a user of Oracle software to qualify. This is all about broadening the success of Linux. To get Oracle support for Red Hat Linux all you have to do is point your Red Hat server to the Oracle network. The switch takes less than a minute."

"We think it's important not to fragment the market," said Oracle's Chief Corporate Architect Edward Screven. "We will maintain compatibility with Red Hat Linux. Every time Red Hat distributes a new version we will resynchronize with their code. All we add are bug fixes, which are immediately available to Red Hat and the rest of the community. We have years of Linux engineering experience. Several Oracle employees are Linux mainline maintainers."

At the same time reliability and scalability of both the platform and underling OS mean more then support of one, even extremely important for large enterprise environment, database vendor.  On new Intel hardware linux is scalable on midrange and can outperform Solaris on Sparc by a large margin. But this is not true for Solaris on X86 platform. Sun claims 3 min downtime per year for its midrange servers.  While this is probably a marketing exaggeration there is some truth in it: Sun servers  have a very good reliability record.  That's why they are used by telco.

Late in 2006 linux again became strategic platform for Oracle as it became a competitor of Red Hat for RHEL support. The current attitude of  Oracle to Solaris is probably less certain after this move. Still being conservative has its own advantages in large enterprise environment. As Paul Murphy aptly put it 16 Mar 2005 paper Face-off SPARC-Solaris vs. Intel-Linux :

If you're running Solaris, particularly if it's on older SPARC gear, you may be under pressure to switch to Linux on "industry standard" hardware. Here's why you should resist that pressure and what you should do instead.

Listen carefully to what you're being told, and you'll notice that public acceptance of Linux on Intel as cheap, effective and gaining in popularity often has the implicit corollary that Solaris on SPARC is expensive and out of date. That reasoning comes from the fact that people update their certainties about systems pricing only when someone draws a related issue to their attention. Recently, the price reductions and advantages in the world of Linux on Intel is front and center all the time, while corresponding changes in RISC/Unix are not.

Suppose, for example, that back in mid-2000, you installed a pair of Sun 450s running Oracle, each with two 327 GB T3 workgroup arrays, 4 CPUs at 450 MHz, and 8 GB of RAM. Taken together that might have cost you around $700,000 initially, and something like $2,900 a month for hardware support after the first year. Now, five years later, users are complaining about those servers' performance. Also, some of your colleagues are saying you'd get a six-month payout just on support costs by replacing this stuff with a couple of Dell boxes running Linux. They add that you'd get an enormous performance boost into the bargain.

Your colleagues are about half right on the money, absolutely right about the performance opportunity and dead wrong about the implicit exclusion of SPARC/Solaris from the operation of Moore's Law.

That dual 450/T3 combination gave you a good bang for the buck in its day. Do a detailed review of the options available to replace it today, and you'll find that vendor pricing relationships haven't changed much on, say, Dell's 6600 quad Xeons, Sun's V40z quad Opteron, IBM's newest PowerRISC 720, or Sun's 440 UltraSPARC.

Back in 2000, both Compaq and IBM had gear that could compete on performance, but not on price, with the Sun 450/T3 combination. Today that range is narrower with Sun, Dell and IBM all clustered around $40,000 for a single system capable of handling the entire load. Even so, the cheapest and fastest options are still from Sun.

The fastest option, built around Sun's four-way Opteron V40z, is also the cheapest from a hardware perspective. At about $38,000 inclusive, it should handle the load formerly shared by the two 450s while cutting response time in half. The most expensive option is probably the Dell 6600 (I could not get fully configured pricing on the IBM 720 from IBM's Web site) at about $45,300 inclusive.

The most reasonable option, the Sun 440, is neither the cheapest nor the fastest at about $41,000 and including a 3310 external disk array. It has, however, the benefit that you can put it in place without procedural, technical or software change.

It's the overall magnitude of the change in Sun's pricing, however, that's stunning. The SPARC/Solaris solution that cost $700,000 in 2000 comes in at about $41,000 now; a 17 to one change in cost over less than five years with an approximate doubling of performance thrown in.

That, of course, is the problem: Everyone knows that Sun's servers have been getting cheaper, but they haven't put two and two together. The people pushing you to go to Intel-based solutions just haven't internalized the reality that a comparable bang now costs about 6% of the bucks it used to, whether you go with Linux on Intel or Solaris on SPARC.

... ... ...

As I mentioned before on the front scene comes such factors as scalability (first of all in the amount of memory supported by the server, and only then the number of CPUs supported; actually the less CPUs you can use the better), speed of memory (and size of CPU cache), disk I/O speed,  compatibility and, of course, reliability.

Other things equal for midrange servers faster problems resolution can be achieved if the vendor of hardware simultaneously a vendor of the OS.

All-in-all on midrange servers Solaris remains competitive to alternatives due to better handing of SMP configurations with 8 or more cores (which translates into just two quad-core CPUs),  better compatibility and, especially, better stability.

As database applications usually benefit from high-end SMP configurations (4-way and above) and from large pools of  memory Solaris still hold its own against competitors (for UltraSparc only with T1 and Fijutsu CPUs) and on dual core Opteron benchmarks Solaris 10 should be very close with linux.

In other words better hardware now is available for Solaris too and on this hardware linux does not have noticeable advantages other then the fact that it is firmly associated with Intel CPUs and is available on Dell servers.

ZFS proves unique advantages and in many cases permit using local storage instead of SANs. Linux with proper tuning can also play the same role of "SANs eliminator" cutting the costs in cases were SAN was deployed due to fashion considerations or due to the limitations of the older servers.

Created Jan 2, 2005.  Last modified: September 08, 2008