2007-03-28

Spring 2007 PC Assembler's Quick Guide ...

2007 March -- work in progress (WIP) / quick, conceptual dump

There's not a day that goes by without someone e-mailing me about PC components and assembly. The overwhelming majority of these are home consumers, although some are for some high I/O throughput server designs. I've addresses many basic server concepts in both my print and blog articles, so I will offer this blog article for the home consumer crowd. I'll try to release a new blog article every six (6) month with major changes, and these articles will be -- more or less -- a "Quick Guide" and not too in-depth.

NOTE: This article is not trying to sell you on PC self-assembly or why you should do it. If you're not sure you should, don't read this article, it's for those who already self-assemble their PCs. In fact, I personally use a sub-$1,000 17" 'desktop replacement' notebook (with a dual-core CPU, GeForce Go 7600 256MB and 1680x1050 video display, 2GB of memory and 160GB disk) as my primary PC now, so my self-assembled system is really for when I need more power.

Overview:
  • Rule #1: Seek at least 2 reviews for every product
  • Everything starts with the mainboard (aka motherboard)
  • GPUs (video/graphics) change twice as fast as CPUs
  • Power, cooling and enclosures
  • The lesser known things to know about memory
  • Storage options, what not to do, and redundancy
  • The extra frills, and budgetary mistakes
Side Discussion:
  • Free/Fake RAID (FRAID) dos, donts and conversions
Rule #1: Seek at least 2 reviews for every product

With the exception of warranty and service (or reseller in the case of DoA, Dead on Arrival), vendor brand name means squat today. Everyone outsources, even Sony to OEM Chinese fabs, Seagate to Maxtor (now the same company), etc... That means you need to research the exact model (and sub-model/revision if need be) and I recommend you seek at least 2, in-depth reviews for each and every component, possibly combinations of components if at all possible.

This also means you don't merely want to rely on "favorable reviews" on the reseller's site. You want to see more technicial expertise at times, or at least look at the explanations in the negative reviews to find some lesser know facts. After all, you're going to find some "lesser known facts" in this article, among others out there. So look for those "lesser known" facts that -- "hey, it works!"

Everything starts with the mainboard (aka motherboard)

We're not talking servers or even workstations here, so throw those out. We're talking about desktops, systems that will run 8 hours a day, maybe 14 hours in a full day (unless you have a serious gaming addiction). So we're talking about mainboards (aka motherboards) as inexpensive as consumers want to pay for their needs.

Key mainboard attributes:
  • Size and expansion
  • Integrated features and video
  • AMD v. Intel, single or multiple core
Size and expansion

In the PC self-assembly world, ignoring Intel's OEM-centric BTX, there are two (2) common standards: ATX and MicroATX. ATX has seven (7) expansion slots spread over a 12" width, and is typically 9.6" deep (Extended ATX is 13" and typically found in only workstation or server mainboards), but can be as shallow as 8". MicroATX has four (4) expansion slots spread over is 9.6" wide and can be as deep as a square 9.6", but is commonly 8" or less. Both ATX and MicroATX share a common 6" port cutout, so they really only differ in size and expansion.

At first glance, MicroATX and ATX seem to only have a 2.4" difference in size. But in reality, the actual, total "area" as typically implemented really adds up. MicroATX cuts corners not merely in expandability, but advanced feature set because it may be a single chip chipset, resulting in a smaller area and fitting into some near small form-factor (SFF) cases (e.g., MicroATX towers, desktops and cubes can be quite small, and almost as small as a more limited expansion chassis, but with 2 extras slots and drive bays). ATX tends to sport additional expansion, including newer multi-video cards options, more ports, extra on-board options, etc...

Integrated features and video

Features are really about what you are going to do with your system. If you're like me, I like either the on-board video or a single video card (we'll get into this later on GPUs), two hard drives, one optical drive and maybe one "device bay" (see the final frills section). That's my typical desktop. I don't like something that is big, I prefer something that is portable, so I've gone MicroATX since 2004 because the feature sets in MicroATX systems now sport almost everything. I still put a MicroATX board into a chassis that can take a full ATX power supply (and not a more limited MicroATX, which will talk about when we get to power). And despite the common belief, smaller cases can actually house solid cooling, power and have other, positive considerations besides just portability.

The more features lacking, the more you may end up going to an expansion board for them if you need them. If you don't need them, then you don't need the expansion, so MicroATX becomes more of a consideration. Some key, integrated features include ...
  • Audio: 2, 4 or 6 channel, maybe even optical output
  • Network: 10/100 or 10/100/1000 (although I wouldn't use the 1000Mbps in most desktop systems for a server)
  • SATA and other expansion ports
  • Integrated video: ATI, Intel, nVidia, SiS or ViA, PCIe slot or dual/tripple PCIe (aka "SLI" or "CrossFire")
Integrated audio varies wildly. More and more mainboards today, even MicroATX, come with a 6-channel audio solution. The question is how much CPU off-load (or increased CPU load) does the on-board DSP/codec give you. A lot of on-mainboard audio, regardless of board size, leaves much to be desired, and you may look to a PCI or, fairly new for audio, PCIe x1 audio card. I have personally put in Creative SoundBlaster Audigy 2ZS PCI cards even on expansion-limited MicroATX mainboards, because most audio DSPs/codecs put a lot of strain on the I/O-CPU for more than 2-channel output. It is very rare for a mainboard to have a quality DSP/codec, and most go for low-cost RealTek ALC6x0/8x0 solutions.

Network is in the same boat as audio, you can get mixed results on either ATX or MicroATX. Many desktop GbE (gigabit ethernet, 1000Mbps) would croak under a server load and result in little better than FE (fast ethernet, 100Mbps). Others actually put a good amount of SRAM (48-64KiB/port) and have decent performance. It all depends on the MAC+PHY combination.

SATA and other ports can be a consideration. Most mainboards are now coming with at least four (4) SATA ports and one (1) ATA channel. You'll want at least (1) ATA channel for an optical drive (see the storage section on why), more than one if you have legacy ATA hard drives. If you really need more than four (4) SATA ports, you might want to invest in a real hardware controller anyway, although I won't tell you what you need.

Video is really the main differentiator, and for home consumers (and gamers), probably the #1 consideration.

For people that want cheap -- around $50 -- the nVidia C51/C61 with integrated NV44 (GeForce 61x0) also known as the nForce 410/430 chipsets have an integrated video powerful enough to run Vista and even older DirectX 8 games (although they support Shading 2.0 for DirectX 9, their performance will not do well). In fact, these integrated Graphics Processor Unit (GPU) chipsets will beat the GeForce FX 5200/5500/5700LE, so it's cheaper to upgrade to a new $50 C51/61 mainboard (with a future PCIe upgrade option) than pay $50-100 for a GeForce FX AGP card. These chipsets are only available for AMD Socket-754/939 and newer Socket-940/AM2, although AMD's new (ATI-based) G690 chipset for Socket-AM2 is also comparable in performance.

If you want integrated graphics, Intel, SiS and ViA are lackluster, and not Vista capable. Yes, the newer Intel G955 options say they are Vista ready, have partial Shader support and look good in artificial benchmarks, but they are not up to the standard of the nVidia C51/C61, the AMD G690 or even the aged ATI Xpress 200/1100 integrated graphics.

If you want to go with the upper-end, then you need a mainboard with a PCIe graphics slot -- which is everything today. You may want to look towards the newer nVidia nForce 500 series (than the older nForce4 or GPU-integrated nForce4x0), which is available for either AMD or Intel. Go SiS or ViA at your own risk. Intel's own chipsets work well with an add-on PCIe graphics too -- the 975X is the premier professional, although the 965 is solid for consumers too. If you want to use more than video card, then you'll definitely need to look to a full ATX mainboard for either nVidia's SLI or ATI's CrossFire (MicroATX options exist, but I don't recommend them as they wipe out virtually all other expansion). And the price of those options will be over $100, sometimes close to $200.

In all honesty, I don't think much of SLI/CrossFire unless you are going to buy two (2) of the top-end graphics cards. The reason why is because two (2) lower-end graphics cards rarely equal the price-performance of one (1) higher-end graphics card, as I'll discuss later. That's the reason why I've stuck with MicroATX personally, but if you're willing to pay for the top-end mainboard and top-end pair of video cards, then you'll want ATX.

AMD v. Intel, single or multiple core

First off, forget the Intel Celeron D, it is not dual-core, and it's a slouch, period. there are far better options unless you find a really cheap mainboard+CPU deal (I'm talking under $50) with AGP and you have a very high-end AGP card (e.g., GeForce 6800) that you want to reuse. Pentium D, although dual-core, not much more desirable either, despite the over-clocking potential (for you enthusiasts out there), with the new "price war" between AMD and Intel.

In fact, the reason why this is under the "mainboard" section has everything to do with cost and performance and your choice of CPU as a result of the mainboard.

If you want "cheap," get a $50 nVidia C51/61 (GeForce 61x0/nForce 4x0) mainboard, put a $50 Sempron/Athlon 64 in it, and you'll get damn fine performance for most desktop applications and even aged (pre-DirectX 9/Shader) games, as well as Vista. Upgrade to dual-core for $50 more, which will be more responsive in multiple applications, although will help gaming very little. Choose Socket-939 if you want to reuse any existing DDR SDRAM memory, or just go Socket-AM2 with DDR2 SDRAM if you have to buy memory.

If you want to put in a solid game performer, buy a $100 nForce 570 or Intel 945 series, put in an $200 Intel Core 2 Duo -- preferably a 6000 series (the 4000 series isn't holding itself against AMD Athlon x2 at the price points) -- and at least a GeForce 7600GT or ATI X1950Pro or faster (see the forthcoming video section). Intel is matching AMD's price cuts, even if delayed, and the Intel Core 2 Duo 6000 series is clearly the "bang for the buck" on performance. Quad core gives you little, unless you are really building software and want to throw "make -j8" or something (if you don't know what that means, don't worry about it).

Your mainboard really defines your video and CPU choices. For cheap, Vista (or even Linux Compiz) capability, the nVidia C51/61 (or newer AMD G690, although for Linux it's still an "unknown" quantity) with a cheap Athlon 64 or entry Athlon x2 is a $100 combo -- especially in a smaller MicroATX form-factor. If you want to game, look to a $100+ mainboard with an nForce 500 series or Intel 965/975, add your own accelerator, and you can still often find then in MicroATX, although you'll definitely need to go ATX for SLI/CrossFire.

Linux Notes: Of all chipsets, I've found the nVidia nForce4/4x0 series to be the most Linux compatible, GPL/MIT driver ready chipset. As far as integrated video, as long as you don't need 3D acceleration, the C51/61 (NV44 GPU) is also excellent for X-Windows releases in the last 2 years. If you want 3D acceleration without binary drivers, Intel is an option, but understand the "open" drivers lag Intel's own proprietary drivers heavily in performance and features (especially the total lack of hardware accelerated OpenGL 2.0 support, shaders, etc...), and are only available on Windows anyway (unlike nVidia's which are a "shared object" for all OSes -- FreeBSD/Linux/X, MacOS X, Windows).

GPUs (video/graphics) change twice as fast as CPUs

With the mainboard+CPU question out-of-the-way, we can talk graphics. The problem with graphics is that they double in performance twice as fast as CPUs. If Moore's (Intel's) Law is that CPUs double in performance every 18 months, nVidia's Law is that GPUs (graphical processor units) double in performance every 9 months. So in 3 years, where CPUs increase in performance 4x over, GPUs will increase in performance 16x over.

Which is why your graphics is what you will upgrade the most, or just not care about the most. ;)

If you're still looking at or want to "recycle" a nVidia GeForce FX 5200/5500/5700LE or ATI Radeon 9200/9600 (pre-X) card, forget it. With nVidia's C51/61 chipset-integrated GPU in a $50 mainboard, you're already behind it. Today, you can get a nVidia GeForce 7600GS for $50 when on-sale, or even select ATI Radeon X1600 products for not too much more. And with the forthcoming nVidia G81+ products and ATI R600 series, prices will drop again.

Here's the deal ...
- Under $50 -- get out of here, you might as well use chipset-integrated (and save money)
- $50-60 -- the GeForce 7600GS on-sale, is also often passively cooled (0 noise)
- $90-100 -- the GeForce 7600GT or possibly ATI Radeon X1650Pro
- $130-150 -- the GeForce 7900GS or ATI Radeon X1950Pro (some would recommend the latter)

Things to avoid ...
- nVidia GeForce 7300LE (junk, maybe 50% better than chipset integrated)
- nVidia GeForce 7300GT (mega-underclocked 7600GS)
- ATI Radeon lower than X1600 (except maybe earlier X800/850 products)

If you're wanting to spend over $150, I would wait. nVidia is going to release its G81/82/etc... GeForce 8600/8900 and other series products shortly. These are die-shrinks of the already "big/costly" G80 GeForce 8800GTS/GTX. I don't like the performance of the 8800GTS compared to a cheaper GeForce 79x0GTO/GTX (or ATI's high-end options for that matter), and the 8800GTX is big and power-hungry at its 130nm size for $600 (ouch). In all cases, prices will come down, so the most I advocate is the 7900GS/X1950Pro at this time, and the 7600GT is a good start (and faster than an older GeForce 6800GT).

Okay then, what about SLI or CrossFire? What about them?

If you're going to be pushing 400W just for the video and dropping $500+ on video cards, you might as well just get an 8800GTX for $600 and save a little power usage. After all, you could go SLI by buying a 2nd 8800GTX at a latter date when they come down in price.

In fact, that's a repeat theme I see. Don't buy two (2) cards for SLI (or CrossFire) immediately. One strategy is to buy a top-end card now, and then another later when it drops in price because it's no longer the best. But then again, even when you do that, the top-end card tends to be just as good as your SLI configuration on its own in many cases. Hence the problem with SLI/CrossFire. It seems to be only the "best" when you literally spend $1,000 just for two (2) of the very top end video cards.

I mean, one (1) GeForce 7900GS typically bests two (2) GeForce 7600GT cards in SLI (let alone the 7600GS or laughable 7300GT) for less money. And one (1) GeForce 79x0GTO/GTX will typically best two (2) GeForce 7900GS cards in SLI. So I have trouble recommending SLI at all myself. Which means I have difficulty putting in an ATX mainboard instead of a MicroATX, hence my preference for the latter. But that's just myself.

Power, cooling and enclosures

Power is all about delivering current on specific voltage rails. When you don't, because your power supply is inadequate or inefficient, then you get voltage drop, which results in instability. Modern mainboards require ATX 2.0, which has two sets of +12V inputs on its standard 24-pin (not 20-pin) connector, as well as the additional +12V of the 4-pin "P4" connector separate from the "additional 4 pins" on the 24-pin mainboard connector. Your ATX 2.0 power supply should have at least two, independent +12V "rails" -- many, even ones that offer a 24-pin connector -- sometimes only have a single +12V rail. This will be noted with a +12V(1) and +12V(2), each with their own current, possibly as well as the total aggregate offered between them.

Note this is only looking at ATX 2.0, not the EPS12V or EEB SSI standards for servers/workstations (which have various other 6-pin workstation and 8-pin server connectors that are not ATX/PCIe or otherwise compatible power connectors for consumers).

Additionally, you need to consider the following, additional connectors ...
  • PCI-Express (PCIe), 6-pin, dual +12V for video cards
  • SATA, 15-pin, +12V, +5V and +3.3V for hard drives
The PCIe slot on the mainboard is supposed to deliver up to 150W of power, far more than the 25W of the AGP slot (and still less than even 25-50W or 50-100W of the lesser known "AGP Pro" slot). Prior to PCIe, many AGP cards used an extra 4-pin Molex connector for additional +12V input. With the advent of PCIe, and the reality that some video cards are pulling 250W+ -- which is over 20A on a +12V rail -- and it's not uncommon to pull more current on a Molex line than the wire gage is rated in some cases when its attached to other devices. The dedicated PCIe connector is designed for this, and you'll need two (2) for SLI/CrossFire.

Even more greatly misunderstood is the 15-pin SATA power connection. Although some SATA drives offer a Molex connector, the 7-pin data + 15-pin power SATA design is a fixed location, single connector attachment (SCA), with staggered pins for power transient. The 15-pin power connector contains three (3) power rails, +3.3V in addition to +5V and +12V. As such, it is best not to use Molex adapters with newer SATA drives as their logic boards may not be regulated to adapt +5V or +12V to +3.3V. Get newer Y-cables that split SATA to SATA itself, including providing the +3.3V that power supplies with SATA connectors do.

Cooling comes next, and it's often a major issue. Bigger heatsinks and more fans doesn't always mean better, just more protruding and louder. Retail CPU fansinks and stock GPU fansinks aren't necessarily bad, and they are warrantied. The bigger issue is the ambient temperature in your case. How much air you can allow in your case, flow over the warmest components and push out of your case is the main issue, as well as the location of components that may or may not be cooled.

Large fans spinning slower can move more air with less noise than smaller fans spinning louder. 120mm at 1000rpm has become the universal ideal, even in smaller MicroATX designs. Dual ball bearing fans are quieter than sleeve fans, and last longer.

The three (3) biggest issues in your case are in order of importance ...
  • Hard drive
  • CPU
  • GPU
Despite more recent marketing, you should never let the ambient temperature around your hard drive exceed 40C (104F). It should be warm to the touch, hot is a bad sign and to singe your finger on mere touch is deadly. Airflow must occur over your hard drives. It doesn't matter if the drive is enterprise rated for "24x7," it will die rather quickly without airflow. Your case design and options for cooling the hard drive bays (unless you relocate them to external 5.25" with a mount option, taking up those slots) is of the utmost concern.

Next is the CPU, which needs to get cool air as well. Intel addressed this in the BTX mainboard design by putting the CPU at the middle-front of the mainboard, and having tower cases blow air from the middle over the CPU first. Ironically, the best cases in the ATX are those that have little more than a side "fan duct" directly over the CPU tower's fan, and then with a 120mm or at least a 92mm exhaust fan out the back. It's effective, doesn't rely on any cooler air from the front of the case (which may have been warmed by hard drives or expansion slots below it in a tower), and just works.

Lastly is the GPU. The GPU is quickly becoming the biggest power gobbler and thermal generator in the system, but the PCIe expansion slot -- by its very nature -- forces GPU packagers and thermal designers to accomodate. As such, it is typically designed to run at much higher temperatures and with less airflow in its far more constrained spaces than your hard drive or CPU. Even just a set of air vents above the expansion area lets more cooler air in. And the hottest GPUs take up two slots, using the headroom to enclose the card and push warm air directly out the back of the second slot.

Without radical enclosures like BTX or the "flipped ATX" design, there's not much we can do about the GPU in a PCIe expansion slot. Although these solutions do lower thermals by as much as 10C. The BTX flips the mainboard orientation so the expansion card's "main IC side" are "facing up" as heat rises. The "Flipped ATX" design also does the same, and can even put the CPU at the lowest part of the case for the ultimate cooling. There are even "Flipped MicroATX" towers.

Furthermore, there is a newfound interest in the "normal standing" desktop or "cube" case, especially for MicroATX. The becoming common 9" x 11" x 14-15" MicroATX "cube" (e.g., Chenming 118 series or the $39 Ultra Microfly) offers some excellent airflow designs. First, they have a 120mm outtake fan just above the CPU area, which is also directly behind the hard drive area (which may have some "side slits" for cooler intake air). Second, they have the power supply right above the card area, and its outtake fan (especially if mounted internally, like some power supplies with large 120mm fans do) can "pull up" warm air directly from the slot area. With two 120mm outtake fans (one in the power supply, another directly above the CPU area), these new MicroATX "cube" designs can effectively cool all components, even though the designs are tight and cramped for their components.

Other variations exist as well.

I have covered both the "Flipped ATX" and "MicroATX cubes" in my earlier blog articles, and the options have increased since them. Enclosures are always a personal taste, but the key is to look for options, and price compare. Even just a cheap ATX enclosure with a 120mm outtake option near the CPU area, the CPU duct/openings and a 80mm intake option around the lower hard drive area is adequate for most users. The Ultra Wizard cases have these options and can be had for free after rebate regularly (provided you add the fans, possibly some felt feet as well).

The lesser known things to know about memory

Memory is probably the most misunderstood aspect of PC assembly. To start, the synchronous timing/signaling used is hardly the performance metric. To continue, virtually all vendors promote violation of JEDEC specifications.
  • JEDEC banking limits for PC (DDR) and PC2 (DDR2)
  • Timing is everything
  • Over-voltage requirements and JEDEC spec violations
First and foremost, know the JEDEC maximum number of DIMMs per channel for PC (DDR) and (DDR2).
DDR   JEDEC Spec  DIMMs    DDR2  JEDEC Spec  DIMMs
---- ---------- ----- ---- ---------- -----
200 PC-1600 3 400 PC2-3200 1
266 PC-2100 2 533 PC2-4200 1
333 PC-2700 2 667 PC2-5300 1
400 PC-3200 1 800 PC2-6400 1
Seeing a repeat theme here? Only 1 DIMM per channel for any DDR2 speed as well as PC-3200 DDR400. On any JEDEC compliant DDR memory controller, using more than 1 PC-3200 (DDR-400) DIMM per channel will result in the signaling slowing down to PC-2700 (DDR-333), which people regularly complain about. These are doubled for "Registered/Buffered" DIMMs, but desktop systems rarely offer them (let alone the newer, "Fully Buffered (FB)" DDR2 DIMMs in newer servers/desktops).

Socket-754 (DDR) is a single channel, direct, 184-pin trace from the CPU, so Socket-754 runs into this limitation directly. Socket-939 (DDR) offers two, 184-pin traces from the CPU, so has limits that are double. Socket-940/AM2 offers two DDR2 channels, so it can only support two (2) DIMMs, period, under JEDEC specification (for stability). Despite marketing, Socket-478 (DDR) runs into single channel issues. LGA-775 is dual channel, so it can only support two (2) DIMMs, period, for JEDEC like Socket-940/AM2.

Timing of the DRAM cells in DRAM ICs on a DRAM DIMM is the biggest performance factor, by far.

The first thing to understand about simple, Dynamic Random Access Memory (DRAM) cells, even those on Synchronous DRAM packages, is that they do not have a clock that runs synchronously with the channel. The MHz signaling does not correspond to the actual cell "latency." That is, a 667MHz (PC2-5300) DDR2 SDRAM module does NOT have 1.5ns (1/666,666,667 of a second) timing. In fact, it doesn't mean it has 3.0ns timing (since two bits are transferred per clock) either.

DRAM cells are typically between 20 to 60ns latency (only 16 to 50MHz -- yes sixteen to fifty). That means it takes several cycles when reading to even get the "first bit of data." Synchronous DRAM attempts to mitigate these great latencies by bursting transfer in pages, typically 4K on x86/x86-64 (what Intel calls IA-32/IA-32e) processors.

Writes are virtually synchronous because they are "send and forget." They will move to the SRAM (static RAM) cache of the processor and/or board, and eventually be sent out to memory. No real latency impact. But reads are devastating if they are not in the SRAM cache already, as the great latency difference between the processor/board and its SRAM and the system DRAM adds great delay. A "cache miss" in modern, superscalar microprocessors and their board interconnects causes an exponential performance hit (of typically over a magnitude).

The SPD chip on each DIMM reports to the channel its various CAS, RAD, RP and RAS timings for the synchronous clock rate. For any DIMM product, you will typically see four (4) timings like 2.5-3-3-6 DDR400/PC-3200 or 5-5-5-15 on DDR2-800/PC2-6400 SDRAM modules (which are pretty good timings). Taking in the synchronous timing, you can figure out the CAS, RAD, RP and RAS of the SDRAM module. If you're intersted more on each definition, see the Wikipedia page (I will not go into them here).

A RAS of 6 on DDR400/PC-3200 means 6 cycles at 200MHz DDR = 6 * 5ns = 30ns (33.3MHz equivalent) maximum delay to access a select row of data in a DRAM IC.

A RAS of 15 on DDR2-800/PC2-6400 means 15 cycles at 400MHz DDR = 15 * 2.5ns = 37.5ns (26.6MHz equivalent) maximum delay to access a select row of data in a DRAM IC.

A RAS of 15 on DDR2-667/PC2-5300 means 15 cycles at 333MHz DDR = 15 * 3ns = 45ns (22.2MHz equivalent) maximum delay to access a select row of data in a DRAM IC.

Yes, latency is not the only factor, as increased, synchronous singling means increased Data Transfer Rate (DTR) -- especially since pages are fetched 4K at a time. But if you're reading more than writing, have a smaller CPU cache (not just L2, but also L1 -- remember, AMD processors have 4x the L1 cache of Intel processors -- long story), etc... latency can have a far greater impact than DTR.

This is why even cheap DDR2 memory can be outperformed by quality DDR, resulting in overall system performance even being 10-20% better. Having directly memory channels on the CPU (e.g., AMD Athlon 64/x2/Opteron), let alone a larger L1 cache (which is completely synchronous with the CPU), can make a huge latency difference as well. This is why AMD did not gain much performance in moving from the DDR Socket-939 platform to the DDR2 Socket-940/AM2 platform.

Over-voltage requirements of DDR2 DIMMs is becoming an increasing issue. It's beyond just many DDR2 DIMMs requiring greater than the JEDEC spec 1.8V -- such as 1.9V, 2.0V, 2.1V or even 2.2V in some cases, to reach the optimal/fastest timing. Many DDR2 DIMMs -- especially higher PC2-6400 (DDR2-800), but even some PC2-5300 (DDR2-667) DIMMs require them to even function at all. So beware when purchasing higher signaling DDR2 DIMMs these days, and verify they will work at the speeds and timings at the stock JEDEC DDR2 1.8V specification. Check the reviews on forums, or even just the negative reviews on reseller sites for feedback with those key discoveries. Over-voltage significantly reduces product life, assuming you even have a mainboard that can offer over-voltage for your DIMMs in the first place.

It's rare, but older DDR DIMMs can require over-voltage as well. Virtually all are stock 2.8V, although newer DIMMs may only require lower voltages of 2.6V (or even 2.1V in a few cases). Putting 2.6V DIMMs in a mainboard that only supports 2.8V is typically not an issue, as over-volting often improves stability (although it can shorten the life of the product), unlike the issues with DDR2 (and its commonly required over-volting). Again, it's far less common with DDR than DDR2, but there are still select products out there.

Storage options, what not to do, and redundancy


The extra frills, and budgetary mistakes


Free/Fake RAID (FRAID) dos, donts and conversions

2007-03-25

My Sys Admin Magazine Articles Now On-line

CMP Media's Sys Admin Magazine, the #1 UNIX and Linux magazine in print circulation, has now made all of my past articles available on-line, and no longer just the ones that were "web features." For easier browsing, I have provided a direct index to any associated figures, listings, sidebars and/or tables for each article.

An index with abstract in reverse chronology ...

2005 September
Dissecting Virtual Tape Libraries (VTLs)
For a company that will never have a data loss event, backup is optional. For companies that will experience a data loss event, recovery is not optional. So, invariably, backup of any and all data that may need to be recovered is not optional. Backup strategies are still widely debated, with many beginning to claim tape is a legacy solution that can and should be completely replaced by disk. Keeping recovery clearly in mind, while disk may solve many backup-time issues, disk still has serious environmental, longevity, and portability issues.

2005 February
Licensing Risks, Not Revolutions: Part II
Part I of this article focused on redefining software licenses beyond the one-dimensional, non-descriptive values of "open" and "closed" (proprietary). It expanded the categorizations, applying the values to axes of "source code" and "standards compliance."

NOTE: These articles were hastily done (per Editor request for copy in these two issues), and my last revision and set of corrections (mainly grammar) were somehow not included by editing.

2005 January
Licensing Risks, Not Revolutions
The primary role of IT is to mitigate risk to corporate investments in information. Simple labels on software like "open" or "proprietary" do little to classify risk to an organization's data and IP. In this article, I'll examine these labels and suggest some new ways to define software types.

Servers are all about I/O. Commodity PC servers have rarely offered high-performance I/O, until now. In this article, I'll dissect PC server design and discuss how to evaluate the performance of AMD Opteron solutions.

2004 April
Dissecting ATA RAID Options
Advanced Technology Attachment (ATA) is a basic, block transfer interface. In recent ATA specifications (e.g., the UltraDMA modes for parallel ATA or newer Serial ATA), this is little more than a copy of a fixed range of data from primary storage (volatile memory) to secondary storage (non-volatile ATA disk), or vice versa, using direct memory access (DMA). There is minimal system processor involvement in the transfer because the ATA controller arbitrates the DMA transfer on the I/O bus (e.g., PCI) and the integrated device electronics (IDE) of the ATA drive to copy data to and from system memory.

2004 February
Securing Intranets With IPcop

An in-depth defense requires both host and network auditing and detection, in addition to basic host and network perimeter security. Because any security can be defeated, it is essential to be alerted when it has been. There is no greater business liability than an intranet that has become unknowingly compromised.

NOTE: The section on the Snort Intrusion Detection System (IDS) was gutted from the published version.

2002 April
CD Backups with Easy File Access
Smith introduces the "back2cd" script, which maximizes CD capacity while preserving CD's inherit random-access advantage.

NOTE: Also works for DVD, DVD-DL, etc... as it produces a standard ISO9660 Yellow Book (data) track image file (commonly known as the .iso extension).

2007-03-19

Boot Linux, NEVER Windows, after a memory upgrade!

Stupid me ... I should know better ...

After a memory upgrade on my HP Pavilion dv9000z notebook (see the following, related "review" below) failed to start Linux correctly, I stupidly assumed it could be -- even though this is virtually impossible -- somehow something to do with the Fedora Core 6 x86-64 (specifically Linux kernel 2.6.20 Fedora build 1.2925.fc6) system was booting, configured, etc... So without thinking, I figured I'd just boot Windows to see if I had the same errors and faults, etc...

Stupid me ... I should know better ...

You see, when Linux systems boot, they boot the disk read-only, meaning they do not write to the disk at all -- not until very late into the boot process. And if not everything looks correctly -- let alone if the init (initialization) process or one of its sub-processors or init scripts don't remount the filesystems as read/write (normal usage) -- the disk will not be corrupted.

All Microsoft OSes (including DOS-based like 95/98/Me and NT-based like 2000/XP/Vista), on the other hand, boot read/write from virtually sector #2 (after the master boot record, MBR, on PCs, although NT 5.1/6.0 aka XP-2003/Vista often does a series of reads/writes to hidden sectors in the MBR after sector #1). So what I got was a nice and toasted C: drive as a result of my memory issues.

Thank you Microsoft! No, I should know better, this is on me. And now you know.

Boot Linux never Windows when you upgrade your memory. Your memory may have compatibility issues resulting in corruptions and you do not want blocks from memory flushed to disk when memory is corrupted due to hardware issues! Linux doesn't mount its filesystems read/write until late in the boot process, assuming no issues (much less it doesn't write to "hidden" areas of the disk label aka partition table -- and even MD/LVM avoid doing such until later in the boot process too), and will "crap out" to a prompt with anything still mounted "read-only" if there are any issues.

For those that don't have Linux installed, you can use a Linux "Live CD" (or DVD) like Knoppix (Debian) or other distro (like the new Fedora Live) CD/DVDs.


Luckily ... I backup my C: drive every 2 weeks or so. ;)

You see, it only wiped out my small, 32GB NTFS C: partition. I learned long, long ago (I've been using NT 3.1 since the original Beta in 1992-1993, and OS/2 with HPFS before, as well as after, that) that keeping the C: drive small has saved my bacon so many times. I put everything non-system/non-boot (MS terminology: system=NTLDR bootstrap including any NTBOOTDD.SYS driver, boot=NTKERNEL.EXE and C:\WINNT directory -- yes, the system=boot and boot=system seems opposite, but that's MS' official NT -- system then boot -- terminology for start-up) on the D: drive -- data, programs, even relocate profiles and Documents to D: via registry changes for my user(s) -- everything except system/boot (and that would include driver/hardware programs installed on C: too, because they could be used at boot). I then I keep a dd image (a raw, sector-by-sector, image) of that C: drive on an external FireWire/USB hard drive (on a Linux Ext3 filesystem), which I re-dd about every 2 weeks so I can recover it if NT (including newer 2000/XP) self-toasts.


Related review of PNY 1GB PC2-5300 (DDR2-667) SO-DIMM (CompUSA SKU 340840) ...
First off, PNY's web site's own memory configurator does NOT yet even
offer ANY HP Pavilion 9000 series choices (only through the 8000 series),
neither Intel Core (dv9000t series) nor AMD Turion (dv9000z series).
But stupid me, I didn't check the web site before buying, and figured
anything that says PC2-5300 (aka DDR2-667) must be at least JEDEC
PC2-5300 compliant.

I could NOT get the SO-DIMMs to boot my HP Pavilion dv9000z without
throwing out errors. At first, I thought it might be a memory controller
issue because each 1GB SO-DIMM had 16 ICs, whereas the original 512MB
SO-DIMMs had only 8 ICs per unit. But then not even a single 1GB SO-DIMM
would work correctly on its own (yes, the AMD Socket-S1 does support
single channel use, including on the newer Mobile Sempron / Turion x2
systems). I reseated the SO-DIMMs a half-dozen times to no avail, even
though putting the originals back in a half-dozen times worked
flawlessly each and every time so it wasn't the SO-DIMM sockets.

There's clearly an issue with the IC sizing, bit-width/IC number or
something else and they are possibly not even fully JEDEC PC2-5300
compliant -- not even for a single SO-DIMM on its own. I thought
everything DDR2-667 these days was JEDEC PC2-5300 and there were no
platform-specific issues unlike DDR2-800 (as JEDEC PC2-6400 is far less
standardized), but I guess not!

Again, since PNY's own memory configurator (as of March 2007) doesn't
even offer any HP Pavilion 9000 series in its list, I'd *AVOID* any
PNY PC2 (DDR2) SO-DIMMs for the time being. I wouldn't even try the
2GB kit with 2 pieces for $10 more (SKU 344987)


And yes, the original memory (2x512MB) is back in this very unit, working flawlessly.

2007-03-14

PC Gaming Dying at 25, Part I: Why?

In this multi-part blog, I will explore the state of PC Gaming after a quarter century, including ...
  • Part I: Why and how is PC gaming dying? (this entry)
  • Looking back at the various genres ... both dead and alive (and sometimes both)
  • The future of smaller commercial and community PC gaming development
Part I: Why and how is PC gaming dying?

The reasons for the slow, but stead death of PC gaming -- at least on a major commercial developer scale -- has everything to do with the set-top console. Specifically, in descending order of relevance from most to least significant ...
  • Hardware entry and development costs v. volume and redesign costs
  • Software entry and development costs v. volume and support costs
  • Consumer preferences of "load and forget" and "reduced limitations"
Hardware entry and development costs v. volume and redesign costs

The first "wake up call" to the changing industry should have been IBM's virtual standstill on the Power4-based PowerPC -- especially the common PowerPC 970 sold under the marketing name G5 by Apple. IBM is the world's largest, independent semiconductor foundry -- i.e., a company that helps layout and actually fabricate large volumes of integrated circuits (IC) including maintaining its own, "core" and "peripheral" set of logic chips -- essentially forced Apple to Intel. Before Apple negotiated its new, preferred status with Intel -- replacing Intel's previous favorite, Dell -- IBM did not deliver on its promises of a low-power PowerPC 970, and failed to ramp up clock and performance on the series.

This had little to do with the actual design, capability and scalability of the PowerPC 970, and everything to do with profit. The PowerPC 970 was a brilliant, powerful, scalable and even very low-power (while still being very fast) PowerPC-compatible, 64-bit Power 4 core. But, as always, economies of scale is everything, especially for a large foundry.

For any given PowerPC 970, IBM did design, layout, tape-out and fabrication of a release of the IC for a 12 to maybe 15 or 18 month (in the best case) run. And during that run, they'd sell maybe 1M+ units to Apple for G5 units (G4 products are Motorola) and, to a far lesser extent, various other IBM microelectronics licensees. In the end, this "hardware entry" was quite steep compared to the "volume pay-off," especially considering the limited lifetime of the design before a revision was required. Hence why IBM eventually reached the real conclusion that a mobile G5 was not really worth their bother.

After all, Motorola had let it's PowerPC (G4 and earlier) line essentially die years earlier, leaving Apple with only IBM. Luckily for Apple, unlike Microsoft, their OS codebase was designed for portability (NT 3.1-4.0 not-withstanding on select Alpha, MIPS and PowerPC, of which the Win32 codebase did not port well because it was mutated by MS-DOS 7+Windows 4 aka Windows 95+, hence dropped as of NT 5.0 aka Windows 2000). It was constantly built and tested and even cross-linked against the most commodity platform on the planet, x86. And to that end, Apple found itself using the ICs from the only one of two companies that ship millions of consumer-grade, desktop/server ICs for general usage with a limited 12-18 month revision cycle, Intel.

So what does this all have to do with PC gaming?

IBM is a foundry. Power, not PowerPC, is their "core" offering (and they have even sold exclusive licenses of PowerPC lines to select companies, like the 400 to AMCC). They have countless design capabilities, plenty of "peripheral" support logic options and many other, unique offerings in their combination of "building block" design and fabrication volume. As such, short of select consortiums (like ARM or MIPS), IBM has a huge market swing when it comes to embedded mindshare. So, why does that matter again?

Imagine you are a design and fabrication firm. Would you like to design (and redesign) a product every 12-18 months, and only see 100,000+ units? Or would you like to design a product once for 6 years, with one die-shrink/ cost-reduction redesign mid-lifespan (i.e., 3 years), with 10+ million units shipped over that lifespan. One design, one customer, 100x the volume, over 6 years. Simple microeconomics. And with a solid design in the Power architecture, with a sprawling set of options, it was hard to look beyond IBM, or at least Power itself.

Which is exactly what Microsoft, Nintendo and Sony did. They all use Power cores, Nintendo and Sony with specialized peripherals, and Microsoft with a more mainstream, tri-core design. IBM wasn't always involved with every process in each design, but they are reaping some major royalties in all of their designs -- direct and indirect.

This resulting economies-of-scale "cost swing" is two-fold.

Not only does IBM reach profitability, let alone avoid reoccurring design costs, very early due to volume, but the cost of the IC units themselves go down. They go way in fact, at 100x the volume, meaning the consumer can buy more power for less money -- especially at the console's initial release, when the "equivalent PC power" is far more costly. Most consumers are far more willing to pay $300-500 for something that is "current" for 3-6 years (even if that is only from a marketing aspect), than a PC that is only 1.5-3 years.

The same argument also holds for the Graphics Processor Unit (GPU) and, thus, companies like ATI (now AMD) and nVidia, as much as it does for the CPU with IBM and other, select licensees of Power involved with Microsoft, Nintendo or Sony. Instead of having to rev new PCI-Express (PCIe, or even the lesser known/implemented HyperTransport eXtension, HTX, for AMD) GPUs every 12-18 months for performance, a product can be rev'd for 3-6 years, with only a die shrink and reduced cost mid-life.

In fact, one could argue that this is merely the commodization of the console 15 some years after the same happened for the PC OEM at the start of the '90s. And it's just beginning.

Software entry and development costs v. volume and support costs

But hardware alone isn't the major cost to the gaming software itself. Software development houses have their own realities and resulting issues. But their metrics are little removed from hardware -- entry and development costs v. volume are the staples, with support costs replacing redesign costs.

To start, let's remove the similarities. Most game development is done the same, regardless of the target platform -- PC, console, etc... You have PCs, some mid-range, many high-end, possibly clustered, and with 3D absolute, definitely either rendered or graphics workstation-class for object modeling. Another aspect that is common to any target is post-release support -- i.e., patching -- especially with consoles now being Internet connected. In fact, now that consoles are Internet connected, games don't have to be completely bug-free before release -- something game development houses won't admit used to differ. With the complexity of console gaming (see the following section) rivaling PC games -- and often the same game ships on both, at least when the game/series began on the PC -- development houses can assume users will now download updates on their console as standard procedure, much like they do for PCs.

Now that's where the direct similarities end. More indirect, there are a few, higher initial costs for the console development kits, although depending on the game, it's not uncommon to have matching (or even higher) kit costs for PC targets, although there is clearly an equilibrium forming there too as most PC gaming engines are now available on the console out of sheer market desire. So even the "development kit/licensing" costs are indirectly relative these days. Furthermore, unlike hardware, there is also an indirect similarity in lifetime, as games are only good for 18-36 months on the console before they are "bargain bin" much like the more popular PC games as well. So no major difference there.

Which brings us back to the two major differentiators, volume and support costs.

A console title often sells in the hundreds of thousands, even for a lesser title, and millions are not uncommon for the top few dozen games on a console, even still holding some value some 2-3 years after release. PC games, on the other hand, are awarded "Game of the Year" titles when they cross hundreds of thousands of sales, and only where games are clearly not favored on consoles (which is becoming less and less of a difference these days), millions is very rare. In fact, a good testament to a best selling PC game that people assume only could be on a PC is when a console port is still made later, and still somehow sells well enough to make the port profitable (although not always).

But even volume alone is not the major issue for game development houses. Volume combined with support costs are.

Support costs are measured beyond just direct user support, which is virtually non-existent today. Commercial software support is abysmal even in the commercial/business application world where you sell at 10x the price and fee-per-incident or support contracts are considered "added cost." So when it comes to games, virtually no software development houses offer much other than forums or other places where users can help themselves, along with the occasional support moderator. No, the support costs are at every aspect of the game's history -- from development and release target to supporting the various iterations and variations of hardware in the PC platform. Add one part predicting the future with one part combinational probability and you've got a really, really unsolvable problem.

It starts in development by trying to predict the PC that will exist to run a title some 1-2 years before release. While some would say that's easier to do for a PC than a console, since a console is fixed and a PC can be upgraded, it's still the key issue for those "top 10 titles" that will sell hundreds of thousands or even break a million copies. Those titles will target the PC consumers will have at the mid-to-higher end (those willing to pay for the hardware and your software) and they will expect it to be at least usable for that generation at actual release, in addition to "look even better" when they upgrade in 18+ months. In reality, this double-set of requirement is much, much more difficult than targeting a "known quantity" of a fixed console platform, even when the platform is just specs some 18 months before it's actually available. It's still much easier from the standpoint of a title's development as it exits initial design and enters actual coding.

The support issue is then compounded by the myriad of PC configurations -- beyond just the CPU and GPU, but the countless factors, as well as the "general use" of the PC that tends to bring in spyware and various other, unexplained and unquantified factors. Yes, consoles and other embedded systems are not immune to such issues either, especially now that they are Internet-connected. But it's much, much easier to bring a console -- or any other embedded system for that matter -- back to a "known state," especially when the hardware is the exact same across millions of units. In contrast, a PC may be unique down to thousands or even hundreds of units in just end hardware configuration -- especially when it comes to targeting those "higher-end" consumers.

No, the major game development houses don't want to deal with a PC if they don't have to. Sure, there are many, many "name brand" PC gaming genres and series that are still selling, and will still sell, on the PC in the future. But these are clearly becoming the mainstay of only those established, well-known and return customer PC gaming genres and their select series (or even those newer titles that clearly rip off ideas from those series). More and more newer gamers are console gamers -- 90% according to many studies, as well as the fact that consoles are now outstriping PC sales in general (and not merely just PC gaming rigs) in the past year.

The reason? The "support" issue is more than just a developer problem ...

Consumer preferences of "load and forget" and "reduced limitations"

The greatest, most well-known -- to the general consumer -- system that runs the allegedly "hard-to-use OS" known as Linux is the common Digital Video Recorder (DVR), even more commonly referred by its most popular brand name, "TiVo." Virtually all major DVR solutions today run Linux, as do countless other embedded devices. Despite other attempts, Microsoft's Media Edition Windows OSes have utterly failed to "catch on" other than being "just another edition" of their general purpose OS (as well as direct, Linux-based open source toolkits other than for fringe, technical users like myself), because 90%+ of consumers just don't want to deal with a PC.

One basic hardware configuration. One "known quantity" software installation. No need to have a technically proficient kid, neighbor's kid or at least yourself to deal with it when it becomes unusable. Consoles, like set-top boxes, are what 90%+ of consumers want. "Load and forget."

It used to be that consumers had to accept select limitations with consoles over PCs. No 3D, typical side-scroller or other, simple gaming titles, countless other limitations including -- the biggy -- lack of resolution. Ugly, 640x480 interlaced -- one 640x240 frame one 1/60th of a second, another 640x240 frame another 1/60th of a second -- equivalent to a sight-disrupting 30 frames per second (fps). Even if the television was capable of providing 480p (progressive, 60fps) for a 480i input signal -- i.e., the game is still rendered interlaced, but the TV display avoids the eye-strain -- it's still not the same.

But that's no longer the case anymore.

I've already talked about the fact that sheer volume is now giving consoles the ultimate "price-for-performance" advantage at least at console release, if not even a year later, versus PCs. A consumer can also get a powerful gaming console for half (or even one-third) the price of a PC at release, especially since the vendor is selling below cost and will recoup later as hardware costs come down (or just via software, if they never do on hardware).

But the other flip of the coin, now in 2006+, is the fact that 720p -- that's 1280 x 720 resolution of progressive scan or (close to) 60fps -- is affordable. As such, most Americans, Europeans, Japanese, Chinese, etc... consumers who game, would buy a higher-end PC for gaming, etc... are putting in HDTV, commonly 720p or close, and have the capability for high-definition now. And with even yesterday's (i.e., 2005) ATI R400 and nVidia NV40 series GPUs, standard full screen anti-aliasing (FSAA) and other effects (such as shading) really makes higher resolutions less of a desire. That's not to say you won't have a significant number of gamers jump to 1080p (1920 x 1080 progressive), or even just 1080i (1920x1080 interlaced, 30fps), but 90%+ of consumers will see little difference, or at least not see the difference to pay for it for now.

720p gaming is what people can get for under-$500 in a console, and another sub-$500 in a 27", sometimes even 32" after rebate, LCD TV that also seconds as their main viewing device. In fact, with so many gamers either being singles, couples without children or children with their own "second TV" from the parents (not that I believe it is so wise), this sub-$1,000 configuration is quickly becoming "standard issue" more than PCs in homes. Indeed, one could argue that the console's quick volume increase in 2006+ has much to do with the fact that HDTV has become affordable, all while the PC need be little more than $400 for e-mail and web surfing, and no longer gaming.

Let alone do the games need to be taking up disk space and possibly "mucking up" the PC -- daddy's (or mommy's) PC for that matter, especially when you want to keep your teenagers off of it. Heck, even I -- an absolute Linux user for the past decade -- don't want to reboot into Windows just to game. When I can't find a title for Linux (which is actually the best platform for gaming, especially when you use the PC for work -- e.g., you don't have to be "root" to run a game, among other security advantages), I'd rather just have a console -- regardless of the underlying OS (even Windows-based) -- if it keeps me from having to deal with Windows on any PC because of the support issues (and not out of any dislike or political-based distain of anything Microsoft).

Enough pro-console? What do I think PC games offer? Does it matter?

After reading all this, you might think I'm pro-console, you just have a hard time accepting much of what I say is reality or you really differ with me because of some points you think I didn't cover. Trust me, I don't like it any more than you do, and I do think there are a few points left in the PC game industry that will keep it running (as I'll cover in the last part of this series). But that doesn't escape the reality that ...

Most major, commercial game companies don't care about us anymore, becauase we're not volume.

Of course, there will always be select houses with select games that cater to us. The big one is any game that requires the precision of a mouse. As much as Halo might have made first person shooters "playable" on a gamepad, it still sucks in my book (yes, I've played and finished it too) compared to the first person shooters we've all been accustomed to playing with a mouse on the PC. And that would include games you could enter and leave vehicles and what-not, PC games that pre-date Halo on the Xbox. I would argue it's popularity is really the result of people who never experienced PC gaming, and grew up with consoles.

With that said, Microsoft basically guaranteed that PC gaming would continue with select, major development houses for select, major product titles/series/genres for at least a short while longer -- purposely perhaps? (given PC gaming is a staple of the Windows OS versus other options, and not because of capability differences, but existing consumer volume) -- since their #1 mistake with the Xbox 360 (other than not offering digital video output options, only digital audio), was not offering a mouse input option. The only 3rd party options available poorly implement emulation for mouse input. And if you think that's bad, it gets worse for us old PC gamers ... consoles are purposely crippling games even on the PC!

One of my favorite games available for both the Xbox and PC (and I own both) was Elder Scrolls III: Morrowind. In Morrowind -- first/third-person Role Playing Game (RPG) -- where you strike someone with an arrow, a sword slash, etc... made a difference in how they were wounded, possibly fatally (especially when instantly). Showing the clear bias towards console players, and removing the expectations of the typical PC player, this was regressed out of Elder Scrolls IV: Oblivion, crippling "any hit makes the same difference." This basically reversed the entire evolution of first person shooters in the PC world -- localizing damage -- and it's not the sole example. But as I said before ...

Most major, commercial game companies don't care about us anymore, becauase we're not volume.

Just look at the popularity of Halo. 90%+ of console gamers -- which means 90%+ of gaming consumers -- don't care because they don't know. They aren't familiar with the Quake and latter engines that actually localized damaged and all those other small, subtle, but "world of difference" changes in almost 15 years of evolution in of the first-person shooter. That lesson is utterly lost, as 90%+ of console gamers -- now the majority far over PC gamers -- don't care, so, again ...

The major, commercial game houses don't care about PC gamers.


This is the reality people. I don't know if Microsoft's leaving the mouse out of their console was intention or not (because of their ties to the PC gaming platform with Windows). But since 90%+ of consumers don't care, and are willing to "play like Halo" essentially "forced them" to -- all without their knowing, and largely because of it -- it really didn't matter if they did. And, sadly enough, I have a Xbox 360 and even I have to agree that I don't miss the mouse for what I do play on the console (which is not first person shooters, of course, that's why I still have a PC).

What about Flight Sims and Space Sims and other things? Oh, perfect examples of why PC gaming is very much dying at age 25, as I will explore in the coming parts. Perfect examples they are, even though some of you reading this are just like me -- you "grew up" on the PC with flight sims and space sims and you loved games like the Jane's series to Wing Commander and latter incarnations like LO-MAC or iL-2 or Freespace and the X series. -- we're old dogs, and our genres' followings are not in those 90%. I.e. ... (my now broken record) ...

The major, commercial game houses don't care about PC gamers, especially our clearly PC-only genres with their "advantages."

Remember this reality as I revisit the history of PC gaming and its various genres in the blog posts to be continued here shortly ... ;-)

Additional thanx to Brian Ashe for feedback.

2007-03-11

Who writes Linux? Been obvious to me all along ...

Who writes Linux?

That's a broad question that is extremely difficult to quantify or even qualify. But a Linux Weekly News article by corbet tries to quantify the contributions -- from many different metrics and angles -- who works actively on the kernel.

Ironically enough, even though IBM maintains almost entire platform ports and architectures on Linux, along with Freescale, Intel and select others -- and that accounts for anywhere from 2-10% in various metrics -- it's not the #1 maintainer in any major metric. It's easy to get a lot of lines of code, patchsets, etc... when you maintain the hardware for your platform. Even HP gets far less credit for this, or it comes bundled with Intel contributions.

In fact, the overwhelming contributor in virtually all metrics -- excluding unknown (which is significant in several metrics) -- is the same company I've been trying to educate people about for years. They have no other interest but Linux and Linux alone. They have no other products. They have no proprietary solutions at all. They sell a 100% GPL-centric product line, giving away everything to all its competitors, hording not product line (let alone its code) from any other competitor (let alone the community).

I have said this over the last 10 years, and a majority of Linux advocates (especially newer) have just looked at me like I'm dumb -- along with the other, brave few that joined me -- because such an overwhelming majority couldn't be wrong in their eyes. The self-fulling prophecy. Sigh, even now people are saying this is a flawed and inaccurate set of metrics because of the results.

The only bias in this article is that it is biased towards vendors who produce their own hardware, and need to donate that knowledge to the kernel in the form of code so it is well supported. Take the 390 and other ports out and you'd see IBM's share drop tremendously. Same deal why you see vendors like Freescale and Intel, although one could argue in these latter two cases that most of their contributions are affect products others than their own.

The list is dominated by the various, anonymous contributors, select organizations and institutions and the -- at the top -- the one, sole commercial entity that does not sell one single, proprietary product. Makes you think about who is really and truly about 100% Linux, and those entities who have other interests than just Linux. That's not to demonize these other companies at all, and I'm happy to see their involvement. In fact, it's a testament to their wish to involve the community with the support of their products, because they realize the value of the community.

But it is something to keep in mind the next time you want to rant about Red Hat. Especially since Red Hat doesn't have a piece of hardware they only sell or design, or a select subsystem or component that they alone are the only one that uses (even if some would argue otherwise).

People have complained for years that Red Hat's "influence" has changed the path of Linux -- from the readoption of GLibC 2 in Red Hat Linux 5 to the push to GCC 3 in Red Hat Linux 7 / Enterprise Linux 2 to the implementation of Native POSIX Threaded Libraries (NPTL) in Red Hat Linux 9 / Enterprise Linux 3 to SELinux Mandatory Access Controls / Role Based Access Controls (MAC/RBAC) in Fedora Core 3 / Enterprise Linux 4. Now you know how Red Hat is able to "influence" Linux, especially the kernel.

Just like the sign at the animal shelter that reads ...
"If you don't like what you see here ... Volunteer!"

True community involvement, by individual choice, in a free-market, capitalist society -- the ideal form of "public good," instead of forcing everyone which quick becomes differing views of "what is right." But what do I know? I mean, the majority of Linux advocates know far more than me, they've been telling me, among other Linux developers and consultants, that we're just wrong. After all, Red Hat is the Microsoft of Linux -- although now more and more think Novell is just as evil too, possibly more so.

Even I, a man who has built 90% of his career over the last 10 years on Linux solutions with Red Hat, don't have anything negative to say about Ubuntu, KDE, etc... (not at all), unlike the seemingly majority of "advocates" who blast each other back and forth with commercial-like marketing sewage. In fact, leadership in the Linux community (note, I didn't say "leader" or even "leaders") isn't about saying anything negative about your competitors, or even preventing them from integrating with your solutions. That's something we -- the Linux community as a whole (putting many advocates aside) -- left the commercial software world for, so why would we drag it here?

I thank God everyday that Linux is a meritocracy and not a Democracy. Funny, the Fedora Project tends to have the same organization too, and not everyone gets a vote. And they get things done, not always as everyone wants them, but in the way that moves Linux forward. Because while you can't make everyone happy, many of us who have been implementing Linux as an enterprise solution (not just for web services) for the past decade are getting what our clients say they need.

If you disagree, remember, Linux can use volunteers just as much as the animal shelter. That's how things get done, and how you get your "influence."

Related Blog Entry:
The Five Types of 'Linux' Corporations

2007-03-07

"Common Sense" Disaster Recovery Fundamentals

Most people in IT get away with not having a disaster recovery plan every year. Why? Because they don't have a disaster. For those that do, they have ensured over 2/3rds of their employers do not recover. Don't be a statistic -- or rather -- don't let your employer be a statistic.

There are basic fundamentals to disaster recovery. They are:
1. The 3 basic levels of data redundancy
2. The time crunch, especially recovery time
3. Test restores are an ongoing duty

I've heard arguments of budget, simplicity and countless other reasons why various IT support departments fail to have a disaster recovery. I've also seen many, many false disaster recovery setups. They all make my shake my head. Especially when disaster recovery can be very inexpensive (and in free) in most cases.

1. The 3 basic levels of data redundancy

There are 3 basic levels of data redundancy:
A. On-line: immediate recovery of data on the system itself
B. Near-line: near-immediate recovery from another system on the network
C. Off-line: disaster recovery when you don't have the system, let alone the network

On-line includes more than redundant power and disk, which merely saves you time (and money, which is a good reason to pay an extra $100 for a 2nd disk in even desktop systems these days) and avoids downtime. Today servers should use filesystem snapshots so accidentally deleted files can be recovered by users without bothering system administrators. It's worth the time and effort to implement filesystem snapshots on servers, or even the free VMWare Server product to run servers as guest OSes and do snapshots of the entire system (which is great for backups too). There are countless options here to help you, and you don't need to buy a NetApp hardware solution to get them.

Near-line means disk on the network, recoverable within a short period of time, let alone provides you a way to backup quickly in the first place. This could be as simple as a server that receives rsync updates from systems every night (ideal this is also your off-line backup server, as it can put data to tape or ruggedized 24x7, and not merely just during the backup window), rotating backup sets or maybe using filesystem snapshots to do the same. Near-line systems can be as simple as an extra $500 PC on the network with two very large disks in a RAID-1 configuration. Again, there are countless options here to help you, and you don't have to go out and buy a full-up Virtual Tape Library (VTL) product (of which my article, in the previous link, dissects and isn't really much different than what you can do on your own) to get them.

Off-line can mean a combination of things, but it usually means tape or ruggedized disk, and it means only periodically. When you have a near-line system, you only need to off-line every week or two for off-site disaster recovery -- that's it! As such, off-line does not mean a 3.5" disk in a hot-swap bay or an USB or IEEE1394 FireWire connection. 3.5" disk is not meant to be thrown around. Consider at least a 2.5" disk -- which takes 20-50x the shock -- and is as low as $0.75/GB today (160GB for $120), which is being even being used more and more in an "enclosed cartridge" by various vendors. There are even hot-swap SATA bays now that fit four (4) 2.5" drives in a 5.25" bay. Of course, if you're putting TBs of data off-line, just invest $500 into an LTO-1 (or later) tape drive which will quickly pay for itself, as well as has an extensive upgrade path through multi-TB cartridges. In either case, these "off-line" options always go in your "near-line" server, not your main servers up 24x7, so if they cause an issue (e.g., when removing 2.5" disks), no one is down!

If you're not seeing a repeat theme here -- near-line disk and off-line ruggedized disk or tape complement each other, but you need both near-line and off-line. You want to keep near-line disk so you can easily backup/restore your actual systems within the backup window and off-line media so you can restore after a disaster. People dismiss tape because they incorrectly implement tape -- you should never backup directly to tape from end-systems, but use a near-line system to directly backup near-line to disk during your backup window, and then off-line to tape from that near-line system at your leisure. That solves the alleged "tape problem," which isn't a "tape problem" but people not removing the disadvantages of tape (linear access) leaving only the advantages (cartridge life, portability, etc...) over 3.5" disk.

If you can't afford tape or don't have enough data to justify it's cost then get some 2.5" drives, which are designed for the torture of being moved around in laptops. Don't look to external 3.5" drives, especially not via USB or FireWire, as they are often not designed for hot-plugging with servers (despite marketing to the contrary), and can bring down your servers (especially overnight when there is a "bus disconnect"). Use 3.5" drives as they were designed, as fixed disks, in a near-line server as above. This is much better of a solution -- often costing less than $500 with a PC and two (2) redundant disks -- than a few external drives that aren't redundant and not as fast over the external buses either.

As countless people have thanked me (among others who put forth the same), recognizing near-line is the answer for nearly all restores -- because companies typically don't have a disaster event -- is why you don't need (let alone want) to use removable disk for off-line. You only need off-line for that disaster chance, so you only need to off-line every 2 or so weeks. Use commodity 3.5" disk in its natural, fixed configuration on the network as your staple, near-line solution, and then complement that with infrequent off-line to either tape or ruggedized disk from that near-line server which can go up and down, unlike your main servers (which you don't want to be hot plugging things on).

2. The time crunch, especially recovery time

Time is money. The time crunch after any disaster -- whether it's a single, server melt-down or the destruction of a full office -- is money down the drain tied linearly to the duration of the downtime. Consider the following concepts:
A. Consistency/Reproducibility
B. Boot to Restore
C. Network Reconfiguration

Consistency and reproducibility are the staples of configuration management in any environment. "One-off" systems can be a real PITA for configuration management, especially in enterprises of multiple servers. When you go to install any capability, try to do as many systems at the same time in the same configuration -- and consider buying 1 extra system for immediate replacement or spare parts -- especially if the systems will in use for more than 2 years as older parts will be no longer available. And at least make it so you can install the next system the exact same way as you did earlier -- be it cloning (often required for Windows), or formal/proper package management (easily to do with modern Linux distros), etc... And another option today, which is free, is to always install your servers as guest OSes in VMWare -- which can be moved to any system running VMWare Player/Server -- ala "reinstall" in 15 minutes.

Put these images on bootable DVDs and make several, extra copies to go with both your near-line systems and off-line media. That way you're ready to bring any type of system back-to-life when you need it. Again, the more you standardize your installs, the easier it is to consistently reproduce them.

Boot to restore time is yet another reality. The longer you can't bring a server back to life, the more users are down and it costs your company. Your bootable DVDs with your system images should have a way to reinstall those images. It's worth paying for 1 license of a professional Windows recovery system for each Windows server. And with Linux, you should know how to recover any Linux system from a "rescue" or other CD, and merging that boot with the images is the best thing you can do. And, again, if you really don't want to deal with much, just having the images as VMWare guest makes it cake, as you completely separate the hardware aspects from the OS run-time, and any VMWare Player or Server will get you back and running.

In all cases, again, your system images are useless or a major PITA if you don't have a way to boot them and get them back on the system -- especially when it might not be the original hardware.

Network reconfiguration is the last aspect people overlook. I don't know how many times I've seen a router or switch fail, and no one backed up the firmware. To make things worse, you need to have not mere network topology documentation, but an extensive document listing all the services required on your network -- especially when a major disaster takes down your office. Your desktops and servers could be ready, but select network devices or networking services could be the massive "reverse engineering effort" that keeps them from running.

Especially since sysadmins have a nasty habit of looking at networking separate, possibly even outsourcing support. Make sure you get that vital, reconfiguration information from your ISP, network support, etc..., and put copies on your restore DVDs.

3. Test restores are an ongoing duty

Lastly, and definitely not least, test restores are not optional. Specifically:
A. On-line: Check options daily
B. Near-line: Daily logs, check restore before you off-line
C. Off-line: As you off-line, as well as when you rotate, from near-line

On-line restores, such as filesystem snapshots, will pretty much be checked by your users. That's the power of filesystem snapshots and letting your users help themselves, your users will let you know if they don't work. And if you are using VMWare or other virtualization snapshots, every now and then test your VMWare snapshot from the previous day on a spare system (if you have one -- you should if you have at least 3 servers), separate from the network.

Near-line restores will also be fairly natural. Check your logs daily to see what nightly pulls from systems didn't complete. Otherwise you really only need to check one other time, when you off-line. Because if and when you are going to put to off-line tape or ruggedized disk, you might want to check that the backup was complete before you do. This is also where you want to make use of that spare server, or at least some spare PC.

Off-line is really the only time anything is a PITA. You at least want to use "write after verify" to your tape or ruggized disk. People who incorrectly backup directly to tape, instead of using a near-line disk solution, often turn this off because it won't finish in their backup window. Just more reason to get a near-line solution, because you can off-line data for off-site storage at any time, from any recent backup that's on your near-line server! That's why near-line is your staple backup/restore solution, especially when complementing off-line!

Furthermore, you need to periodically check your previous off-line backups.
I normally recommend 2 off-line check a month, or at least 1 per month, and it's normally a real pain. People loathe this, but that's where near-line comes to the rescue yet again. Instead of restoring to the original server, which most people do when they don't have separate near-line -- only directly to off-line (or, worse yet, a removable/portable disk right on the server), you can directly compare the off-line media to the near-line server's storage. You can even restore to the near-line server, not bothering your main servers. And you could even store file lists and checksums of your off-line backups on your near-line server, which gives you a quick "sanity check" if a prior off-line has gone bad or not.

Conclusions

This article is about "common sense." When people start talking about proper backup, they think big bucks or big efforts. No, that's so far from the truth it's not even funny. The "common sense" is all in the restoration. And it's often very cheap to do (even for small businesses) and saves sysadmins countless hours compared to the common (and poor) procedures in place.

For the most part this is realizing a near-line backup server is the key. It gives you the fastest 3.5" disk performance available, in their natural, fixed, "on" (not stored) environment, right on your network, ready to use. You don't need every backup to be removed from your office, especially since that's only an issue if and when you have a disaster.

From there you only need to do an off-line backup ever 2 weeks or so, and most companies only retain data for longer-term store on a monthly basis. That's where the near-line backup server is ideal again -- because it has backups of all your systems you can then put to off-line media (tape or ruggedized disk) at your leisure, and not under the constraints of the backup window. It also gives you a system to test restores from older, off-line backup media too, without bothering the servers.