Brian's Brain

EDN Senior Technical Editor Brian Dipert exposes, analyzes and
opines on diverse topics in technology.

Profile

About Me

RSS Feed

Add this blog to your RSS newsreader!

By Category

Consumer Electronics Design Articles

Blog

Brian's Brain

« Penny Arcade Takes On Apple TV... | Main | Apricorn's DriveWire: A Hard D... »

Friday, January 25, 2008

System (Un)Reliability: The Microsoft Xbox 360 Case Study

In yesterday morning's weekly EDN editorial teleconference, we were discussing (among other things) Mike Santarini's upcoming cover story on IC reliability in the March 6 issue. The topic of the Xbox 360 and its various reliability woes came up, in part (I suspect) because Mike's a contented owner of this particular console as of Christmas and may be wondering if (and if so, when) he's going to need to tap into the system's recently extended three-year warranty.

Coincidentally, several particularly interesting Xbox 360 reliability-related bits recently found their way into my RSS reader, which I'll pass along for your Friday-and-weekend inspection.

The Seattle Post-Intelligencer recently snagged an interview with an anonymous Microsoft 'insider' (more commentary from DailyTech), who passed along some detailed (albeit, of course, officially unsubstantiated) information on the console's RROD (red ring of death) debacles. As I've also suggested several times in the past, the insider reveals that Microsoft made a conscious decision to rush the Xbox 360 to market, which resulted in an all-important one-year-plus availability lead in this particular round of the console wars. However, by (for example) relying on 90 nm-fabricated ICs versus waiting for (potentially) cooler-running 65 nm chips, Microsoft is now dealing with customers' system failures. I should point out that none of my Xbox 360s has (yet) had problems, even though I acquired one of my systems shortly after the November 2005 Zero Hour launch event. Granted, I'm not much of a gamer, but I've played plenty of CPU-intensive HD DVD titles on the system...
...however, the CPU isn't the system's primary Achilles' Heel, as my system teardown from last year pointed out, and as the always-excellent Andrew 'bunnie' Huang also suggests in his recently-published blog entry. The 'weakest link' is the GPU, specifically the solder joints that connect it to the PCB and the progressive degradation of those joints as high internal temperatures cause the PCB to flex. The culprit here is coefficient of thermal expansion mismatch between the IC package and PCB, a concept that I showcased in the last introduction I managed prior to my departure from Intel (the late-1996 µBGA flash memory package). The photos accompanying Huang's writeup graphically communicate the result of this mismatch, along with showing evidence of trapped-gas 'voiding' that occurred during initial soldering as part of the system manufacturing flow. In Huang's particular case the red ink-based 'dye and pry' analysis didn't reveal any flat-out failed solder joints, but the degraded links he discovered certainly could combine with other system compromises to increase the likelihood of breakdown...a statistical probability that of course increases over usage time.
In my particular case, I'm not sure of what percentage of the overall HD DVD decoding-and-rendering flow runs in software on the CPU, versus being hardware-accelerated on the GPU, so it's not clear how much my systems' GPU solder joints are being stressed. And pragmatically, anecdotal evidence suggests to me that only a small percentage of the Xbox 360 system failures are 'infant mortality' (a term that causes me to cringe every time I type it) in nature. By the time most consumers' consoles exhibit a RROD, the owners have already amassed a library of game titles, effectively 'hooking' them. Shipping in a system for replacement with someone else's refurb is a frustrating hassle, granted, and the reputation damage may hurt Microsoft in the next round of the console wars, but for now Microsoft can rest assured that the lucrative revenue flow from its installed console customer base will in most cases continue unabated.

How's your Xbox 360 working?

Reader Comments

(Add your comments)

at 1/25/2008 12:34:13 PM, Steve M said:
This isn't a case study, its just a bunch of words that circle around like water going down the drain. Was there a point here? I'm not going to reread it in case it was a waste of time.

at 1/25/2008 12:48:43 PM, LRW said:
You stated “The 'weakest link' is the GPU, specifically the solder joints that connect it to the PCB and the progressive degradation of those joints as high internal temperatures cause the PCB to flex.” The Mac G4s, w/dual processors mounted on daughter boards, suffered from a similar fate, requiring utility software to power-down the 2nd processor whenever possible. Is it possible that XBOX’s GPU/BGA/PCB joint failure, comes from excessive temp excursion(s) caused (in part) from excessive current draw from the GPU's output pins ? As in connecting to a monitor which draws current in excess of the design ?

at 1/25/2008 1:09:56 PM, Gwansho said:
Owner of the first X-Box 360 sold at my Gamestop. It went RROD just a month and a half ago. THe problem is getting MS to fix it. After hours on the phone, they promise to send a box to ship it in (in " a week or so"). Then 3 weeks later, no box, and no history of ever talking to them. Started over, now waiting almost 2 more weeks, still no box.

at 1/25/2008 1:33:09 PM, Brian Dipert said:
Dear LRW, yes I posted a clip of the G4 'repair' back in January: www.edn.com/blog/400000040/post/520006252.html. Enjoy!

at 1/25/2008 1:34:32 PM, Harvey Miller said:
You should have mentioned that the "questionable" solder joints use lead-free alloys, a non-eutectic mix of tin (95%), silver, and copper. Unlike 63-37 solder, the new solder does not wet, is brittle, subject to many failure mechanisms like tin whiskers and Kirkendall effect, leading to voids and open circuits. There is an alternative lead-free assembly approach that eliminates solder. It's called Occam--ref <www.verdantelectronics.com>. harvey@fabfileonline.com>

at 1/25/2008 1:40:36 PM, Tom G said:
The issue of "weakest link" can be solved by using some sort of an Epoxy under fill material, which might also help with the CTE mismatch issue. Are they using any under fill material?

at 1/25/2008 2:47:13 PM, Rocketman said:
I think the YouTube frying an egg in 3 sec on the GPU heatsink say it all.

at 1/25/2008 2:48:49 PM, lashtx said:
I had no issue with Microsoft. Very professional on the phone, tried all the steps they recommended and then had a box within 3 days. I had my XBOX 360 back with in 1 Week of sending. Works perfect.

at 1/25/2008 2:50:19 PM, PS3_is_Better said:
What about the connectors? I tried to take my son''s 2nd xbox back to GameStop after it died after 3 weeks and they wouldn''d take it back and gave me the "We don''t want to be responsible for the products we sell after you had them for more than a week..." line. They said "Try Microsoft and they will ship out a box to mail it in..." By the way, my son''s first xbox had intermittent connector problems. I eventually got it to work, but the next day it wouldn''t start up again. Played with the connectors some more and got it work and them it failed again. So I took it back to Gamestop and they gave me my 2ed xbox. (which was an elite and cost me more money.) I can''t help but wonder if they are just curning the same old intermittent boxes... Makes me such a happy customer! After about a week, my other, and oldest, son played with the connectors again and got it working (for about a week so far). My oldest had asked for and got a PS3 for Christmas and we''ve had no problems with it).

at 1/25/2008 2:54:53 PM, Davide Cogliati said:
My opinion is that the problems of early failures (also some months of work are of the family) can be caused from production accidental problems, especially soldering process with lead free pastes. I want to think that the engineers who looked after the acelerated life tests on this product in order to find weakness in the product have done a good job. If this is not the case, well...I am suprised how some products (weak from the design ) are sold on the maketplace.

at 1/25/2008 3:12:17 PM, Ben Leaf said:
My 360 broke after 1 year and 1 month of ownership - and yes, I had amassed about 10 games or so in that time. It took me about 3 weeks to get a refurbed system back from their repair facility in McAllen, Texas. Overall I was pleased with the customer service - although at one point I called their help line to check the status of the repair and they could find no information on me - no customer history, no repair order, nothing - which was a little scary. Something tells me that the actual failure percentage of all 360's released in the first year after lauch is more likely around 60%... Microsoft knowingly gave consumers a faulty product...and I can't say much for Windows Vista either...

at 1/25/2008 4:16:13 PM, nogameboy here said:
I would love to see the stats on how many first gen xbox 360''s failed. By first gen I mean the ones before microsoft started adding there fixes. I have been a PC gamer traditionally, but do to the lack of games released on the PC, I have turned to consoled systems. I fell in love with the orignal Xbox. What a great sytem! So of course I jumped when Microsoft made a new system. Every person I know including myself, who bought first gen xbox 360 has had it die on them. One friend has had 5, I kid you not 5 xboxs fail on him. Of course this was back when they were recycling broken xboxs though the system. I really have been surprised at both Microsoft and Sonys reponse to the market

at 1/26/2008 8:37:38 AM, smoothn00dle said:
M$'s strategy is like drinking poison to cure thirst. They push their install base to 17min but PS3 is going to catch up and Wii already surplus it. Many Many 360 owner are changing to PS3 because of price and reliable. 360 will profitable this two years but from 2010 will be back to the red.

at 1/26/2008 10:08:14 AM, Paul Rako said:
Interesting to see Microsoft's hardware ethics match their software ethics. They have been rushing crap out before it's done for twenty years. Why expect them to change now (Ohhhh they forgot that with hardware Micro$soft gets to pay for the failures, not us, oppsee.) Microsoft is evil because anytime they are faced with a decision to either do what is right for the long-term interests of the community or do what puts a little more filthy lucre in Bill Gates pocket, they always choose the latter. Serves them right to suffer, wait til people get familiar with Linux and see you don;'t have "reboot for settings to take effect" much less reboot your ASP servers once a day to zero out all the memory leaks.

at 1/26/2008 4:12:57 PM, M.C. said:
What a shame really. The first Xbox, of which I have three, two which I modded, are still running like lightning. Now THAT was a great console to me. I have a 360 I bought just to finish off the Halo story. I can honestly say that were it not for this game, I would NOT have purchased a 360. Luckily for me so far, no problems with the 360.........yet! But I just know somewhere near the end of my one year warranty I am going to be watching it very, very closely. Too bad MS, you has something good going, then you pulled your tpycial f#@k the consumer attitude out. I admit the PS3 is looking better choice for the future.

at 1/27/2008 11:56:57 PM, supernaut said:
my 360 works perfectly with no downtime although it is running approx. 2-3 hours a day for more than a year now.

at 1/28/2008 4:23:30 AM, VV said:
Funny how Bill Gates was running after me with his bugfixes while Ken Kutaragi has me running after him for more nice feature upgrades. I am so done with MS crap. When I saw Vista I almost panicked. Luckily there came Leaopard OSX update. Since last week I got my new MacBook and I dumped my last blue-sceen and red-ring crap with the garbage. My house is now officialy MicroSoft free!

at 1/28/2008 4:43:12 AM, JE said:
Worst customer service I have ever experienced. I have been on the phone with them 7-8 times over the last 3 months. Still no working Xbox. If you have not yet purchased an Xbox, I would recommend you get something else!!

at 1/28/2008 8:44:44 AM, Tender Vittles said:
Interesting story. We had the 360 about 9 months and the RROD appeared. I somehow was not surprised by this, in fact I was sort of expecting it. The customer service was apparently handled by an off-shore company where English was alledgedly a second language. In the course of our discussions, it sounded like MS wants to warrant the replacement for 90 days. If you want more, they want to sell you an extended warranty for 1 or 2 years, good deal for them but not so hot for you. The goverment forced their hand to honor a three year warranty period. Apparently they have discovered a way to sidestep the whole issue with this little trick. I have no idea if what they propose is legal, but then what the heck are my options with this piece of crap. MS ought to actually try designing their next product rather than turning out half baked designs like this. Unlike their lousy SW you can''t just make downloads available for the latest patches to please people. You actually have to fix the thing. I hesitated buying the PS3, seems I made a mistake.

at 1/28/2008 9:27:22 AM, TomN said:
The Verdant Electronics sounds good but highly impractical. Inablility to rework faulty parts - can''t test the assembly until it''s completed. Show me a part that doesn''t get reworked and I will show you a highly mature product or it''s an extremely low cost product. Products that are Z-height constrained would be penalized. More specialzed suppliers would have to be developed. Doesn''t lend itself well to lower cost consumer products.

at 1/28/2008 2:20:33 PM, Lol said:
Paul Rako is a ledgend, so true buddy. Greed rules the world! Except in my world. I use a 'Doesnt work, dont pay' policy with my designs. Microsoft use a 'Doesnt need to work, you will pay' policy. Makes me laugh every time I remember that Microsoft thinks people should pay for defective software and hardware hahaha. When they have a working operating system (not one that gets a virus in under 5 mins on the net after a fresh install) then I will pay (by then computers will be out dated). I dont buy new cars that dont work, why should I buy a defective operating system? Microsoft insults the term 'Software Engineer'

at 1/30/2008 1:37:56 PM, SteveS said:
Solder joint reliability of BGA packages is an issue manufacturers of mil-aerospace equipment have had to face for a long time. In fact, it's not just BGA packages, but any type of package that does not have some type of "compliance" in the attachment method. I can remember this problem first surfacing many years ago, going back to the use of LCC (Leadless Chip Packages) on boards. When the solder joints will fail depend on several factors, including 1) the size of the package, 2) the number of temperature cycles, and 3) the temperature swing.

at 1/30/2008 2:12:57 PM, 2512 said:
We have had issues with 2512 resistors cracking as they are very large and thin. I dont think glue will help them.

at 1/30/2008 2:23:09 PM, Prime said:
FOR REAL GAMERS TO READ ONLY!!! For all the Rookie gamers stop crying, just give your 360 some breathing room and maybe a "intercooler depending on the environment. I''ve got the XBOX360, PS3, and the Wii. Out of all three system I can honestly say I play the 360 more than the rest of my consoles. The graphics are a slight step down from the PS3 true but the 360 is still over all a better console, for right now. For one the PS3 doesn''t have many games for it. For two, most of the good games they have they are also available on the 360. For three the PS3 doesn''t even have the dual shock controller available for the console. How am I supposed to really get into a game when I can''t even feel the action. I mean it''s so basic. I might as well be playing PS1. As far as the Wii goes, it''s a fun unique system, but it gets old real quick. As far as graphics go I feel like I''m playing a game off the GameCube. Actually the Game Cube had a over better library of games than the Wii. All you have to do is just give your 360 some space to breath and get a "intercooler" not one of the cheap ones either and you''re good.

at 1/30/2008 2:25:53 PM, SF said:
I've had my console replaced 7 times now, all through the mail. 3 times for RRoD, 2 times for the drive scratching discs, and 1 time for a replacement console being sent to me with the "Wireless Connect" button on the front being jammed. Including that one, 3 have been DoA, and including those, 5 have lasted me less than a month. Although I love having it when it works (it's my primary DVD player and gaming console, and I use the Media Center Extender when at home and home from school), Microsoft has paid dearly for having me as a customer. Not only in shipping, but with a free month of Xbox live every time something goes wrong, as well as a couple of "free gifts as a way of expressing their apologies" - I've gotten a free wireless controller and a quick charger. I've tried demanding a brand new non-refurb console, but they won't budge and tell me I'm already at the top level of support when I ask to be elevated. I've had my current console for about a month now and so far no problems, but they're not going to like dealing with me if I have problems again.

at 1/30/2008 3:08:45 PM, AST said:
1) Buy Xbox; 2) Plug in and turn on; 3) Run a game in demo mode for 90 days straight; 4) Enjoy

at 1/30/2008 7:12:26 PM, KDL said:
I had my system for 1 year and 2 months before the dreaded "ring ring of death." Thank goodness for the extended warranty. I found the replacement process relatively painless. Getting it started was as easy as filling out an online form with the serial number. I was afraid I would be without my console for months, but the entire process of filling out the form, receiving the shipping box, sending it in, and then getting the replacement only took two weeks from start to finish. Of my five friends that have xbox 360's, four of them have had problems which required replacements. Ouch.

at 1/31/2008 7:38:03 AM, JB said:
I am aware of the many problems the 360 has, specially the 1st generation systems. That''s why I waited a year before I got one. So far so good, and I bought the extended warranty, just in case. As a gamer the system sees lots of use, including long sessions over the weekends. I got it because the many good games I didn''t want to pass up. I also have a PS3 that doesn''t see a lot of use (might change in the future). Will I buy the next system from MS? Depends. I''ll wait (probably longer than I did with the 360) before I bite.

at 3/25/2008 1:14:38 PM, your boss said:
at least tell me something i don't already know.

at 8/6/2008 4:38:00 PM, people who say its your fault you shoul said:
16 percent is a phenomenal rate saying that there are about 20 million created. Arguing about the cooling system makes a valid statement . why should someone have to buy a cooling system for thier XBOx when it already cost about 300 dollars. So what xbox is saying that they shouldnt put much effort into there cooling system pretty much making this product insufficient for its self. its cooling system is just their to fail wtf... Thier is so much profit to be made i ve never seen such a scam for people who do not have thier xbox under this extended warranty. It is believed that xboxs break down more than 16 percent this is obviously true.. why ?having the report only drawn for a limited time period doesnt give a valid statistic. Think about it your xbox isnt going to last like an n64 which still works today..Its DOOmed for failure so u can buy xbox 720 THink about any/ all the other systems that require an added on cooling system... thats right noneee. cooling systems shouldnt be relied on besides the one that is built in to the actual product unless you expect to have your xbox on neglectly 24/7.

Brian's Brain

Profile

RSS Feed

Recent Posts

Recent Comments

Most Commented On

Archives