A running log of all the boots and kernels that fabbione and I have been doing

Initial boot failure with ubuntu boot.img-19
prtconf output
an oops that's mostly like the initlal boot failure, but different
log of the post process through opm boot prompt
linux smp kernel /proc/cpuinfo contents
the submitted work around change

Ubuntu dapper release time. I spent several hours tonight downloading the about to be official boot images and running through a full re-install. Things went through without issue. Sweet. I look forward to really playing with linux on this hardware.. and being able to compare it to solaris.

Over the last several days/nights, David and Fabbione have been busy little beavers. They've been trying numberous new kernels and doing some binary searching to narrow down the underlying cause. With the release deadline for Ubuntu Dapper looming scarily close, it looks like a 'good enough' work around for the problems booting the linux/non-smp kernel has been found. Work was made considerably easier when David Miller was able to finally reproduce the problem on his own box once some of the triggering events were narrowed down on my box. There's apparantly a tiny block of memory just shy of the 8G mark that tickles some issue that's yet to be well explained. However, by ignoring any region of memory smaller than 256Kb the issue goes away. That's the fix that dapper is going to ship with until a better fix can be worked out. For those interested, the change.

With this fix in place, my box is, for the first time, happy with a standard boot.img file.

Setup access for fabbione to be able to upload images to my tftp server and work remotely while I'm asleep. The fun of living on opposite sides of the planet. Technology is great, isn't it?

Spent a good bit of time updating gdc to the recently released 0.18 and re-testing both some local patches and the original source. The t2000 shined here once again. Test runs in the 2 hour range. Filed a few bugs, but generally a pretty solid release and much needed, syncing up with modern versions of dmd. Subsequent commits to the now public svn repository bring it up to fully current with regard to dmd.

In anticipation of having to return the box soon, sadly, I need to start feeding patches back to dvdf. The t2000 hardware is certainly very nice to work with, but I can't afford the over $8k it'd cost to keep it. If I could, it'd be tempting.

Whoo hoo.. the install made big progress while I snoozed. I stayed home longer than I should have to finish the install prompts and headed to work quite late, but with a nice working bootable t2000/linux box.

Time passes and work is over for the day (at least mostly..).

After playing around for a bit with the uni-processor kernel, seeing that the box seemed stable, fabbione suggested booting into the stock -smp kernel that was supplied by the install process. I had doubts, but shouldn't have. As usual, he was right.. it booted, saw all 16 processors, and is generally mostly happy.

Now, the fun.. some box stressing to see how it holds up. I threw a 'make -j' at it. Full parallelism, no holds barred. It oopsed. Then it hit another kernel bug that's apparantly already known dealing with SLAB and fd releasing.

The testing continues....

Well, with my trial period nearing the end, and David Miller being swamped, it was time to take things into my own hands. I almost did this back a month or so ago, but at the time it seemed like David was gonna be available and his experience and skill with sun hardware and kernel issues so vastly trumps mine, that I didn't bother. I don't want to loose a known broken config since that'd make debugging the problem essentially impossible. So, the choice, dig in!

I hopped back on to #ubuntu-ports and got some hand-holding from fabbione, which has made things _so_ much easier. He was doing builds with varous kernel debug options and I was booting, and rebooting, and rebooting.. all trying to descern a pattern. Some boots would work, some wouldn't. We finally got through into user space and hit some known kernel module abi issues. So, fabbione built one more boot.img that had a bunch more builtin modules, enough to get through the install. The download time is huge.. so... sleep.

Been too long since I updated this log of my status. I'll fill in some past data this weekend but for now, a quick update. Feelings are mixed. As advertised, the box is pretty decent when all cores and threads are kept fed with work. My test workload is two fold:

• Compilation of gdc (gcc + the d language frontend)
• DStress, the d language compiler stress test system.

Both tend to keep most of the cpu's busy much of the time, but both are also somewhat disk intensive. I'm fairly sure that I'm hitting disk io limits which is keeping me from keeping all the cpu's busy. For compilations, my old Athlon 1700 outpaces the T2000. Numbers will be posted this weekend after I can run some more scientific tests (ok, so I can actually write down the numbers). One set of numbers I can share right now are dstress run times. On the Athlon 1700, with no attempt at parallelization, a full run would take 8-10 hours. On the T2000 with 'make -j32' (with 16 cores), a full run takes approximately 3-3.5 hours. Now, an important caveat, a whole lot more tests fail on solaris/sparc than on linux/x86 so the numbers aren't 100% comparable. But regardless, over twice as fast is very nice.

More soon.

One of the reasons I felt confident with this try/buy program is that David S. Miller had already gotten the linux kernel booting and a full, normal, Ubuntu Linux installed and running on his box. I assumed that I'd be able to do the same and work with an os I was comfortable with. So, I set about reading up on network booting and locating the boot image. I finally got all the pieces together after much reading and some rapid learning about rarpd and tftp. Shortly after the elation of that success, I hit a wall, the kernel wouldn't make it through boot (log).

I proceeded to subscribe to the sparclinux mailing list. After checking the archives I didn't see any traffic about T2000 boot issues, so time for a new thread. Luckily both David (long time sparc kernel expert) and Fabio (Ubuntu kernel guy and also a sparc guy) were both up and quickly engaged on the problem. After hopping on irc to short-circuit the communication lag of email, we tred a few things: an older image, some command line args, comparing hardware (prtconf output) between the three of us. Nothing obviously different except for cpu counts. The working theory before we break for the night is that it's either buggy hardware, though solaris doesn't seem to have any problems, or a bug that's triggered with 16 cpus but not 24 or 32 which is what they have. Unfortunatly David is going to be busy for a couple days but will get back to this probably wednesday. During some of the testing, I did get past the above error and got a different one that doesn't halt the boot process. The Ubuntu install process starts, but stalls after the second screen, almost certainly due to the underlying problem causing both the boot failure and the oops. More later this week I expect.

If you're paying way too much attention, then you might have noticed the mention of 16 cpu's in the above paragraph and that I'd asked for the 8 core model. Yup, I was shipped the wrong configuration, 4 cores instead of 8. If the problem with booting the linux kernel is related to # of cpu's, it's somewhat serendipitous, since catching the bug is good. However, it's half the cpu's I want for my testing. Mail has been sent to the guys at Sun. I'd like to get the right config, but until this linux boot issue is resolved, I don't want to give up the currently breaking configuration either.

Sorrow and frustration ensues. There's a required cable that wasn't included in the shipment. It's the SC serial management cable. It's how you actually get into the box to perform the initial setup. You only need it once after which you can use ethernet to get to both the management port and the standard login port. I don't have one laying around. What I do have, though, is spare network cables and serial cables and can cobble one together. That done I can successfully, for the first time, connect and figure out how to boot the machine.

Word to the wise, including some clueless newbie instructions on the website or with the hardware would be invaluable. A side by side comparison with linux would do wonders. I haven't used solaris as a user in a decade and never really as an administrator. After getting the initial boot and setup of solaris done, I feel rather lost. Few of the normal admin tools I've come to expect under linux are present. I know I can download a lot from the net and will, but it's like a deep form of culture shock.

Delivery! Alas, work comes first, so play time will have to wait.

Lacking a contact with the shipper, I make calls to sun to get the status of my shipment and try to track down the shipper. After a few attempts, apparantly the DHL they use and the DHL the rest of the world thinks of as DHL are two different entities. I finally talk with the dispatcher for the right DHL and arrange for a friday delivery.

Waiting at home all day didn't accomplish anything. No calls from the shipper. No visits from the shipper. I'm lost and anxious to get down to some serious progress.

The shippers attempted to deliver a pallet to me but not being home, they didn't succeed. The phone number they left in voice mail is incorrect, grf. I'll work from home tomorrow to receive the shipment.

Sweet, I got an email today about the box being shipped. The 10 day countdown has started.

My credit limit has been raised, yay. A quick mail sent off to the gentleman I've been talking to have him call me back.

I talk with another gentleman on the phone for a bit. Apparantly there are no options for individuals other than authing a credit card. For companies, however, the doors are much wider. As long as their finance team determines that the company is worth or worth the risk than chances are very good the application would sail through. Choices: up my credit limit, form a company, find one willing to back me. I opt for #1, as the most appropriate and on the up and up.

A major wrinkle forms in this little saga. Apparantly before Sun can/will ship me the trial hardware, I have to be able to provide them with a credit card they can auth for the cost of the box. This is a problem as I've never needed a credit limit this high. I ask them to call me to see what the options are.

Worried that I haven't heard from anyone in several days, I send off a ping to the gentleman I've been exchanging mails with. He assures me that it'll ship soon and I should have it about 10 days later.

Success, a flurry of emails back and forth with a couple hardware config quotes and I picked the medium model: Medium: 8 cores, 16 x 512MB (T20-108A-08GA2C) -- US$13,395. Pricy compared to the x86 hardware I'm used to dealing with, but it's a trial, so roll with it.

Attempt #2: After several days with no response, I decide it's time to try again. Still no company, but I said I had a product, as I was highly confident that gdc could be ported to run on sparc/linux and sparc/solaris without too much trouble, certainly within the 60 day period leaving time to take real advantage of the hardware.

Well, I got wind (thanks slashdot) of a wonderful promotion that sun is running for their new T2000 hardware line. It specializes and seems to be specifically targeted at the large niche of parallel tasks such as web servers or software compilation. I happened to have recently become involved with The D Programming Language, specifically the compiler itself. Building the compiler doesn't take super long, but running the test suite does (approximately 12 hours on my fastest box, an athlon 1700 w/ 3 gigs ram). Both are tasks, being perfectly suited for this hardware so I just had to sign up for the try and buy program.