Disaster. NOW.

While I was writing this blog post yesterday I had no idea how accurate picture I chosen to adorn it. I had absolutely no idea that I was actually playing a game with fate and obviously he hit me badly today.

So thanks to ASUS I got my replacement mainboard delivered today. I gave it a couple of hours to get used to conditions inside my flat given it most likely spend last few days in a cold warehouses or trucks. Here we go! Disassembly was quite fast but I've noticed something interesting - my heatsink didn't seem to be in proper contact with CPU. Then I just started to think what if the heatsink is too heavy and it pushes the cpu so hard that it does not have proper contact with pins on upper side? Sounds crapy. 

Nevertheless; I assembled everything back again. Just to be on the safe side - I put discoverer on side just to make sure that heatsink is unable to make any unbalanced pressure on CPU. Held my breath, power-on. It's b7 again. Damn it...

Looks like there is some serious issue around, but who to blame? Second motherboard same fault? Not really much possible from the probability point of view. So I decided to completely remove heatsink from the CPU itself. Well, it may be a little bit risky but it shouldn't burn in flames after a minute and I'll make sure that it's not heatsink fault. Okay, power-on... b7. Its beginning to be ridiculous. CPU fail? Is that really possible? Time to change CPU. Being proactive, I equipped myself with additional E5-2630L-v2. Power-On... b7 again. What's wrong with that stuff?

Having all this experience I decided to get in touch w/ASUS support again. It's more likely to have two broken motherboards than two broken CPUs. It was easy to replace the mainboard (but anyway I'm grateful for ASUS to be such a professionals and I'm impressed by the way they handled it). Are we gonna start blame game now? Let's see what's ASUS support is made from. My best guess? Both MBs are good, there may be some issue with support for E5v2 CPUs, hopefully to be corrected by BIOS update. Oh, by the way I've just taken this little screenshoot from ASUS Web page. :-)

Double trouble

This week I was quite excited. As briefly mentioned here, recently I've ordered last piece of the puzzles: hard drives and memory.  I was expecting delivery of everything next couple of days and my configuration should be finally completed. Yup, it took me more than month from first order up to today. Amazing, how fast time flows, huh?

Just to be on the safe side I've ordered 4x 16GB DDR3 from motherboard HCL. I didn't think about it too much, I took cheapest ones - Kingston KVR16R11D4/16. At very beginning I thought that I'm gonna go for 2x32, but apparently 4x16 costs as much as 1x32. With 16GB DIMMS my mainboard would have 128G capacity, which seems to be good enough for foreseeable future. Interestingly enough, one delivered DIMM was slightly different then another, but at the end I con't care about colors as long as it works. Added to my current 16GB will be 80GB in total. Sweet.

Having in mind my previous experience with memory seating I opened up my manual and started to populate slots. Have to say that this single-notch dimm sockets are crazy Idea. Done, power up, and? You got it. Stuck in b7. Again. After a few hours of playing with DIMMs I gave up. Looks like every single one works perfectly on its own. I managed to have working 64G configuration, but I've noticed that there are some slots on my motherboard which simply does not accept memory at all - when something is inserted it ends up with b7/b0 code.

I reported this to ASUS and to be honest - I'm very satisfied how they handled this issue. After a few e-mails exchanged we agreed on motherboard replacement. Just to make it short - I'm expecting delivery of new mainboard tomorrow and I reported it two days ago. It should be normaln - why I am so happy? Well, I'm not business customer and I was afraid that they push me back through normal warranty process meaning talking to supplier and sending motherboard back, then waiting for at least two weeks for replacement. Special thanks to Pawel from ASUS for helping me out. And - by the way - ASUS suspects damaged CPU pin or DIMM slots.

All in all after I gave up with memory I ended up with working 64G configuration and I decided to go ahead and install hard drives - hoping that I will solve this memory issue later. Disks arrived wrapped up with bubble envelope and it was quite surprising to me but well, I explained it to myself - they know what they send and how to handle it. I couldn't be more wrong. After connecting I noticed excessive vibrations and noise. 1st thought was OMFG - I've never seen so noisy hard drives before!

Then I realized that it must be something wrong with drives. I pull it out of the case again. See it yourself...

Damaged badly. Again, back to the supplier. They agreed on replacement. Thats good. No delivery blame game. Bad thing is that I'm kinda stuck for at least another week now.

In the mean time I'll try to finally setup or at least design some basic virtual lab topology. And yeah, test my SSD, cause it seems to be a little bit slower than promised. Hopefully not another subject to replacement. Stay tuned!

Unleash the power of SSH tunneling!

This time I will describe something practical. Nothing really exciting or new, but I always forgot this syntax and it's boring to read SSH manual over and over again... So let this be my memo - if it is also useful for you - great!

Imagine you have a server without internet connectivity at all. But you can - of course - connect there via SSH. And you need to download something from the internet on the server itself. Badly.

That's easy! - you say. You can download it to your host then scp to the server. Done! 

Yup, you're totally right. Imagine however, you have to install perl module with tons of dependencies. Downloading all of them one-by-one, then compiling by hand just to reveal that there is another dependency in place.. nightmare. But here comes the solution! SSH Tunneling!

Step #1

Go to ftp://ftp.perl.org/pub/CPAN/MIRRORED.BY and select your favorite CPAN mirror. For purpose of this tutorial I'd use fist one mentioned - http://mirror.23media.de/cpan/

Step #2

Connect to the server and tell SSH to create tunnel starting at remote SSH connection endpoint (-R) port 65432 and ends at selected CPAN mirror (mirror.23media.de:80). (This example uses OpenSSH, but you can use it with Putty or any other SSH client if you want)

ssh username@remote.server -R 65432:mirror.23media.de:80

This command basically establish SSH connection to remote.server and create tunnel from that host, (127.0.0.1:65432) with endpoint at mirror.23media.de:80. You can test it if you want by typing 

telnet 127.0.0.1 65432

just after you authorize yourself at remote server correctly. If connection succeeds, you know that you are good to go (you can always follow HTTP protocol and use GET method to do further testing).

Step #3

Launch CPAN. If it is already configured pick your favorite editor and edit your ~/.cpan/CPAN/MyConfig.pm file. Find urllist and change it to:

'urllist' => [ q[http://127.0.0.1:65432/cpan/]],

(If you haven't got CPAN configured previously, dont panic - run cpan normally and answer default to the most of the questions except first one and at very end don't agree for automatic probing and just provide mirror path like above)

There you go! Now you can launch CPAN

$ perl -MCPAN -e shell

and you can enjoy automatic installation of all required perl modules.

cpan> install Net::SSH::Perl

This method, however can lead to issues when during installation process CPAN wants to download something from other url than expected. But now you know how to overcome that issue, doesn't you?

Ahh, almost forgot. What about unleashing full power of SSH tunneling? This example was good for babies. Imagine you need to jump through another 2 servers. Yes, you can join multiple tunnels together. All in single command! The important thing is that respective sockets must match. This is a little bit tricky at the beginning, but once you get to the speed...

ssh username@remote.server1 -R1025:google.com:80 ssh username@remote.server2 -R1026:127.0.0.1:1025 ssh username@remote.server3 -R1027:127.0.0.1:1026

telnet 127.0.0.1 1027 ... :-)

And oh, by the way. Net::SSH:Perl has about 35 dependencies...

$ ls -la ~/.cpan/build | wc -l

35

Know a better way? Share!

Pink pendrive story

Last few weeks passed under the sign of pink pendrive which I use it to boot discoverer. I've got many questions asking "do you recommend this setup?" Yes, I do - short answer (by the way, I'm not sponsored by anyone :-)). Long answer below. I decided to write this quick summary even though I'm not 100% complete yet still missing a few components (mostly disk drives, BMC controller and memory), but I think that I've gone through the hardest part and finally there is something to share.

CASE

I can say only good things about the case I've bought. Not too heavy (like my previous one, Chieftec Dragon), but still top quality. Looks good even in living room :-)
Air filters at every inlet do a great job - even though after the month its still not 24/7, there is already quite a bit of dust on them. Otherwise It would be collected inside.

Next great thing about this case I'd like to emphasize is that it has great cable management - you don't have almost any cables inside the chassis (they're all on the back). This is also good for cooling. PSU placement on the bottom is definitely great idea.

        

PSU
So far I haven't put heavy load on this box, but with all my VM up & running (ESXi, vCenter, FreeNAS, 2x nested ESXi) doing nothing it consumes about 52W of energy. I've never seen more than 120-130W. Looks like my 550W PSU is a little bit overkill, but to be honest, today I'd go for same one. It has all safety measures, plus modular cabling - this is definitely worth its price! One more thing about PSU itself- before you buy it - it's definitely worthwhile to checkout the length of the cables - I got lucky and they were just as long as I needed. 1cm less and I'd be in trouble.

COOLING

Case itself was equipped with 2 140mm coolers - one on the front and second on the back of the chassis. Quite silent, but airflow takes its sound. 

Additionally I have one cooler inside PSU and obviously my favorite second one on the CPU - cooling not only CPU itself but as you can see on the image - memory too! There are air filters in every inlet of the case, and surprisingly there is a lot of dust on them, provided that It does not run 24h yet. In terms of silentness - everybody is different and have different expectations. As I said, I didn't have the opportunity to make this cpu hot yet. Anyways, so far its quite silent - normally you wouldn't notice it is even running. In the night when it's really quiet - you will hear the fans, but it's not really annoying. My MBPR is actually more noisy when it starts its coolers.

MAINBOARD

Reliable server building block? So far so good, but story here is a little bit more complicated. Overall - I think I'm pretty much happy with this mainboard even it's not perfect (do you know perfect one? :-))

One of my biggest disappointments was management NIC because i don't (and I don't wanna have) monitor/keyboard to manage it locally when needed and it wasn't stated clearly that you need to buy additional module to support it. Another one is that only 2 out of 6 SATA ports are SATA3 interfaces.
On the flip side - mainboard is fully supported by VMware. I managed to run nested ESXi with 64bit nested VMs. I run Wake on Lan and PXE boot and both works just fine.

Pros:

+ Small form factor
+ 256GB of memory support w/single CPU on 32G DIMMS
+ two RAID controllers on-board
+ management ethernet on-board (but works only with external BMI add-on)
+ two real GE nics (connected to PCI-E) w/PXE support
+ additional mangement GE NIC
+ support for BMI
+ serial console support

Cons:
- BMI must be purchased separately (about $50) and without this your management NIC doesn't work
- Only two SATA3 out of 6 total slots
- Only RAID1/0 support on-board, for RAID5 you must buy additional Asus PIKE card (fits only into special slot on mother board)
- Really stupid and annoying one-side dimm latch. Might be OK if you want to add your memory once, but provided my troubles with memory support - a nightmare
- No external USB3.0 connector - forget about your front panel blue USBs


CPU
So far most expensive part of all this venture. Most likely to be dethroned by memory in the future. Just to mention - It's Intel Xeon E5-2690-v2 doing really only NOP's here. Nothing really much to say except looks like it's huge overkill and only one thing which could possibly justify this CPU is 768G (YAY!) memory support. Currently there is no more than 64GB memory support in any Intel high-end desktop CPU. My main board supports 256G and I bought it for a reason. Hitting this limit with this CPU is only a matter of time and budget.


MEMORY
Nothing really much to say here except after a little bit of struggling it works (all 16igs). But unfortunately it looks like 16Gb of memory is sufficient enough only for ESXi, vCenter and FreeNAS. I just ordered another 64Gigs. Watch out for update.

STORAGE
So far it doesn't really exists. My old good 500GB usb drive turned out to be as slow as pendrive. So I've gone for another solution - 256Gig SSD for critical data and 2x4TB SATA3 drives in RAID0 for capacity. Trying not to spoil my next story - SSD was already delivered and I have decided to test its performance having some interesting results while I still wait for SATA delivery.

SUMMARY
After all - Looks like it was worth it and everything seems to work nicely. I'm gonna also order this BMC kit as I don't like the idea to connect via the serial console every time I want to change bios setting (local console doesn't work through this serial console at all) and I'd love to have remote KVM in case of disaster. Still waiting for memory & drives, to be finally able to run something useful. And yeah. I owe you pink pendrive true story. Well, I bought it few years ago for my wife. Apparently she haven't used it even once. :-)

Any thoughts or questions? Feel encouraged to comment!

A new hope

Okay, it's not good, but it's not very bad either. Looks like my worst case scenario just came to play. What to do? What to do? Okay, let's start with something. b7. Where's my manual? Here we go. It has to mean something. "b7. NVRAM configuration reset". Great. Google, you are my last resort. Still nothing? Somebody mentioned main board replacement? I'm so screwed...

Going back to the very beginning of the story - a month ago (yeah, I promised to put it next day, sorry about that) I finally decided to order all missing parts and put together my new toy^H^H^Htool. It was quite funny, but at the end it failed. Ended up with dark screen (didn't show even a thing) and this mystery b7 code on led indicator. I had to figure this out. It could be anything. So the first one and most obvious, easiest to tshoot was memory. I removed all three DIMMs and put back only one. First one... Nothing. Second... Like a charm! So after a bit of experimenting it revealed that both 4GB banks works perfectly, but third, 8GB one does not. 

First thought was that it was damaged, but then I realized that it may be some compatibility issue and after confirming that I don't have most recent BIOS version I've gone for upgrade. After that I was able to run every single memory bank, but only two of them at the same time. Weird, isn't it? After a little bit of further experiments I discovered that my manual is wrong and I changed the placement of memories in the slots for a few times. Then I finally managed to have my server up & running.

Ready-Steady-Go?

Oh no, wait. I still need some storage. Here is the great part. This mainboard has on-board USB slot, so you can plug in any USB device you want directly into motherboard. How cool is that? Of course I'll run my ESXi from usb and have all my disk space dedicated to the data stores. Okay, usb stick plugged in, drives mounted and connected. Onboard RAID controller enabled and assigned to all SATA ports.

Power-on, go to RAID controller ctrl-m, ctrl-m, ctrl-m... There we go! Crash course mode on, learning by doing. After a bit of time walking around the RAID controller BIOS i found out, that my 500-GB Seagate Constellation actually reports more than 2TB of capacity. I got these drives somehow second-hand, so my first thought was "Great, I thought that they are 0,5TB and they are actually 2.5!", but then I realized that I cannot initialize array on them. Moreover, I already noticed strange sounds from them just after powering on the machine. Another failure...


Then I recalled that I have my good old 500GB transcend drive. It spent last few years in the cabinet, so without any regrets I got it outside of case and put it into the server. Maybe its not long-term solution, but at least its gonna allow me to have some start. Then I'll move to new disks once I'll have it.




At the end? Thats it? Up & Running. 16GB Memory. Booting from USB Stick. 500GB of Storage. Good to go! Let's setup PXE and finally install ESXi!. In the mean time, let's make some review.

Fix your lens. Yourself.

Things tends to be malignant. That's it. I learned it once more on the very first day of my vacation when I lost autofocus in my camera. I bought it about 8 years ago, so I'm not complaining. It served me well, but couldn't that happen a few days later?

At the end it wasn't so bad. My lens was still turning automatically in one direction. With that being said, I was able to use semi-auto-focus - just turn my lens manually in counter-focus direction, then half-press the shutter to focus. Sometimes I had to do this multiple times, but still better than nothing. And the pictures I took were not too bad anyway.

When I got back home I googled a little bit around and realized that I have three options:

1) Sell this camera body with broken lens as is. I was thinking about replacing it anyway.

2) Fix it in service - it should be no more than $70
3) Fix it myself - sounds like a challenge. And should cost around $7

What choice did I really have? :-)

Found this great tutorial on youtube and disassembly went pretty smooth. I identified broken flex cable and I ordered one on the internet. I put every part of my lens to the box for a week waiting for delivery. Once I got it I was waiting for a good moment to assembly things back again. Then I realized that I screw up. I should have wait with disassembly of my lens until the cable arrives, because during the week I just forgot order of operations and which screw I should use to secure which part. 

So I was kind of stuck with something like this on the table:

Ok, where to start? Again, I used second part of great tutorial I've found. The only problem I had was that disassembly went so smooth, that I decided to go a few steps further than tutorial and I just disassembled everything. Good news - it was much easier to replace the cable, because I had interesting part completely removed from lens body. Bad news - I had no idea how to put it back together. So I just started with broken cable replacement.

        

Then I spent another two hours trying to find out how to put all the things together to reflect the state of the lens from the beginning of second part of tutorial. It was quite frustrating, because every time I managed to put things together so I was able to turn around focus or zoom and lens behaved like it should, later on it revealed that parts were not aligned properly and I had to start from the beginning. Go figure.

When I finally understood logic of all moving parts and put it together properly, I just followed that lovely tutorial. It might be a little better quality, but it's good enough if you watch it carefully.

Good progress

Keep it up!

Almost there.

Yup, almost there. Just put back PCB, connect flex cables without damaging them, close the corps and... good to go?

Ready, steady?

You can only imagine how good I felt once I realized that it actually works!

If your lens ever happen to fail, you should fix it yourself. It's quite tricky and complicated to disassembly it and then put it back together, but it's definitely not a rocket science. 

Stay tuned, I have some significant progress on my Whitebox project to report!

PXE: Freedom with capital F!

During my entertainment with my new virtualization box here, here and here one of obstacles coming into my way was lack of cd-rom drive. Putting installation image on bootable USB stick was tempting, but its all about fun, isn't it? Besides, I always wanted to have network boot environment set up and ready to use for number of purposes, so it looks like perfect occasion to me.

So what's the big deal with PXE? It allows you to boot operating system directly from the network with zero-touch configuration on the machine that you want to boot itself. Its quite easy to setup and with this guide you'll be able to set it up within 15 minutes.

What you need:

• dhcp server
• tftp server
• pxelinux distribution

How it works?

1. Machine boots up, and its set to PXE network boot
2. It issues DHCPDISCOVER and waits for the reply
3. DHCP server assigns IP address, tftp server and tftp image path to it
4. It configures network stack using provided informations
5. Then it download specified boot image from tftp server and fire it up

Easy, huh? Let's go straight ahead to the configuration.

I tried two different dhcp servers for this purpose - IOS and (of course) ISC dhcp server. I have also used static dhcp binding to my MAC address.

IOS DHCP:

ip dhcp pool pool-name

 host 192.168.102.20 255.255.255.0

 hardware-address aaaa.bbbb.cccc

 bootfile pxeboot/pxelinux.0

 next-server 192.168.102.10

 domain-name domain.name

 dns-server 8.8.8.8 8.8.8.4

 default-router 192.168.102.1

ISC DHCP: (dhcpd.conf)

allow booting;

allow bootp;

option domain-name "domain.name";

default-lease-time 600;

ddns-update-style none;

authoritative;

log-facility local7;

subnet 192.168.102.0 netmask 255.255.255.0 {

  host hostname {

    hardware ethernet aa:bb:cc:dd:ee:ff;

    fixed-address 192.168.102.20;

    filename "pxeboot/pxelinux.0";

    next-server 192.168.102.10;

  }

}

Setting up IOS dhcp is as simple as entering those commands provided you have IOS device on your network. ISC DHCP may be a little more complicated - you may need to download and configure it from source package but it's a part of probably every single linux distribution. I set it on my macbook using osx ports:

port install dhcp

<edit config file>

launchctl load -F /opt/local/etc/LaunchDaemons/org.macports.dhcpd/org.macports.dhcpd.plist

launchctl start org.macports.dhcpd

Note that if you have any issues with your dhcp.conf it will just silently fail, so it's always a good idea to test it on the very beginning using 

/path/to/dhcpd -cf /path/to/dhcpd.conf -d

It will run in foreground and show you what dhcpd is actually doing.

Ok, so far so good. Now its time for tftp! 

Again, most linux distributions should include one so you need to refer to your documentation to set it up, on os X its a part of the system, so its just a matter of enabling it:

launchctl load -F /System/Library/LaunchDaemons/tftp.plist
launchctl start com.apple.tftpd

Now it listens for incoming connections and servers files found under /private/tftpboot directory.

Okay, so our infrastructure is basically ready for PXE applications. You will find required package here: http://www.syslinux.org. You just need to download latest package and thats it. Then it's time to setup tftp files required to netboot your station.

Without going too much into details, this is what I've done and works for me:

./pxeboot/ldlinux.c32

./pxeboot/libcom32.c32

./pxeboot/libutil.c32

./pxeboot/mboot.c32

./pxeboot/memdisk

./pxeboot/menu.c32

./pxeboot/pxelinux.0

./pxeboot/pxelinux.cfg

./pxeboot/pxelinux.cfg/default

./pxeboot/VMware

./pxeboot/VMware/VMware-VMvisor-Installer-5.5.0.update02-2068190.x86_64.iso

First and most important file is pxelinux.0. Like all of the files in /pxeboot directory, it was copied out from syslinux distribution package. It is boot loader for your host. Then you need to create pxelinux.cfg directory and put config file for your host inside it. If you boot a host from PXE it will look for its configuration file at this directory (details). I want universal configuration, so I just put my config into "default" filename, so it can be eventually loaded by all PXE-enabled hosts.

This is the content of this file:

DEFAULT menu.c32

MENU TITLE PXE Boot Menu

NOHALT 1

PROMPT 0

TIMEOUT 80

LABEL hddboot

 LOCALBOOT 0x80

 MENU LABEL ^Boot from local disk

LABEL install

  KERNEL memdisk

  APPEND iso initrd=VMware/VMware-VMvisor-Installer-5.5.0.update02-2068190.x86_64.iso raw

  MENU LABEL ^ESXi-5.5U2 Install ISO

Again, this is still quite simple if you have any linux boot loader (grub, lilo) background. There are two options specified - local boot and VMware ESXi installer-iso boot.

And this is it! All you need to make sure now is to copy all required files into required places and it's done. Set up your test box for PXE booting and enjoy all new possibilities. I just booted & installed vmware on my WhiteBox using this setup.