Jan
9

My new test lab

By kellabyte  //  Random  //  15 Comments

I recently started building my own test lab so that I can run much more reproducible tests and have dedicated hardware that has sufficient capacity to do some larger testing and experimentation. I’ve gotten a lot of requests to publish what I’ve built, so here it is!

This is my private test lab that I funded myself to continue to improve my skills. This was a necessary step for the kinds of things I work on.

The whole setup cost me about $2,000 Canadian dollars so far. I wasn’t as cost effective as I could have been since I bought a lot of cabling over the counter at a local store.

Hardware

1 Dell PowerEdge Server XS23-SB 4 Node 8x L5420 Quad Core 96GB RAM 4x HD Caddy (about $980 shipped)

  • Each node has 2 Intel® Xeon L5420 2.5ghz quad core CPU’s for 8 cores for a total of 32 cores.
  • Each node has 24GB of RAM for a total of 96GB of RAM. Each has a free slot open that allows me to upgrade them to 32GB for a total of 128GB of RAM.
  • Each node has 3 hot swappable drive bays for a total of 12.
  • Each node has 2 Intel® (ESB2/Gilgal) 82563EB gigabit ethernet controllers for a total of 8.
  • Each node has 1 ADATA XPG SX900 256GB SSD (about $100 each).

I’ve plugged in all 8 gigabit LAN controllers into a TP-Link TL-SG1024 24 port gigabit switch ($120). I went cheap on the switch for now. I debated buying a used Cisco switch but I figured this is fine for now. I may use the PCI-Express slots to upgrade to 10GbE or Infiniband in the future.

Some questions will probably be raised why the SSD’s I picked do not have very good IOPs. I want to have 3 different drives per node for different testing. I want each node to have a mediocre SSD, a high IOP SSD and a rusty spinning HDD. This will allow me to do different kinds of tests. Since I needed some drives to install the OS and get the cluster functioning, starting with the mediocre SSD’s for the OS drive seemed like a fine starting point.

Power cost is reasonable. I estimate power cost for this system at peak CPU usage 24 hours a day is around $30/month. I won’t be slamming it that hard 24/7 (well maybe sometimes!).

Software

Right now each of the 4 nodes are running Ubuntu Server 12.04 Precise. I plan to dual boot the ADATA OS drives with Windows Server as well since I do a lot of cross-platform development and testing.

A goal I had early on was to automate installing and configuring the machines because I would likely be doing this many times over and keeping all 4 nodes consistent with each other is important. I decided to use Puppet for this because I am familiar with it already. I created 1 puppet master on a small Ubuntu virtual machine that I host on Windows Azure. I put my Puppet master on Azure because I didn’t want to use up a computer or run a virtual machine taking up resources. Since this isn’t required to be running all the time and Azure only charges me for use, I can safely shut this down when I don’t need to sync changes or re-configure a machine. Every time I want to install a package on one of the 4 nodes I add it to the Puppet configuration.

I am using Docker and Linux containers to partition the physical machine nicely without the virtualization performance overhead.

With 32 total cores and 96GB of RAM, the likely configurations for tests depends on the resources required in the tests and what the software requires for CPU and RAM.

  • 25 containers per node for a total of 100 containers (for tests that don’t use a lot of CPU or a lot of RAM).
  • 8 containers per core for a total of 32 containers.
  • 4 containers per node for 2 cores per container for a total of 16 containers.
  • 2 containers per node per CPU for 8 total containers.
  • 1 container per node for 4 total containers.

For Windows tests there isn’t a good technology for containers so I might just find ways to run multiple copies of the same software at once and in some cases only run 4 node tests. Virtualization is an option if I need to.

Final thoughts

This will allow me to do performance benchmarking, experimentation as well as distributed systems testing. I can do interesting things like treat each hardware node as a data center and mess with the gigabit connections to introduce latency or partitions and some other interesting tests.

I like this setup because in one 2U server I basically have 4 physical machines with 2 quad core CPU’s each with decent amount of RAM (which I can upgrade to double later). I feel for around $2,000 (which could have been cheaper if I was more diligent) that this is a pretty damn good setup.

I am really happy with this setup. With all the varying drives I want to eventually buy filling every drive bay, I expect the final build cost to be around $2,500 – $3,000. Even without those drives this is a pretty capable setup already. Considering the cost, I get good CPU and RAM capacity and a dedicated lab for experimentation and testing.

The one drawback to this setup, noise! This server is meant to be in a rack with many other servers. Yeah, it is pretty loud.

  • http://wallmarkets.ru/ Albert Nikanorov

    Wow.

  • http://wallmarkets.ru/ Albert Nikanorov

    Are you getting to establish something like GoDaddy?

  • http://wallmarkets.ru/ Albert Nikanorov

    Lots of abs and unknown words in this post, :( But I had an idea to do the same a couple days ago, so I guess it’s destiny that I found you on twitter then your website. Internet is miracle :) ))

  • Obdurodon

    If I were to do something like this, I’d put it in the basement just because of the noise. It’s worth running the extra cable. Then again, most of the time I’d be even happier renting servers by the hour from someone like Rackspace. For example, their 15GB “Performance 2″ servers offer dual SSDs and 1.25Gb/s (though that’s not a guarantee). That works out to $21.76 per work day for a four-node cluster. Since I actually spend most of my time designing, writing, or reviewing code rather than testing, I figure I’d actually only need to be on those machines half time at most, so that’s approximately a year to break even.

    I’m not saying that’s the way you should have gone. Maybe you need more control and/or consistency than that approach would provide. On the other hand, it’s certainly an option to consider for many people. With good automation to minimize the time spent with machines actually spun up, the financials can look pretty good.

    • Kelly Sommers

      Yeah totally agree! For some people renting time in the cloud could make more sense.

      In my day job, we ship physical hardware and as you suggested I need more control since some of my experimentations will be with different hardware setups.

  • aarongreenlee

    Thank you. I have been awaiting this post as we all followed your progress on Twitter. Keep it up!

  • http://www.phrasememe.com Greg Bray

    Very nice! I been trying to convince my boss to let me expense a Dell PowerEdge VRTX as a mobile test lab. They have 4 blades and a mini-SAN that would be perfect for POC testing and tweaking. Details at http://fw.to/6FAfJRG

  • Marcel Popescu

    Damn… I’ve been threatening my wife with a 128GB server for about a year now… I have NO clue what I would use it for, I just like the idea. Good job :)

  • Ofir Manor

    that’s very cool!
    Maybe my post on my LXC/Ubuntu/Hadoop setup will interest you, regarding hacking static IPs to specific containers (I needed it for Hadoop)
    http://ofirm.wordpress.com/2014/01/05/creating-a-virtualized-fully-distributed-hadoop-cluster-using-linux-containers/
    (also, when you upgrade to 14.04, maybe consider BTRFS for the containers – I use it and love it)

  • Kapil Thangavelu

    Cool article, i’ve been looking for a chasis to do some openstack testing on. I’m curious which vendor you picked up the chasis at? Are you doing ipmi/bmc for power management, wasn’t clear if the nodes powered on separately or its a single bmc for the box. thanks.

  • JJ Welch

    So glad you posted this, it has inspired me. I think it would be fun for my kids to see this thing in action and connect what they are doing on their iDevices to what happens on the other end. I think I will be trying this out myself and then finding a way to cool it without the noisy fans as a “maker” project.

  • http://www.mcwtech.com Brian A. Randell

    Kelly, how do you like this box so far
    Did the machine come with HDDs or did you add your own?
    Did you buy locally or order online? Looks likes an older model.

  • Pingback: Hanselman's Newsletter of Wonderful Things: January 21st, 2014 - Scott Hanselman

  • Pingback: Hardware | Pearltrees

  • Pingback: Hanselman's Newsletter of Wonderful Things: June 2nd, 2014 - Scott Hanselman