Several questions have been posed about my 40-node cluster design (e.g., “Why build a Raspberry Pi cluster?”). I’ve decided to provide a little of the history of my project, and some thoughts on its possible future.
I first had the idea of building microcontroller-based cluster in early 2002. At the time, I was working with Motorola 68k microcontrollers in my senior design project for my BSEE at SIUC. I knew that such a cluster would have been nothing more than a novelty, as microcontrollers were entirely insufficient for running the software used by the supercomputers of the day. But the idea stuck; since that time, I occasionally checked the market for tiny platforms that might be sufficient for running industry-standard software. At one point, I even considered building a cluster of smart phones.
I think the first really promising mini-computer platform I saw was either the BeagleBoard or Pandaboard, either of which still costs over $100. While these were both awesome hobby platforms, they were pricier than what I’d hoped for and they had a lot of hardware I didn’t need, which also made them unnecessarily large. On the other hand, the Pi’s circuit board was about the size of a credit card and only cost $35. If you wanted a tiny, inexpensive platform like this with an Ethernet port (not wireless) in early-to-mid-2012, the Raspberry Pi was the best, hands down.
I knew about the Raspberry Pi in early 2012, but initially held off on ordering because I didn’t want the hassle of getting around the one-per-customer restriction. I didn’t learn that they’d lifted the restriction until 6/29/2012. After some debate, I ordered my first 5 Pis on 7/1. By then, Allied Electronics’ site already “estimated delivery time will be 10–12 weeks”. At the time, I had only planned to build a 5-node cluster, or maybe even 16-node if they were really suited to it.
While I was waiting for my first 5 Pis to arrive, I learned of the now-infamous 64-node Lego Pi cluster. And this is when it really started to sting that my 5 Pis hadn’t arrived yet. My Pis were not shipped until the day before Allied started shipping with 512 MB of RAM. (Allied, seriously, Y U NO ship the 512 MB version to the customers who patiently waited almost 4 months?) But I already knew by that time that I wanted at least 16 nodes in my cluster; I immediately ordered 11 more.
Had I received my Pis in July, I would have had several months to work with them before things turned busy for me in the fall. I had 5 Pis in late October of 2012, but didn’t get to play with them much over the following 11 months. The rest of that time was eaten up by internship responsibilities and thesis work. Of course, I don’t regret prioritizing that stuff; it had to come first. I didn’t get to seriously work on finishing my cluster until September 2013. Pi clusters were all over the Internet by then, some of them very well-constructed.
In spite of how long it took for my Raspberry Pis to arrive, I don’t regret choosing it. I admit, Broadcom’s announcement that they had open-sourced the BCM2835 video drivers has really sweetened the deal, because this should mean there will eventually be OpenCL drivers. However, this is not to say I’d still choose the Pi if I were starting this project now. If I had to make the same choice right now, I might choose the BeagleBone Black or wait for the official release of the Hummingboard, both of which should be compatible with my case design. There are also a number of other quad-core i.MX6 platforms on the market now. None of them with Gigabit Ethernet are near the Pi’s price point and size yet, but I expect that to change in the future. The i.MX6 has already had OpenCL support for a while though, which makes any suitable platform based on the i.MX6 appealing.
Many have asked why I would building a cluster based on smaller platforms, such as the Pi, suggesting that I could test distributed algorithms with hundreds of virtual machines, such as Docker or EC2 instances. Obviously, this is true. In the future, I’m sure I’ll set up a virtual cluster as well. But a Pi cluster is what I’ll be using for now, as there are benefits to testing on real hardware, even if it isn’t the target hardware for the software in question.
I built my cluster primarily for software testing, but bigger players, such as Dell, HP, and Intel are preparing for the day when data centers are filled with microservers. In a setting where redundancy is important, this makes a great deal of sense.
When I planned my cluster, I intended to create something upgradeable. The case I designed is easily adaptable to other platforms that are roughly the size of the Pi and it would also be easy to use Gigabit switches (e.g., the Cisco SG102-24-NA) in place of the 10/100 switches I used. With a better power card design and more powerful mini-computers (e.g., the Hummingboard), it would be much faster. This means the same case could support 192 processor cores and 80 GB of RAM. Just imagine what will be possible when similar platforms based on 8-core or 16-core ARM or Atom SoCs are available.