Monthly Archives: March 2014

SD Card Issues

I’ve had 16 Pis running for about a week (with 16 GB SD cards) with no trouble. I’ve been running them with only 2 of 4 intake fans to maintain a power-to-airflow ratio that’s close to what I should expect from having all 40 of them running. I’ve been seeing case temperatures of roughly 31 °C (88 °F) and core temperatures of up to 77 °C (171 °F). 77 °C seems kind of hot to me, but it’s still acceptable for a BCM2835 core. I’ve read that 85 °C is supposed to be their limit, but this announcement by the Raspberry Pi foundation seems to indicate that 85 °C is an acceptable steady-state operating temperature.

Two nights ago, I decided to bring the remaining 24 Pis online. This required that I use 24 of my 8 GB SD cards . I didn’t anticipate any problems. I flashed 24 fresh 8 GB cards, brought the other 2 intake fans back online, and booted the remaining 24 Pis.

I went to the work of assigning static IPs to all of them. (Managing 40 machines through SSH without assigning static IPs is a hassle.) But, an hour or so in, I had to hard reset the whole system. As it came back online, I noticed that some machines didn’t appear to be booting up properly. (Their indicator lights were staying red.)

After a lot of trial and error to try to get these machines to boot, I realized that a whole bunch of the SD cards had warped, as shown in the photo below:
Warped SD Cards

 

Since the biggest group of them failed, I’ve noticed that a couple of stragglers have also warped to the point that they no longer work. This is important to note, because I’ve been running the system with no filters and with the left side of the case off, which greatly increases airflow. So the warping is continuing, in spite of the fact that they’re probably at no more than 55 °C now.

I’m not really sure what to blame for this. The Pi’s SD card socket doesn’t back the card when it’s fully inserted, which seems like a weakness in the design that would allow this to happen. However, Kingston rates these SD4 series cards for 85 °C and I have a hard time believing that any of them got over 70 °C at any time. There’s also the fact that the 16 GB cards haven’t had the same problem, which leads me to believe this is specifically an issue with the SD4/8GB. Either way, I’m pretty sure the solution is to add a clip to the socket to back the SD card so it doesn’t warp.

Progress at Compiling Mesos

I’ve been in the process of trying to get a good compile of Mesos on the Raspberry Pi for about 3 1/2 weeks now. I decided to document the challenges of this, in case it might help anyone else who is trying to do the same thing.

My first approach was to try compiling on a single Raspberry Pi. I used the recommended “make -j”, which uses an exorbitant amount of memory. (It was a big mistake to not look up the “-j” option.) I frequently got the nearly-useless error “g++: internal compiler error: Killed (program cc1plus)” mixed in with what appeared to be continued attempts to compile. After a great deal of searching, I found this was caused by the system running out of memory.

Memory Error

My impression at the time was that simply adding more swap memory wouldn’t be enough to allow the job to proceed to the end, so I decided to try to cross-compile instead. I spent about  a week trying to get Crosstool-NG to build a cross-compile toolchain that would run under Windows. It appears to be unable to do so, in spite  of the fact that it tries so very hard. I managed to get it to run to completion with settings that should have generated a working toolchain, but the files it generated were garbage. I intend to pick up this effort again later, after I get a few other things compiled. If I get it to work, I intend to document the process from start to finish, which apparently no one has done yet. (There are a lot of bits-and-pieces sets of instructions out there, but even the best of them skip steps that are necessary to make it work.)

After giving up on Crosstool-NG, I went back to trying to compile on a single Pi. I expanded the swap memory available to it. After some experimentation, I arrived at the conclusion that a 6 GB swap file was necessary. This seemed to be enough for the compile to run. I allowed it to continue to run for 3 1/2 days like this, but finally decided it was taking too long.

Memory While Compiling Mesos

In hindsight, adding a 512 MB swap file probably would have been enough if I’d used “make -j2″ instead of “make -j”.

This is when I decided to try getting distcc going on a subset of my 40 Pis. Even with the troubles I’ve had with distcc, I probably should have done this sooner. I set it up on 16 Pis (1 master + 15 slaves). Using this distcc cluster, I succeeded in compiling mesos for the first time on 3/6. It can compile mesos-0.16.0 in about 3 hours. At the time, this was a wonderful thing to see:

Mesos 0.16.0 Compile Finished

Before this will even work, the mesos source code has to be edited in several places to remove/replace Intel-specific assembly code. I also had to install gcc-4.7 (it is not necessary to remove gcc-4.6). I will document code changes in greater detail at a later date.

A pure distcc build results in the following errors during “make check”:

Python Remote ReplicaTest.Promise Error

I discovered that it’s possible to get ExamplesTest.PythonFramework to pass by deleting the mesos-0.16.0/build/python directory and re-running “make -j2″ without distcc:

Python Local

It only takes a few hours to finish it again. But it still baffles me that this would make any difference at all. With every instance of distcc running on a Pi (with the same OS, compiler, and configuration), I should get the same result regardless of which machine is compiling which part. At present, my only guess is that distcc is deciding not to pass along my CFLAGS and CXXFLAGS. But the only documented precedents I found for this would not apply in this situation.

I’m currently trying to find the source of the ReplicaTest.Promise test failure. I believe this failure is also related to discrepancies in how distcc compiles code remotely vs a purely local compile. However, I don’t know yet which parts of mesos will need to be compiled locally before it will work, or if compiling it all locally will even be enough.

Why the Raspberry Pi?

Several questions have been posed about my 40-node cluster design (e.g., “Why build a Raspberry Pi cluster?”). I’ve decided to provide a little of the history of my project, and some thoughts on its possible future.

Past

I first had the idea of building microcontroller-based cluster in early 2002. At the time, I was working with Motorola 68k microcontrollers in my senior design project for my BSEE at SIUC. I knew that such a cluster would have been nothing more than a novelty, as microcontrollers were entirely insufficient for running the software used by the supercomputers of the day. But the idea stuck; since that time, I occasionally checked the market for tiny platforms that might be sufficient for running industry-standard software. At one point, I even considered building a cluster of smart phones.

I think the first really promising mini-computer platform I saw was either the BeagleBoard or Pandaboard, either of which still costs over $100. While these were both awesome hobby platforms, they were pricier than what I’d hoped for and they had a lot of hardware I didn’t need, which also made them unnecessarily large. On the other hand, the Pi’s circuit board was about the size of a credit card and only cost $35. If you wanted a tiny, inexpensive platform like this with an Ethernet port (not wireless) in early-to-mid-2012, the Raspberry Pi was the best, hands down.

I knew about the Raspberry Pi in early 2012, but initially held off on ordering because I didn’t want the hassle of getting around the one-per-customer restriction. I didn’t learn that they’d lifted the restriction until 6/29/2012. After some debate, I ordered my first 5 Pis on 7/1. By then, Allied Electronics’ site already “estimated delivery time will be 10–12 weeks”. At the time, I had only planned to build a 5-node cluster, or maybe even 16-node if they were really suited to it.

While I was waiting for my first 5 Pis to arrive, I learned of the now-infamous 64-node Lego Pi cluster. And this is when it really started to sting that my 5 Pis hadn’t arrived yet. My Pis were not shipped until the day before Allied started shipping with 512 MB of RAM. (Allied, seriously, Y U NO ship the 512 MB version to the customers who patiently waited almost 4 months?) But I already knew by that time that I wanted at least 16 nodes in my cluster; I immediately ordered 11 more.

Had I received my Pis in July, I would have had several months to work with them before things turned busy for me in the fall. I had 5 Pis in late October of 2012, but didn’t get to play with them much over the following 11 months. The rest of that time was eaten up by internship responsibilities and thesis work. Of course, I don’t regret prioritizing that stuff; it had to come first. I didn’t get to seriously work on finishing my cluster until September 2013. Pi clusters were all over the Internet by then, some of them very well-constructed.

Present

In spite of how long it took for my Raspberry Pis to arrive, I don’t regret choosing it. I admit, Broadcom’s announcement that they had open-sourced the BCM2835 video drivers has really sweetened the deal, because this should mean there will eventually be OpenCL drivers. However, this is not to say I’d still choose the Pi if I were starting this project now. If I had to make the same choice right now, I might choose the BeagleBone Black or wait for the official release of the Hummingboard, both of which should be compatible with my case design. There are also a number of other quad-core i.MX6 platforms on the market now. None of them with Gigabit Ethernet are near the Pi’s price point and size yet, but I expect that to change in the future. The i.MX6 has already had OpenCL support for a while though, which makes any suitable platform based on the i.MX6 appealing.

Many have asked why I would building a cluster based on smaller platforms, such as the Pi, suggesting that I could test distributed algorithms with hundreds of virtual machines, such as Docker or EC2 instances. Obviously, this is true. In the future, I’m sure I’ll set up a virtual cluster as well. But a Pi cluster is what I’ll be using for now, as there are benefits to testing on real hardware, even if it isn’t the target hardware for the software in question.

Future

I built my cluster primarily for software testing, but bigger players, such as  DellHP, and Intel are preparing for the day when data centers are filled with microservers. In a setting where redundancy is important, this makes a great deal of sense.

When I planned my cluster, I intended to create something upgradeable. The case I designed is easily adaptable to other platforms that are roughly the size of the Pi and it would also be easy to use Gigabit switches (e.g., the Cisco SG102-24-NA) in place of the 10/100 switches I used. With a better power card design and more powerful mini-computers (e.g., the Hummingboard), it would be much faster. This means the same case could support 192 processor cores and 80 GB of RAM. Just imagine what will be possible when similar platforms based on 8-core or 16-core ARM or Atom SoCs are available.