This site has lately been plagued with mysql crashes. After lots of procrastination, I finally decided to look under the hood rather than just restart the mysql daemon. Here’s what I found in the mysqld.log:

Yep, that’s a fatal error.

Did you see the fatal error in there? Looks like this micro instance it having difficulty allocating memory. After doing a little Internet research, I discovered that AWS micro instances with Linux do not have a swap space configured by default. So, without swap space, the database was crashing whenever memory became unavailable (which was becoming more frequent). Boo!

There’s an easy solution, however — set up swap space on the instance. Here’s how I did that

  1. Run dd if=/dev/zero of=/swapfile bs=1M count=1024
    ** this says, read from file /dev/zero, write to file /swapfile, reading 1m of data at a time, copying only 1024 input blocks.
    ** For more on dd utility, see the wikipedia dd entry.
  2. Run mkswap /swapfile
    ** make the swap area
  3. Run swapon /swapfile
    ** turn on swapping
  4. In the /etc/fstab file, add this line: /swapfile swap swap defaults 0 0
    ** this sets up the swapfile automatically after a restart

I then decided to take a closer look at total memory usage on the box by running free -m.

Yikes.

Whoa, that’s no good. Only 7M available? How much of that memory is actively being used by mysql? To check that out, I got the mysql pid by running ps -ef | grep mysql. Then I looked at the process status file at /proc/$pid/status.

Looks OK, but I just restarted this process.

Humm, not much memory being used here, but I just restarted the process not that long ago. To be safe, I set a bound on the amount of memory that mysql can allocate. I updated the /ect/my.cnf file to set a buffer pool size:

datadir=/var/lib/mysql
 
socket=/var/lib/mysql/mysql.sock
 
innodb_buffer_pool_size = 256M

I then went back to looking at how memory is being used across the system by running top, shift+m.

well it's not using too much memory

Looks like apache might be a bit of a pig too for this low memory machine. To limit the amount of work apache is doing on this little box, I made a special optimizing_memory.conf file in my /etc/httpd/conf.d directory; my http.conf file in /etc/http/conf automatically includes all config files from the httpd directory. Keeping my apache config changes in a separate file helps me keep track of my modifications versus the defaults. The content of the optimizing_memory.conf file are below, ripped directly out of this helpful apache tuning article.

Timeout 30
KeepAlive On
MaxKeepAliveRequests 50
KeepAliveTimeout 10

<IfModule prefork.c>
    StartServers          3
    MinSpareServers       2
    MaxSpareServers       5
    MaxClients            10
    MaxRequestsPerChild   1000
</IfModule>

I don’t think I need to adjust the memory usage in my php.ini file — it looks pretty reasonable.

The php memory limit seems ok.

Ok, so how is memory usage now? One more time with top, shift+m.

Now apache is doing less work.

And how about free -m?

Sweet, not there is more memory available system wide!

Looks like things are a little more under control. Now that I have a gig of swap set up and some bounds on memory usage, it’s time to sit back and hope that the site stays up more reliably :)

More Reading