I received the error “The change you wanted was rejected” after I moved a rails app from one host to another.

Looking into the log file, the following error appears:

ActionController::InvalidAuthenticityToken (ActionController::InvalidAuthenticityToken):
/usr/lib/ruby/1.8/mongrel/rails.rb:76:in `process'
/usr/lib/ruby/1.8/mongrel/rails.rb:74:in `synchronize'
/usr/lib/ruby/1.8/mongrel/rails.rb:74:in `process'
/usr/lib/ruby/1.8/mongrel.rb:159:in `process_client'
/usr/lib/ruby/1.8/mongrel.rb:158:in `each'
/usr/lib/ruby/1.8/mongrel.rb:158:in `process_client'
/usr/lib/ruby/1.8/mongrel.rb:285:in `run'
/usr/lib/ruby/1.8/mongrel.rb:285:in `initialize'
/usr/lib/ruby/1.8/mongrel.rb:285:in `new'
/usr/lib/ruby/1.8/mongrel.rb:285:in `run'
/usr/lib/ruby/1.8/mongrel.rb:268:in `initialize'
/usr/lib/ruby/1.8/mongrel.rb:268:in `new'
/usr/lib/ruby/1.8/mongrel.rb:268:in `run'
/usr/lib/ruby/1.8/mongrel/configurator.rb:282:in `run'
/usr/lib/ruby/1.8/mongrel/configurator.rb:281:in `each'
/usr/lib/ruby/1.8/mongrel/configurator.rb:281:in `run'
/usr/lib/ruby/1.8/mongrel/command.rb:212:in `run'

Rendering /vol/sites/site/public/422.html (422 Unprocessable Entity)


Some further investigation revealed that there was an issue with how rails 2.3.8 was handling cookies.  I stumbled upon the following fix at https://rails.lighthouseapp.com/projects/8994/tickets/4690.

I used Ken Collins’ fix of adding his mongrel.rb (http://gist.github.com/471663) to config/initializers and now all is well.

I have been moving applications from my Rackspace host over to Amazon’s EC2 service.  Moving simple LAMP apps has been fairly cut and dry, but while moving a RoR app and accessing it through apache w/ mod_proxy, the html is delivered to the browser as plaintext.  If I access the app directly on mongrel’s port, it renders correctly.   So it appears that something with either my mod_proxy configuration, or something about how my EC2 instance is set up is causing the problem.


> curl -I http://linux.local:3000
HTTP/1.1 200 OK
Connection: close
Date: Thu, 15 Jul 2010 14:27:27 GMT
Content-Length: 0

> curl -I http://linux.local
HTTP/1.1 200 OK
Date: Thu, 15 Jul 2010 14:27:35 GMT
Via: 1.1 linux.local
Content-Type: text/plain


As you can see, when accessing the page through Apache/mod_proxy, the Content-Type that is returned is explicitly set as text/plain.

Update
This is a bit shoe-horned, but it works for right now. In the apache2.conf file, I changed the DefaultType from text/plain to text/html. The reason this solution “concerns” me is that on my other servers, (where this problem is not occurring) the DefaultType is also set to text/plain, but the html is output correctly.

Note: I have ~/.gem set as my GEM_HOME and GEM_PATH to avoid needing to sudo


> gem install mysql
Building native extensions. This could take a while...
ERROR: Error installing mysql:
ERROR: Failed to build gem native extension.

/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/bin/ruby extconf.rb
checking for mysql_query() in -lmysqlclient... no
checking for main() in -lm... yes
checking for mysql_query() in -lmysqlclient... no
checking for main() in -lz... yes
checking for mysql_query() in -lmysqlclient... no
checking for main() in -lsocket... no
checking for mysql_query() in -lmysqlclient... no
checking for main() in -lnsl... no
checking for mysql_query() in -lmysqlclient... no
checking for main() in -lmygcc... no
checking for mysql_query() in -lmysqlclient... no
*** extconf.rb failed ***
Could not create Makefile due to some reason, probably lack of
necessary libraries and/or headers. Check the mkmf.log file for more
details. You may need configuration options.


The workaround was fairly simple. Since I had installed MySQL from source the situation was resolved with a few command line options:


> export ARCHFLAGS="-arch x86_64"
> gem install mysql -- \
--with-mysql-dir=/usr/local/mysql \
--with-mysql-lib=/usr/local/mysql/lib \
--with-mysql-include=/usr/local/mysql/include \
--with-mysql-config=/usr/local/mysql/bin/mysql_config

Building native extensions. This could take a while...
Successfully installed mysql-2.8.1
1 gem installed


Getting Ready
I always set up a src directory under my user account for these installations.

> cd ~
> mkdir src
> cd src


Download MySQL Community Edition
At the time of this writing, it could be found at http://mysql.mirrors.hoobly.com/Downloads/MySQL-5.1/mysql-5.1.45.tar.gz

Download this file into your src directory.

> curl -O http://mysql.mirrors.hoobly.com/Downloads/MySQL-5.1/mysql-5.1.45.tar.gz


Untar, configure, and install

> tar -xvzf mysql-5.1.45.tar.gz
> cd mysql-5.1.45
> ./configure --prefix=/usr/local/mysql --with-extra-charsets=complex \
--enable-thread-safe-client --enable-local-infile --enable-shared \
--with-plugins=innobase

> make
> sudo make install

> cd /usr/local/mysql
> sudo ./bin/mysql_install_db --user=mysql
> sudo chown -R mysql ./var
> cd ..


MAMP specific issue

Given that I had installed MAMP previously, I had modified my /etc/my.cnf to specify that the mysql.sock was located in /Applications/MAMP/tmp/mysql/mysql.sock. I modified this line to point to /usr/local/mysql/var/mysql.sock, and then set the sticky bit on the var directory.

> chmod 1777 /usr/local/mysql/var


There are other steps that can be added so that mysqld launches on startup, but I prefer to start it manually when it is needed.

The site that it referenced was a very rarely used site, that only housed a custom built app (more of a script than an actual app) so I was puzzled as to how this site could have become infected.

I logged into the host, and did a quick check of the directory structure, and immediately noticed some suspicious things.


-rw-r--r-- 1 rgardner pg1704804 18716 2010-04-17 13:07 rargq.php
drwxr-xr-x 13 rgardner pg1704804 4096 2010-04-17 13:07 .
drwxr-xr-x 2 rgardner pg1704804 36864 2010-04-17 23:30 .files


I had never placed these here, so it was obvious that these were the culprits.  I quickly chmod’d the file so I could investigate it further, and then looked at the contents of the .files diectory.  This directory contained 692 static html files, all with the same general format/content:


-rw-r--r-- 1 rgardner pg1704804 10874 2010-04-17 23:17 kelly pavlik boxrec.html
-rw-r--r-- 1 rgardner pg1704804 11016 2010-04-17 23:17 mayhem miller twitter.html
-rw-r--r-- 1 rgardner pg1704804 9987 2010-04-17 23:17 the good shepherd cast.html
-rw-r--r-- 1 rgardner pg1704804 10456 2010-04-17 23:18 edison miranda.html
-rw-r--r-- 1 rgardner pg1704804 11071 2010-04-17 23:18 mayhem miller vs gsp.html
-rw-r--r-- 1 rgardner pg1704804 11297 2010-04-17 23:19 the good shepherd quotes.html
-rw-r--r-- 1 rgardner pg1704804 11949 2010-04-17 23:19 mayhem miller next fight.html
-rw-r--r-- 1 rgardner pg1704804 10394 2010-04-17 23:19 the good shepherd trailer.html
-rw-r--r-- 1 rgardner pg1704804 11427 2010-04-17 23:21 mayhem miller ninja warrior.html
-rw-r--r-- 1 rgardner pg1704804 10769 2010-04-17 23:21 kelly pavlik record.html
-rw-r--r-- 1 rgardner pg1704804 6261 2010-04-17 23:30 mayhem miller wiki.html


To identify where these were coming from, i looked at the apache access log for the time right around 2010-04-17 13:07 (the rarqp.php timestamp.)

I found the following lines from logs/access.log to be of interest:


69.89.31.119 - - [17/Apr/2010:13:07:49 -0700] "POST /images/moselle_sorcha.php HTTP/1.1" 200 275 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.2; ru; rv:1.9.1.2) Gecko/20090729 Firefox/3.5.2 (.NET CLR 3.5.30729)"
69.89.31.119 - - [17/Apr/2010:13:07:49 -0700] "GET /rargq.php?m=test HTTP/1.1" 200 10259 "-" "Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)"


Somebody had managed to place a PHP script in my images directory, and then posted to that script, which then kicked off thousands upon thousands of requests like the following:


67.195.112.60 - - [17/Apr/2010:13:16:12 -0700] "GET /robots.txt HTTP/1.0" 200 305 "-" "Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)"
67.195.112.60 - - [17/Apr/2010:13:16:12 -0700] "GET /rargq.php?m=prom%20hairstyles%202010%20down%20dos HTTP/1.0" 200 5472 "-" "Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; http://help.yahoo.com/help/us/ysearch/slurp)"
67.195.112.60 - - [17/Apr/2010:13:17:02 -0700] "GET /rargq.php?m=norfolk%20island%20pine%20poisonous%20to%20cats HTTP/1.0" 200 3954 "-" "Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; http://help.yahoo.com/help/us/ysearch/slurp)"
67.195.112.60 - - [17/Apr/2010:13:17:08 -0700] "GET /rargq.php?m=lawrence%20wilson%20md HTTP/1.0" 200 4491 "-" "Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; http://help.yahoo.com/help/us/ysearch/slurp)"
67.195.112.60 - - [17/Apr/2010:13:17:17 -0700] "GET /rargq.php?m=salt%20lake%20city%20marathon%20reviews HTTP/1.0" 200 4661 "-" "Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; http://help.yahoo.com/help/us/ysearch/slurp)"
67.195.112.60 - - [17/Apr/2010:13:17:25 -0700] "GET /rargq.php?m=toyota.ca HTTP/1.0" 200 4470 "-" "Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; http://help.yahoo.com/help/us/ysearch/slurp)"
67.195.112.60 - - [17/Apr/2010:13:17:37 -0700] "GET /rargq.php?m=sweetwater%20420%20fest HTTP/1.0" 200 4029 "-" "Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; http://help.yahoo.com/help/us/ysearch/slurp)"
67.195.112.60 - - [17/Apr/2010:13:17:42 -0700] "GET /rargq.php?m=tx%20lottery%20live HTTP/1.0" 200 3809 "-" "Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; http://help.yahoo.com/help/us/ysearch/slurp)"


I found the file and saw that it had been placed on the server several months ago:

-rw-r--r-- 1 rgardner pg1704804 28116 2009-11-26 13:09 moselle_sorcha.php


Around this time, a client that I was hosting a blog for was notified that the blog software she was using was outdated and had a known vulnerability. Apparently, I was now a victim of her vulnerable software.

Interested to see if there were other scripts, I ran the following:

> find -type f -wholename "*/images/*.php" | wc -l
30


This meant that there were 30 of these files that had been placed on my server. I removed the files with another command:


> find -type f -wholename "*/images/*.php" -mtime -150 -exec rm {} \;

I added the mtime switch to exclude some known php scripts that I wanted to exclude
Morals of the story

1. People are jerks
2. Potentially vulnerable software should be quarantined to reduce the impact of situations like this
3. Access logs should be routinely analyzed to watch for things like this

I am experimenting with the django web-framework for a few projects at work.  I do the majority of my work in PHP or Java, but occasionally I like to play with Ruby on Rails, and decided that django deserved to have a look as well.

The first step in this process is to get django and the various components installed.

• Python
• EasyTools
• MySql
• MySqlDB
• Django

Python – I do my development from a relatively new Macbook Pro, and as such, Python 2.6.1 is already installed.  This is confirmed by typing ‘python’ from a terminal window.

> python

Python 2.6.1 (r261:67515, Jul  7 2009, 23:51:51)

[GCC 4.2.1 (Apple Inc. build 5646)] on darwin

Type “help”, “copyright”, “credits” or “license” for more information.

>>> ^D

EasyTools – Download the OS X of setup tools from http://pypi.python.org/pypi/setuptools#files, then run the .egg file as if it were an executable:

sh setuptools-0.6c9-py2.4.egg

MySql - I use MAMP (http://www.mamp.info) for my local PHP development (not sure how anybody could live without MAMP running locally.) MAMP provides a drop-in MySQL installation that I use for all of my local development purposes.  The one caveat to this is that it is not uncommon to run into scenarios where something is looking for a mysql_* executable that isn’t being found in the default location.

I previously was using MAMP for these installations, but found that at times, header files were needed for other installations, and MAMP isn’t really intended for this type of usage.  I still think MAMP is a great tool, but in this case, I needed to Install Mysql from source.

Next, add the path to the mysql_* executables to your PATH.

> export PATH=/usr/local/mysql/bin:$PATH

MySqlDB – In order for python/django to be able to talk with MySQL, the MySQL DB driver needs to be installed.  This is an area that caused me a little bit of delay, as the installation of MySQL-python 1.2.3c1 was complaining about my lack of gcc.  I encountered a few posts that described how to get around this limitation with earlier versions of MySQL-python, but given that I do a lot of development, it felt like the right thing to do would be to install the Mac OSX Developer Tools.

> curl -O http://pypi.python.org/packages/2.6/M/MySQL-python/MySQL_python-1.2.3c1-py2.6-linux-i686.egg
> sudo easy_install MySQL_python-1.2.3c1-py2.6-linux-i686.egg

If you receive errors about mysql, or gcc, then you need to re-visit your environment setup.

Django – Once these pieces were in place, installing the actual django source code was trivial.

> svn co http://code.djangoproject.com/svn/django/trunk/ django-trunk

Next, make sure that the Python interpreter can load Django’s code. There are various ways of accomplishing this. One of the most convenient, on Linux, Mac OSX or other Unix-like systems, is to use a symbolic link:

> ln -s `pwd`/django-trunk/django SITE-PACKAGES-DIR/django

Finding SITE-PACKAGES-DIR -
To find the location of the SITE-PACKAGES-DIR on your system, execute the following command from the command line:

> python -c “from distutils.sysconfig import get_python_lib; print get_python_lib()”

Add Django commands to PATH – I find it helpful to add the django bin directory to my PATH environment variable so that I have convenient access to the framework’s various commands:

> export PATH=SITE-PACKAGES-DIR/django/bin:$PATH

Earn $$$ – Now that these pieces are in place, you can create your first django project by issuing the command:

> django-admin.py startproject mysite

Overall the installation was a snap, on par with a Ruby on Rails installation.  Over the next few weeks, I will start developing some simple django projects and write a post describing the experience.

When I first started working at my consulting firm, they had an eye on continuity, and even had nightly backups to an external hard drive. This was reassuring, but still left something to be desired. First, the router, wireless router, and backup drive were all located precariously close to the coffee maker. It doesn’t take much imagination to envision an employee stumbling in for their morning brew inadvertently taking down the office connectivity, and our backup drive as well.

The solution for this problem was low tech and cost about $50.  A portable laptop desk allowed us to move the electronics elements to a safer temporary location (we’re moving to a new office in a month where we will be able to correctly house all electronics in a dedicated location.)

This also forced us to visit the question about what would happen if the backup drive were to be needed, but damaged.  An off-site solution was needed in order to reduce the risk of losing our data, or losing access to our data.  After some research, we opted for mozy as an offsite storage solution.    All local workstations now backup to the external drive nightly, and also push their changes to the mozy service.

Our online servers will require a slightly different solution.  While Mozy does offer remote backup for servers, this is only offered for Windows and Mac OS X servers, whereas our servers are both running Ubuntu Linux.  The solution for this was to buy a cheap desktop workstation (locked in a computer cabinet) and have it pick up nightly backups from the servers.  This backup workstation would then back itself up to Mozy on a nightly basis.  The backup server is attached to a UPS unit in case there are power issues.  Now, each machine is backed up to one external hard drive, and backed up to offsite storage as well.  In an upcoming post, I will post a script that I have used previously for backing up webservers.  This will need to be modified a touch to support our alfresco/liferay/cas installations, but the basics will still be the same.

Good luck in your backup efforts.  Don’t wait to get this set up.