err: Could not request certificate: Retrieved certificate does not match private key; please remove certificate from server and regenerate it with the current key

Apparently the root cause of this error is that the client’s ssl certificates have been messed up.

To fix it you have to remove all of the client’s ssl stuff – cd into the directory containing all the ssl info – /etc/puppet/ssl for me running a manual install of puppet 2.6 – and remove all files, in all sub-directories, apart from ‘ca/serial’, which should contain 0000.

Then on the server revoke the client’s ssl certificate using:

sudo pupetca --clean {client hostname}

Then restart the client, resign it on the server and you’re good to go!

This is a fairly simple process and there’s lots of information on how to do it.  The key steps in the process are emptying magento’s cache directory and updating the base urls in the core_config_data table so that magento will generate correct urls.

When I setup my copy of the site locally I changed local.xml & deleted the cache before modifying the db, just to make sure that magento was connecting ok.  Once that was done I went off to delete the new cache files from siteroot/var/cache.  Only, there weren’t any cache files.  I took no note of it figuring magento had just decided not to cache the files and so proceeded to load the site, only for it to redirect me to the live site.

I spent the next few hours trying in vain to work out where magento was getting the idea that it should redirect to the main site.  After wiping the db / files and re-extracting them several times I noticed that it was still redirecting even if the database was empty and the cache dir (var/cache) didn’t exist.  Obviously this is one helluva wtf, especially as there weren’t any references to the live site’s address in the files.  The opinion on IRC was that somehow my local site was talking to the production site’s db and was using it’s database instead of my local one, however this couldn’t be the case as magento started complaining if I dropped the local database entirely.

In a last ditch attempt I tried grepping for the site’s address across the entire filesystem (using grep -R “sitedomain.com” / ) and came across a load of files in /tmp/magento.  After deleting the directory magento started working perfectly again, though it continued to store cache files in /tmp/magento rather than var/cache/ for some reason.

One problem we ran into whilst setting this up was that php scripts running through apache were having trouble connecting to the mysql server.

What was even stranger though, was the fact that this problem only appeared when the php script(s) were run through apache – running them through the command line / shell worked absolutely fine.

After a couple of hours of debugging, head bashing and confusion we found the solution at the bottom of one of those very very long experts exchange threads.

It turns out that some linux distros has a neat little access control system called SELinux which was blocking communication by apache to remote database servers.

The aforementioned exchange thread suggests disabling SELinux entirely by executing

sudo setenforce 0

but this isn’t a permanent solution and won’t persist through a reboot without changing a config file.

This can be accomplished by changing a line in /etc/selinux/config. Change the line that says:
SELINUX=enforcing
to
SELINUX=disabled

However, if you’re willing to do a bit of digging there are SEL options you can change to grant apache access to remote database servers, and if you have a few hours to kill, there’s also the fedora documentation.

Hopefully this’ll save someone else the headache we had!