I guess this is something of a cautionary tale.
The weekend before last we decided that it was time to update the WordPress installations on two of our five sites. Both sites had been using version 3.4.2 which was now a year old. Days earlier, WordPress had released 3.6.1, urging all users to update due to some serious security issues. Although it wasn’t clear that this affected the version we were using, we decided to go ahead and update. It was time.
That Friday night, in the wee hours of Saturday morning actually, I upgraded If This Be Treason, our less trafficked site. I began by checking all of the plugins to make sure they were good-to-go with WordPress’ latest and greatest and then updated all that had newer versions available. Except for one, all of the plugins used by that site indicated they worked with at least 3.6.0, which was good enough I figured, since 3.6.1 was only days old and was primarily a bugfix and security release, otherwise no different than the earlier point version.
The plugin that wasn’t vetted for the 3.6 series was Ad Injection, which isn’t officially cleared for any version past 3.4.2. The plugin hasn’t been updated since last summer and the developer seems to be no longer dealing with support issues on the WordPress forums. However, just because a plugin hasn’t been officially cleared for a new version of WordPress doesn’t mean it won’t work, it just means that it either hasn’t been tested or that no one’s willing to give it an official thumbs-up. Because of the nature of Ad Injection, I was betting it would perform just fine. However, I made a mental note that it appears as if this plugin has been abandoned, meaning I’d eventually need to find a new way to serve ads on our sites. Pity. Ad Injection has been a perfect fit for us.
After checking and upgrading the plugins I did a backup of the WordPress installation, downloading the functioning WordPress install via FTP and dumping the database using phpMyAdmin. At that point I turned off all of the plugins, crossed my fingers and installed the latest and greatest that WordPress has to offer. I installed from the dashboard, using the auto-install feature that’s been part of WordPress for quite a while now. As long as WordPress hasn’t been highly modded, which would necessitate the modification of files before updating, there’s no reason not to install this way. It’s easier and there’s probably less likelihood of something going wrong than with a manual upgrade. Just remember to have a good backup in place just in case.
WordPress installed in less than a few seconds and everything was fine. I reactivated all of the plugins and they all seemed to be performing normally, even Ad Injection. I breathed a sigh of relief and went about the business of updating our theme.
Since our theme stores all of its configuration options in the database, I was able to test it first to make sure the latest and greatest had no trouble with our server configuration, which has been a problem in the past. I downloaded the newest version of the theme, changed its name so the version we were using wouldn’t be overwritten, and uploaded it to the server. I then activated it. Our logo and favicon were missing but that was expected since they were inside a directory in the old installation. I then reactivated the old version of the theme, blew up the version I’d just installed and performed an upgrade from the dashboard.
Everything went fine, except the logo and favicon were still missing. I quickly found the problem to be an issue known to the theme’s developers. When I’d automatically updated from the dashboard, WordPress had overwritten the theme directory containing the logo and favicon images. Luckily, I had both safely resting on our main computer here at FOSS Force. I uploaded them to their proper places on the server and all was right with the world.
On the next night I upgraded FOSS Force. To upgrade WordPress, I went through the same steps I’d gone through the night before. Everything went according to plan. When I upgraded the theme, I moved the logo and favicon to a directory outside the theme itself and linked the theme to them after the upgrade process. It was easy. The site worked fine. I went to bed happy.
The next morning there were two emails from site visitors informing me that FOSS Force was serving blank pages, one even sending a screen shot that showed our logo up top with nothing but white space below. However, our site looked fine on the company desktop and on my laptop so I figured they’d been attempting to access the site while the upgrade was in process and paid them no mind. I also noted that when I checked our AdSense account we’d only earned a couple of cents so far for the day, a fraction of a percent of what I would expect for the amount of traffic we were having, but I chalked that up to the inconsistency of online advertising.
On Monday morning I was surprised to see that our traffic was way up, about four times what we normally get in an entire day. I went straight to work, writing and posting the article Oracle Losing Its MySQL Grip to MariaDB.
About three that afternoon, just as I was getting prepared to leave for my “day” job, I checked with AdSense again. Given the traffic we were having that day, I was expecting to see some decent earnings on the board; instead, Google was again showing earnings of a couple of cents. I changed the product view to display that day’s stats. Google was showing a total of 6 ad impressions for the day. The figure should have been in multiple thousands.
Google AdSense represents money to us. Not a lot, but more than enough to pay for our server with some left over to defray a lot of the costs involved with running our sites. Natch, I spent every spare minute at work that night trying to figure-out why Google wasn’t seeing the thousands of ads we were displaying.
I reasoned the problem had to be connected to the upgrade we’d performed two nights earlier, so I concentrated my investigation there. It would’ve been nice if I could’ve contacted someone with AdSense, but Google doesn’t like to be bothered with small fry client sites like FOSS Force, so that wasn’t an option. I posted a detailed query on the AdSense user support forums, but the only replies I received were along the lines of “have you included your site in your list of sites approved to display your ads?” This, despite the fact that I included in the forum post that we’d been using AdSense on our sites since May of 2004 and that everything had worked fine until two days previous.
Finally, at about 9 PM I found the problem. In the Ad Injection support forum on wordpress.org I found a long thread, unanswered by the developer, about the exact issue we were experiencing. It seems that a lot of users of Ad Injection found themselves serving-up blank pages after upgrading WP Super Cache, a plugin we use here at FOSS Force to lighten our server load. The problem was with Ad Injection, not WP Super Cache. Evidently Ad Injection inserts ads using obsolete “mfunc” code that’s no longer supported.
Suddenly it became clear to me what was going on. I was seeing perfect pages because as a “known” user, meaning I was logged-in as an admin, Super Cache was serving me uncached pages. Visitors to the site, however, were being served blank cached pages.
To check this theory, I went to the settings page for WP Super Cache and unticked the box “Don’t cache pages for known users” so I’d be served a cached page like everyone else. Sure enough, when I went to the site I was served a blank page. Then I went back to the dashboard and disabled Ad Injection; our site was back to normal. AdSense was showing no impressions for good reason–we weren’t serving any. And our super high traffic for the day was probably the result of people refreshing their pages, trying to get an article they wanted to read to come up. I scrambled and found another plug-in to serve our ads temporarily until I could find something better and we were back up and running. By the time I finished work and got back home, AdSense was showing several hundred impressions.
I’m not sure what the moral of this story is. Perhaps it’s merely a reminder that Murphy’s law is always at work–you know, if something can go wrong, it will. Or maybe it’s to instill in my own mind that the next time I upgrade, be sure to look at a cached page to make sure the public is seeing what I see. Then again, it could be an opportunity for you to learn from my mistakes.