Upgrading to Confluence 3.1 was rougher than I expected

I went in early to work today to upgrade our Confluence server. I had been running the latest version on a test server with great success, and we had a lot of really exciting changes to roll out. Ever since I first set up our wiki around 2.5+ years ago, no one from our West Plains campus could log in (their accounts are in a separate Active Directory), and only a select number of AD groups could be used for permissions sets. What was worse, the operating system it was running on (the first iteration of Ubuntu JeOS) was pretty much broken and couldn’t be updated.

So we were moving to Windows Server 2008 (at my boss’s request–migrating from Ubuntu about broke my heart), and we also transitioned from a locally hosted PostgreSQL database to a remotely hosted Microsoft SQL Server. Lots and lots of changes, all to the good.

Instead of the hour I had an anticipated (moving virtual machines on our VM server, swapping MAC addresses, renaming the machines, voila), it ended up taking eight hours and forty-five minutes. Random issue after random issue got in my way, bugs that could neither be reproduced nor explained cropped up, and I strained against my torment without a break and with great intensity. First it was database errors, so I had the web DB guy drop all the tables and started anew. Then it was a Java error, then an unlabeled error, then backup restoration errors, with each error taking 30-60 minutes each to overcome. Once everything was OK, we discovered that all the attachments were broken, and after a frantic hour or so including some frenzied communications with Atlassian, it dawned on me to just copy the attachments over from the old server. If you knew how complicated attachments are in Confluence, you’d understand why I didn’t do that to begin with.

About fifteen minutes before I was going to leave, I started getting emails that people couldn’t see the content they needed. Permissions were broken, and that took another hour to figure out. A helpful person who runs our Active Directory was on the phone with me the whole time, and between us we were able to narrow down the issue: From within Confluence, I could see the permissions groups, but none of the members of those groups. AD groups can be created as Local Domain or as Universal, and Confluence 3.1 requires the latter. The former is the default, though, and most every group created before he started (about 2 years ago) was set to the default. Unfortunately, this included all of the permissions groups we needed. A quick fix on his end, but it took us a while to figure out what was going on.

I got no confused, frustrated, or angry emails on the way home, and five hours later I’ve had nothing but a few Thank You notes for the upgrade, so I’m feeling pretty good about it. Unfortunately, I was also exhausted and ended up having to take a 3-4 hour nap, and my throat’s sore, and my sinuses are all stuffed up. Hopefully sleep will cure what ails me.

2 thoughts on “Upgrading to Confluence 3.1 was rougher than I expected

  1. Hey Matthew,
    Sorry to hear about the tedious process, sounds like you were going through a whole lot of changes at once. Glad to hear it finally went okey.

    If there is any specific feedback you have of the upgrade process, or suggestions of how the upgrade process could have been easier – please do let us know. I’m always keen to hear any suggestions you may have.

    Sherif Mansour
    Confluence Product Manager

    1. The problems I encountered were only partially due to Confluence. The jump from 3.0.0_01 to 3.1 was a lot larger than I had expected, so some things we had expected to work simply hadn’t. As for the continual botched restorations, the only thing I can think of is that the server just couldn’t handle it. I noticed afterwards that I hadn’t applied my usual Java tuning, so it was maxed at 512mb of RAM >.>

      The Active Directory glitch was rather unexpected, since Confluence had been working fine for 2.5 years with this setup. But since Atlassian support has told me they don’t support dynamic AD groups used for permissions/authentication to begin with, I don’t hold too much of a grudge over that change.

      Hopefully the next 2.5 years will be as uneventful as the last 2.5. Then again, we’re on Windows now, so it’s probably foolish of me to expect it to be as rock solid as my little Linux box was 😛

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s