Adobe CQ Tar PM data file rotation bug

At $WORK this week we encountered an extremely strange and annoying set of bugs in older versions of the Adobe CQ CMS system – specifically, in all CQ systems using CRX versions before 2.2. We’re still using CQ4 for some sites, which has CRX 1.4.2, and is affected by this.

The bugs are documented in the Adobe CQ Help knowledge base, but let me explain it a bit more.

The CRX repository system by default uses a persistence mechanism based on *NIX tar packages. Every time a change is written to the repository, that change (I’m not sure what exact details) is appended to a numbered tar file. At a defined size, the tar files are rolled over. Also, at some point older tar files are removed.

The bugs are as follows…

• If the number in the filename of the data file exceeds 32768, the CRX will fail to start, or will start in a broken state
• There is no checking in the tar file rolling system for data file numbering in excess of 32768
• The default file size limit for the tar file rolling is quite low (though higher for CRX 2.x than 1.4)

The result of these bugs is that changes will be continually appended to files that have invalid numbering, and the next time the CQ system is restarted it will either fail to start entirely or, more confusingly, will start with incomplete or broken repository contents.

The more changes you have in your system, the quicker you will hit this limit, especially if you’re using CQ4 which has a 64Mb tar file limit. Older CQ5 versions with CRX 2.x have a 256Mb limit. Luckily we hit the limit on our staging environment first, as it had some automated content activations that generate many changes each day, and not on production.

This issue is fixed in CRX 2.2 and above, so doesn’t affect recent or current CQ5 versions, but I’m sure there are still many companies using older versions.

The workaround described in the documentation is to stop your CQ system, rename all the tar files (but preserve the order), remove the tar index files, and increase the configuration of the tar persistence management system to use larger files. Note that this doesn’t actually fix the problem, it just means it will take longer to reach the limit. The only real fix is to upgrade to a newer version of CQ5.

Here’s a simple hacked-together bash function I wrote to rename the data files and remove the index files.

do_fix_repo () {
COUNT=0
if [-e $1]; then
        echo "Fixing CRX in $1"
        cd $1
        DATA=$(ls data*tar)
        for i in $DATA; do
                NEWDATA=$(echo $COUNT | awk '{printf("data_%05s.tar", $1)}')
                mv $i $NEWDATA
                COUNT=$((COUNT+1))
        done
        rm index*.tar
else
        echo "CRX $1 not found"
fi
}

You just supply it with the path to the data files, for example for a CQ4 author repo:

do-fix-repo $CQDIR/crx-1.4.2/crx-quickstart/repository/crxauthor/shared/workspaces/live_author
do-fix-repo $CQDIR/crx-1.4.2/crx-quickstart/repository/crxauthor/workspaces/live_author/copy

Also, since the documentation doesn’t include them, here are some example CQ4 author paths to the repository.xml and workspace.xml configuration files, where you need to increase the tar file rolling size parameter.

$CQDIR/crx-1.4.2/crx-quickstart/server/runtime/0/_crxauthor/WEB-INF/repository.xml
$CQDIR/crx-1.4.2/crx-quickstart/repository/crxauthor/workspaces/live_author/workspace.xml