Dobrica Pavlinušić's random unstructured stuff
WebGUI upgrade: Revision 48
Upgrade from WebGUI 6.2.11 to 7.3.33
{toc: } ^ Install dependencies .pre apt-get install apache2-mpm-prefork apt-get install libapache2-mod-perl2 apt-get install perlmagick imagemagick .pre Don't bother with installation of perl modules from packages. Debian has modules which are sometimes a bit too old for cutting-edge WebGUI, so just let CPAN install it for you. ^ Basic configuration All operations on both hosts (mjesec, zemlja) are done as `root` user. This is not strictly necessery (especially for WebGUI tasks), but it's easier in combination with MySQL setup. We want to get running quickly, YMMV. ^^ Mysql binary log Comment out following in `/etc/mysql/my.cnf` .pre log_bin expire_logs_days max_binlog_size .pre To preserve disk space if you are not using replication! And restart server: .pre /etc/init.d/mysql restart rm /var/log/mysql/mysql-bin.* .pre ^ Cleanup target machine If this is not first installation attempt, you might want to cleanup target machine first. .pre # root@mjesec cd /data/WebGUI grep dsn etc/*.conf | cut -d: -f4- \ | xargs -i echo 'drop database {};' \ | ssh zemlja 'cat > /data/drop_all_dbs.sql' .pre Now switch to target machine and finish cleanup: .pre # root@zemlja cd /data xargs -i mysql -e '{}' < /data/drop_all_dbs.sql rm -Rf WebGUI .pre ^ Copy existing installation In this step, we will copy existing WebGUI 6.2.11 installation to new (virtual) host (zemlja) to test upgrade. ^^ Copy MySQL databases .pre # root@mjesec cd /data/WebGUI grep dsn etc/*.conf | cut -d: -f4- \ | xargs mysqldump --databases \ | ssh zemlja mysql mysql .pre ^^ Copy existing WebGUI installation .pre # root@mjesec cd /data tar cfp - WebGUI/ | ssh zemlja 'cd /data && tar xvfp -' .pre This is much faster than scp in my expirience... ^^ Finish database setup Now, on zemlja, let's grant webgui user access to those databases: .pre # root@zemlja cd /data/WebGUI mysql -e 'create user webgui' grep dsn etc/*.conf | cut -d: -f4- \ | xargs -i mysql -e "grant all privileges on {}.* to webgui@localhost identified by 'password';" mysql -e "flush privileges" .pre ^ Upgrade steps First, test current enviroment for any mis-configuration: .pre # root@zemlja cd /data/WebGUI/sbin perl testEnvironment.pl .pre If there are missing perl libraries, install them. Some will have to be forced, but all-in-all it goes well. And run upgrade... .pre # root@zemlja cd /data/WebGUI/sbin perl upgrade.pl --doit .pre Now, extract new version of webgui on which you want to upgrade (I'm assume that you downloaded required versions in /data): .pre cd /data/ ls tar xvfz webgui-6.6.4-beta.tar.gz cd /data/WebGUI/sbin/ perl testEnvironment.pl perl upgrade.pl --doit .pre This will take a while. After that you might want to remove backup of databases: .pre rm /tmp/backups/* .pre ^^ Upgrade scripts Here are list of upgrade scripts: * {file: z-1-backpan.sh} * {file: z-3-mysql-perms.sh} * {file: z-4-webgui-6.6.5.sh} * {file: z-5-webgui-6.8.10.sh} * {file: z-6-webgui-7.0.9.sh} ** {file: upgrade_6.8.9-6.8.10.pl} ** {file: webgui-7.0.diff} * {file: z-7-webgui-7.2.3.sh} * {file: z-8-webgui-7.3.22.sh} * {file: z-9-webgui-7.4.38.sh} ^^ Helper scripts Backup/restore whole WebGUI instance (data+database) so that you can re-try upgrade > This was probably one of most useful ideas enabling me to re-try upgrade until it finish without errors! * {file: webgui-snap.sh} * {file: webgui-restore.sh} * {file: webgui-recover-db.sh} ^ Tweaks Here are some commands to help you with various changes which break WebGUI: .pre # root@zemlja cd /data/WebGUI rm lib/HTML/Template.pm cd /data/WebGUI/etc/ mv log.conf.original log.conf mv spectre.conf.original spectre.conf .pre ^^ MySQL encoding My data has `iso-8859-2` while MySQL thinks it's in `latin-1`. .pre mysqldump --all-databases --add-drop-database --default-character-set=latin1 > all.sql cat all.sql | sed \ -e 's/DEFAULT CHARSET=latin1/DEFAULT CHARSET=utf8/g' \ -e 's/character set latin1 collate latin1_bin/character set utf8 collate utf8_bin/' \ -e 's/DEFAULT CHARACTER SET latin1/DEFAULT CHARACTER SET utf8/' \ -e 's/SET NAMES latin1/SET NAMES utf8/' \ | iconv -f iso-8859-2 -t utf-8 \ | mysql --default-character-set=utf8 mysql .pre > *This should work, but it doesn't* So, we are using much simplier solution which keep encoding in database in wrong latin1 encoding, but it displays correctly in WebGUI: .pre cat all.sql | iconv -f iso-8859-2 -t utf-8 | mysql mysql .pre ^^ Fix configuration Upgrade procedure isn't still perfect. Here is a list of tweaks to fix configuraion: .pre cd /data/WebGUI/etc/ perl -p -i -n -e 's/WebGUI::Asset::Wobject::WebGUI::Asset::Wobject::Survey/WebGUI::Asset::Wobject::Survey/g' *.conf .pre ^^ Code cleanup You might want to replace `WebGUI/lib/WebGUI` which is result of upgrade with clean version from distribution. I have accumulated some cruft there, and it was spitting errors in `/var/log/apache2/error.log` and `/var/log/webgui.log` ^^ Fix permissions .pre cd /data/WebGUI sudo chown -R www-data */uploads .pre ^^ Spectre I didn't have `spectre.conf` after upgrade. Test connectivity with: .pre cd /data/WebGUI/sbin/ perl spectre.pl --test .pre Add following to `/etc/rc.local` .pre cd /data/WebGUI/sbin/ perl spectre.pl --daemon .pre ^^^ Cleanup versions Since I created few workflows which always returned commerr (because of mis-configuration) I needed to find way to manually erase them. .pre select tagId from assetVersionTag join WorkflowInstance on WorkflowInstance.instanceId = assetVersionTag.workflowInstanceId ; .pre ^^^ Delete workflow instances .pre delete from WorkflowInstance ; .pre This will erase *all* pending workflow instances. So, be sure to check if they are all stuck before running this command! ^^^ /etc/resolv.conf Make sure your `nameserver` entries point to existing servers. If you have one of DNS servers which doesn't respord (I had 127.0.0.1 for DNS and there wasn't any DNS server on machine running WebGUI) you will get Errors 500 from spectre without any indication where the problem is. ^ Critical verision you have to upgrade to *This part is obsoleted by scripts above which provide correct and workable upgrade path* ^^ 6.6.4 last 6.6.x version required for upgrade to 6.7.0 ^^ 6.8.10 Upgrade to this version will die at one point with: .pre Cannot open config file '../../etc/log.conf' at /usr/share/perl5/Log/Log4perl/Config.pm line 612. .pre .pre # root@zemlja cd /data/WebGUI/etc/ cp log.conf.original log.conf cp spectre.conf.original spectre.conf .pre And then, re-run upgrade procedure: .pre cd /data/WebGUI/sbin/ perl upgrade --doit .pre Unit it succeeds. ^^ 6.99.5 this verison will convert config files to JSON format First, install *old* JSON or you will be in pain: http://backpan.perl.org/authors/id/M/MA/MAKAMAKA/JSON-0.991.tar.gz You might have problems upgrading to version 6.99.0 or later because scripts expect configuration in JSON format, and upgrade won't convert files for you (because it created config files from tar). So, change your `upgrade.pl` beginning to something like this: .pre ## Find site configs. print "\nTesting site config versions...\n" unless ($quiet); opendir(DIR,"../etc"); my @files = readdir(DIR); closedir(DIR); foreach my $file (@files) { next unless ($file =~ m/\.conf$/); # next if ($file eq "spectre.conf" || $file eq "log.conf"); my $configFile = "../etc/".$file; open(FILE,"<".$configFile); my $line = <FILE>; close(FILE); unless ($line =~ m/JSON 1/) { print "\tConverting ".$file." from PlainConfig to JSON\n" unless ($quiet); convertPlainconfigToJson($configFile); } } exit; .pre Note commented out line and added exit. Now you can run `upgrade.pl` once to convert configuration files to JSON, then run `testEnvironment.pl` and finally run `upgrade.pl` once more to do actual update. ^^ 7.3.22 requirement for upgrade to 7.4.0 To prevent `perl upgrade.pl --doit` die with: .pre Cannot open config file '../etc/log.conf' at /usr/share/perl5/Log/Log4perl/Config.pm line 612. .pre fix it: .pre cd /data/WebGUI/etc/ mv log.conf.original log.conf .pre and *then* run upgrade! ^ Version related notes This are just notes. You can ignore them for normal upgrade described above. ^^ 7.3.4 Re-run upgrade procedure if it breaks ^^ 7.4.17 Attack of JSON again: .pre Can't locate object method "relaxed" via package "JSON" .pre Install http://backpan.perl.org/authors/id/R/RI/RIZEN/Config-JSON-1.1.0.tar.gz ^^ 7.4.33 It will install newer JSON and Config::JSON packages from CPAN ^ Hint For testing, I created `*.zemlja.example.com` DNS entry, so that all sites become `old.site.name.zemlja.example.com`. |