Server Move - What's happening?

As you might have seen, I’m in the process of changing the server that weather-watch.com runs on. The current server (as of 09 Nov 2006) is in Reading, UK. The new servers (there will be two) are in Dallas, Texas. The new servers are more powerful, have more storage, more bandwidth quota available and they’re cheaper too :smiley: I’m going to try to partition the functions of all my domains across the two servers, with the ability for each server to act as a warm backup device for the other one. This should mean that if there’s ever a server failure it should be quicker to get my sites back up and running as quickly as possible.

I’ve been setting the servers up for about a week now and they’re reaching the point where I’ll be able to start transferring my domains (there’s more than weather-watch.com on the server) across to the new servers. I’m starting this thread to keep you up to date on what’s happening or what’s going to happen in the next 12-24 hours. There will be outages, but I’ll try to keep them as short as possible. There will probably be breakages, but I’ll try to test things as much as possible and fix any problems as quickly as possible, but please bear with me on this because some things might not be fixable very quickly.

I’ll try to post an update a couple of times per day until the migration is complete, so please keep on watching this thread for information. If you find you can’t access the server, please keep on trying…it will come back eventually!

09 Nov 2006 - 22:45 (UTC)

  • I’ve shortened the TTL and other parameters on the DNS records for all my domains. The current settings won’t expire from the various DNS caches around the world until Saturday evening. I have to do this to keep the server switching downtime to the minimum on the day I want to move. The downside of this is that you might find the server slower at times, partly because you will need to do more DNS lookups when accessing the site and partly because of a greater DNS load on the server. The short TTLs will stay in place until at least a couple of days after the server migration.

  • Given the DNS expiry timescales, the earliest that any domains will move across is Sunday (12 Nov 2006). I’ll be moving my test and less regularly used domains first, so unless things go really well on Sunday I suspect it will be early next week before the main weather-watch.com domain (including the forum, wiki, bug tracker, MML MC and MML data feed) moves.

10 Nov 2006 - 10:15 UTC

I’m on leave today so thought I’d be able to spend some time working on the server. Unfortunately it looks like I’m suffering from the purchase of my ISP (Telewest/Blueyonder) by NTL. They seem to have changed the inter-UK routings to use the NTL network and it seems to have problems. Perhaps I’m now learning why people call them NTHell. The route from here to Dallas is…

Blackpool->Manchester
Manchester->Timed out hops
Timed out hops->Paris
Paris->Amsterdam
Amsterdam->Virginia
Virginia->Dallas

That suggests the NTL have lost their UK->US peering link and have some packet loss on the links into Europe (probably due to them carrying all the US traffic.

This was a very rare occurrence with TW/BY but it seems to be the future with NTL. Grrrrrrrr :x

10 Nov 2006 - 21:00 UTC

My bad day got worse :frowning: The problems I was having earlier might not have been NTL routing problems. I reached the point where the management software I’m using (Webmin) started to behave very erratically. I ended up uninstalling it along with all the domain structures I’d already set up. I then had to re-install Webmin, re-configure it and then re-load all the domain info, reset the DNS records, etc. However, I’m now past the point where I was this morning (having taken 8 hours to get here) because I’ve just migrated my first domain across to the new server :slight_smile: It’s going to serve as my test domain to prove that everything is set up correctly.

Those of you who provide clientraw/METAR/VWS data for MML will get a few extra hits on your data in the next few days as I exercise the MML data collection routines. I won’t hit you too hard though.

11 Nov 2006 - 11:30 UTC

I’m making better progress now. My test domain transfers seem to be working well and I’m just running through the MML batch scripts to make sure they’re working OK. So far METARs, clientraw.txt and amateur METAR downloads are OK. The VWS reader seems to be broken so I’m just looking at that. I’ve got the databases set up (but I’ll need to do a final data copy when the migration takes place) and also email/spam scanning is running OK.

VWS data collection now works. It’s flushed out a bug which has been in the code for some time :oops:

11 Nov 2006 - 17:00 UTC

I’m making steady progress. I’ve exercised most of the MML batch scripts now and they’ve all worked fine so far. I’ve taken the opportunity to upgrade the code that generates the maps. It’s a lot quicker now, although I don’t know if that’s the bigger server or the new code or a combination of the two.

It’s possible that I might be ready to migrate tomorrow, although that will depend on how well things go this evening. I’ll post again later on and decide whether tomorrow might be Move-Day.

11 Nov 2006 - 23:20 UTC

I’ve hit a rock or two during the evening and it’s set me back a few hours. The good news is that the MML batch scripts all work now and it’s looking like the MML Management Centre scripts will also work OK. I can’t check all the MML MC scripts just yet because the subscription system has been playing up (it didn’t like me upgrading PHP and MySQL and Zend Optimizer and probably a dozen other things all at the same time!)

At the moment it’s looking less likely that there will be downtime tomorrow for the switch over, but I’ll let you know how things go.

Time for bed…I’ve spent too many hours today at this keyboard.

12 Nov 2006 - 23:15 UTC

I’ve not had as long to work on the migration today as in the last couple of days. However, I’ve made some good progress. I’ve got the email up and running on the server and have upgraded the membership software and checked that my test copy of the MML Management Centre works OK (both on the new server and with the upgraded membership software.

I’ve got a few more things to work on tomorrow, but the list of completed tasks is getting longer and the list of ToDo’s shorter so I’m getting there. I suspect that I’m going to need 2-3 hours of downtime to complete the migration, so that means it’s probably going to be at least Thursday evening before that’s scheduled in and I might decide to leave it until next weekend.

13 Nov 2006 - 22:40 UTC

Nailed a few more problems this evening. They were little niggles that needed fixing before going live, so it’s good that they’re out of the way. I’ve also run up a copy of Mantis/Bug Tracker on my test domain and it seems to be working OK. I’ve just got to test out the FTP functions and if they’re working then I’ll start migrating my lesser used domains across tomorrow and Wednesday.

14 Nov 2006 - 23:00 UTC

The move has started, but you won’t notice it yet. I’ve now move four of my domains over with two more just pending their DNS TTLs timing out before they switch over as well. I’ve got one more small sub-domain to shift before I hit the weather-watch.com domain. There is a possibility that I will do the weather-watch.com move tomorrow evening because the usual ‘kids taxi service’ has had some changes to the itinerary! The final decision on the move will depend on how tired I feel when I get home tomorrow. If you find that you can’t access the forum/MML/wiki/Mantis/etc tomorrow evening (UK time) then the move has started. The new server is performing well, so I’ve no concerns about reliability or speed of access. I just hope that I’ve not missed anything significant in my migration method.

15 Nov 2006 - 22:00 UTC

Weather-Watch.com is now on the new server. The forum is the first thing back up to allow me to let you know what’s going on. Things seem to have gone pretty well so far, but there’s still more work to do to bring all the services back up. The only thing that I know has gone wrong so far is that I made a mistake with a DNS entry which affects the images used in FAQs, so you might not see the images until the correct DNS entry has propagated round the various servers. It’s on a short TTL so it shouldn’t take long.

The MML data collection batch jobs are just being brought back up one at a time to check they’re still working and the next priority is getting MML completely back up.

15 Nov 2006 - 23:00

MML should be back up and running now. I’ve tested the Management Centre, back-end batch jobs and the MML Flash client and they’re all working for me. Please let me know if they’re not working for you. Post a message in the forum for now (there’s a server thread in Chatter) because the bug-tracker isn’t back up and running yet.

15 Nov 2006 - 23:10

Oops…MML Management Centre really will be back up and running now :oops: Mantis Bug Tracker is also back.

15 Nov 2006 - 23:15

I think we’re nearly there now. Wiki, Gallery and the main www.weather-watch.com site are all back now.

It’s been a long evening and I’m pretty tired now, so that’s all that I’m doing for now. I’m sure there’s a lot of tidying up to do and probably tweaks to improve perfomance, but that can wait for now. There seems to be a problem with FTP, so my own weather site is off air, but that can wait until tomorrow. Please let me know if you find anything behaving oddly and I’ll take a look tomorrow.

thanks for all your Hard work on this Chris!!! :slight_smile: :slight_smile:

17 Nov 2006

If no-one finds any more problems than I think we’ve finished moving into our new home :smiley:

I still have to do some background tidying up, but that’s something I can sort out over the next few weeks.