Site Downtime (ongoing)

[CaleyD]

We experienced some downtime today due to our host moving our servers without giving us any advance notice so as we could prepare for it. This resulted in some parts of the site going on the blink.

I think I've managed to kick everything back in to life, but if you happen upon any part of the site that is not working, please flag it.

[tm4tj]

Any issues with the site the last couple of days Don/Scotty

The reason I ask is that since the changes were made after this issue, three or four times the site was unavaiable. Today at approx 11:12 I was knocked off for a few minutes.

[CaleyD]

We do have some ongoing issues and are battling with the provider to try and get to the bottom of them.

They've generally been pretty good and efficient when dealing with the (very) few problems we've had in the past, but they seem to have dropped the ball on this one and are trying our patience.

[old caley girl]

Don just to report was bumped off last evening and couldnt get back on kept saying Google link broken.ok today though.

[Scotty]

Yup I had the same problem myself - seems to have been a DNS error last night.

We submitted several tickets to our supplier last week and based on the latest response from them yesterday I am going to be tearing someone a new one today !!!

We apologise for these errors but unfortunately they are well and truly beyond our control

[Scotty]

Just to keep everyone up to speed - the response from our website hosting provider today was as follows:

I apologize for the huge inconvenience this poorly executed move has caused you.

I've been in touch with <name removed> and he's merged all of your databases over to the correct MySQL server, so there should no longer be any inconsistencies with your databases.

As far as the webserver move, the server has been surprisingly unstable as the moves were being performed, however it does now seems to have stabilized, it's been looking good for about the last 21 hours straight.

These moves are being run in order to decomission old, unstable hardware, and move our customers to our new machines which not only run more up-to-date packages, but also have much better hardware, including gobs of memory, quad 64bit processors and locally mounted drives which don't

suffer from networking issues like the old NFS drives did.

While <your server> does look stable now, I can't promise that things won't be a bit shaky over the next few days as the moves complete, but when they're finished you should have increased stability and performance, though I know it seems reduced if anything at this point.

So now you guys know as much as we do. We do have a monitoring service in place that checks if the web address works and sends us email when the site cannot be reached but if anyone experiences any sustained problems over the next few days (or at anytime really) we would like to know about it.

[Jay_7]

Just to keep everyone up to speed - the response from our website hosting provider today was as follows:

I apologize for the huge inconvenience this poorly executed move has caused you.

I've been in touch with <name removed> and he's merged all of your databases over to the correct MySQL server, so there should no longer be any inconsistencies with your databases.

As far as the webserver move, the server has been surprisingly unstable as the moves were being performed, however it does now seems to have stabilized, it's been looking good for about the last 21 hours straight.

These moves are being run in order to decomission old, unstable hardware, and move our customers to our new machines which not only run more up-to-date packages, but also have much better hardware, including gobs of memory, quad 64bit processors and locally mounted drives which don't

suffer from networking issues like the old NFS drives did.

While <your server> does look stable now, I can't promise that things won't be a bit shaky over the next few days as the moves complete, but when they're finished you should have increased stability and performance, though I know it seems reduced if anything at this point.

So now you guys know as much as we do. We do have a monitoring service in place that checks if the web address works and sends us email when the site cannot be reached but if anyone experiences any sustained problems over the next few days (or at anytime really) we would like to know about it.

There was the dreaded IPS driver error about an hour ago. Not noticed much since then.

[CaleyD]

That could have been them completing the manual shift of our databases because when I left to go out earlier we were still on the old hardware.

Our provider has generally been pretty good, but trying to get an answer from them this time round was like trying to get blood from a stone. If only they had explained things up front then we would have been prepared and wouldn't have wasted hours trying to diagnose a known problem.

Hoping this is the start of a less stressful week for me B)

[Guest Mahonio]

We experienced some downtime today due to our host moving our servers without giving us any advance notice so as we could prepare for it. This resulted in some parts of the site going on the blink.

I think I've managed to kick everything back in to life, but if you happen upon any part of the site that is not working, please flag it.

I don't know if this is the same as your saying here but since Saturday i have struggled to load this page especially today when it took me over an hour to load the page. A message came up saying that the Site was still here but the server wouldn't let it load up.

[Scotty]

yes it is an extension of the same problem .....

in layman's terms, we are about a ba' hair away from telling our current provider to go **** themselves and move the site over to a new host (we already have one available to us). I have yet another support ticket in right now with our current hosts and if they do not have it resolved in a decent amount of time, then we are offski .....

Obviously, something like that would be a last resort and would also be a major major move so we wont do it lightly but we cant allow the service to be as degraded as it is right now.

If we do move then in the short term it will mean more disruption (which is why we would like our current guys to sort things) as not only would we need to move every single file across to the new hosts (no small task) but we would also have to deal with several days of disruption as we configure, format and point the site to a new host and then let the new (host) address be updated in the internet DNS system .........

[tm4tj]

Unable to get onto site tonight from 18:35 to 19:05 approx.

It is back on now................obviously as I have just posted this message.

[Scotty]

yup - I noticed

I have my own kit now .....

The bottom line is that we have been moved to a new server and the company is in the process of moving many other sites to the same server too ..... as they work on it, the "server load" goes through the roof and everything slows down or stops ..... an acceptable server load in this environment is in he 0-10 range, it starts getting dodgy at 30+ and over the last 24 hours we have seen it go as high as 468 550 whilst using all 16Gb of RAM on the server!!!

from our monitoring software, the load in the time you mention appears to have been 0 and is now about 15 ..... there is a gap in the load which suggests the server crashed or was rebooted.

Having spoken to CaleyD we are going to give our providers until the weekend to resolve all our issues or we are going to move all of our websites to another provider. That would be painful, but we will have no option, we cannot accept the current level of (dis)service.

[Guest Mahonio]

This ******* problem is getting out of hand and to be honest i would consider really hard about moving because i have problems EVERYDAY getting in!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

By the way this is not aimed at this website it is the people who caused this problem because it is doing my head waiting 20-30mins+ to get in :( :( :( :( :( :(

[Scotty]

oh we are considering it, but to give you an idea of whats involved we would have to transfer around 50Gb of files from one host to another (download then re-upload elsewhere), initiate transfers of our domain names (which can take a few days) and do all sorts of other stuff that would mean myself and Don would have literally hundreds of hours of work to do just to be back where we started .....

seems the box they moved us to finally died today so now they are using new hardware and hopefully this will mean some more stability.

[Yngwie]

Probably worth persevering with the current guys, given that prior to these server move glitches it seems to have been a very stable service for years.

[Heilandee]

oh we are considering it, but to give you an idea of whats involved we would have to transfer around 50Gb of files from one host to another (download then re-upload elsewhere), initiate transfers of our domain names (which can take a few days) and do all sorts of other stuff that would mean myself and Don would have literally hundreds of hours of work to do just to be back where we started .....

seems the box they moved us to finally died today so now they are using new hardware and hopefully this will mean some more stability.

Don't envy yer task lads,i got an e-mail on Sunday from the folks that host my wee website saying the raids on the server my sites on had failed and to just upload the site again! "sorry for any inconveniance caused" :( took me several hours with my limited knowledge,still some pics and things missing that i will need to re-do so i can appreciate the magnitude of the task on a site this size to new servers :(

[Scotty]

October 8th - 3:30PM Inverness time: We know its down again .... support ticket submitted yet again.

[+Gringo]

Coventry Nil time 4.52 p.m. Slow as fork to load pages... :(

[Johnboy]

Not much sign of any improvement....

If anything it's getting worse.

If you do have to transfer to another host is there anything any of the site users could do to help?

[Scotty]

appreciate the offer JB, but honestly, there is not really much anyone can do to help unfortunately ... not much we can do either other than move files about and change configs ... the real killer is the time it takes to do that while you are sitting twiddling your thuimbs ......

I do have an idea which I am going to discuss with Don in a short time and hopefully it may get us up and running elsewhere until this blows over ... might take a few hours though

[Oz647]

Well all I can say is I take my hat off to you and Don having the patience to sort this. It's frustrating enough for us mere mortals trying to log into the site. God knows I would have lost it big style if I had to sort out the host problems as well. :(

PS: This is only valid if you sort it. :mellow:

[Scotty]

having spoken to Don, here is what we have decided.

We are going to create a copy of this site (the forum part at least) and put it on our other hosting service. Once we have it setup, anytime we have an issue like this, we will give you an alternative web address you can go to to get your forum fix. As both should be able to use the same database, all your posts whould be fine.

It may take a while to setup as we have to download files from this host and upload them to the new one, but we should be up and running with a plan 'B' later today or tomorrow. until then, all we can ask is for your continued patience.

[Scotty]

IMPORTANT NOTICE:

As many of you are aware, we have been experiencing issues with our current web hosting provider. The level of service we are receiving is simply not acceptable and the site has been going down several times a day as a result.

As an emergency measure - while we explore our options - we have made a full copy of the forum on another host and you should be able to utilise this if/when the current forum experiences downtime issues. The URL for the "emergency" site is

http://www.caleythistleoffline.com

Please note that the "emergency" site should only be used when this one is down

We will disable it when this one is working normally.

A few points to note:

• Note 1: Your account (ie. login and password) is the same on both sites
  

• Note 2: When you make a post on the "emergency" site it will automatically be added to both sites as they use the same database.
  

• Note 3: Most things should work normally but some areas of the emergency site that rely on internal addressing such as blogs/gallery and even adding attachments may throw up errors when used. please keep the use of images and such like to a minimum when the "emergency" site is operational.
  

• Note 4: We have encountered the odd "ips driver" error on the emergency site - its to do with our host not liking another host accessing their database. We believe we have it fixed, but if it re-occurs we will investigate further.
  

• Note 5: Because they are on two different domains (online.com and offline.com) you may experience issues with unread posts showing up incorrectly when you switch between sites. this is because of the fact that this is cookie based and you have different cookies for each site. its a quirk you will need to put up with for a little while. sorry.
  

[SMEE]

I have to say, i had always found the site to be very relaible...until the last week or so. Dont worry tho Scotty and Don....I "upgraded" my Acer to Vista, worst move of all time...so i know all about frustration! Your not alone. :thumb04:

(ps, after putting about 20 hrs of my life into the upgrade and all that goes with it, i called it quits and did a factory restore to XP)

[Scarlet Pimple]

Good luck boys. I had hoped that you would be able to hold on a bit longer before trying any alternative but you know best.

This evening at 3.20 a.m. on 9 October ,U. K time, I got in first time I tried. I was surprised and agreeably pleased.

Such a load of work for you! Many thanks for all your patience and work just to allow us to enjoy our daily fix. Great! .