Page 1 of 7 123 ... LastLast
Results 1 to 15 of 98

Thread: So... What the hell happened to the Forums?

  1. #1
    Toymods Club Member #194 Conversion King Lambolica's Avatar
    Join Date
    Jul 2005
    Location
    NSW
    Posts
    2,109

    Default So... What the hell happened to the Forums?

    Just to answer any questions that you may have in regards to the recent outage of the Toymods forums / Website(s)

    Mid-Last week we got an email notifying us that Mediatemple (our web host) were End of Life-ing our Server and Migrating us to their latest and greatest.

    Friday around Lunchtime, another email notifying that the migration was about to start, around 2pm the forums went down.

    Several hours later (4.5 hours) we were notified that the migration had been completed sucessfully. So I logged in.... error fest. A reply to the support Ticket was made indicating that all sites on the server were down, and for them to look into the issue.

    Saturday Lunch (~18 hours later), No response to the ticket, another reply with the same request made.

    About 14 Hours later, they replied with:

    "We have restored the files and databases for the three domains. The toymods.org.au/forums, however, is redirecting to "http://www.toymods.org.au/.php", which throws an error. The files are in place, and so is the database, so it is not an issue with the server configuration, but rather something in the coding or .htaccess file. At this point, I recommend having your developer check the configuration and make sure it's compatible with PHP 5.4."

    One would think that restoring files & databases was part of the migration process?!? Regardless, the Frontpage site came back, but the forums were still a no-show.

    Much of Monday was spent tinkering, looking for the obvious settings didn't resolve anything, then we realised there were random files missing, and the database was not exporting as expected. Monday night a reply to the ticket was sent requesting the old server be bought back online (to which the reply was No) and/or supply a backup of the data from the old server be supplied.

    About midnight after the board meeting a call was put out to Sam (which we really didn't want to do as he has saved our butts countless times) to have a look see at what the hell was happening. Sam went at it on Tuesday retriving the old files and basically got the forums up and running in a limp mode Tuesday evening. I was privy to the conversations on Skype during the process, and Sam was less than impressed with the state of the migration.

    As Server admin is well over my head, I'll post the ticket response that Sam provided, so you more technically minded can see things in Sam's words.

    "Dear Mediatemple,

    Calling this migration sloppy would be too kind - incompetent is the word that comes to mind. It was only after four days did we manage to get our applications up and running again, at a considerable cost to us in advertising revenue, administration, and development costs. Our application was migrated from apache to nginx, which it doesn't support, and our subdomains, web_user directories, and SSL certificate were not even migrated at all! Let alone the disregard for our scripts, cron jobs, as well as performance tuning which still needs to be done again. Just dumping all our old files in a directory and leaving the cost of cleaning up the mess that was left behind by your technicians is completely unacceptable.

    We would like to lodge a formal complaint on this matter, can you please pass on the contact details to do so. An apology is the very least that should be offered in this case, and without some sort of assurance that such an occurrence is not likely to happen once again then I can't see why we wouldn't take our business elsewhere. "

    That was 12 hours ago, and there hasn't been a response from MT. Personally I think Sam's reply is still too kind, I would have borrowed some colourful language from Old Man River and at least threw a few "Arsehats" in there.

    The Toymods Community owes a great deal of thanks to Sam's work.

    Where things stand at the moment. The forums are running, but seem to be building towards another fallover, or server reboot, usage stats are being collected to resolve the problems further.

    Coincidently, we were about to do a review / upgrade / optimisation of the forum software, and this is still planned in the very short term to better use the available resources. This will occur after Sam gives the OK on the Server stability. So we expect there to be more down time in the near future. We will however have more control of when this will happen and will provide warming to upgrade windows, and intermittant outages.

    Suffice to say, we are looking into other hosting options ATM.

    Cheers
    Simon
    Beige.... The new Black!!!

  2. #2
    Your mum is a Conversion King
    Join Date
    Mar 2007
    Location
    a
    Posts
    2,230

    Default Re: So... What the hell happened to the Forums?

    Thanks Sam!

  3. #3
    Toymods Net Nazi Too Much Toyota river's Avatar
    Join Date
    Sep 2005
    Location
    NSW
    Posts
    7,039

    Default Re: So... What the hell happened to the Forums?

    Hi,

    Yes, Sam's words were too kind.

    I had to do a feedback response today to our engineering division at work. I responded to their feedback form... "Waste of time. Waste of space. Waste of money. Waste of energy. Waste of oxygen. Unable and unwilling to understand the Financial Markets Division. Glaciers move more quickly. The best thing the Engineering Division can do is go back to Enterprise/Retail and let the Financial Markets get on with their own work, like they used to."

    But, back to Media Temple... they showed all the worse traits of a company that doesn't give a toss about their customers, and they have shown their technical expertise is, at best, minimal.

    I am looking at other Hosting providers, because I can't stomach having these forums under the control of those Media Temple retards.

    seeyuzz
    river
    The thinking man's clown and the drinking woman's sex symbol
    RA25GT - There is no substitute | 18R-G - Toyota's Dependable Masterpiece
    Toymods Car Club Treasurer, assistant Historic Plate Registrar & Forums Admin

  4. #4
    Junior Member Automotive Encyclopaedia
    Join Date
    Jan 2011
    Location
    NSW
    Posts
    960

    Default Re: So... What the hell happened to the Forums?

    Thanks Sam, thanks River and thanks to everyone else that had to pitch it to fix a shit fight of someone else's making.

    MT sound like a right pack of cockforeheads.

  5. #5
    Junior Member Conversion King timbosaurus's Avatar
    Join Date
    Feb 2006
    Location
    Melbourne
    Posts
    2,037

    Default Re: So... What the hell happened to the Forums?

    Well done Sam, and the others who were stewing over this since the Media Temple fuckup on Friday!
    Current rides...
    2) White RA25ST
    1) Red RA28LT (NOW WITH 1G )

  6. #6
    Forum Member 1st year Apprentice
    Join Date
    Oct 2012
    Location
    Vic
    Posts
    20

    Default Re: So... What the hell happened to the Forums?

    Having worked in the IT/IS area for most of the last 30 years I have yet to experience a major migration that went exactly as planned. We warned our users, well in advance, that an upgrade was about to take place and followed up with reminders before any changes were made.

    This approach allows users to plan their use of our systems, or rather prepare for the inevitable down time. It also meant that we were not flooded with calls asking what was wrong with the system. Keeping users "in the dark" is a disservice to them and will cause unnecessary confusion.

    MT plainly failed to plan the migration appropriately. Leaving the system to display an error message instead of redirecting users to a page that explained the problem is, frankly, unprofessional.

    I would advise moving to another hosting service and adopting a policy of advising Toymods users that maintenance is about to occur and some disruption may be experienced.

    Cheers.

  7. #7
    Junior Member Conversion King
    Join Date
    May 2010
    Location
    ACT
    Posts
    2,510

    Default Re: So... What the hell happened to the Forums?

    If the hardware was being EoL'ed, but no mention of software itself etc. Why the hell not just image your running server and chuck it on new hardware?

    Linux kernel images etc. are generic enough these days to run on new hardware no problems. Gone are the days of renaming IDE to SATA compliant device names in /etc/fstab etc.

    I agree with moving to another service. Also whilst I'm not a sysadmin like Sam is, I am very passionate about Linux (and BSD) and webservers as well as very security focused (Again BSD is usually better here), so I'd be glad to help out whenever possible. My expertise in this area far extends beyond my current job description.

    moving to nginx is never a simple process, so I agree with the arsehat remarks needing to be made!
    Also from my previous recommendations to Riv about looking into nginx, whilst Apache still has a place when using a mix of PHP, CGI and SSL, nginx can support some of these and also makes a very good frontend for speeding up Apache.

    Nginx listening on port 80/443, and caching requests going to Apache on an alternate, firewalled port, such as 3128 etc. Reverse proxy
    Autodub - 1987 AW11 G-Limited, Dark Blue Mica 4AGZE T-Top 4EAT

  8. #8
    Toymods Net Nazi Too Much Toyota river's Avatar
    Join Date
    Sep 2005
    Location
    NSW
    Posts
    7,039

    Default Re: So... What the hell happened to the Forums?

    Hi,

    The issue we have with nginx is more to do with our software reference/standards policy, then getting the best software for the task.

    To get nginx we have to go through a lot of hoops, where security and enterprise engineering teams shake their head and say no, as it's not a tested product within our systems. However, there is light at the end of the tunnel where our small agile servers are out performing the enterprise systems by a factor of 4 to 10.

    Hopefully our current tests on a spare $13k blade will show it outruns the $8M dollar enterprise system. There will be blood on the streets when these stats come out, as it basically throws a knife in the hearts of those people who pushed for the big expensive systems, only to find they cannot perform as fast as ours. When the blood dries there will be changes, and then we may end up being left alone (like we were until 2 years ago) so we can get nginx and seek other more suitable solutions that are specific to our business unit, rather than being forced into a large enterprise solution that is not only expensive, but slow to respond (for updates and changes), as well as slower in performance.

    Our enterprise support and change control teams make Media Temple look good.

    seeyuzz
    river
    The thinking man's clown and the drinking woman's sex symbol
    RA25GT - There is no substitute | 18R-G - Toyota's Dependable Masterpiece
    Toymods Car Club Treasurer, assistant Historic Plate Registrar & Forums Admin

  9. #9
    Junior Member Conversion King
    Join Date
    May 2010
    Location
    ACT
    Posts
    2,510

    Default Re: So... What the hell happened to the Forums?

    *mutters something about nginx being used to cache 600mbit of Internet traffic for a significant government document that was released the other week*
    Autodub - 1987 AW11 G-Limited, Dark Blue Mica 4AGZE T-Top 4EAT

  10. #10
    Forum Member 1st year Apprentice
    Join Date
    Jul 2005
    Location
    Amman
    Posts
    37

    Default Re: So... What the hell happened to the Forums?

    I have nothing against nginx... but how would you feel if MediaTemple broke into your garage overnight and replaced your toyota engine with something out of a Lada

  11. #11
    Junior Member Conversion King
    Join Date
    May 2010
    Location
    ACT
    Posts
    2,510

    Default Re: So... What the hell happened to the Forums?

    Yeah wasn't the context I meant. I casually mentioned it well before the server migration happened.

    Totally agree their deciding to completely overhaul your VPS's software unannounced is totally uncalled for, and could potentially breach your agreement and maybe allow for compensation?
    Why the hell didn't they just save a snapshot of your whole system and transfer it? No messing about with backing up DB etc.
    Autodub - 1987 AW11 G-Limited, Dark Blue Mica 4AGZE T-Top 4EAT

  12. #12
    anti blasphemy ! Carport Converter
    Join Date
    Jan 2006
    Location
    perth
    Posts
    1,346

    Default Re: So... What the hell happened to the Forums?

    Haha this place is terrible, gota be the most unreliable site ever.
    What you guys doing wrong ? Haha
    i dont have a funny or cool signature.

  13. #13
    Lord of InterpretiveDance Chief Engine Builder tomvale13's Avatar
    Join Date
    Jul 2006
    Location
    Convictania
    Posts
    3,093

    Default Re: So... What the hell happened to the Forums?

    It was the people they pay to host the site that caused the outage, which was almost 3 weeks ago..

    edit: apparently db errors today.. which I have luckily missed.
    Last edited by tomvale13; 12-06-2013 at 01:07 PM.
    Quote Originally Posted by Cuzzo View Post
    They cash in the empty can of whoopass they received in SA and receive 10c's per punch...
    They have to accrew enough for nose reconstructions as well as feeding his mime family. The dont talk much so they never need much water for their dry mouths.

  14. #14
    Building Corollas Chief Engine Builder Cuzzo's Avatar
    Join Date
    Oct 2005
    Location
    NSW
    Posts
    4,628

    Default Re: So... What the hell happened to the Forums?

    and if Tom was as much a toyknobs whore as he says he is, he would have noticed all of the database errors through out today.
    KE30 4agte - Bullet - Rick Rolled
    UZX83 1UZ W58- Cruiser - Tow Car - Fun Car
    Quote Originally Posted by -GT- View Post
    You had an oil and plastics engine bay fire, with flames that reached at least to the roof - of course shit got hot, it wasn't burning jiffy firelighters back there.

  15. #15
    Lord of InterpretiveDance Chief Engine Builder tomvale13's Avatar
    Join Date
    Jul 2006
    Location
    Convictania
    Posts
    3,093

    Default Re: So... What the hell happened to the Forums?

    I've been here all day..

    Haven't noticed anything?
    Quote Originally Posted by Cuzzo View Post
    They cash in the empty can of whoopass they received in SA and receive 10c's per punch...
    They have to accrew enough for nose reconstructions as well as feeding his mime family. The dont talk much so they never need much water for their dry mouths.

Similar Threads

  1. Where the hell...
    By Stockmx83 in forum Tech and Conversions
    Replies: 6
    Last Post: 02-05-2012, 07:21 AM
  2. What happened to 'lem'?
    By RObErT_RaTh in forum Toymods Car Club Membership and Feedback
    Replies: 4
    Last Post: 25-01-2007, 07:27 PM
  3. What the hell is this....
    By Biggo in forum Tech and Conversions
    Replies: 2
    Last Post: 09-04-2006, 01:03 PM

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •