There was a problem and the downtime lasted more than what was expected. In fact LOTRO, Dungeons and Dragons Online and Asheron’s Call were all affected by this problem.

The SAN upgrade that went awry was intended to reduce the length of downtime’s during future patches, by a pretty significant amount.

The games were down for almost 24 hours and there were plenty of angry people. Most were understanding and were patient. I went on our monster play ventrilo server to find out what everyone else was doing. We all just shared fun web pages and talked about what things we liked to see changed in the game. Most everyone else waited for any word from the developers about the status of the game.

On behalf of the entire staff at Turbine we would like to apologize for yesterday’s extended downtime. The outage was a unique occurrence involving a combination of the hardware and software that supports our login servers and other related web services. This problem prevented players from successfully logging into the games, although the game servers themselves were unaffected and remained up.

Most everyone was happy that the servers finally came up and wasn’t expecting anything from Turbine but an apology. We were surprised with a free day and some special items. Very nice way to make their customers happy. Hope this never happens again and I have heard of other games with longer downtime’s so this one isn’t that bad. One big mess-up their first year is good for a new mmo game I think.

Here’s the official statement from turbine

Last night I noticed a RL friend who I also have in my SL friends list sending a Twitter message about some technical problems. I logged into SL and planned to send him a message to ask for more information and possibly help if I could. Problem is my friend list didn’t load, it just stayed in “waiting…” mode. While writing this it occurred to me I could have searched for his avatar and contacted him that way. Where was that line of thinking last night?

When the friends list finally loaded last night, he wasn’t online any more. Has anybody else been having problems with their friends list loading? Is there a tip/trick/workaround to deal with this somehow?

But who am I to complain? All in all between JIRA, phones, and the support portal, we’re only looking at about … eight times it’s been down without scheduled maintenance.  That’s only a total of 2.66 times a month.  Not bad for premium support service that was supposed to take care of all of our support needs and help make it more efficient for contacting Linden.  I think.  Or something like that.

Kind of amusing really.   When the support line goes down that is.   You don’t hear of technical support actually needing technical support often.

The latest scheduled maintenance was done at 2AM-3AM Pacific time.   This is norm for those of us that have actually supported live twenty four seven systems, but definitely one of the first for Linden being that in the past they’ve always had scheduled windows on Wednesdays during the wee hours of the morning.  Well, at least when most other people start drinking coffee, type hours.

So is this a break from the norm?   It definitely will be a lot more professional from a maintenance window standpoint.  Or is it just a fluke and we’ll be back to our regularly scheduled programming?

Here’s to hoping that this is a new beginning for support staff.

Click any image for larger image

One of the major changes is the ability to purchase credits on Kaneva. There are several packages of credits being offered. 1,500 credits for $9.99, 3,150 credits for $19.99, 6,600 credits for $39.99, 14,000 credits for $79.99, or…

The One time only offer of 9,000 credits plus and exclusive home deed, –The Manor– the largest living space offered by Kaneva — a three story building that is 20% larger than the previous largest living space,  20 building blocks, 1 big TV, 4 exclusive lights, 5 picture frames, 1 exclusive staircase, 10 wall units, and 13 floor tiles for the reduced price of $49.99.

I went through the steps of purchasing credits by clicking on the begin purchase button which carried me to a screen to add my billing address (see image on left) –which must be the billing address on the credit card used– when I had that information filled in I clicked save and continue.

This carried me to the page (image on the right) where you will enter your credit card information and actually submit all that information for the purchase of your new Kaneva credits.

Another aspect of Kaneva that is in the process of changing is the avatars! There was a loud outcry of the Kaneva users about how their avatars did not quite match their expectations when they saved their new character. Kaneva is now stating, “As we optimize and improve the avatars and the avatar creation process, you will soon be able to re-create your avatar anytime and as many times as you like.”

Kaneva has also started allowing you to fully integrate YouTube videos into their 3D environment. Inside your media library, the YouTube videos can be managed and ordered within your Kaneva playlist. This simplifies how individuals and communities put together content into their own playlists regardless of where the video originates.

By simply placing the 3D TV’s inside the virtual world, videos from Kaneva and YouTube can play seamlessly and synchronized across all members in the same virtual space at the same time. This ensures that the community can experience the same videos at the same time to maximize social experience.

Kaneva today at 12:01 PM ET announced the Grid would be brought down for some “quick maintenance.  No estimated time for the grid to be brought back online.

Being as I have a little space in this post I thought I would post a few more pictures from the new Kaneva Catalog.

Yesterday was the day new avatars came to Kaneva!

They can actually be made to look somewhat better than these examples, by working with a few clickable arrows and a slider or two, but much diversity is still allowing user created content.

I logged into the Kaneva’s world upon completion of the download and nearly had a heart attack when I reached my apartment and found every texture I had placed had been replaced with the texture of a new Kaneva T-shirt. Thank goodness I lost server connection and had to log back in again…. and my L.S.U. Textures were back in place.

I decided to go to the mall and spend some of my reward credits on all the new men’s clothes, as I do not particularly enjoying wearing the same thing everybody around me is wearing. I teleported to the mall, which only had 58 people in it when I arrived, only to move about three steps then freeze up and after a few moments lost access to the server again.

At this point I figured I may as well check on the Kaneva web page to see if they were mentioning anything going on with the servers. Couldn’t find out anything about the wold servers, but found this shopping catalog page, where there were listed samples of the new women’s clothing.

They also have pictured in the shopping catalog pictures of the empty apartments so you might get a very limited idea on what you are wanting to buy for an apartment…. hardly anyone would want the closet they start you off in.

I wanted to go back to the mall and buy some clothes, and I tried again.  I was heading for a clothing store when I noticed there was a Kaneva Bank over in one of the corners of the center of the mall.  I went to see how much Kaneva Credits were going to cost, but found it was a depository for items you didn’t want to carry around with you, rather than a place to exchange dollars for credits and visa-versa.

I turned and headed for the clothing store  and again loss the server.  When I tried to get back into Kaneva I received an error message…. basically showing me the servers were down.

I can only assume that the good folks at Kaneva are pulling their hair out about now… trying to work out their “bugs”.  I will make an attempt here in a little while to go back into Kaneva and see what has changed, and try to buy some clothes with these worthless reward credits I have.

In any case, so as you can see… there’s some gestures trying to load. No clue what it is or what the deal is with it. There’s no way to actually kill them or disable those particular gestures either.

So after about five to ten minutes, your client blows up. Just dies. Kaput. Blam. Pop. Boom. nyah, nyah nyah nyah….. Batman! Oh wait.

Reproducible? Definitely. Just take a look at what happens right before it dies. What the heck is this?

And I seriously doubt that these two fixes should have messed with gestures:

• VWR-830 Muting Residents and\or Objects does not work reliably
  
• MISC-157 Searching for groups fails with certain keywords

We’re not doing any software updates – this is purely hardware work needed to ensure critical things like electricity are delivered to all of the machines that make up the grid, and also to increase the bandwidth into our Dallas facility. We should be open again by 10am PDT.

Four hours for non-software and only hardware checks should be enough right? Along with bandwidth increase? I guess not. It looks like they didn’t schedule enough time for the maintenance window… again. Really. It’s better to have more maintenance window and finish early, than finish late and not come through. As of this post, there isn’t an update on why there’s a delay either.

Maybe there needs to a lesson on how mainstream corporations operate for maintenance windows. This isn’t your average small business anymore. Let’s act like it a bit and do things the way the big leaguers do it… m’ kay?

[12:16] Jeska Linden shouts: Kylix Petit: Where does stability and performance fall in the priorites for LL development…. There is a problem literally every other day and it seems like there is VERY much room for improvement in this regard.
[12:16] Cory Linden shouts: As I spoke about in the blog post, 69% of the development staff at LL are currently on scaling and stability…
[12:17] Cory Linden shouts: and that percentage is rising over the next few weeks. It is the highest priority and the focus of the majority of our design, coding, and QA work.
[12:17] Cory Linden shouts: We are also working on the next gen architecture to allow far more scaling thean the current design.

Analysis: The point here isn’t that a lot of people are working on scaling and performance. It’s the fact that over the last few releases, it’s pretty obvious that the process of release is skewed and not a good process at all. Personally, I would suggest someone perform the Six Sigma process on the development/QA testing as much as I hate running those types of things, but throwing out a number of 69% doesn’t help anyone. Microsoft has thousands of people in testing, but it doesn’t stop their blue screens of death, now does it? Or how about Apple back in the day, when they had General Error 1 or 11? The process needs to be defined better here in how things are done and what sort of process you go through to fix a problem.

[12:17] Jeska Linden shouts: Vienna James: I am concerned about releases that have blatant issues, why are they being missed in Beta?
[12:18] Cory Linden shouts: Many problems only appear when we hit high load. Part of the solution to this is better synthetic testing, but the beta grid exists for people to expose problems…
[12:18] Cory Linden shouts: You can all help by exploring the beta grid and posting bugs and repro cases to the public jira.

Analysis: To a degree, I agree with Cory here. Real data will always beat out synthetic data. But the issues here are the fact that you don’t even tell your users if you use synthetic data. This here says that better synethetic testing would help. Do you guys have a test suite? Does it do load testing? Have you tried to do maximum load testing or any type of stress test? The reason this is a question from my end is because many of the issues seem to happen due to high stress. This is without having to write a test suite, or even doing anything. Any experienced tester or developer would have seen this as a red flag if they even participate in the slightest in the production world. To my knowledge, there is no compensation for participating as a Beta Grid tester, nor is there any reason for it. First, do the synthetics. Then when you have everything and need more detailed definition, move to real data. It’s only my opinion, but it doesn’t even seen the larger issue has been resolved so no need to delve into details.

[12:19] Jeska Linden shouts: Spank Lovell: I’d like to ask what Quality control measures LL has in place to police the testing and release of both client updates and patches the the SL client and backend systems. It seems to me that each release introduces issues or breaks previously stable features?
[12:20] Cory Linden shouts: As I mentioned, the challenge with a system as complex as SL is testing w/o full load. As we continue to builkd out the dev team…
[12:20] Cory Linden shouts: We will be working on better synthetic tests and benchmarks. In addition, nearly all new/refactored code is being …
[12:21] Cory Linden shouts: written with unit tests, whcih help. But the best way for you all to help make SL better is to use the beta grid and report repro cases when you find them.

Analysis: More of the same. This question is a very good question because it asks what metrics QA is using to pass clients and server code into production. Cory basically regurgitated the previous analysis’ answer with working on better tests and benchmarks and telling the users to use the Beta Grid.

[12:23] Jeska Linden shouts: Sascha Vandyke: Can’t the patches be tested on a beta grid, because they always render the whole sim unstable?
[12:23] Cory Linden shouts: Patches are tested on the beta grid, but often times bugs do not appear until full load testing.

Analysis: This is good to know, because this means they do test the patches in a pseudo-real environment.

[12:54] Jeska Linden shouts: tx Oh: over 90% open bug reports are unassigned. why?
[12:55] Cory Linden shouts: Torley is on this, so we are trying to provide better feedback on when bugs are being worked on. Not all will show the specific Linden who is on it, but we will indicate thos we are on.

Analysis: This is good to know, but realistically just like any type of support, someone has to take it on. In every single customer service type support scenario, there is always a point of contact (POC). That POC is always the assignee. To say that there are bugs being worked on, but no specific Linden who is on it truly is a cop out in support services.

The filtering from this Town Hall wasn’t that great. Even though Jeska Linden said that duplicate questions would not be answered, there was plenty of duplicates in the transcript. I suppose that shows that most people were concerned with the following:

• inventory disappearing (answer: terms of service, and JIRA)
• stability and scalability (answer: JIRA, Beta Grid, better metrics)
• voice and other new features (answer: comes in time, beta Grid)

Besides the MySQL questions, most of these answers were very non-technical and public relations oriented. Throwing out numbers seems to satisfy the mob usually. At least that’s what you’re taught in PR class. While it answered some of my issues with Second Life, it also skipped out on certain very interesting things that I’ve pointed out before and a couple new questions:

• Why is maintenance window during the day when any other support service has it during less busy hours?
• Why is there this direction of pushing bugs to the JIRA (which is fine) but there’s still less communication on a more personal scale. Everyone knows that from a customer service stand point, the personal relationship always trumps all.
• Six Sigma!!?!!! Get a Black Belt in there!
• Terms of service? So basically you’re saying that while you provide an economic exchange, if the stock market crashes you’re not at fault? That’s kind of below-the-belt punch when the money conversion is very real.

So far, *knock on wood* I’m at least happy that the client hasn’t crashed since the new server patch for Group IMs went in. I also find it absolutely amusing that the Town Hall was supposed to be a communications channel and everyone was “supposed to be able” to listen in on Group IM. Unfortunately, you couldn’t so it basically locked out a good number of other participants and kept it only with the avatars at Pooley Stage. I also didn’t like the fact that in the transcript, there was this one little bit about inventory loss, where Cory pointed out TOS. Basically it was… if you play here, you take some risk. While that’s all good and all, that’s not very satisfactory for business owners. You literally are playing a high risk stake by doing anything in SL based on the stability of the software. If you can’t count on the stability, how do you expect to do business? It’s like if you went to the store and bought a Coke for $1USD and cash register happened to be on the fritz and said… hey.. your Coke is now $10.00USD. Usually a store will apologize and take the loss at this point, but not in SL. In SL, it’s basically a …. you’re screwed. Doesn’t really make people feel all warm and fuzzy when it comes to upgrading their accounts.Maybe it’s just me, and perhaps I’m just as cynical as ever. But the Town Hall really didn’t provide the answers we as users were looking for. The old adage of: “Time will tell” is reinstated, but truthfully, it’s sort of getting old.