A breakdown analysis of the Cory Linden Town Hall

It looks like I couldn’t make it to the Town Hall. By the time I was trying to teleport in, and even tried to fly in, the region was full. I couldn’t read the Closed Captioning (Cory mentions there’s issues here) due to more Group IM issues. But it’s interesting that I see a lot of answers in the transcript that doesn’t really answer … well, anything. As I have previously written, the issue is the process of development and testing, not how many people you throw at the procedure. Process needs to be refined. You can find the full transcript here. I’ll be frank and do most of the analysis on actual code issues and not on inventory or policy type issues. So let’s do some dissecting.

[12:16] Jeska Linden shouts: Kylix Petit: Where does stability and performance fall in the priorites for LL development…. There is a problem literally every other day and it seems like there is VERY much room for improvement in this regard.
[12:16] Cory Linden shouts: As I spoke about in the blog post, 69% of the development staff at LL are currently on scaling and stability…
[12:17] Cory Linden shouts: and that percentage is rising over the next few weeks. It is the highest priority and the focus of the majority of our design, coding, and QA work.
[12:17] Cory Linden shouts: We are also working on the next gen architecture to allow far more scaling thean the current design.

Analysis: The point here isn’t that a lot of people are working on scaling and performance. It’s the fact that over the last few releases, it’s pretty obvious that the process of release is skewed and not a good process at all. Personally, I would suggest someone perform the Six Sigma process on the development/QA testing as much as I hate running those types of things, but throwing out a number of 69% doesn’t help anyone. Microsoft has thousands of people in testing, but it doesn’t stop their blue screens of death, now does it? Or how about Apple back in the day, when they had General Error 1 or 11? The process needs to be defined better here in how things are done and what sort of process you go through to fix a problem.

[12:17] Jeska Linden shouts: Vienna James: I am concerned about releases that have blatant issues, why are they being missed in Beta?
[12:18] Cory Linden shouts: Many problems only appear when we hit high load. Part of the solution to this is better synthetic testing, but the beta grid exists for people to expose problems…
[12:18] Cory Linden shouts: You can all help by exploring the beta grid and posting bugs and repro cases to the public jira.

Analysis: To a degree, I agree with Cory here. Real data will always beat out synthetic data. But the issues here are the fact that you don’t even tell your users if you use synthetic data. This here says that better synethetic testing would help. Do you guys have a test suite? Does it do load testing? Have you tried to do maximum load testing or any type of stress test? The reason this is a question from my end is because many of the issues seem to happen due to high stress. This is without having to write a test suite, or even doing anything. Any experienced tester or developer would have seen this as a red flag if they even participate in the slightest in the production world. To my knowledge, there is no compensation for participating as a Beta Grid tester, nor is there any reason for it. First, do the synthetics. Then when you have everything and need more detailed definition, move to real data. It’s only my opinion, but it doesn’t even seen the larger issue has been resolved so no need to delve into details.

[12:19] Jeska Linden shouts: Spank Lovell: I’d like to ask what Quality control measures LL has in place to police the testing and release of both client updates and patches the the SL client and backend systems. It seems to me that each release introduces issues or breaks previously stable features?
[12:20] Cory Linden shouts: As I mentioned, the challenge with a system as complex as SL is testing w/o full load. As we continue to builkd out the dev team…
[12:20] Cory Linden shouts: We will be working on better synthetic tests and benchmarks. In addition, nearly all new/refactored code is being …
[12:21] Cory Linden shouts: written with unit tests, whcih help. But the best way for you all to help make SL better is to use the beta grid and report repro cases when you find them.

Analysis: More of the same. This question is a very good question because it asks what metrics QA is using to pass clients and server code into production. Cory basically regurgitated the previous analysis’ answer with working on better tests and benchmarks and telling the users to use the Beta Grid.

[12:23] Jeska Linden shouts: Sascha Vandyke: Can’t the patches be tested on a beta grid, because they always render the whole sim unstable?
[12:23] Cory Linden shouts: Patches are tested on the beta grid, but often times bugs do not appear until full load testing.

Analysis: This is good to know, because this means they do test the patches in a pseudo-real environment.

[12:54] Jeska Linden shouts: tx Oh: over 90% open bug reports are unassigned. why?
[12:55] Cory Linden shouts: Torley is on this, so we are trying to provide better feedback on when bugs are being worked on. Not all will show the specific Linden who is on it, but we will indicate thos we are on.

Analysis: This is good to know, but realistically just like any type of support, someone has to take it on. In every single customer service type support scenario, there is always a point of contact (POC). That POC is always the assignee. To say that there are bugs being worked on, but no specific Linden who is on it truly is a cop out in support services.

The filtering from this Town Hall wasn’t that great. Even though Jeska Linden said that duplicate questions would not be answered, there was plenty of duplicates in the transcript. I suppose that shows that most people were concerned with the following:

• inventory disappearing (answer: terms of service, and JIRA)
• stability and scalability (answer: JIRA, Beta Grid, better metrics)
• voice and other new features (answer: comes in time, beta Grid)

Besides the MySQL questions, most of these answers were very non-technical and public relations oriented. Throwing out numbers seems to satisfy the mob usually. At least that’s what you’re taught in PR class. While it answered some of my issues with Second Life, it also skipped out on certain very interesting things that I’ve pointed out before and a couple new questions:

• Why is maintenance window during the day when any other support service has it during less busy hours?
• Why is there this direction of pushing bugs to the JIRA (which is fine) but there’s still less communication on a more personal scale. Everyone knows that from a customer service stand point, the personal relationship always trumps all.
• Six Sigma!!?!!! Get a Black Belt in there!
• Terms of service? So basically you’re saying that while you provide an economic exchange, if the stock market crashes you’re not at fault? That’s kind of below-the-belt punch when the money conversion is very real.

So far, *knock on wood* I’m at least happy that the client hasn’t crashed since the new server patch for Group IMs went in. I also find it absolutely amusing that the Town Hall was supposed to be a communications channel and everyone was “supposed to be able” to listen in on Group IM. Unfortunately, you couldn’t so it basically locked out a good number of other participants and kept it only with the avatars at Pooley Stage. I also didn’t like the fact that in the transcript, there was this one little bit about inventory loss, where Cory pointed out TOS. Basically it was… if you play here, you take some risk. While that’s all good and all, that’s not very satisfactory for business owners. You literally are playing a high risk stake by doing anything in SL based on the stability of the software. If you can’t count on the stability, how do you expect to do business? It’s like if you went to the store and bought a Coke for $1USD and cash register happened to be on the fritz and said… hey.. your Coke is now $10.00USD. Usually a store will apologize and take the loss at this point, but not in SL. In SL, it’s basically a …. you’re screwed. Doesn’t really make people feel all warm and fuzzy when it comes to upgrading their accounts.Maybe it’s just me, and perhaps I’m just as cynical as ever. But the Town Hall really didn’t provide the answers we as users were looking for. The old adage of: “Time will tell” is reinstated, but truthfully, it’s sort of getting old.