Genii Weblog

I wish Lotus knew quality assurance better

Fri 4 Sep 2009, 02:27 PM



by Ben Langhinrichs
It has been a grim few days, testing out the new Notes/Domino 8.51 (Beta 8) Discussion database template, based on XPages and the Dojo editor.  The bugs in Dojo compete for attention with the bugs in XPages.  The regression bugs clamor for eminence over the new bugs.  It is not really a question of looking for bugs, it is more a question of swatting as many as possible in hopes of getting out alive.
 
This may sound like hyperbole or gross exaggeration, but it isn't.  I maintain a list of free and commercial web editors which is widely referenced even by those who don't know a thing about Notes/Domino, and there is hardly a single editor in that list with fewer features than the Dojo editor used with XPages, yet even with this highly limited feature set, there are bugs that practically jump out at you.  The rich text to HTML/MIME rendering has bugs similar to those in Domino 8 that were fixed in Domino 8.5, but are back in 8.5.1, but you barely notice them because they are blocked by new bugs.

I plan to do a series of posts next week highlighting individual bugs (it takes a LOT of work to set up fair tests, do screen prints that show clearly what the bug is, etc.), but I'll show one because it is a perfect combination of a bug I have blogged about that has reappeared, along with a new bug.

Original created in Notes 8.5.1  (Beta 8) client (simplification of example I blogged as Will Sam (and Rick) get fired tonight?)  Note nested ordered lists with the last number as 5 and the inner lists as lowercase alpha.

Inline JPEG image


Same document displayed by Domino 8.5.1 (Beta 8) in the new XPage Discussion template.  Note how the nested ordered lists have merged, with the last number now 12, but the former inner lists still lowercase alpha.

Inline JPEG image


The cause for this bug (which was fixed in 8.5.0, at least for MIME emails) is that the nested lists are not indented far enough, so next, let's indent one set further.


Original document in Notes client (never saved from web), but with deeper indent on second nested list  Note nested ordered lists with the last number as 5 and the inner lists as lowercase alpha.

Inline JPEG image



Document displayed again by Domino 8.5.1 (beta 8) in the new XPage Discussion template.  Note how the first nested ordered list has merged (regression bug), but the second nested list shows as numbers rather than lowercase letters (new bug).

Inline JPEG image



Document edited by Domino 8.5.1 (beta 8) in the new XPage Discussion template.  Note how the first nested ordered list has merged (regression bug), but the second nested list shows as lowercase alpha in edit mode unlike in read mode.

Inline JPEG image



Saved by Domino 8.5.1 (Beta 8) and then redisplayed from Notes.  The merged list and mixed number/alpha are preserved as in the corrupted wb version, so the original numbering is lost.

Inline JPEG image


To be absolutely clear, this is beta code, it is only beta code, and anything you see or I report may change before release!

Copyright © 2009 Genii Software Ltd.

What has been said:


868.1. Ed Brill (09/04/2009 08:52 PM)

I believe the standard disclaimer is meant to be that this is beta code and may or may not be representative of the final product.

More to the point, I don't think that your public blog is really the way to get these bugs addressed. I see you have posted some of them in the design partner and/or managed beta forums. We are working with you to see which ones can be addressed in 8.5.1 and which ones will be post-8.5.1.

And even more to the point, these are in the template, not in the core Notes/Domino engine. If we, hypothetically, looked to iterate on the template faster and through alternative distribution means, these might not be as "fatal" as you seem to make them out.


868.2. Erik Brooks (09/04/2009 10:03 PM)

@Ed - These look to me as rich text rendering problems, not anything with the template. I'm guessing that's just a rich text field, and the content is being handled differently version-to-version.

I'm in the 8.5.1 beta now and am uncovering a significant performance issue that needs to be addressed. I've posted about it in the Managed Beta forum, more details to come there.

I posted about it before on my blog, but the QC effort obviously needs to be HUGE for 8.5.1.


868.3. Ben Langhinrichs (09/04/2009 10:13 PM)

@Ed - Right, but the standard disclaimer tends to be about features, not bugs. As for reporting this in the DP forum, I am cataloging them there. While I cannot comment publicly on the responses in that forum, do you think they are indicative of this being fixed in 8.5.1? And Erik is correct, these are rich text rendering issues that are not related to the template (I created my own to test and be sure)


868.4. Ed Brill (09/04/2009 10:24 PM)

Ben, the terms under which you are allowed to blog about 8.5.1 include the requirement to indicate that it is beta software and may change by release. I ack'd that in this case, these might or might not.

Erik - please don't use phrases like "significant performance issue" without qualification. Your post in the design partner forum says an agent that normally runs in 10-12 hours is taking double that in the 8.5.1 beta. That is indeed significant in the context of your agent, but it does not support a blanket statement.

Guys, I'm hyper-sensitive about this stuff. We have fixed a ton of issues in the 8.5.1 code. There are always more to go, but I feel like we've made a lot of great progress and don't want to see that diminished. Ben, you posted in our forum about how happy you were with 8.5.1 CD8, but your first public posting on it is a slam.


868.5. Robert Laing (09/05/2009 01:09 AM)

Bugs in beta code ? And you're surprised ??? Isn't that one of the points of beta code -- to surface the bugs ?


868.6. Erik Brooks (09/05/2009 03:26 AM)

Ed, you don't think that an agent running twice as slow is qualification of a "significant performance issue"? Hours aside, even if the agent went from 5 *seconds* to 10, that's a pretty big concern with regards to server scalability and load.

I'm not too worried about that though - I'm sure it will get handled. After all, that is the point of beta code... things change before release!

I agree, the forward progress starting with 8.0.x has been flat-out awesome, and what I've seen in some of 8.5.1 is truly (and I don't use this word lightly) fantastic. The LS editor alone makes me weep tears of joy.

But my point was to echo Ben here - due to the massive (and *growing*) vertical-ness of the platform, regressions are absolutely killer. And it absolutely breaks our hearts to see them appear, *especially* when it's something that was either reported before or should have been caught with internal IBM alpha testing.

I mean, I'm running the most complex Domino web app in the world. Nobody, I mean NOBODY wants XPages more than I do. But we can't even touch them because of a few 8.5.0 crashes and 8.5FP1 memory leak bugs. It's truly a shame.

I understand completely that you're sensitive about this, Ed. I feel the exact same way about my products, and your passion has always been one of the things I respect about you. But even before new features roll out, the ones that currently exist have *got* to be solid.


868.7. Volker Weber (09/05/2009 04:55 PM)

Ben, you should really only blog about the awesomeness that is 8.5.1 :-)


868.8. Erik Brooks (09/06/2009 05:16 PM)

Great (though slightly embarrassing) news -- it turns out that our 8.5.1 Beta vs. 7.0.3 testing wasn't a true apples-to-apples comparison.

8.5.1 definitely seems to NOT be slower than 7.0.3. I posted a "False Alarm" update to the beta forum.


868.9. Erik Brooks (09/06/2009 07:53 PM)

Argh -- I take that back. The wrong log times were referenced... 8.5.1's definitely running this code slower than 7.0.3 was.