Daily Builds 1.0 - 1.4

[Ondra]

we tried longer testing earlier, but it seemed pointless - it seems that people no longer use dailies, as there are not that big improvements as before (most of useful features are already implemented). We waited weeks with 1.0 release only to discover major bugs at day 1.

We are open to suggestions on how to do better testing in this conditions... anyone? ;)

[Ludvik Koutny]

Yep. It would not be much of a problem to let RC rest for a few weeks. The problem is most of the people will simply wait for final release and won't bother to test RC much.

[karnak]

I think that the users who can test RC builds are only the ones that are not working with Corona on projects with deadlines.
It is quite insane to jump to a new version of the software in the middle of the process.

[Ondra]

yes. So nobody will test the unstable version. But then they will expect it to become stable all by itself ;)

[racoonart]

I think this will always be a problem after all. Unless you use it in daily production you will never find the nasty bugs. We used finalRender betas for years in production and it often was devastating - we only kept doing it because we could (usually) count on getting a fix a couple of hours after the report and bugging the developers.
If the response times are long, no one will risk their project to be delayed by a bug. Look at max, how many people in the beta are actually testing it? Most are just installing the new beta, rant about how bad a new feature seems to be designed and wait for the next installer and that someone else finds all the bugs. In the end, beta testing is expensive for the people doing it - it costs time, be it for testing or waiting for a fix, and thus it costs money.
Imho, the only way to "somehow" solve the problem is to have software developed "in-house", meaning while working on projects and getting instant fixes by the developers. Or having a core testing team that's using the software on real projects.

[karnak]

yes. So nobody will test the unstable version. But then they will expect it to become stable all by itself ;)

Don't be so negative, I don't think that all Corona users are every single day on a deadline.
With a little more testing time and a committed group of testers I'm sure we can get a stable enough release.

[pokoy]

Let's say you settle on a 2-4 weeks timeframe to take the RC through rough waters and encourage people on the forum to test as much as they can and let it do some heavy lifting internally as well.

It's not that there was no feedback about problems with the 1.2 dailies and RCs ;) There were, in fact, issues and I think it's clear that the new displacement for example failed on simple tests for some people.

I totally understand how hard it is to track down bugs or even come across unusual bugs, more so if those who run into problems are not able to supply any scenes. Still, releasing a final version means that people will take the 'final' tag as s sign of a stable and tested release. With this in mind, 2 days of testing of RCs is not enough, no matter what.

Make a subforum for each RC. Supply scenes that people can test on their machines (even if simple or exotic), encourage people to upload their own scenes for testing for others, let them discuss new features (the new sun was an unwelcome surprise for many users, for example) and get some feedback. But, please, take the time to make sure it's stable and a broad range of users had the chance to test it. Once a final release is marked as unstable by the user base, no one will want to run it in production.

[lacilaci]

Have you thought about providing some simple demo scenes with every new (major or not) features when they get implemented?

So that even people who may not have time or don't really need that feature right now could still just run and see and report,
if it was working in their hw/sw config and report about the performance or possible issues in a more relevant test scenario...

Then maybe add a poll with different versions of max and windows so that you could see how many people tested with which config.

I'm thinking, just about developing some easy and streamlined and relevant testing process to rule out some basic issues and usage fails and then focus on more extreme scenarios and so on..

I use dailies all the time, but many times I don't even know what's new so... what to test if all seems working. There's a bugfix for some issue I never had.
Just adding a test scene that supposedly didn't work before and should now and adding a poll to see how it goes would provide much more relevant results I think even from people who don't have time.

put a simple demo to test in a daily build folder so that the moment i install it i can run it and at that point I know the new feature is stable for me and I'm relevant to you even If I might not need it
or have no time for playing with it and testing.

All I'm saying is that you don't necessarily need more people or more time for testing. You need a simple but effective, relevant and targeted testing in a first place to rule out random basic problems.

Btw, like I said I use daily builds almost allway no matter if on tight deadline or not, and I don't even remember when was the last time I had problems, not even after upgrading to win10 at day one of release.
So I'm a daily user that you have no use for.

Just a bit random thoughts while waiting for food... sorry if stupid :D

[Ludvik Koutny]

I would say that vast majority of bugs are scene setup specific. There are few extremely rare that are HW/SW specific, like those VFB wxWidget asserts, but overwhelming majority of the bugs is found by users running it on their scenes with their specific setups and workflows. So providing premade scenes for users to test would be really bad way to search for bugs.

[Ondra]

....

we have a set of (currently) 165 scenes that we run automatically every other day. It helps a lot, but it cannot substitute user testing. When I look back at problems at launch, there are things we cannot catch with our testing, such as:
- problems caused by obscure 3dsmax behavior we were never able to reproduce at any of our machines (~6 different PCs at least) - for example crashes when opening scene with old sun
- things that were not a bug, but resulted in user confusion (for example people cannot read version numbering with zeroes - instead of 1.00, 1.01, 1.02, ... 1.09, 1.10 people were expecting numbering of 1.00, 1.20, 1.30, ...)
- incorrect workflows and users relying on obscure functionality from Corona resulting in problems when opening new scenes - such as setting displacement quality extremely high and relying on "max subdivs" parameter to limit the subdivision rate. When the parameter gets removed because it is no longer needed (when using the displacement properly), incorrectly set up scenes will blow up.
- problems caused by old hardware (old AMD CPUs)

[Ondra]

yes. So nobody will test the unstable version. But then they will expect it to become stable all by itself ;)

Don't be so negative, I don't think that all Corona users are every single day on a deadline.
With a little more testing time and a committed group of testers I'm sure we can get a stable enough release.

of course it is not black and white, but the problem is there - everybody wants to use stable software, but software is born unstable and becomes stable only through being used.

[karnak]

Having coded a few small programs myself I definitely see your point and I feel the same.
I suppose there has to be a little bit of effort and pain on both sides, developers and users.

I'm here to help by the way, let's just figure out how do you want me* to help you. :)

edit.
* not just me of course, but everyone else that want to help.

[maru]

Would it be a good idea to create some kind of newsletter that would inform only the interested ones about new daily builds availability?

[Ondra]

BTW: hotfix 1 is released.

[pokoy]

Would it be a good idea to create some kind of newsletter that would inform only the interested ones about new daily builds availability?

Good idea. Or a blog entry calling all eligible users to test RCs, to focus on specific areas that were changed/improved, provide feedback, ask questions.
Basically all the stuff that happens anyways, but too late. Like I said, once it's out and marked as unstable, chances are you'll get to know of bugs way later because less people will install the latest version.

I'd also like to encourage the dev team to talk about upcoming changes that affect workflows so users know upfront of these or even influence the original course through open discussion if needed. Even if this ends up in endless debates and you have to cut it off eventually - at least people are informed.

Take your time! I guess most of us don't care if the next release takes two months longer as long as we know it's not going to blow up our work and schedule.