VE 1.9.5 RGB conversion still needs improvement

[Maccara]

I've been doing some extensive testing (was going to test different deblocking algorithms in relation to SR processing, but noticed VE already misbehaves so started testing that alone instead) and it seems YV12-RGB conversion still could use some improvement, or VE does not respect the setting "always work in RGB". Avisynth ConvertToRGB still gives much better results in the end.

Otherwise, I'm getting fairly good results with select material from VE (when compared to spline36/64/nnedi etc).

Testing method:
orig.avi : 720x576 Ut Lossless Video Codec YV12 (1300 frames re-encoded from cleaned VHS capture stored with matrox mpeg2 i-frame only codec)
small.avi : 360x288 Ut Lossless Video Codec YV12 (from orig.avi with avisynth bilinearresize)
small.avs : AVISource("small.avi").ConvertToRGB()

I used MSU Video Quality Measurement Tool v2.7.2 to confirm what I can see visually (VQM statistics, but replicable with PSNR etc too, but results here show VQM statistic).

Since VE does not give any info whatsoever what it is doing (what colorspace used to open file, does some conversion happen etc) I can only guess what happened, but the results are clear nevertheless.

All SR processing on max quality.

All other parameters/codec choices kept exactly same between tests and only source file changed between tests (with restarting VE in between)

orig_VE_RGB_int.avi: 720x576 Ut Lossless Video Codec YV12 (from small.avi VE-SR, "always work in RGB" checked)
orig_VE_RGB_avs.avi: 720x576 Ut Lossless Video Codec YV12 (from small.avs VE-SR, "always work in RGB" checked)

comparing to orig.avi:
RGB_int avg VQM: 1.37918 RGB_avs avg VQM: 1.35804
(smaller is better)

I know the difference isn't very big, but I can visually easily see the artifacts when comparing zoomed frames - looks like a small checkerboard pattern (similar to what you show this version was supposed to fix).

I get practically the same results if I let VE work in YV12 mode (small avi and not forced RGB), so it may be that the artifacts are from YV12-SR processing and VE is not respecting the settings?

Nevertheless, using Avisynth to convert the colorspace to RGB prior to feeding the video to VE is clearly superior, whether VE YV12-SR or VE YV12-RGB conversion is used, and VE just does not work too well with other than properly prepared RGB sources yet.

Could you please check this? And could you PLEASE add some diagnostics to processing, so users wouldn't have to guess what is happening... How about an option to FORCE pixeltype (i.e. no guesswork at all)? (just fail if source codec does not allow that pixel_type, no ifs or buts - would reveal bugs in how you try to open files / enumerate codecs)

[Dee Mon]

Thanks! We'll do some more testing and measurements.

What I can guess from your description is that you have a codec capable of outputting both YV12 and RGB32, you have "Always work in RGB32" mode turned on, so VE asks the codec to provide video in RGB32 and the codec performs internally YV12-RGB32 conversion which it does poorly (without interpolating U and V). VE's high quality YV12-RGB32 conversion is not used in this scenario.
If you remove ConvertToRGB32() from your AviSynth script, make it provide YV12 and open .avs in VE with Always RGB on, then VE's colorspace converter will be used.

I agree, it would be nice to see in what colorspace SR is working, we just didn't decide how to implement it in the UI.
I guess we'll add more options regarding colorspaces and make forcing certain colorspace possible.

[Maccara]

Yes, I think this codec can input and output "any" format requested and does internal conversion if non-native requested.

Didn't think of that possibility - was expecting VE to request "native" format, but now that I think of it, was there even an indicator through VfW/DS which format was preferred by the codec? (this codec has VfW and DS filters - which does VE prefer and default to?)

Seems I need to be careful how I test when there's no diagnostic from the tool - I always forget something.

I'll re-test doing everything through avisynth (including measurements, although it showed correct pixel_type) as I suspect that's why I'm seeing difference also between YV12-SR vs RGB-SR (was expecting measurable difference, not visible due to colorspace).

I'll report those results back shortly.

[Maccara]

Ok. Now this seems better.

One imposed limitation in this quick test: output codec was YV12 (rationale is that this would be the end format anyway) and I didn't take any special measures to account for that. But, since this is same for all methods, this might only favor YV12 processing where there's a "direct chain" (everything in YV12).

Avisynth convertToRGB: 1.35804 Avisynth YV12 SR force RGB: 1.35946

Hardly measurable advantage for avs converttorgb, but there were a couple of problem-frames where VE conversion produced some artifacts (in boundary regions) which is visible (and probably skews the results slightly). Otherwise the difference is not even visible.

Interesting is Avisynth source YV12 with YV12-SR processing: 1.39261

I wasn't expecting YV12-SR to bee "so much" worse than RGB - the difference is clearly visible when zoomed 2x. I was expecting measurable difference only and maybe a few problem-frames, so I'm a bit surprised the difference is that much - especially since source format and end format are YV12 so if anything, RGB processing should see diminishing returns.

Is that expected? I understand YV12-SR works in limited colorspace, but should that make so much a difference? (of course 2x speed boost is nice still )

Note: seems the differences are mostly visible in moving regions so it may be a "bad" lanczos (still use that as a fallback?) implementation at work. Worth to check out?

Nevertheless, I now have a benchmark and workflow to test against what I was originally going to test - deblocking in relation to SR processing.

[Dee Mon]

Yes, I think this codec can input and output "any" format requested and does internal conversion if non-native requested.

Actually it's impossible to know what format is native for a codec, only codec itself knows this.

was there even an indicator through VfW/DS which format was preferred by the codec? (this codec has VfW and DS filters - which does VE prefer and default to?)

VE lets DirectShow choose a codec. I suppose if both DS and VfW are present, DS one will be chosen. It's easy to check: just render your file in GraphEdit or GraphEditPlus. If you see AVI Decompressor, it is a DS wrapper for a VfW codec.

1.39261

What metric was used?

I wasn't expecting YV12-SR to bee "so much" worse than RGB - the difference is clearly visible when zoomed 2x.

Don't forget that YV12 SR still outputs YV12 which is 4:2:0, so color resolution is still twice lower than luminance and when you zoom video you'll always see some artifacts comparing to honest RGB SR (which is 4:4:4).

[Maccara]

Actually it's impossible to know what format is native for a codec, only codec itself knows this.

Yes, and this lossless codec has specific variants for YV12,YUY2 and RGB24/32. Just forgot they still accept other formats too.

What metric was used?

VQM. And confirmed visually.

Don't forget that YV12 SR still outputs YV12 which is 4:2:0, so color resolution is still twice lower than luminance and when you zoom video you'll always see some artifacts comparing to honest RGB SR (which is 4:4:4).

I didn't forget. If you noticed, all of the results were in YV12 - all comparisons done from YV12 sources. That's why I asked if it is normal to see the difference in processing when the output was in the same limited colorspace (YV12) for both. I guess so, just didn't expect it to be a visible difference. So YV12-SR speed doesn't come for free.

For best quality workflow for the moment: source (YV12) - avis converttorgb (or force RGB in VE) - RGB-SR - temp (RGB) - avis converttoYV12 -> (assuming you need to end up in YV12 anyway)
For speed (and 2x speed is nothing to sneer at): source (YV12) - YV12-SR - temp (YV12)

[Dee Mon]

Yes, exactly.