vc1dec: fix invalid memory access for small video dimensions
authorJohn Brooks <john.brooks@bluecherry.net>
Sun, 11 Dec 2011 09:37:21 +0000 (02:37 -0700)
committerMichael Niedermayer <michaelni@gmx.at>
Thu, 5 Jan 2012 01:23:08 +0000 (02:23 +0100)
commite540446f6c5ca1613eea4f9e8c5db12f1ec3925c
tree8be072d9e5de3639f935b763afe6d868561c2479
parentfc9c5ad9ea70479d09a5d37c800665746ff4710d
vc1dec: fix invalid memory access for small video dimensions

For small video dimensions, these calculations of the upper bound
for pixel access may have a negative result. Using an unsigned
comparison to bound a potentially negative value only works if
the greater operand is non-negative. Fixed by doing edge emulation
when the upper bound is probably negative, everywhere that this
pattern appears.

Signed-off-by: Kostya Shishkov <kostya.shishkov@gmail.com>
(cherry picked from commit d209c27b09234cc40bbdbd680aa502b493edf595)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
libavcodec/vc1dec.c