rv34: fix invalid memory access for small video dimensions
authorJohn Brooks <john.brooks@bluecherry.net>
Wed, 4 Jan 2012 06:27:35 +0000 (23:27 -0700)
committerMichael Niedermayer <michaelni@gmx.at>
Thu, 5 Jan 2012 01:23:02 +0000 (02:23 +0100)
commitfc9c5ad9ea70479d09a5d37c800665746ff4710d
tree0a837994d088cb332d408715deff284ab6bbf852
parent68f121269694d61c137f046ea8f51ce717e57d7c
rv34: fix invalid memory access for small video dimensions

For small video dimensions calculations of the upper bound for pixel
access may result in negative value. Using an unsigned comparison
works only if the greater operand is non-negative. This is fixed by
doing edge emulation explicitly for such conditions.

Signed-off-by: Kostya Shishkov <kostya.shishkov@gmail.com>
(cherry picked from commit aacf6b3a2fd8bc8603e3deaa6e612ea03cf08707)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
libavcodec/rv34.c