diff options
| author | Jorge E. Moreira <jemoreira@google.com> | 2020-05-08 15:23:48 -0700 |
|---|---|---|
| committer | Jorge E. Moreira <jemoreira@google.com> | 2020-05-29 12:41:57 -0700 |
| commit | 1911e63d5f62606eb5ca7cf86f3dd6ba02fa822a (patch) | |
| tree | 42eebe62078a454e92286adad5c15282a76e850b | |
| parent | 2a305a3beda0221530e996ca3d01274b45bd0f9f (diff) | |
| download | libvpx-1911e63d5f62606eb5ca7cf86f3dd6ba02fa822a.tar.gz | |
Temporarily convert to 64 bits to avoid overflows
In the vp8_cost_branch function a couple of unsigned int are being
multiplied by integer coeficients and added to later be divided by
256. While the end result most likely fits an unsigned int, the
intermediary result of multiplying and adding sometimes doesn't (I was
able to reproduce it by leaving the encoder running at 60 fps for a
while). To avoid the multiplication overflow (which is undefined
behavior and causes a wrong result anyways) the calculation is
performed using unsigned long long instead and cast to unsigned int
for return.
Bug: 154172422
Test: run cuttlefish with webrtc enabled for an hour
Change-Id: If7ebbda38b2450a59ed3c99ffbb59dc62431a324
Merged-In: If7ebbda38b2450a59ed3c99ffbb59dc62431a324
| -rw-r--r-- | libvpx/vp8/encoder/treewriter.h | 6 |
1 files changed, 5 insertions, 1 deletions
diff --git a/libvpx/vp8/encoder/treewriter.h b/libvpx/vp8/encoder/treewriter.h index c02683a58..4e9ed6af1 100644 --- a/libvpx/vp8/encoder/treewriter.h +++ b/libvpx/vp8/encoder/treewriter.h @@ -14,6 +14,8 @@ /* Trees map alphabets into huffman-like codes suitable for an arithmetic bit coder. Timothy S Murphy 11 October 2004 */ +#include <stdint.h> + #include "./vpx_config.h" #include "vp8/common/treecoder.h" @@ -48,7 +50,9 @@ static INLINE unsigned int vp8_cost_branch(const unsigned int ct[2], vp8_prob p) { /* Imitate existing calculation */ - return ((ct[0] * vp8_cost_zero(p)) + (ct[1] * vp8_cost_one(p))) >> 8; + return (unsigned int)(((((uint64_t)ct[0]) * vp8_cost_zero(p)) + + (((uint64_t)ct[1]) * vp8_cost_one(p))) >> + 8); } /* Small functions to write explicit values and tokens, as well as |