Cleanups in the video encoding decoding code. Reenable VP8.

Cleanups in the video encoding decoding code. Reenable VP8.

1. Moved video-related protobuf messages from event.proto to video.proto. Removed those that we don't need anymore
2. Fixed naming for enums and some types.
3. Reenabled VP8.
4. Proper RGB-YUV converter for VP8 encoder.
5. Changed the capturer_fake to show more meaningful picture.

BUG=57374
TEST=unittests

Committed: http://src.chromium.org/viewvc/chrome?view=rev&revision=64700

[Sergey Ulanov, 2010-10-29 00:37:39]

http://codereview.chromium.org/4136010/diff/4001/5031
File remoting/proto/video.proto (right):

http://codereview.chromium.org/4136010/diff/4001/5031#newcode64
remoting/proto/video.proto:64: message VideoPacket {
I intent to use this for RTP too. Will probably need to make this a regular
struct instead of protobufs to avoid memcpy's

[Alpha Left Google, 2010-10-29 08:12:05]

looks pretty good. This change is a bit too big, we should also separate the yuv
conversion to a util function.

http://codereview.chromium.org/4136010/diff/4001/5010
File remoting/base/encoder_vp8.cc (right):

http://codereview.chromium.org/4136010/diff/4001/5010#newcode128
remoting/base/encoder_vp8.cc:128: y_out[j] = clip_byte(((pixel[0] * 66 +
pixel[1] * 129 +
Please separate this part into a util function in media/. This can be left as a
TODO for now.

http://codereview.chromium.org/4136010/diff/4001/5018
File remoting/host/capturer_fake.cc (right):

http://codereview.chromium.org/4136010/diff/4001/5018#newcode63
remoting/host/capturer_fake.cc:63: memset(buffers_[current_buffer_].get(), 0xff,
What is this going to look like, please put a comment here.

http://codereview.chromium.org/4136010/diff/4001/5027
File remoting/host/simple_host_process.cc (right):

http://codereview.chromium.org/4136010/diff/4001/5027#newcode39
remoting/host/simple_host_process.cc:39: 
remove this line.

http://codereview.chromium.org/4136010/diff/4001/5031
File remoting/proto/video.proto (right):

http://codereview.chromium.org/4136010/diff/4001/5031#newcode64
remoting/proto/video.proto:64: message VideoPacket {
On 2010/10/29 00:37:39, sergeyu wrote:
> I intent to use this for RTP too. Will probably need to make this a regular
> struct instead of protobufs to avoid memcpy's

I think we want to retain this as a protobuf container for any encoding format.
For RTP we probably want to have something else.

http://codereview.chromium.org/4136010/diff/4001/5034
File remoting/remoting.gyp (right):

http://codereview.chromium.org/4136010/diff/4001/5034#newcode141
remoting/remoting.gyp:141: 'base/decoder_vp8.cc',
Seperate these into a non-ARM section. ARM build doesn't have vp8.

[Sergey Ulanov, 2010-10-29 19:46:56]

http://codereview.chromium.org/4136010/diff/4001/5010
File remoting/base/encoder_vp8.cc (right):

http://codereview.chromium.org/4136010/diff/4001/5010#newcode128
remoting/base/encoder_vp8.cc:128: y_out[j] = clip_byte(((pixel[0] * 66 +
pixel[1] * 129 +
On 2010/10/29 08:12:05, Alpha wrote:
> Please separate this part into a util function in media/. This can be left as
a
> TODO for now.
Added TODOs

http://codereview.chromium.org/4136010/diff/4001/5018
File remoting/host/capturer_fake.cc (right):

http://codereview.chromium.org/4136010/diff/4001/5018#newcode63
remoting/host/capturer_fake.cc:63: memset(buffers_[current_buffer_].get(), 0xff,
On 2010/10/29 08:12:05, Alpha wrote:
> What is this going to look like, please put a comment here.

Done.

http://codereview.chromium.org/4136010/diff/4001/5027
File remoting/host/simple_host_process.cc (right):

http://codereview.chromium.org/4136010/diff/4001/5027#newcode39
remoting/host/simple_host_process.cc:39: 
On 2010/10/29 08:12:05, Alpha wrote:
> remove this line.

Done.

http://codereview.chromium.org/4136010/diff/4001/5034
File remoting/remoting.gyp (right):

http://codereview.chromium.org/4136010/diff/4001/5034#newcode141
remoting/remoting.gyp:141: 'base/decoder_vp8.cc',
On 2010/10/29 08:12:05, Alpha wrote:
> Seperate these into a non-ARM section. ARM build doesn't have vp8.
There is third_party/lib/linux/arm, so I suppose it should work fine on arm. No?

[Alpha Left Google, 2010-10-29 20:45:53]

VP8 is not build on ARM as far as I know. Even if it does, they symbols are not
exported in FFmpeg, see ffmpeg.gyp. So this code won't link on ARM.

[awong, 2010-10-29 21:27:21]

Yayy!!!!

Thanks for cleaning up the mess I left.  I owe both you and Gary one.

Comments added.

http://codereview.chromium.org/4136010/diff/19001/20005
File remoting/base/decoder_vp8.cc (right):

http://codereview.chromium.org/4136010/diff/19001/20005#newcode34
remoting/base/decoder_vp8.cc:34: const gfx::Rect& clip, int bytes_per_src_pixel)
{
This API was originally written to expect that we've got a subsection of the
frame that we're writing in to.  Do we want to DCHECK ehre that clip is the same
size as frame_ for VP8?  Even more generally, do we even want to use this API
for the VP8 decoding?

http://codereview.chromium.org/4136010/diff/19001/20005#newcode60
remoting/base/decoder_vp8.cc:60: codec_ = NULL;
Move state_ to kError?

http://codereview.chromium.org/4136010/diff/19001/20005#newcode68
remoting/base/decoder_vp8.cc:68: vpx_codec_err_t ret = vpx_codec_decode(
I can't see where we're telling VP8 what the frame size is.  Am I just missing
it somewhere?

http://codereview.chromium.org/4136010/diff/19001/20005#newcode69
remoting/base/decoder_vp8.cc:69: codec_, reinterpret_cast<const
uint8*>(encoded_bytes.c_str()),
Use .data() for things that are not null-terminated.

http://codereview.chromium.org/4136010/diff/19001/20010
File remoting/base/encoder_vp8.cc (right):

http://codereview.chromium.org/4136010/diff/19001/20010#newcode129
remoting/base/encoder_vp8.cc:129: y_out[j] = clip_byte(((pixel[0] * 66 +
pixel[1] * 129 +
uh...I'm just going to assume this is right.

Add a TODO that this should get pulled out into a utility class and possibly
shared with media.

http://codereview.chromium.org/4136010/diff/19001/20010#newcode166
remoting/base/encoder_vp8.cc:166: DCHECK(ret == VPX_CODEC_OK)
Isn't there a DCHECK_EQ?

http://codereview.chromium.org/4136010/diff/19001/20010#newcode200
remoting/base/encoder_vp8.cc:200: message->set_flags(VideoPacket::FIRST_PACKET |
VideoPacket::LAST_PACKET);
We're guaranteed not to span a max packet size?

http://codereview.chromium.org/4136010/diff/19001/20018
File remoting/host/capturer_fake.cc (right):

http://codereview.chromium.org/4136010/diff/19001/20018#newcode17
remoting/host/capturer_fake.cc:17: static const int kSpeed = 20;
Add comment explaining difference between Width/Height and BoxWidth/BoxHeight? 
And Comment explaining Speed?

Also, COMPILE_ASSERT that kBoxWidth < kWidth and kBoxHeight < kHeight?

http://codereview.chromium.org/4136010/diff/19001/20018#newcode64
remoting/host/capturer_fake.cc:64: // This draws a ractangle of size
kBoxWidth*kBoxHeight.
ractangle -> rectangle

http://codereview.chromium.org/4136010/diff/19001/20018#newcode68
remoting/host/capturer_fake.cc:68: //     blue.......red
The box also bounces around the screen doesn't it?  Explain that here too?

http://codereview.chromium.org/4136010/diff/19001/20019
File remoting/host/capturer_fake.h (right):

http://codereview.chromium.org/4136010/diff/19001/20019#newcode13
remoting/host/capturer_fake.h:13: // A CapturerFake always output an image of
640x480 in 24bit RGB. The image
Update comment?  Or co-locate it with the actual constants?

http://codereview.chromium.org/4136010/diff/19001/20019#newcode34
remoting/host/capturer_fake.h:34: int seed_;
Do we still use seed_?

http://codereview.chromium.org/4136010/diff/19001/20024
File remoting/host/session_manager.cc (left):

http://codereview.chromium.org/4136010/diff/19001/20024#oldcode403
remoting/host/session_manager.cc:403: ChromotingHostMessage* message,
Encoder::EncodingState state) {
YAY!!!! Thanks for killing EncoderState here.

http://codereview.chromium.org/4136010/diff/19001/20026
File remoting/host/session_manager_unittest.cc (right):

http://codereview.chromium.org/4136010/diff/19001/20026#newcode24
remoting/host/session_manager_unittest.cc:24: VideoPacketFormat::ENCODING_NONE;
Can we rename this to ENCODING_VERBATIM so our names match?

http://codereview.chromium.org/4136010/diff/19001/20027
File remoting/host/simple_host_process.cc (right):

http://codereview.chromium.org/4136010/diff/19001/20027#newcode126
remoting/host/simple_host_process.cc:126: LOG(INFO) << "Using the verbatim
encoder.";
log info is wrong.

Also, should this be a VLOG(1)?

http://codereview.chromium.org/4136010/diff/19001/20031
File remoting/proto/video.proto (right):

http://codereview.chromium.org/4136010/diff/19001/20031#newcode4
remoting/proto/video.proto:4: //
Remove // on line 4 so there's just a newline between copyright and the rest of
the stuff.

http://codereview.chromium.org/4136010/diff/19001/20031#newcode5
remoting/proto/video.proto:5: // Protocol for video messages.
I wonder if the file should be called screen_cast.proto, or media.proto, or
something.  Eventually, we'll want to send audio, and it's possible that this
will have more than video.

The concept we're trying to represent is a stream of updates about the remote
desktop.  Maybe name it based off somethign like that?

http://codereview.chromium.org/4136010/diff/19001/20031#newcode26
remoting/proto/video.proto:26: PixelFormatInvalid = 0;
Since we're changing the world, make these PIXEL_FORMAT_INVALID, and so forth to
comply with the chromium style guide.

http://codereview.chromium.org/4136010/diff/19001/20031#newcode64
remoting/proto/video.proto:64: message VideoPacket {
What if we call this whole thing a MediaPacket, or ScreenPacket?  Or maybe
FrameBufferUpdate?

It's not 100% clear to me that we won't also be sending static images at various
points.

[Sergey Ulanov, 2010-10-30 01:43:09]

On 2010/10/29 20:45:53, Alpha wrote:
> VP8 is not build on ARM as far as I know. Even if it does, they symbols are
not
> exported in FFmpeg, see ffmpeg.gyp. So this code won't link on ARM.
disabled vp8 on arm

[Sergey Ulanov, 2010-10-30 01:43:35]

http://codereview.chromium.org/4136010/diff/19001/20005
File remoting/base/decoder_vp8.cc (right):

http://codereview.chromium.org/4136010/diff/19001/20005#newcode34
remoting/base/decoder_vp8.cc:34: const gfx::Rect& clip, int bytes_per_src_pixel)
{
On 2010/10/29 21:27:21, awong wrote:
> This API was originally written to expect that we've got a subsection of the
> frame that we're writing in to.  Do we want to DCHECK ehre that clip is the
same
> size as frame_ for VP8?  Even more generally, do we even want to use this API
> for the VP8 decoding?

Yes, I still want to use the same API for VP8 and Zlib encode, but we will need
to change this API a bit. I'll do this in a separate CL. I think DecodeBytes()
should be replaced with Decode(VideoPacket* packet) where |packet| contains one
packet with all additional info (e.g. rect).

http://codereview.chromium.org/4136010/diff/19001/20005#newcode60
remoting/base/decoder_vp8.cc:60: codec_ = NULL;
On 2010/10/29 21:27:21, awong wrote:
> Move state_ to kError?

Done.

http://codereview.chromium.org/4136010/diff/19001/20005#newcode68
remoting/base/decoder_vp8.cc:68: vpx_codec_err_t ret = vpx_codec_decode(
On 2010/10/29 21:27:21, awong wrote:
> I can't see where we're telling VP8 what the frame size is.  Am I just missing
> it somewhere?
We don't. I guess it is encoded in the frame.

http://codereview.chromium.org/4136010/diff/19001/20005#newcode69
remoting/base/decoder_vp8.cc:69: codec_, reinterpret_cast<const
uint8*>(encoded_bytes.c_str()),
On 2010/10/29 21:27:21, awong wrote:
> Use .data() for things that are not null-terminated.

Done.

http://codereview.chromium.org/4136010/diff/19001/20010
File remoting/base/encoder_vp8.cc (right):

http://codereview.chromium.org/4136010/diff/19001/20010#newcode129
remoting/base/encoder_vp8.cc:129: y_out[j] = clip_byte(((pixel[0] * 66 +
pixel[1] * 129 +
On 2010/10/29 21:27:21, awong wrote:
> uh...I'm just going to assume this is right.
> 
> Add a TODO that this should get pulled out into a utility class and possibly
> shared with media.
There is already TODO.

http://codereview.chromium.org/4136010/diff/19001/20010#newcode166
remoting/base/encoder_vp8.cc:166: DCHECK(ret == VPX_CODEC_OK)
On 2010/10/29 21:27:21, awong wrote:
> Isn't there a DCHECK_EQ?

Done.

http://codereview.chromium.org/4136010/diff/19001/20010#newcode200
remoting/base/encoder_vp8.cc:200: message->set_flags(VideoPacket::FIRST_PACKET |
VideoPacket::LAST_PACKET);
On 2010/10/29 21:27:21, awong wrote:
> We're guaranteed not to span a max packet size?
No. The RTP layer will split this into multiple UDP packets if necessary.

http://codereview.chromium.org/4136010/diff/19001/20018
File remoting/host/capturer_fake.cc (right):

http://codereview.chromium.org/4136010/diff/19001/20018#newcode17
remoting/host/capturer_fake.cc:17: static const int kSpeed = 20;
On 2010/10/29 21:27:21, awong wrote:
> Add comment explaining difference between Width/Height and BoxWidth/BoxHeight?

> And Comment explaining Speed?
> 
> Also, COMPILE_ASSERT that kBoxWidth < kWidth and kBoxHeight < kHeight?

Done.

http://codereview.chromium.org/4136010/diff/19001/20018#newcode64
remoting/host/capturer_fake.cc:64: // This draws a ractangle of size
kBoxWidth*kBoxHeight.
On 2010/10/29 21:27:21, awong wrote:
> ractangle -> rectangle

Done.

http://codereview.chromium.org/4136010/diff/19001/20018#newcode68
remoting/host/capturer_fake.cc:68: //     blue.......red
On 2010/10/29 21:27:21, awong wrote:
> The box also bounces around the screen doesn't it?  Explain that here too?

Added comments in the top of the file

http://codereview.chromium.org/4136010/diff/19001/20019
File remoting/host/capturer_fake.h (right):

http://codereview.chromium.org/4136010/diff/19001/20019#newcode13
remoting/host/capturer_fake.h:13: // A CapturerFake always output an image of
640x480 in 24bit RGB. The image
On 2010/10/29 21:27:21, awong wrote:
> Update comment?  Or co-locate it with the actual constants?

Done.

http://codereview.chromium.org/4136010/diff/19001/20019#newcode34
remoting/host/capturer_fake.h:34: int seed_;
On 2010/10/29 21:27:21, awong wrote:
> Do we still use seed_?

Done.

http://codereview.chromium.org/4136010/diff/19001/20026
File remoting/host/session_manager_unittest.cc (right):

http://codereview.chromium.org/4136010/diff/19001/20026#newcode24
remoting/host/session_manager_unittest.cc:24: VideoPacketFormat::ENCODING_NONE;
On 2010/10/29 21:27:21, awong wrote:
> Can we rename this to ENCODING_VERBATIM so our names match?

Done.

http://codereview.chromium.org/4136010/diff/19001/20027
File remoting/host/simple_host_process.cc (right):

http://codereview.chromium.org/4136010/diff/19001/20027#newcode126
remoting/host/simple_host_process.cc:126: LOG(INFO) << "Using the verbatim
encoder.";
On 2010/10/29 21:27:21, awong wrote:
> log info is wrong.
> 
> Also, should this be a VLOG(1)?
LOG(INFO) is not banned. I think it is appropriate to use it here.

http://codereview.chromium.org/4136010/diff/19001/20031
File remoting/proto/video.proto (right):

http://codereview.chromium.org/4136010/diff/19001/20031#newcode4
remoting/proto/video.proto:4: //
On 2010/10/29 21:27:21, awong wrote:
> Remove // on line 4 so there's just a newline between copyright and the rest
of
> the stuff.

Done.

http://codereview.chromium.org/4136010/diff/19001/20031#newcode5
remoting/proto/video.proto:5: // Protocol for video messages.
On 2010/10/29 21:27:21, awong wrote:
> I wonder if the file should be called screen_cast.proto, or media.proto, or
> something.  Eventually, we'll want to send audio, and it's possible that this
> will have more than video.
> 
> The concept we're trying to represent is a stream of updates about the remote
> desktop.  Maybe name it based off somethign like that?

I don't think we would need to use the same protobuf messages for audio, so they
will probably be in another file. Screencast is longer, but doesn't add much.

http://codereview.chromium.org/4136010/diff/19001/20031#newcode26
remoting/proto/video.proto:26: PixelFormatInvalid = 0;
On 2010/10/29 21:27:21, awong wrote:
> Since we're changing the world, make these PIXEL_FORMAT_INVALID, and so forth
to
> comply with the chromium style guide.

Done.

http://codereview.chromium.org/4136010/diff/19001/20031#newcode64
remoting/proto/video.proto:64: message VideoPacket {
On 2010/10/29 21:27:21, awong wrote:
> What if we call this whole thing a MediaPacket, or ScreenPacket?  Or maybe
> FrameBufferUpdate?
> 
> It's not 100% clear to me that we won't also be sending static images at
various
> points.

We will most probably need to change in many times anyway. Let's rename it later
if we need to.

[awong, 2010-11-01 20:11:58]

LGTM w/ nits

We should talk about the decoder/decompressor structure.

http://codereview.chromium.org/4136010/diff/19001/20005
File remoting/base/decoder_vp8.cc (right):

http://codereview.chromium.org/4136010/diff/19001/20005#newcode34
remoting/base/decoder_vp8.cc:34: const gfx::Rect& clip, int bytes_per_src_pixel)
{
On 2010/10/30 01:43:35, sergeyu wrote:
> On 2010/10/29 21:27:21, awong wrote:
> > This API was originally written to expect that we've got a subsection of the
> > frame that we're writing in to.  Do we want to DCHECK ehre that clip is the
> same
> > size as frame_ for VP8?  Even more generally, do we even want to use this
API
> > for the VP8 decoding?
> 
> Yes, I still want to use the same API for VP8 and Zlib encode, but we will
need
> to change this API a bit. I'll do this in a separate CL. I think DecodeBytes()
> should be replaced with Decode(VideoPacket* packet) where |packet| contains
one
> packet with all additional info (e.g. rect).

I'm not entirely sure I agree with that.  The thing that would be nice to avoid
is leaking the state flags, etc.s from the packet upwards.  In the previous
code, the decompressor and decoder were tightly coupled on the network packet
state.  If we could make a strong separation there, I think things will be
cleaner.

http://codereview.chromium.org/4136010/diff/30001/31022
File remoting/host/capturer_fake.cc (right):

http://codereview.chromium.org/4136010/diff/30001/31022#newcode11
remoting/host/capturer_fake.cc:11: // CapturerFake generates a white picture of
size kWidth*kHeight with a
try kWidth x kHeight, and kBoxWidth x kBoxHeight.  Using a * makes it look like
you're muliplying.