More Chromium Blob hacking

More Chromium Blob hacking to switch from bloburls to blobuuids.

This cr patch is needed prior to continuing work over on the blnk side. One of many in multi-side-patch-tango.

This change introduces a compile time flag, USE_BLOB_UUIDS, to switch over to the new model. The old and new style of identifying these things are currently in place at the webkit api level. This flag tells chromium which to use. Blink is currently still using the old style urls and USE_BLOB_UUIDS is not defined. The flag will be set in the Blink patch that switches it over (it will be defined in WebBlobData.h).

There is one blob use case (feedback extension api) up in /chrome that is handled a little differently. Instead of the compile time flag, it relies on a runtime check due to layering/deps rules. There is nothing in the contentAPI about blobs yet and that consumer relies on reading the blob data by urlfetching given only the internalurl/uuid.

Also in this change are some minor cleanup to better indicate when we're working with a FileSystemUrl in particular.

TBR=kalman
BUG=174200
Committed: https://src.chromium.org/viewvc/chrome?view=rev&revision=224906

[michaeln, 2013-09-14 00:56:24]

@rkc, ptal at the feedback api related stuff and there are a couple of questions
in there for you.
@kinuko, ptal at the other stuff

[michaeln, 2013-09-14 01:36:54]

I've started work on the blink patch to switch things over...
https://codereview.chromium.org/23992003/
... but need to grease the skids so to speak with this cr patch prior to being
able to complete that and test it and such.

[kinuko, 2013-09-15 15:51:52]

content and fileapi change looks good, I have one question about feedback API's
usage of blob URL.

https://codereview.chromium.org/23898006/diff/34001/chrome/renderer/extension...
File chrome/renderer/extensions/feedback_private_custom_bindings.cc (right):

https://codereview.chromium.org/23898006/diff/34001/chrome/renderer/extension...
chrome/renderer/extensions/feedback_private_custom_bindings.cc:29: // a
BlobDataHandle given the uuid value.
Hmm... I wonder if we could just call URL.createObjectURL (instead of using this
private native method) and use public URL in
feedback_private_custom_bindings.js?  We should be able to call revokeObjectURL
after necessary work is done in the custom binding too.

[michaeln, 2013-09-16 20:02:13]

https://codereview.chromium.org/23898006/diff/34001/chrome/renderer/extension...
File chrome/renderer/extensions/feedback_private_custom_bindings.cc (right):

https://codereview.chromium.org/23898006/diff/34001/chrome/renderer/extension...
chrome/renderer/extensions/feedback_private_custom_bindings.cc:29: // a
BlobDataHandle given the uuid value.
On 2013/09/15 15:51:52, kinuko wrote:
> Hmm... I wonder if we could just call URL.createObjectURL (instead of using
this
> private native method) and use public URL in
> feedback_private_custom_bindings.js?  We should be able to call
revokeObjectURL
> after necessary work is done in the custom binding too.

If you have a blob that needs to be processed somewhere somehow, please use the
id to identify whats being referred to. 

Setting up a temporary urlmapping could be made to function i suppose, but i'd
rather identify blobs by id than depend upon obscure layers of indirection. Once
the switch to using uuids in webcore is complete, what you would do to extend
the lifetime of a blob given its uuid is addBlobDataRef(uuid) and when you no
longer need it removeBlobDataRef(uuid). There's a class in WebCore that will do
that for you in child processes, and there's a class in chromium code that will
do that for you in the browser process.

What's needed for the feedback api case is a BlobReader class in the browser
process that will accept a uuid as input. This CL modifies that class to do just
that. Ultimately, that class probably wants to move into the content lib and to
be exposed to /chrome via content API. I'm not mucking with content API just
yet.

[michaeln, 2013-09-16 22:43:19]

https://codereview.chromium.org/23898006/diff/7001/content/common/page_state_...
File content/common/page_state_serialization.cc (right):

https://codereview.chromium.org/23898006/diff/7001/content/common/page_state_...
content/common/page_state_serialization.cc:303: void WriteStdString(const
std::string& s, SerializeObject* obj) {
Added these helpers that take std::strings as inputs/outputs so it compiles when
USE_BLOB_UUIDS is defined.

[rkc, 2013-09-17 02:01:02]

https://codereview.chromium.org/23898006/diff/7001/chrome/browser/extensions/...
File chrome/browser/extensions/api/feedback_private/blob_reader.cc (right):

https://codereview.chromium.org/23898006/diff/7001/chrome/browser/extensions/...
chrome/browser/extensions/api/feedback_private/blob_reader.cc:20: //
TODO(michaeln): remove support for deprecated blob urls
We probably don't need to have support for these. The only user for this class
is the feedback API; since this CL updates the feedback API to use uuid's, there
shouldn't be anything that would use this class with the deprecated blob URL's.

https://codereview.chromium.org/23898006/diff/7001/chrome/browser/extensions/...
chrome/browser/extensions/api/feedback_private/blob_reader.cc:47: // TODO: ask
rkc where fetcher_ is deleted?
This is a leak in my code, the fetcher should have been deleted right before the
'delete this' line.

https://codereview.chromium.org/23898006/diff/7001/chrome/renderer/extensions...
File chrome/renderer/extensions/feedback_private_custom_bindings.cc (right):

https://codereview.chromium.org/23898006/diff/7001/chrome/renderer/extensions...
chrome/renderer/extensions/feedback_private_custom_bindings.cc:19: // do
something with it. Are there multiple consumers of this funtion?
There shouldn't be any other consumers of this function other than the file I've
mentioned in the file below.

https://codereview.chromium.org/23898006/diff/7001/chrome/renderer/extensions...
chrome/renderer/extensions/feedback_private_custom_bindings.cc:27: // TODO: find
where that generic extension plumbing happens? I'm looking for
asargent@ would know more about how this plumbing works.

https://codereview.chromium.org/23898006/diff/7001/chrome/renderer/extensions...
chrome/renderer/extensions/feedback_private_custom_bindings.cc:43: //
feedback_private_custom_bindings.js. Ask rck where to do that?
You'll need to update
http://src.chromium.org/viewvc/chrome/trunk/src/chrome/renderer/resources/ext...

[kinuko, 2013-09-17 03:13:56]

https://codereview.chromium.org/23898006/diff/34001/chrome/renderer/extension...
File chrome/renderer/extensions/feedback_private_custom_bindings.cc (right):

https://codereview.chromium.org/23898006/diff/34001/chrome/renderer/extension...
chrome/renderer/extensions/feedback_private_custom_bindings.cc:29: // a
BlobDataHandle given the uuid value.
On 2013/09/16 20:02:14, michaeln wrote:
> On 2013/09/15 15:51:52, kinuko wrote:
> > Hmm... I wonder if we could just call URL.createObjectURL (instead of using
> this
> > private native method) and use public URL in
> > feedback_private_custom_bindings.js?  We should be able to call
> revokeObjectURL
> > after necessary work is done in the custom binding too.
> 
> If you have a blob that needs to be processed somewhere somehow, please use
the
> id to identify whats being referred to. 
>
> Setting up a temporary urlmapping could be made to function i suppose, but i'd
> rather identify blobs by id than depend upon obscure layers of indirection.
Once
> the switch to using uuids in webcore is complete, what you would do to extend
> the lifetime of a blob given its uuid is addBlobDataRef(uuid) and when you no
> longer need it removeBlobDataRef(uuid). There's a class in WebCore that will
do
> that for you in child processes, and there's a class in chromium code that
will
> do that for you in the browser process.

To clarify-- when a user javascript code needs to get a blob URL to process on
it it still calls regular createObjectURL() method, right?  You mean we don't
want to introduce new 'internal' code which uses public URL, am I right?

> What's needed for the feedback api case is a BlobReader class in the browser
> process that will accept a uuid as input. This CL modifies that class to do
just
> that. Ultimately, that class probably wants to move into the content lib and
to
> be exposed to /chrome via content API. I'm not mucking with content API just
> yet.

This sounds good, but current code (uses special internal prefix to encode uuid
in a url just for this private API) feels like it's introducing another
complexity.

[michaeln, 2013-09-17 20:25:22]

Thnx for looking i'll upload a new patch soon.

> To clarify-- when a user javascript code needs to get a blob URL to process on
> it it still calls regular createObjectURL() method, right?  You mean we don't
> want to introduce new 'internal' code which uses public URL, am I right?

Yes, nothing about this changes the scriptable contract in any way.

> This sounds good, but current code (uses special internal prefix to encode
uuid
> in a url just for this private API) feels like it's introducing another
> complexity.

It is, until this can be better addressed at the content api level, I need to
retain support for url fetching of the bytes given only a string identifier.
Since the identifier no longer happens to be a url, i've introduced this
internal prefix to allow this impl of BlobReader to continue to function.

https://codereview.chromium.org/23898006/diff/7001/chrome/browser/extensions/...
File chrome/browser/extensions/api/feedback_private/blob_reader.cc (right):

https://codereview.chromium.org/23898006/diff/7001/chrome/browser/extensions/...
chrome/browser/extensions/api/feedback_private/blob_reader.cc:20: //
TODO(michaeln): remove support for deprecated blob urls
On 2013/09/17 02:01:03, Rahul Chaturvedi wrote:
> We probably don't need to have support for these. The only user for this class
> is the feedback API; since this CL updates the feedback API to use uuid's,
there
> shouldn't be anything that would use this class with the deprecated blob
URL's.

These need to be here until the blink changes are in. The "uuids" here contain
internal urls until that happens... its a multi-sided patch tango.

https://codereview.chromium.org/23898006/diff/7001/chrome/browser/extensions/...
chrome/browser/extensions/api/feedback_private/blob_reader.cc:47: // TODO: ask
rkc where fetcher_ is deleted?
On 2013/09/17 02:01:03, Rahul Chaturvedi wrote:
> This is a leak in my code, the fetcher should have been deleted right before
the
> 'delete this' line.

Ok, I'll change the cl to put the fetcher in a scoped ptr.

https://codereview.chromium.org/23898006/diff/7001/chrome/renderer/extensions...
File chrome/renderer/extensions/feedback_private_custom_bindings.cc (right):

https://codereview.chromium.org/23898006/diff/7001/chrome/renderer/extensions...
chrome/renderer/extensions/feedback_private_custom_bindings.cc:43: //
feedback_private_custom_bindings.js. Ask rck where to do that?
On 2013/09/17 02:01:03, Rahul Chaturvedi wrote:
> You'll need to update
>
http://src.chromium.org/viewvc/chrome/trunk/src/chrome/renderer/resources/ext...

Ok, in this CL, i'll update that script too and change this binding layer to say
GetBlobUuid().

[rkc, 2013-09-17 20:40:12]

https://codereview.chromium.org/23898006/diff/7001/chrome/browser/extensions/...
File chrome/browser/extensions/api/feedback_private/blob_reader.cc (right):

https://codereview.chromium.org/23898006/diff/7001/chrome/browser/extensions/...
chrome/browser/extensions/api/feedback_private/blob_reader.cc:20: //
TODO(michaeln): remove support for deprecated blob urls
On 2013/09/17 20:25:23, michaeln wrote:
> On 2013/09/17 02:01:03, Rahul Chaturvedi wrote:
> > We probably don't need to have support for these. The only user for this
class
> > is the feedback API; since this CL updates the feedback API to use uuid's,
> there
> > shouldn't be anything that would use this class with the deprecated blob
> URL's.
> 
> These need to be here until the blink changes are in. The "uuids" here contain
> internal urls until that happens... its a multi-sided patch tango.

This might get complicated then; since the binding script will call either
GetBlobUrl or GetBlobUuid; but depending on what's checked in to Webkit, only
one of those will exist, correct?

[michaeln, 2013-09-17 20:55:06]

> This might get complicated then; since the binding script will call either
> GetBlobUrl or GetBlobUuid; but depending on what's checked in to Webkit, only
> one of those will exist, correct?

There will be only one function, GetBlobUuid(). The return value of the function
will be either an internal url or a uuid depending on the revision of blink.
Once blink rolls, it'll be using uuids. Welcome to multi-sided patch
engineering.

[michaeln, 2013-09-17 22:22:40]

new patch, ptal

[asargent_no_longer_on_chrome, 2013-09-17 22:29:20]

https://codereview.chromium.org/23898006/diff/7001/chrome/renderer/extensions...
File chrome/renderer/extensions/feedback_private_custom_bindings.cc (right):

https://codereview.chromium.org/23898006/diff/7001/chrome/renderer/extensions...
chrome/renderer/extensions/feedback_private_custom_bindings.cc:27: // TODO: find
where that generic extension plumbing happens? I'm looking for
On 2013/09/17 02:01:03, Rahul Chaturvedi wrote:
> asargent@ would know more about how this plumbing works.

Mini "life of an extension/app API call":

Bindings get set up by chrome/renderer/resources/extensions/binding.js based on
the schema files in chrome/common/extensions/api/*.{json,idl}. Additionally
there can be custom bindings set up by
chrome/renderer/resources/extensions/<APINAME>_custom_bindings.js.

The JS function
sendRequest(chrome/renderer/resources/extensions/send_request.js) handles most
regular API calls, and it saves a reference to the callback (if any), packages
the arguments and function name called and jumps into C++ code which sends a
ExtensionHostMsg_Request or ExtensionHostMsg_RequestForIOThread IPC to the
browser process. 

In the browser process, the processing of the IPC ends up in some concrete
subclass of ExtensionFunction (created based on the string with the function
name in it). There are a few subclasses of ExtensionFunction for which thread
you want to handle the request on, including UIThreadExtensionFunction (most
APIs use this) and IOThreadExtensionFunction (most do not use this). Also
UIThreadExtensionFunction has some more subclases (SyncExtensionFunction and
AsyncExtensionFunction) for the 2 cases of when you can immediately respond in
the current spin of the message loop or need to respond later. We even have a
set of helper classes that make it easy to declare where you want your work to
take place - it's "ApiFunction" defined in
chrome/browser/extensions/api/api_function.h. All you need to do is inherit from
that and call set_work_thread_id() with the right thread id in your Prepare
method. 

Anyhow, typically work is done for the API call by this subclass and any result
is put into the results_ member variable, and we end up sending a
ExtensionMsg_Response IPC back to the renderer, that ends up in the JS function
handleResponse (again in chrome/renderer/resources/extensions/send_request.js)
that will pass any result data to the callback saved away in sendResponse.

[michaeln, 2013-09-17 23:52:18]

https://codereview.chromium.org/23898006/diff/7001/chrome/renderer/extensions...
File chrome/renderer/extensions/feedback_private_custom_bindings.cc (right):

https://codereview.chromium.org/23898006/diff/7001/chrome/renderer/extensions...
chrome/renderer/extensions/feedback_private_custom_bindings.cc:27: // TODO: find
where that generic extension plumbing happens? I'm looking for
> Mini "life of an extension/app API call":

Thnx for the pointers!

Not for this CL, but I'm intersted in being able to "do something" with the args
on the IO thread prior to processing on the <whatever> thread the function is
specified to run on. These notes will be useful later.

[michaeln, 2013-09-18 18:15:35]

@kinuko and @rkc... ping

[rkc, 2013-09-18 22:09:39]

lgtm

[kinuko, 2013-09-19 00:33:24]

lgtm

[michaeln, 2013-09-19 23:47:32]

@piman for content owners review

[piman, 2013-09-20 01:11:13]

https://codereview.chromium.org/23898006/diff/14015/content/common/DEPS
File content/common/DEPS (right):

https://codereview.chromium.org/23898006/diff/14015/content/common/DEPS#newco...
content/common/DEPS:15: "+third_party/WebKit/public/platform/WebBlobData.h",
This is not a stricly enum/POD...

https://codereview.chromium.org/23898006/diff/14015/content/common/page_state...
File content/common/page_state_serialization.h (right):

https://codereview.chromium.org/23898006/diff/14015/content/common/page_state...
content/common/page_state_serialization.h:12: #include
"third_party/WebKit/public/platform/WebBlobData.h"
It doesn't look like you're using this.

[michaeln, 2013-09-20 19:17:58]

https://codereview.chromium.org/23898006/diff/14015/content/common/DEPS
File content/common/DEPS (right):

https://codereview.chromium.org/23898006/diff/14015/content/common/DEPS#newco...
content/common/DEPS:15: "+third_party/WebKit/public/platform/WebBlobData.h",
On 2013/09/20 01:11:14, piman wrote:
> This is not a stricly enum/POD...

whats being used from this .h file is the #define

https://codereview.chromium.org/23898006/diff/14015/content/common/page_state...
File content/common/page_state_serialization.h (right):

https://codereview.chromium.org/23898006/diff/14015/content/common/page_state...
content/common/page_state_serialization.h:12: #include
"third_party/WebKit/public/platform/WebBlobData.h"
On 2013/09/20 01:11:14, piman wrote:
> It doesn't look like you're using this.

see #ifdef USE_BLOB_UUIDS

Alternatively, i could #include WebBlobData.h in WebHTTPBody.h.

[piman, 2013-09-20 19:49:46]

On Fri, Sep 20, 2013 at 12:17 PM, <[email protected]> wrote:

>
> https://codereview.chromium.**org/23898006/diff/14015/**
>
content/common/DEPS<https://codereview.chromium.org/23898006/diff/14015/content/common/DEPS>
> File content/common/DEPS (right):
>
> https://codereview.chromium.**org/23898006/diff/14015/**
>
content/common/DEPS#newcode15<https://codereview.chromium.org/23898006/diff/14015/content/common/DEPS#newcode15>
> content/common/DEPS:15:
> "+third_party/WebKit/public/**platform/WebBlobData.h",
> On 2013/09/20 01:11:14, piman wrote:
>
>> This is not a stricly enum/POD...
>>
>
> whats being used from this .h file is the #define


Which #define? I don't see USE_BLOB_UUIDS in my WebBlobData.h


>
> https://codereview.chromium.**org/23898006/diff/14015/**
>
content/common/page_state_**serialization.h<https://codereview.chromium.org/23898006/diff/14015/content/common/page_state_serialization.h>
> File content/common/page_state_**serialization.h (right):
>
> https://codereview.chromium.**org/23898006/diff/14015/**
>
content/common/page_state_**serialization.h#newcode12<https://codereview.chromium.org/23898006/diff/14015/content/common/page_state_serialization.h#newcode12>
> content/common/page_state_**serialization.h:12: #include
> "third_party/WebKit/public/**platform/WebBlobData.h"
> On 2013/09/20 01:11:14, piman wrote:
>
>> It doesn't look like you're using this.
>>
>
> see #ifdef USE_BLOB_UUIDS
>
> Alternatively, i could #include WebBlobData.h in WebHTTPBody.h.
>

Ok, it sounds like it would be better. page_state_serialization.h doesn't
directly depend on WebBlobData.h, but it needs WebHTTPBody.h, and that's
already in the DEPS


>
https://codereview.chromium.**org/23898006/<https://codereview.chromium.org/2...
>

To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].

[michaeln, 2013-09-20 20:23:43]

> Which #define? I don't see USE_BLOB_UUIDS in my WebBlobData.h

(sigh) please read the cl description

[michaeln, 2013-09-20 20:29:02]

here's the next patch in the tango-dance...
https://codereview.chromium.org/23992003/

I've got WebHTTPBody.h including WebBlobData.h in that CL so i'll undo the DEPS
addition for that in this CL.

[piman, 2013-09-20 21:32:27]

On 2013/09/20 20:23:43, michaeln wrote:
> > Which #define? I don't see USE_BLOB_UUIDS in my WebBlobData.h
> 
> (sigh) please read the cl description

A link to the webkit CL would have made it obvious.

[michaeln, 2013-09-20 22:49:02]

removed the DEPS addition

[piman, 2013-09-20 22:50:06]

lgtm

[commit-bot: I haz the power, 2013-09-20 22:57:47]

CQ is trying da patch. Follow status at
https://chromium-status.appspot.com/cq/[email protected]/23898006/81001

[commit-bot: I haz the power, 2013-09-21 01:42:29]

Retried try job too often on win_rel for step(s) base_unittests, browser_tests,
cacheinvalidation_unittests, cc_unittests, check_deps, chrome_frame_net_tests,
chrome_frame_tests, chrome_frame_unittests, chromedriver2_unittests,
components_unittests, content_browsertests, content_unittests, crypto_unittests,
gpu_unittests, interactive_ui_tests, ipc_tests, jingle_unittests,
media_unittests, mini_installer_test, nacl_integration, net_unittests,
ppapi_unittests, printing_unittests, remoting_unittests, sql_unittests,
sync_integration_tests, sync_unit_tests, telemetry_unittests, unit_tests
http://build.chromium.org/p/tryserver.chromium/buildstatus?builder=win_rel&nu...

[commit-bot: I haz the power, 2013-09-23 19:00:56]

CQ is trying da patch. Follow status at
https://chromium-status.appspot.com/cq/[email protected]/23898006/81001

[commit-bot: I haz the power, 2013-09-24 00:08:17]

Retried try job too often on win_rel for step(s) sync_integration_tests
http://build.chromium.org/p/tryserver.chromium/buildstatus?builder=win_rel&nu...

[michaeln, 2013-09-24 00:43:57]

The CQ is not useful. More often than not, i regret trying to use it.

[commit-bot: I haz the power, 2013-09-24 00:46:32]

CQ is trying da patch. Follow status at
https://chromium-status.appspot.com/cq/[email protected]/23898006/111001

[commit-bot: I haz the power, 2013-09-24 04:59:59]

Change committed as 224906