feat(runtime): add a JavaScript evaluation API with named arguments, keeping the old one with warnings

CHANGELOG.md

@@ -50,6 +50,8 @@
   `429 Too Many Requests` for a full remote browser service, and exposes it as `#status`
 
 ### Fixed
+- `Node#property` and `Node#attribute` read any name, a quote in it no longer breaks the script, and `property`
+  returns a `Ferrum::Node` for a property holding one instead of `{}` [#673]
 - `evaluate`, `execute` and friends raise `JavaScriptError` when JS throws something other than an `Error`, e.g.
   `throw "smth"`, instead of returning it. Returning an `Error` object no longer crashes with `NoMethodError`
 - `evaluate` keeps `null` and `undefined` in arrays and returns `NaN`, `Infinity`, `-0` and `BigInt` instead of `nil`,
@@ -117,6 +119,9 @@
   loaded (on CDP's `Network.loadingFinished`, not `Network.responseReceived`, so `exchange.response.body` is always
   available), yielding the request's `Network::Exchange`. Doesn't require `network.intercept` to be set up, and is
   never fired for requests that fail to load [#294]
+- `#evaluate`/`#execute` take named arguments that become the script's parameters, `page.evaluate("a + b", a: 1,
+  b: 2)`, a function declaration as the script with positional arguments, `await`, and `timeout:` [#673]
+- `#evaluate_handle` returning a `Ferrum::RemoteObject`, and `Node#execute`/`Node#evaluate_handle` [#673]
 - `Ferrum::Browser#quit`/`Ferrum::Browser::Process#stop` accept `wait: false` to return immediately and run process
   killing and user-data-directory cleanup on a background thread instead of blocking; the call returns the `Thread`
   so callers can `#join` it if they need cleanup to have finished, e.g. before process exit or before reusing a
@@ -132,6 +137,10 @@
 - `Ferrum::Page::Stream#stream` now closes the CDP stream handle (`IO.close`) once it's been fully read, so streams
   opened for `Ferrum::Browser#pdf` and `Ferrum::Page::Tracing#record` no longer keep their backing storage alive in
   the browser; a failed `IO.close` is raised to the caller.
+- `#evaluate_async`, `#evaluate_func` and `#evaluate_on` are deprecated, and scripts whose result will change in
+  0.21 warn with a rewrite. 0.21 removes the old behavior, and `FERRUM_JS_API=new` runs the new one now, see
+  [upgrading](docs/13-javascript.md#upgrading-to-021) [#673]
+- `Ferrum::ScriptTimeoutError` inherits from `Ferrum::TimeoutError` [#673]
 
 ### Fixed
 - An out-of-process iframe, e.g. Chrome's PDF viewer for an embedded PDF, appearing while a web worker was connected

README.md

@@ -85,6 +85,55 @@ JS
 browser.quit
 ```
 
+Pass arguments by name, so the script can use parameters instead of digging
+through `arguments[0]`:
+
+```ruby
+page.evaluate("a + b", a: 1, b: 2) # => 3
+page.evaluate("el.getAttribute(name)", el: page.at_css("a"), name: "href")
+```
+
+A script that is a function or arrow declaration is used as-is and takes its
+arguments positionally, in the order it declares them:
+
+```ruby
+page.evaluate(<<~JS, 1, 2)
+  function(a, b) {
+    const sum = a + b;
+    return sum * 3;
+  }
+JS
+```
+
+`await` works directly, and a function declaration's returned promise is
+awaited:
+
+```ruby
+page.evaluate("await fetch(url).then(r => r.text())", url: "/api")
+```
+
+Pass `timeout:` (seconds, defaulting to the page timeout) to bound how long the
+script may take before `Ferrum::ScriptTimeoutError` is raised. `timeout:` is the
+only reserved keyword.
+
+Use `#execute` when you only want the side effects, `#evaluate_handle` when you
+want to keep a value in the browser and pass it back in later, and
+`Ferrum::Node#evaluate` to run a script with `this` bound to an element:
+
+```ruby
+page.execute("window.scrollBy(0, 100)") # => true
+
+list = page.evaluate_handle("document.querySelectorAll('li')")
+page.evaluate("Array.from(nodes).map(n => n.textContent)", nodes: list)
+
+page.at_css("input").evaluate("this.value")
+```
+
+Scripts written for the old API keep working, and Ferrum warns, with a suggested
+rewrite, about any whose result will change. The next release, 0.21, removes the
+old behavior, so run your suite with `FERRUM_JS_API=new` to check it's ready. See
+[upgrading to 0.21](docs/13-javascript.md#upgrading-to-021).
+
 Do any mouse movements you like:
 
 ```ruby

docs/13-javascript.md

@@ -4,48 +4,89 @@ sidebar_position: 13
 
 # JavaScript
 
-#### evaluate(expression, \*args)
+#### evaluate(expression, \*args, timeout: nil, \*\*named)
 
-Evaluate and return result for given JS expression
+Evaluate JavaScript and return the result, converted to Ruby.
 
-* expression `String` should be valid JavaScript
-* args `Object` you can pass arguments, though it should be a valid `Node` or a
-  simple value.
+* expression `String` JavaScript returning the value of its last statement, or a function/arrow declaration, which
+  is used as-is
+* args positional arguments, a declaration's parameters or `arguments[0]` and up. Can't be mixed with named ones
+* timeout `Numeric` seconds to wait for the script and any promise it returns, defaults to the page timeout. Raises
+  `Ferrum::ScriptTimeoutError` when exceeded, which inherits from `Ferrum::TimeoutError`
+* named named arguments, which become the parameters of a script that isn't a declaration. A `Ferrum::Node` arrives
+  as the live element, but can't be put inside an Array or Hash. `timeout` can't be a parameter name
 
 ```ruby
-page.evaluate("[window.scrollX, window.scrollY]")
+page.evaluate("[window.scrollX, window.scrollY]") # => [0, 0]
+page.evaluate("a + b", a: 1, b: 2) # => 3
+page.evaluate("el.getAttribute(name)", el: page.at_css("a"), name: "href")
+page.evaluate("await fetch(url).then(r => r.status)", url: "/") # => 200
+page.evaluate(<<~JS, 3) # => 9
+  function(c) {
+    const sum = 1 + 2;
+    return sum * c;
+  }
+JS
 ```
 
 The result is converted to Ruby: objects become `Hash`, arrays and node lists `Array`, DOM nodes `Ferrum::Node`,
 dates ISO 8601 strings, `NaN` and `Infinity` `Float`, `BigInt` `Integer`, `null` and `undefined` `nil`. Values
 without a Ruby counterpart, such as functions, `Map`, `Set` or `window`, become `{}`. A reference back to an
 enclosing object, e.g. `a.self = a`, becomes `Ferrum::CyclicObject`.
 
-#### evaluate_async(expression, wait_time, \*args)
-
-Evaluate asynchronous expression and return result
+#### evaluate_handle(expression, \*args, timeout: nil, \*\*named) : `Ferrum::RemoteObject`
 
-* expression `String` should be valid JavaScript
-* wait_time How long we should wait for Promise to resolve or reject
-* args `Object` you can pass arguments, though it should be a valid `Node` or a
-  simple value.
+Same as `evaluate`, but keeps the value in the browser and returns a reference you can pass back in as an argument.
+Call `#value` to convert it to Ruby, and `#release` to free it.
 
 ```ruby
-page.evaluate_async(%(arguments[0]({foo: "bar"})), 5) # => { "foo" => "bar" }
+list = page.evaluate_handle("document.querySelectorAll('li')")
+page.evaluate("Array.from(nodes).map(n => n.textContent)", nodes: list)
 ```
 
-#### execute(expression, \*args)
+#### execute(expression, \*args, timeout: nil, \*\*named) : `Boolean`
+
+Run JavaScript for its side effects. The script is used as a function body, so statements need no wrapping.
 
-Execute expression. Doesn't return the result
+```ruby
+page.execute("window.scrollBy(0, 100)") # => true
+```
 
-* expression `String` should be valid JavaScript
-* args `Object` you can pass arguments, though it should be a valid `Node` or a
-  simple value.
+`Ferrum::Node` has the same three methods, with `this` bound to the element. Arrow functions don't get `this`, so use a
+`function` declaration when you need it.
 
 ```ruby
-page.execute(%(1 + 1)) # => true
+page.at_css("input").evaluate("this.value")
+page.at_css("input").execute("this.value = text", text: "hello")
 ```
 
+#### Upgrading to 0.21
+
+Scripts written for the old API run exactly as before, and Ferrum warns, pointing at your code, about any whose result
+will change. Calls that only the new API can make, e.g. with named arguments, already run the new way.
+
+* 0.20 runs old-style scripts the old way and warns. Set `FERRUM_JS_API=new` to run everything the new way.
+* 0.21 removes the old way, along with `evaluate_async`, `evaluate_func` and `evaluate_on`. There's no switch back.
+
+Warnings only cover the scripts and results your suite actually runs into, so before upgrading to 0.21, run the suite
+once with `FERRUM_JS_API=new`. Set `FERRUM_DEPRECATION_WARNINGS=raise` to turn the warnings into errors while
+migrating, or `0` to silence them. A library built on Ferrum should pass named arguments or a function declaration,
+which run the new way in both 0.20 and 0.21.
+
+| Script | < 0.21 | >= 0.21 | Notes |
+|---|---|---|---|
+| `evaluate("arguments[0].foo", foo: 1)` | `1` | `nil` | keywords are named parameters now, write `evaluate("foo", foo: 1)` |
+| `evaluate("[arguments[0], arguments[1]]", 1, foo: 2)` | `[1, { "foo" => 2 }]` | `ArgumentError` | positional and keyword arguments can't be mixed, pass `{ foo: 2 }` in braces |
+| `evaluate("a(); b")` | `a()`'s value, `b` never runs | `b`'s value | every statement runs, the last one's value is returned |
+| `evaluate("\n  window.ran = true\n")` | `nil`, nothing runs | `true` | the line break no longer ends the `return`, check it's safe to run |
+| `evaluate("Promise.resolve(1)")` | `{}` | `1` | promises are awaited, `void` keeps one from being awaited |
+| `node.evaluate("this.parentNode")` | `{}` | `Ferrum::Node` | |
+| `node.evaluate("new Date(0)")` | `{}` | `"1970-01-01T00:00:00.000Z"` | |
+| `evaluate_async("arguments[0](1)", 5)` | `1` | `NoMethodError` | `evaluate("await …", timeout: 5)` |
+| `evaluate_func("function(a) { return a }", 1)` | `1` | `NoMethodError` | `evaluate("function(a) { return a }", 1)`, with `on:` use `node.evaluate` |
+| `evaluate_on(node: node, expression: "this.id")` | `"main"` | `NoMethodError` | `node.evaluate("this.id")` |
+| `evaluate_on(node: link, expression: "this.click()", wait: 0.1)` | waits for the new page | `NoMethodError` | `page.wait_for_navigation!(0.1) { link.evaluate("this.click()") }` |
+
 #### evaluate_on_new_document(source, \*\*options) : `String`
 
 Runs JavaScript in every new document of the page and its in-process frames, before the document's own scripts.
@@ -67,7 +108,7 @@ JS
 #### isolated_world(name) : `IsolatedWorld`
 
 Returns the frame's isolated world with this name, its own JavaScript globals over the frame's DOM, so the page's
-scripts can't see or override what runs there. It has `evaluate`, `evaluate_async`, `execute` and `evaluate_func`,
+scripts can't see or override what runs there. It has `evaluate`, `evaluate_handle` and `execute`,
 takes and returns nodes, and comes back empty in every new document. On a page it's the main frame's world, use
 `frame.isolated_world(name)` for an iframe. It's the same world `evaluate_on_new_document` runs in with
 `world_name:`.

docs/21-workers.md

@@ -36,7 +36,7 @@ Returns service worker targets registered in this context. Unlike `workers`, the
 are not connected to -- call `attach_target` first if you want to interact with one.
 
 ```ruby
-page.evaluate_async(%(navigator.serviceWorker.register("/sw.js").then(arguments[0])), 5)
+page.evaluate(%(await navigator.serviceWorker.register("/sw.js")))
 
 sleep 0.1
 
@@ -91,7 +91,7 @@ Same API as `page.console` -- see [JavaScript](/docs/ferrum/javascript#console).
 worker.console.map(&:text) # => ["from worker"]
 ```
 
-#### evaluate / execute / evaluate_async
+#### evaluate / execute / evaluate_handle
 
 Same API as on `Page`/`Frame` -- see [JavaScript](/docs/ferrum/javascript). A worker has a single execution context,
 so there's no frame to target.

lib/ferrum.rb

@@ -7,8 +7,10 @@
 require "ferrum/utils/platform"
 require "ferrum/utils/elapsed_time"
 require "ferrum/utils/attempt"
+require "ferrum/utils/deprecate"
 require "ferrum/errors"
 require "ferrum/browser"
+require "ferrum/remote_object"
 require "ferrum/node"
 
 #

lib/ferrum/browser.rb

@@ -37,7 +37,8 @@ class Browser
                 screenshot pdf mhtml viewport_size device_pixel_ratio
                 start_screencast stop_screencast
                 frames frame_by main_frame
-                evaluate evaluate_on evaluate_async execute evaluate_func isolated_world
+                evaluate evaluate_handle execute
+                evaluate_on evaluate_async evaluate_func isolated_world
                 add_script_tag add_style_tag bypass_csp
                 evaluate_on_new_document remove_script_on_new_document
                 on position position=

lib/ferrum/errors.rb

@@ -87,7 +87,7 @@ def awaited
   end
 
   # Raised when an evaluated script takes too long to return a value.
-  class ScriptTimeoutError < Error
+  class ScriptTimeoutError < TimeoutError
     #
     # Explains that the evaluated script timed out.
     #

lib/ferrum/frame.rb

@@ -167,12 +167,11 @@ def parent
     #   frame.body # => <html><head></head><body><p>lol</p></body></html>
     #
     def content=(html)
-      evaluate_async(%(
+      execute(<<~JS, html: html)
         document.open();
-        document.write(arguments[0]);
+        document.write(html);
         document.close();
-        arguments[1](true);
-      ), @page.timeout, html)
+      JS
       @page.document_node_id
     end
     alias set_content content=