feat: native plugins #3372

afinch7 · 2019-11-18T17:52:58Z

Now that we are using futures 0.3 I should be able to finish this.

Api will look a little different this time to reflect some of the refactors that have occurred since #2385.

I'm targeting something like this on the plugin side:

use deno::CoreOp;
use deno::Op;
use deno::{Buf, PinnedBuf};
use deno::PluginInitContext;
use futures::future::FutureExt;

#[macro_use]
extern crate deno;

fn init(context: &mut dyn PluginInitContext) {
  context.register_op("some_op", Box::new(some_op));
}
init_fn!(init);

pub fn some_op(data: &[u8], zero_copy: Option<PinnedBuf>) -> CoreOp {
  if let Some(buf) = zero_copy {
    let data_str = std::str::from_utf8(&data[..]).unwrap();
    let buf_str = std::str::from_utf8(&buf[..]).unwrap();
    println!(
      "Hello from native bindings. data: {} | zero_copy: {}",
      data_str, buf_str
    );
  }
  let result = b"test";
  Op::Sync(result.into())
}

and something like this on the typescript side:

const plugin = Deno.openPlugin("./path/to/some/lib.so");
const some_op = plugin.ops.some_op;

function doSomeOp(): null | ArrayBufferView {
  const response = some_op.dispatch(new UInt8Array([1,2,3,4]));
  return response;
}

refs #3366 #2473

bartlomieju

There are a few things I have questions about, but I believe this might be very solid MVP for native plugins;
a) it's built on ops
b) this is the simplest iteration, doesn't expose state to op dispatcher in Rust, we can iterate on API safely

cli/ops/native_plugins.rs

ry

I like that this is only +400 lines


        wip: native plugin ops


        implement native plugins in typescript


        fix flags tests and adjust permissions for install


        wip: add test plugin


        rebase fixes


        added tests and fixed a issue with types

afinch7 · 2019-11-20T00:21:43Z

For some reason workers_startup_bench.ts doesn't terminate now. I don't think that I changed anything that would break that.


        refactor `InitContext`

cli/worker.rs

@bartlomieju


        patch by @bartlomieju


        fix issue with `setAsyncHandler` and added simple async test


        make sure test native plugin gets built for tests


        fix debug test

bartlomieju

It is very solid PR, especially because all changes introduced in it are minimal - it allows to iterate on the API before 1.0 release. It's well aligned with our ops APIs and has minimal impact on current architecture 👍 I'd love to see this landed to give users chance to use it and break it 🚀

bartlomieju · 2019-11-22T23:32:41Z

.github/workflows/ci.yml

@@ -129,6 +129,7 @@ jobs:
      - name: Test debug
        if: matrix.kind == 'test_debug'
        run: |
+          cargo build --locked -p test_native_plugin


Can this step be somehow integrated into cli/tests/integration_tests.rs flow?

I tried to add a single test to test_native_plugin, but it still doesn't build automatically when running cargo test --all-targets. Not sure if there may be some other simple workaround here.

We should figure out a way to test this without making changes to ci.yml... I'm looking into it.

cli/flags.rs

cli/js/dispatch.ts

cli/js/native_plugins.ts

cli/ops/native_plugins.rs

cli/permissions.rs

cli/tests/test_native_plugin/src/lib.rs


        review fixes and other misc fixes

* Rename `allow_native`/`allow-native` to `allow_plugin`/`allow-plugin`. This makes more sense with the primary public api function being `openPlugin`. * Fully intagrate the newly added permission type `plugin`. * Add docs to `Deno.openPlugin`. * Generate op names for native plugin ops that avoid potential collision. Lastly this commit adds a delayed future to native plugins async test op. This should properly test the function of contexts/tasks/wakers in async native plugin ops. This would not function correctly with futures 0.1, and thus was a major blocker. This test proves this issue no longer exists in this implementation. The most analogous problem I can think of here is loading two copies of the same TS lib: the share the same type thus are interoperable, but don't share the same module local data(closest comparison to TLS used to store current task in futures). This isn't a problem anymore, since futures now pass `Context` values directly during calls to `poll` with the new API.


        Move test_native_plugin to root - add failing test


        Merge branch 'master' into native_plugins_2

fix

ry · 2019-11-27T16:40:33Z

I've merged this branch with master and moved a few things around. I think the plugin infrastructure shouldn't depend on cli, so the tests shouldn't be in cli. I'll try to do that now...


        test_plugin WIP


        Fix test_plugin


        clean up


        clean up

ry · 2019-11-28T17:18:47Z

So.. I'm still having the same trouble you were having, @afinch7, with getting the plugin to build for the test. I guess moving the test_plugin crate to the root didn't help. I've done some superficial clean ups. I'm pretty much ready to land this if we can get it to go green somehow.


        fix scoping in plugin test


        Merge remote-tracking branch 'upstream/master' into native_plugins_2


        fix another issue in plugin test


        fixes

* Removed "native" wording from plugins * Build `test_plugin` before running debug tests in CI


        build test plugin during test


        fix windows issue

ry · 2019-12-03T23:31:37Z

test_plugin/tests/integration_tests.rs

+fn basic() {
+  let mut build_plugin_base = Command::new("cargo");
+  let mut build_plugin =
+    build_plugin_base.arg("build").arg("-p").arg("test_plugin");


It's annoying that it's not built already when the integration tests run. I wish I knew some cargo expert who could advise us here.

ry

LGTM - Thank you for the long and hard work of getting this in @afinch7 !


        feat: first pass at native plugins (denoland#3372)


        feat: first pass at native plugins (denoland#3372)

afinch7 force-pushed the afinch7:native_plugins_2 branch 2 times, most recently from fef9c0c to 661e5a4 Nov 18, 2019

afinch7 mentioned this pull request Nov 18, 2019

feat: op registration during calls #3375

Merged

bartlomieju reviewed Nov 18, 2019

View changes

cli/ops/native_plugins.rs Outdated Show resolved Hide resolved

cli/ops/native_plugins.rs Outdated Show resolved Hide resolved

afinch7 force-pushed the afinch7:native_plugins_2 branch from 4cdeeb2 to 51910ee Nov 18, 2019

bartlomieju reviewed Nov 18, 2019

View changes

cli/ops/native_plugins.rs Outdated Show resolved Hide resolved

ry reviewed Nov 19, 2019

View changes

afinch7 force-pushed the afinch7:native_plugins_2 branch 2 times, most recently from 7df0cbd to 1fc95be Nov 19, 2019

afinch7 added 6 commits Nov 18, 2019

wip: native plugin ops

4a11ec3

implement native plugins in typescript

6758ea9

fix flags tests and adjust permissions for install

7dc9fed

wip: add test plugin

631b257

rebase fixes

fe8ed8d

added tests and fixed a issue with types

Loading status checks…

112622d

afinch7 force-pushed the afinch7:native_plugins_2 branch from 1fc95be to 112622d Nov 19, 2019

afinch7 marked this pull request as ready for review Nov 19, 2019

refactor `InitContext`

Loading status checks…

cb94110

afinch7 force-pushed the afinch7:native_plugins_2 branch from 45f301d to cb94110 Nov 20, 2019

bartlomieju reviewed Nov 22, 2019

View changes

cli/worker.rs Outdated Show resolved Hide resolved

afinch7 added 3 commits Nov 22, 2019

patch by @bartlomieju

Loading status checks…

2913dc9

fix issue with `setAsyncHandler` and added simple async test

Loading status checks…

496329a

make sure test native plugin gets built for tests

Loading status checks…

62c126c

afinch7 force-pushed the afinch7:native_plugins_2 branch from 5c796ea to 62c126c Nov 22, 2019

fix debug test

Loading status checks…

102831e

afinch7 changed the title ~~[WIP] feat: native plugins~~ feat: native plugins Nov 22, 2019

bartlomieju reviewed Nov 22, 2019

View changes

afinch7 force-pushed the afinch7:native_plugins_2 branch from f780f53 to 0bb50b3 Nov 23, 2019

ry added 3 commits Nov 24, 2019

Move test_native_plugin to root - add failing test

Verified

This commit was signed with a verified signature.

ry Ryan Dahl

GPG key ID: C6F2E2494922A6BA Learn about signing commits

2a38515

Merge branch 'master' into native_plugins_2

Verified

This commit was signed with a verified signature.

ry Ryan Dahl

GPG key ID: C6F2E2494922A6BA Learn about signing commits

Loading status checks…

a2e3783

fix

Verified

This commit was signed with a verified signature.

ry Ryan Dahl

GPG key ID: C6F2E2494922A6BA Learn about signing commits

Loading status checks…

8901da5

ry added 4 commits Nov 27, 2019

test_plugin WIP

Verified

This commit was signed with a verified signature.

ry Ryan Dahl

GPG key ID: C6F2E2494922A6BA Learn about signing commits

e0004c3

Fix test_plugin

Verified

This commit was signed with a verified signature.

ry Ryan Dahl

GPG key ID: C6F2E2494922A6BA Learn about signing commits

Loading status checks…

ae129f7

clean up

Verified

This commit was signed with a verified signature.

ry Ryan Dahl

GPG key ID: C6F2E2494922A6BA Learn about signing commits

e0dcb0b

clean up

Verified

This commit was signed with a verified signature.

ry Ryan Dahl

GPG key ID: C6F2E2494922A6BA Learn about signing commits

Loading status checks…

26bd330

afinch7 added 6 commits Dec 3, 2019

fix scoping in plugin test

Loading status checks…

33b749f

Merge remote-tracking branch 'upstream/master' into native_plugins_2

Loading status checks…

a9df117

fix another issue in plugin test

Loading status checks…

10ad75c

fixes

Loading status checks…

230e926

* Removed "native" wording from plugins * Build `test_plugin` before running debug tests in CI

build test plugin during test

Loading status checks…

b96ec38

fix windows issue

Loading status checks…

ad9f9e2

ry reviewed Dec 3, 2019

View changes

bartlomieju approved these changes Dec 5, 2019

View changes

ry approved these changes Dec 5, 2019

View changes

ry mentioned this pull request Dec 6, 2019

1.0 #2473

Closed

40 of 43 tasks complete

cztomsik mentioned this pull request Dec 7, 2019

support deno #131

Open

nayeemrmn mentioned this pull request Dec 17, 2019

Importing a native module #296

Closed

bartlomieju added a commit to bartlomieju/deno that referenced this pull request Dec 28, 2019

feat: first pass at native plugins (denoland#3372)

8a9b96f

eliassjogreen mentioned this pull request Jan 15, 2020

Writing native plugin in pure C #3598

Open

linhub15 added a commit to linhub15/deno that referenced this pull request Feb 13, 2020

feat: first pass at native plugins (denoland#3372)

Loading status checks…

466032f

kitsonk mentioned this pull request Feb 15, 2020

plugin test is flaky on windows #3473

Closed

rektide mentioned this pull request Mar 9, 2020

Dispatch and native plugins #2987

Closed

This was referenced May 7, 2020

meta: will Deno be used in pipcook? alibaba/pipcook#172

Open

boa: create a new DSL to generate JavaScript native extension alibaba/pipcook#174

Open

denoland / deno

feat: native plugins #3372

feat: native plugins #3372

afinch7 commented Nov 18, 2019

bartlomieju left a comment

ry left a comment

afinch7 commented Nov 20, 2019

bartlomieju left a comment

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

ry commented Nov 27, 2019

ry commented Nov 28, 2019

This comment has been minimized.

ry left a comment

denoland / deno

Join GitHub today

feat: native plugins #3372

feat: native plugins #3372

Conversation

afinch7 commented Nov 18, 2019

bartlomieju left a comment

ry left a comment

afinch7 commented Nov 20, 2019

bartlomieju left a comment

This comment has been minimized.

bartlomieju Nov 22, 2019

This comment has been minimized.

afinch7 Nov 23, 2019

This comment has been minimized.

ry Nov 24, 2019

ry commented Nov 27, 2019

ry commented Nov 28, 2019

This comment has been minimized.

ry Dec 3, 2019

ry left a comment