Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fs: add recursive cp method #39372

Closed
wants to merge 1 commit into from
Closed

fs: add recursive cp method #39372

wants to merge 1 commit into from

Conversation

@bcoe
Copy link
Member

@bcoe bcoe commented Jul 13, 2021

Introduces recursive cp method, based on fs-extra implementation

Refs: nodejs/tooling#98
Fixes: #35880


Opening to start conversation.

TODO:

  • still have many tests to write, to increase coverage. (coverage is at 92%).
  • need to document sync and promise API.

CC: @nodejs/tooling, @jprichardson, @manidlou, @RyanZim

test/parallel/test-copy.js Outdated Show resolved Hide resolved
lib/internal/fs/copy/copy.js Outdated Show resolved Hide resolved
lib/internal/fs/copy/copy.js Outdated Show resolved Hide resolved
@bcoe bcoe force-pushed the bcoe:copy branch from bb5fe4a to 4107b45 Jul 13, 2021
@nodejs-github-bot

This comment has been hidden.

lib/internal/fs/promises.js Outdated Show resolved Hide resolved
lib/internal/fs/promises.js Outdated Show resolved Hide resolved
lib/internal/fs/copy/copy.js Outdated Show resolved Hide resolved
lib/internal/fs/copy/copy.js Outdated Show resolved Hide resolved
lib/internal/fs/copy/copy-sync.js Outdated Show resolved Hide resolved
Copy link
Member

@mcollina mcollina left a comment

lgtm

Copy link
Member

@targos targos left a comment

Just making sure this doesn't land without documentation for the Promises and sync APIs

@nodejs-github-bot

This comment has been hidden.

@nodejs-github-bot

This comment has been hidden.

Copy link
Member

@iansu iansu left a comment

Looks good overall! 🙌

lib/internal/fs/promises.js Outdated Show resolved Hide resolved
lib/internal/fs/utils.js Outdated Show resolved Hide resolved
@bcoe
Copy link
Member Author

@bcoe bcoe commented Jul 15, 2021

@mcollina @targos small, but significant update, I thought force would be a better option than overwrite for when clobbering the destination folder, and have renamed it to this. Also, force now defaults to false.

@RyanZim
Copy link

@RyanZim RyanZim commented Jul 15, 2021

I thought force would be a better option than overwrite for when clobbering the destination folder, and have renamed it to this. Also, force now defaults to false.

While we're rethinking options, should errorOnExist default to false or true? This is another one of those options that has its value for compat reasons.

@RyanZim
Copy link

@RyanZim RyanZim commented Jul 15, 2021

Also, for reference; discussion thread on how fs-extra should handle the forthcoming naming conflict: jprichardson/node-fs-extra#912

Copy link
Contributor

@aduh95 aduh95 left a comment

Further simplifications on the subject of replacing open + futimes + close with utimes.

lib/internal/fs/copy/copy-sync.js Outdated Show resolved Hide resolved
lib/internal/fs/copy/copy-sync.js Outdated Show resolved Hide resolved
lib/internal/fs/copy/copy.js Outdated Show resolved Hide resolved
lib/internal/fs/copy/copy.js Outdated Show resolved Hide resolved
@bcoe
Copy link
Member Author

@bcoe bcoe commented Jul 15, 2021

@RyanZim @iansu

While we're rethinking options, should errorOnExist default to false or true? This is another one of those options that has its value for compat reasons.

I was tempted to go with false, which I believe would make copying to the same folder twice relatively safe, i.e., you don't clobber files, but it's less of a pain in the neck to recover from a partial copy (you don't have to lookup an option).

Thoughts?

lib/internal/fs/utils.js Outdated Show resolved Hide resolved
@nodejs-github-bot

This comment has been hidden.

doc/api/errors.md Outdated Show resolved Hide resolved
doc/api/errors.md Outdated Show resolved Hide resolved
doc/api/fs.md Outdated Show resolved Hide resolved
lib/internal/fs/copy/copy-sync.js Outdated Show resolved Hide resolved
lib/internal/fs/copy/copy.js Outdated Show resolved Hide resolved
lib/internal/fs/copy/copy.js Outdated Show resolved Hide resolved
@bcoe bcoe force-pushed the bcoe:copy branch from f94a533 to 83a403c Jul 19, 2021
@bcoe
Copy link
Member Author

@bcoe bcoe commented Jul 22, 2021

FYI, waiting on a windows machine at work (there are some hiccups with provisioning). Would happily pair with someone on fixing up the failing windows test, if anyone has a system up and running.

@bcoe bcoe force-pushed the bcoe:copy branch from 5988d2c to 86d3eeb Jul 25, 2021
@bcoe bcoe force-pushed the bcoe:copy branch from 6445160 to 0adfd38 Aug 4, 2021
@bcoe bcoe requested a review from targos Aug 4, 2021
@bcoe
Copy link
Member Author

@bcoe bcoe commented Aug 4, 2021

@targos @manidlou please take another look and make sure I've addressed your comments.

@targos
Copy link
Member

@targos targos commented Aug 4, 2021

Just want to make sure: does it behave like Unix cp when dest exists and is a directory or not?

Example, given:

  • input/a.js is a file
  • output/ is a directory

Execute:

  • cp -r input output -> creates output/input/a.js
  • cp -r input output2 -> creates output2/a.js
@bcoe
Copy link
Member Author

@bcoe bcoe commented Aug 4, 2021

@targos there is not an attempt made to nest a directory under anther directory, as demonstrated in your cp -r input output -> creates output/input/a.js example.

Rather, input will be copied over output, which is the existing behavior of fs-extra copy.

Implementing this folder nesting logic would complicate the copy algorithm, and I'm not convinced it's behavior we need to carry over from cp.

@bcoe
Copy link
Member Author

@bcoe bcoe commented Aug 4, 2021

@targos perhaps a compromise is to call out this difference in documentation, and pointing out that we don't support globs, and the when copying folders it's modeled after:

cp test/ test2/

When copying directories, globs are not currently supported and behavior is similar to cp dir1/ dir2/.

@manidlou
Copy link
Contributor

@manidlou manidlou commented Aug 5, 2021

@bcoe @targos I'd also like to point your attention toward my comment here nodejs/tooling#98 (comment). I think it's worth mentioning it here again, our implementation of copy at fs-extra is a bit different than unix cp command, and we are aware of that but we decided to go with explicitness meaning we forced users to explicitly specify where they want their src to copy to, instead of figuring it out (internally in the code) based on the condition of src and dest.

So, I would say, if we keep it like this, it's a good idea to mention that in the docs to avoid confusion for users.

test/parallel/test-fs-cp.js Outdated Show resolved Hide resolved
@targos
targos approved these changes Aug 5, 2021
@targos
Copy link
Member

@targos targos commented Aug 5, 2021

@nodejs-github-bot

This comment has been hidden.

test/parallel/test-fs-cp.mjs Outdated Show resolved Hide resolved
test/parallel/test-fs-cp.mjs Outdated Show resolved Hide resolved
@jasnell
jasnell approved these changes Aug 6, 2021
Copy link
Member

@jasnell jasnell left a comment

LGTM. Let's make sure we follow up on the path-as-buffer issues, however.

lib/internal/fs/promises.js Outdated Show resolved Hide resolved
@iansu
iansu approved these changes Aug 6, 2021
@joesepi
joesepi approved these changes Aug 6, 2021
Introduces recursive cp method, based on fs-extra implementation.
@bcoe bcoe force-pushed the bcoe:copy branch from e52047b to 63b1c80 Aug 8, 2021
@nodejs-github-bot

This comment has been hidden.

@aduh95
aduh95 approved these changes Aug 8, 2021
if (isPromise(shouldCopy)) {
throw new ERR_INVALID_RETURN_VALUE('boolean', 'filter', shouldCopy);
Comment on lines +148 to +149

This comment has been minimized.

@aduh95

aduh95 Aug 8, 2021
Contributor

Non blocking reflection, but probably something worth discussing before calling the API stable:
I'm not sure checking isPromise is the right thing to do here, %Array.prototype.filter% does not handle promises differently (JSON.stringify([0, 1].filter(async()=>false)) === '[0,1]'). Maybe we could emit a warning instead of throwing? That may help users spot if they have passed by mistake an async function, but wouldn't crash if they just happen to return a Promise object here.

This comment has been minimized.

@targos

targos Aug 9, 2021
Member

The reason that Array.prototype.filter doesn't handle promises differently is mostly historical. I can't think of a correct use case where the user wants to return a Promise and expects it to be treated as a truthy value.

Copy link
Member

@mcollina mcollina left a comment

lgtm

@nodejs-github-bot

This comment has been hidden.

@nodejs-github-bot

This comment has been hidden.

bcoe added a commit that referenced this pull request Aug 12, 2021
Introduces recursive cp method, based on fs-extra implementation.

PR-URL: #39372
Fixes: #35880
Refs: nodejs/tooling#98
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Michaël Zasso <targos@protonmail.com>
Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Ian Sutherland <ian@iansutherland.ca>
@bcoe
Copy link
Member Author

@bcoe bcoe commented Aug 12, 2021

Landed in 87d6fd7

@bcoe bcoe closed this Aug 12, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment