Join GitHub today
GitHub is home to over 50 million developers working together to host and review code, manage projects, and build software together.
Sign upUnexpected behaviour of tokenizer.to_disk compared to documentation #5256
Labels
Comments
|
I think you're right, and the documentation needs to change. |
|
take |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
More precisely, if, for some reason, someone uses
tokenizer.to_disk(path)as follows:a
PermissionErroris raised, whereasworks.
Judging from from this and the source code,
pathis opened as if it was a file.The documentation of
tokenizer.to_disk, however, describes it to require a path to a directory as argument. This seems to be inconsistent with its actual behaviour.Your Environment