Caused by: java.nio.file.DirectoryNotEmptyException: /basedir/target/jib-cache/tmp/3227117988474356011 #2318

chanseokoh · 2020-03-09T14:30:19Z

Reported on the jib-users Google group.

Error while caching an application layer. It was trying to atomically move (rename) the directory .../tmp/3227117988474356011 to .../layers/<some sha256 hash>. It may be that atomic move is not supported on the filesystem. Interestingly, it is a Unix filesystem.

    at sun.nio.fs.UnixCopyFile.move (UnixCopyFile.java:498)
    at sun.nio.fs.UnixFileSystemProvider.move (UnixFileSystemProvider.java:262)

Even if atomic move is not supported, it's not clear why it failed to delete the directory after copying.

mvn compile jib:build \
-Dimage=${TARGET_GCR_IMAGE} \
-Dbase-container-image=${BASE_CONTAINER_IMAGE} \
-Dbase-container-image.version=${BASE_CONTAINER_IMAGE_VERSION}

ERROR] Failed to execute goal com.google.cloud.tools:jib-maven-plugin:1.5.1:build (default-cli) on project dataflow-streaming-benchmark: /basedir/target/jib-cache/tmp/6910034298542626851

On running maven in debug mode error is:
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal com.google.cloud.tools:jib-maven-plugin:1.5.1:build (default) on project /basedir/target/jib-cache/tmp/3227117988474356011
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:215)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:156)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:148)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81)
    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:56)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:305)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192)
    at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105)
    at org.apache.maven.cli.MavenCli.execute (MavenCli.java:957)
    at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:289)
    at org.apache.maven.cli.MavenCli.main (MavenCli.java:193)
    at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:498)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:282)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:225)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:406)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:347)
Caused by: org.apache.maven.plugin.MojoExecutionException: /basedir/target/jib-cache/tmp/3227117988474356011
    at com.google.cloud.tools.jib.maven.BuildImageMojo.execute (BuildImageMojo.java:158)
    at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:137)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:210)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:156)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:148)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81)
    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:56)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:305)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192)
    at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105)
    at org.apache.maven.cli.MavenCli.execute (MavenCli.java:957)
    at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:289)
    at org.apache.maven.cli.MavenCli.main (MavenCli.java:193)
    at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:498)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:282)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:225)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:406)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:347)
Caused by: java.nio.file.DirectoryNotEmptyException: /basedir/target/jib-cache/tmp/3227117988474356011
    at sun.nio.fs.UnixCopyFile.move (UnixCopyFile.java:498)
    at sun.nio.fs.UnixFileSystemProvider.move (UnixFileSystemProvider.java:262)
    at java.nio.file.Files.move (Files.java:1395)
    at com.google.cloud.tools.jib.cache.CacheStorageWriter.lambda$moveIfDoesNotExist$0 (CacheStorageWriter.java:91)
    at com.google.cloud.tools.jib.cache.Retry.run (Retry.java:120)
    at com.google.cloud.tools.jib.cache.CacheStorageWriter.moveIfDoesNotExist (CacheStorageWriter.java:97)
    at com.google.cloud.tools.jib.cache.CacheStorageWriter.writeUncompressed (CacheStorageWriter.java:232)
    at com.google.cloud.tools.jib.cache.Cache.writeUncompressedLayer (Cache.java:115)
    at com.google.cloud.tools.jib.builder.steps.BuildAndCacheApplicationLayerStep.call (BuildAndCacheApplicationLayerStep.java:109)
    at com.google.cloud.tools.jib.builder.steps.BuildAndCacheApplicationLayerStep.call (BuildAndCacheApplicationLayerStep.java:37)
    at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly (TrustedListenableFutureTask.java:125)
    at com.google.common.util.concurrent.InterruptibleTask.run (InterruptibleTask.java:69)
    at com.google.common.util.concurrent.TrustedListenableFutureTask.run (TrustedListenableFutureTask.java:78)
    at java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:624)
    at java.lang.Thread.run (Thread.java:748)

I followed up on the Google group thread and asked a few things.

To explain what's going on, Jib attempted to move the directory /basedir/target/jib-cache/tmp/3227117988474356011 to /basedir/target/jib-cache/layers/<some sha256 hash>. On the same filesystem, this usually becomes a simple atomic move that doesn't involve copying. But looks like atomic move was not possible for some reason on your system, so the move operation degraded into deleting after copying. However, for another reason, the system could not delete the original directory (/basedir/target/jib-cache/tmp/3227117988474356011) after copying.

Is there anything special about the filesystem for /basedir/target/jib-cache, such as as NFS or Samba? Running df usually gives you an insight about filesystems.

And for testing, you can change the location of the application cache (/basedir/target/jib-cache) by setting the system property jib.applicationCache. For example, passing -Djib.applicationCache=/some/other/cache/directory on the command line. Worth trying different filesystems, including tmpfs (/tmp).

Finally, what's your Jib version?

prathapreddy123 · 2020-03-09T16:19:46Z

Thanks @chanseokoh and @briandealwis for responding immediately. Here are the details:
Jib version is 1.5.1
Directory is mounted on file system of type fuse.srcfsd.

After pointing the cache directory to different path using -Djib.applicationCache the command completed successfully.

Thanks to both of you for your timely help.

chanseokoh · 2020-03-09T17:48:06Z

Thanks. I guess fuse.srcfsd is referring to FUSE. I don't know anything about FUSE, but I think the /basedir directory is like a (semi-read-only?) central source repo in your organization specially mounted via FUSE. It may be that moving (or deleting) files and directories under /basedir is not really supported? Is this true? If that's the case, there's nothing really Jib can do. (We could improve the error message though.) When using -Djib.applicationCache, keep in mind you need to consistently use the same directory for your project to benefit the cache. (Different Jib projects can you different directories.)

Another option would be to just copy out the project source into another filesystem.

prathapreddy123 · 2020-03-09T19:39:28Z

Yes it's a source central repo mounted on fuse.While raising issue I replaced original root path with /basedir as I don't want to expose original path in GitHub. Since it's more file system specific issue, as you mentioned improving error message may be helpful Thanks, Prathap

…

On Mon, Mar 9, 2020, 10:48 AM Chanseok Oh ***@***.***> wrote: Thanks. I guess fuse.srcfsd is referring to FUSE <https://en.wikipedia.org/wiki/Filesystem_in_Userspace>. I don't know anything about FUSE, but I think the basedir directory is like a (semi-read only?) central source repo in your organization specially mounted via FUSE. It may be that moving (or deleting) files and directories under /basedir is not really supported? Is this true? If that's the case, there's nothing really Jib can do. (We could improve the error message though.) When using -Djib.applicationCache, keep in mind you need to use the same directory for your project to benefit the cache. (Different Jib projects can you different directories.) Another option would be to just copy out the project source into another filesystem. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#2318?email_source=notifications&email_token=ABXSGW4MQFXYOTLJC3IU6G3RGU2WPA5CNFSM4LEKAQYKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEOIJZEQ#issuecomment-596679826>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABXSGW6REE2O7VWN4WIKAG3RGU2WPANCNFSM4LEKAQYA> .

chanseokoh added the good first issue label Mar 9, 2020

GoogleContainerTools / jib

Caused by: java.nio.file.DirectoryNotEmptyException: /basedir/target/jib-cache/tmp/3227117988474356011 #2318

Caused by: java.nio.file.DirectoryNotEmptyException: /basedir/target/jib-cache/tmp/3227117988474356011 #2318

chanseokoh commented Mar 9, 2020 •

edited

prathapreddy123 commented Mar 9, 2020

chanseokoh commented Mar 9, 2020 •

edited

prathapreddy123 commented Mar 9, 2020

GoogleContainerTools / jib

Join GitHub today

Caused by: java.nio.file.DirectoryNotEmptyException: /basedir/target/jib-cache/tmp/3227117988474356011 #2318

Caused by: java.nio.file.DirectoryNotEmptyException: /basedir/target/jib-cache/tmp/3227117988474356011 #2318

Comments

chanseokoh commented Mar 9, 2020 • edited

prathapreddy123 commented Mar 9, 2020

chanseokoh commented Mar 9, 2020 • edited

prathapreddy123 commented Mar 9, 2020

chanseokoh commented Mar 9, 2020 •

edited

chanseokoh commented Mar 9, 2020 •

edited