pantsd hang in CI #10877
Description
When pantsd is enabled for a run of remote-apis-testing in CI, the three sub-builds for Pants have no output and then are killed once the one hour watchdog timeout occurs.
All three jobs report pantsd running at PID 152. Probably requires further investigation whether they are sharing any state between the sub-builds.
https://gitlab.com/tdyas/remote-apis-testing/-/jobs/761469804
client_1 | 18:18:33.96 [DEBUG] pantsd is running at pid 152, pailgun port is 43369
client_1 | 18:18:33.96 [DEBUG] releasing lock: <pants.process.lock.OwnerPrintingInterProcessFileLock object at 0x7ff13e9d1940>
client_1 | 18:18:33.96 [DEBUG] connecting to pantsd on port 43369 (attempt 1/3)
client_1 | 18:18:33.96 [DEBUG] masking tcgetattr exception: error(25, 'Inappropriate ioctl for device')
client_1 | 18:18:34.01 [DEBUG] Logging already initialized.
client_1 | 18:18:34.01 [INFO] initializing pantsd...
client_1 | 18:18:34.46 [DEBUG] File handle limit is: 1048576
client_1 | 18:18:34.48 [DEBUG] initial_polymorphic iteration 1000: 1107 nodes
client_1 | 18:18:34.48 [DEBUG] initial_polymorphic iteration 2000: 2106 nodes
client_1 | 18:18:34.48 [DEBUG] initial_polymorphic iteration 3000: 3105 nodes
client_1 | 18:18:34.49 [DEBUG] initial_polymorphic iteration 4000: 4137 nodes
client_1 | 18:18:34.49 [DEBUG] initial_polymorphic iteration 5000: 5101 nodes
client_1 | 18:18:34.50 [DEBUG] initial_polymorphic iteration 6000: 6109 nodes
client_1 | 18:18:34.52 [DEBUG] rule_graph monomorphize: iteration 1000: live: 3663, minimal: 393, to_visit: 5648, total: 6716
client_1 | 18:18:34.53 [DEBUG] rule_graph monomorphize: iteration 2000: live: 3504, minimal: 512, to_visit: 5189, total: 6962
client_1 | 18:18:34.54 [DEBUG] rule_graph monomorphize: iteration 3000: live: 3451, minimal: 839, to_visit: 4537, total: 7277
client_1 | 18:18:34.54 [DEBUG] rule_graph monomorphize: iteration 4000: live: 3301, minimal: 972, to_visit: 4050, total: 7538
client_1 | 18:18:34.55 [DEBUG] rule_graph monomorphize: iteration 5000: live: 3274, minimal: 1344, to_visit: 3337, total: 7873
client_1 | 18:18:34.56 [DEBUG] rule_graph monomorphize: iteration 6000: live: 3129, minimal: 1479, to_visit: 2875, total: 8130
client_1 | 18:18:34.56 [DEBUG] rule_graph monomorphize: iteration 7000: live: 3045, minimal: 1759, to_visit: 2215, total: 8441
client_1 | 18:18:34.57 [DEBUG] rule_graph monomorphize: iteration 8000: live: 2954, minimal: 1984, to_visit: 1691, total: 8728
client_1 | 18:18:34.57 [DEBUG] rule_graph monomorphize: iteration 9000: live: 2892, minimal: 2313, to_visit: 992, total: 9055
client_1 | 18:18:34.58 [DEBUG] rule_graph monomorphize: iteration 10000: live: 2759, minimal: 2475, to_visit: 495, total: 9310
client_1 | 18:18:34.60 [DEBUG] Changes to /root/.cache/pants/setup/bootstrap-Linux-x86_64/pants.hmxZAk/install/bin, outside of the buildroot, will not be invalidated.
client_1 | 18:18:34.60 [DEBUG] Changes to /root/.cache/pants/setup/bootstrap-Linux-x86_64/pants.hmxZAk/install/bin, outside of the buildroot, will not be invalidated.
client_1 | 18:18:34.60 [DEBUG] Changes to /root/.cache/pants/setup/bootstrap-Linux-x86_64/2.0.0b0+git0c54f5d3_py38/lib/python38.zip, outside of the buildroot, will not be invalidated.
client_1 | 18:18:34.60 [DEBUG] Changes to /root/.cache/pants/setup/bootstrap-Linux-x86_64/2.0.0b0+git0c54f5d3_py38/lib/python3.8, outside of the buildroot, will not be invalidated.
client_1 | 18:18:34.60 [DEBUG] Changes to /root/.cache/pants/setup/bootstrap-Linux-x86_64/2.0.0b0+git0c54f5d3_py38/lib/python3.8/lib-dynload, outside of the buildroot, will not be invalidated.
client_1 | 18:18:34.60 [DEBUG] Changes to /usr/lib/python3.8, outside of the buildroot, will not be invalidated.
client_1 | 18:18:34.61 [DEBUG] Changes to /root/.cache/pants/setup/bootstrap-Linux-x86_64/2.0.0b0+git0c54f5d3_py38/lib/python3.8/site-packages, outside of the buildroot, will not be invalidated.
client_1 | 18:18:34.61 [DEBUG] setting up service <pants.pantsd.service.scheduler_service.SchedulerService object at 0x7f1f90719100>
client_1 | 18:18:34.61 [DEBUG] setting up service <pants.pantsd.service.store_gc_service.StoreGCService object at 0x7f1f90719490>
client_1 | 18:18:34.61 [DEBUG] starting service <pants.pantsd.service.scheduler_service.SchedulerService object at 0x7f1f90719100>
client_1 | 18:18:34.61 [DEBUG] starting service <pants.pantsd.service.store_gc_service.StoreGCService object at 0x7f1f90719490>
client_1 | 18:18:34.61 [INFO] pantsd initialized.
client_1 | 18:18:34.67 [DEBUG] specs are: Specs(address_specs=AddressSpecs(literals=(), globs=(DescendantAddresses(directory='helloworld'),), filter_by_global_options=True), filesystem_specs=FilesystemSpecs(includes=(), ignores=()))
client_1 | 18:18:34.67 [DEBUG] changed_options are: ChangedOptions(since=None, diffspec=None, dependees=<DependeesOption.NONE: 'none'>)
client_1 | 18:18:34.68 [DEBUG] requesting <class 'pants.core.goals.test.Test'> to satisfy execution of `test` goal
client_1 | 18:18:34.68 [DEBUG] Launching 1 roots (poll=false).
client_1 | 18:18:34.68 [DEBUG] Starting: `test` goal
client_1 | 18:18:34.68 [DEBUG] Starting: Find targets from input specs
client_1 | 18:18:34.69 [DEBUG] Completed: Find targets from input specs
client_1 | 18:18:34.71 [DEBUG] Starting: Run Pytest
client_1 | 18:18:34.71 [DEBUG] Starting: pants.backend.python.goals.pytest_runner.setup_pytest_for_target
client_1 | 18:18:34.71 [DEBUG] Starting: Run Pytest
client_1 | 18:18:34.71 [DEBUG] Starting: pants.backend.python.goals.pytest_runner.setup_pytest_for_target
ERROR: Job failed: execution took longer than 1h0m0s seconds
https://gitlab.com/tdyas/remote-apis-testing/-/jobs/761469805
client_1 | 18:33:34.11 [DEBUG] pantsd is running at pid 152, pailgun port is 33193
client_1 | 18:33:34.11 [DEBUG] releasing lock: <pants.process.lock.OwnerPrintingInterProcessFileLock object at 0x7f8cee37a940>
client_1 | 18:33:34.11 [DEBUG] connecting to pantsd on port 33193 (attempt 1/3)
client_1 | 18:33:34.11 [DEBUG] masking tcgetattr exception: error(25, 'Inappropriate ioctl for device')
client_1 | 18:33:34.15 [DEBUG] Logging already initialized.
client_1 | 18:33:34.16 [INFO] initializing pantsd...
client_1 | 18:33:34.68 [DEBUG] File handle limit is: 1048576
client_1 | 18:33:34.70 [DEBUG] initial_polymorphic iteration 1000: 1113 nodes
client_1 | 18:33:34.71 [DEBUG] initial_polymorphic iteration 2000: 2133 nodes
client_1 | 18:33:34.71 [DEBUG] initial_polymorphic iteration 3000: 3098 nodes
client_1 | 18:33:34.72 [DEBUG] initial_polymorphic iteration 4000: 4100 nodes
client_1 | 18:33:34.72 [DEBUG] initial_polymorphic iteration 5000: 5101 nodes
client_1 | 18:33:34.73 [DEBUG] initial_polymorphic iteration 6000: 6096 nodes
client_1 | 18:33:34.76 [DEBUG] rule_graph monomorphize: iteration 1000: live: 3632, minimal: 310, to_visit: 5727, total: 6689
client_1 | 18:33:34.77 [DEBUG] rule_graph monomorphize: iteration 2000: live: 3503, minimal: 508, to_visit: 5174, total: 6960
client_1 | 18:33:34.77 [DEBUG] rule_graph monomorphize: iteration 3000: live: 3473, minimal: 878, to_visit: 4498, total: 7296
client_1 | 18:33:34.78 [DEBUG] rule_graph monomorphize: iteration 4000: live: 3355, minimal: 1060, to_visit: 3969, total: 7567
client_1 | 18:33:34.79 [DEBUG] rule_graph monomorphize: iteration 5000: live: 3278, minimal: 1338, to_visit: 3354, total: 7870
client_1 | 18:33:34.80 [DEBUG] rule_graph monomorphize: iteration 6000: live: 3129, minimal: 1477, to_visit: 2869, total: 8128
client_1 | 18:33:34.81 [DEBUG] rule_graph monomorphize: iteration 7000: live: 3039, minimal: 1720, to_visit: 2309, total: 8421
client_1 | 18:33:34.82 [DEBUG] rule_graph monomorphize: iteration 8000: live: 2951, minimal: 1977, to_visit: 1704, total: 8720
client_1 | 18:33:34.83 [DEBUG] rule_graph monomorphize: iteration 9000: live: 2888, minimal: 2294, to_visit: 994, total: 9045
client_1 | 18:33:34.83 [DEBUG] rule_graph monomorphize: iteration 10000: live: 2759, minimal: 2471, to_visit: 500, total: 9309client_1 | 18:33:34.87 [DEBUG] Changes to /root/.cache/pants/setup/bootstrap-Linux-x86_64/pants.kM4f4H/install/bin, outside of the buildroot, will not be invalidated.
client_1 | 18:33:34.88 [DEBUG] Changes to /root/.cache/pants/setup/bootstrap-Linux-x86_64/pants.kM4f4H/install/bin, outside of the buildroot, will not be invalidated.
client_1 | 18:33:34.88 [DEBUG] Changes to /root/.cache/pants/setup/bootstrap-Linux-x86_64/2.0.0b0+git0c54f5d3_py38/lib/python38.zip, outside of the buildroot, will not be invalidated.
client_1 | 18:33:34.88 [DEBUG] Changes to /root/.cache/pants/setup/bootstrap-Linux-x86_64/2.0.0b0+git0c54f5d3_py38/lib/python3.8, outside of the buildroot, will not be invalidated.
client_1 | 18:33:34.88 [DEBUG] Changes to /root/.cache/pants/setup/bootstrap-Linux-x86_64/2.0.0b0+git0c54f5d3_py38/lib/python3.8/lib-dynload, outside of the buildroot, will not be invalidated.
client_1 | 18:33:34.88 [DEBUG] Changes to /usr/lib/python3.8, outside of the buildroot, will not be invalidated.
client_1 | 18:33:34.88 [DEBUG] Changes to /root/.cache/pants/setup/bootstrap-Linux-x86_64/2.0.0b0+git0c54f5d3_py38/lib/python3.8/site-packages, outside of the buildroot, will not be invalidated.
client_1 | 18:33:34.88 [DEBUG] setting up service <pants.pantsd.service.scheduler_service.SchedulerService object at 0x7f94ab3c4130>
client_1 | 18:33:34.89 [DEBUG] setting up service <pants.pantsd.service.store_gc_service.StoreGCService object at 0x7f94ab3c47c0>
client_1 | 18:33:34.89 [DEBUG] starting service <pants.pantsd.service.scheduler_service.SchedulerService object at 0x7f94ab3c4130>
client_1 | 18:33:34.89 [DEBUG] starting service <pants.pantsd.service.store_gc_service.StoreGCService object at 0x7f94ab3c47c0>
client_1 | 18:33:34.89 [INFO] pantsd initialized.
client_1 | 18:33:34.96 [DEBUG] specs are: Specs(address_specs=AddressSpecs(literals=(), globs=(DescendantAddresses(directory='helloworld'),), filter_by_global_options=True), filesystem_specs=FilesystemSpecs(includes=(), ignores=()))
client_1 | 18:33:34.96 [DEBUG] changed_options are: ChangedOptions(since=None, diffspec=None, dependees=<DependeesOption.NONE: 'none'>)
client_1 | 18:33:34.97 [DEBUG] requesting <class 'pants.core.goals.test.Test'> to satisfy execution of `test` goal
client_1 | 18:33:34.97 [DEBUG] Launching 1 roots (poll=false).
client_1 | 18:33:34.97 [DEBUG] Starting: `test` goal
client_1 | 18:33:34.97 [DEBUG] Starting: Find targets from input specs
ERROR: Job failed: execution took longer than 1h0m0s seconds
https://gitlab.com/tdyas/remote-apis-testing/-/jobs/761469807
client_1 | 18:22:57.22 [DEBUG] pantsd is running at pid 152, pailgun port is 37705
client_1 | 18:22:57.22 [DEBUG] releasing lock: <pants.process.lock.OwnerPrintingInterProcessFileLock object at 0x7fb762ea4940>
client_1 | 18:22:57.22 [DEBUG] connecting to pantsd on port 37705 (attempt 1/3)
client_1 | 18:22:57.22 [DEBUG] masking tcgetattr exception: error(25, 'Inappropriate ioctl for device')
client_1 | 18:22:57.26 [DEBUG] Logging already initialized.
client_1 | 18:22:57.27 [INFO] initializing pantsd...
client_1 | 18:22:57.76 [DEBUG] File handle limit is: 1048576
client_1 | 18:22:57.83 [DEBUG] initial_polymorphic iteration 1000: 1110 nodes
client_1 | 18:22:57.83 [DEBUG] initial_polymorphic iteration 2000: 2105 nodes
client_1 | 18:22:57.84 [DEBUG] initial_polymorphic iteration 3000: 3143 nodes
client_1 | 18:22:57.84 [DEBUG] initial_polymorphic iteration 4000: 4123 nodes
client_1 | 18:22:57.85 [DEBUG] initial_polymorphic iteration 5000: 5131 nodes
client_1 | 18:22:57.85 [DEBUG] initial_polymorphic iteration 6000: 6104 nodes
client_1 | 18:22:57.88 [DEBUG] rule_graph monomorphize: iteration 1000: live: 3648, minimal: 338, to_visit: 5754, total: 6691
client_1 | 18:22:57.89 [DEBUG] rule_graph monomorphize: iteration 2000: live: 3504, minimal: 513, to_visit: 5175, total: 6964
client_1 | 18:22:57.89 [DEBUG] rule_graph monomorphize: iteration 3000: live: 3477, minimal: 890, to_visit: 4488, total: 7298
client_1 | 18:22:57.90 [DEBUG] rule_graph monomorphize: iteration 4000: live: 3357, minimal: 1072, to_visit: 3958, total: 7575
client_1 | 18:22:57.91 [DEBUG] rule_graph monomorphize: iteration 5000: live: 3276, minimal: 1348, to_visit: 3335, total: 7876
client_1 | 18:22:57.91 [DEBUG] rule_graph monomorphize: iteration 6000: live: 3129, minimal: 1479, to_visit: 2883, total: 8127
client_1 | 18:22:57.92 [DEBUG] rule_graph monomorphize: iteration 7000: live: 3047, minimal: 1753, to_visit: 2237, total: 8438
client_1 | 18:22:57.93 [DEBUG] rule_graph monomorphize: iteration 8000: live: 2951, minimal: 1985, to_visit: 1693, total: 8728
client_1 | 18:22:57.94 [DEBUG] rule_graph monomorphize: iteration 9000: live: 2873, minimal: 2278, to_visit: 997, total: 9045
client_1 | 18:22:57.94 [DEBUG] rule_graph monomorphize: iteration 10000: live: 2759, minimal: 2474, to_visit: 498, total: 9311
client_1 | 18:22:57.96 [DEBUG] Changes to /root/.cache/pants/setup/bootstrap-Linux-x86_64/pants.cFkfgR/install/bin, outside of the buildroot, will not be invalidated.
client_1 | 18:22:57.97 [DEBUG] Changes to /root/.cache/pants/setup/bootstrap-Linux-x86_64/pants.cFkfgR/install/bin, outside of the buildroot, will not be invalidated.
client_1 | 18:22:57.97 [DEBUG] Changes to /root/.cache/pants/setup/bootstrap-Linux-x86_64/2.0.0b0+git0c54f5d3_py38/lib/python38.zip, outside of the buildroot, will not be invalidated.
client_1 | 18:22:57.97 [DEBUG] Changes to /root/.cache/pants/setup/bootstrap-Linux-x86_64/2.0.0b0+git0c54f5d3_py38/lib/python3.8, outside of the buildroot, will not be invalidated.
client_1 | 18:22:57.97 [DEBUG] Changes to /root/.cache/pants/setup/bootstrap-Linux-x86_64/2.0.0b0+git0c54f5d3_py38/lib/python3.8/lib-dynload, outside of the buildroot, will not be invalidated.
client_1 | 18:22:57.97 [DEBUG] Changes to /usr/lib/python3.8, outside of the buildroot, will not be invalidated.
client_1 | 18:22:57.97 [DEBUG] Changes to /root/.cache/pants/setup/bootstrap-Linux-x86_64/2.0.0b0+git0c54f5d3_py38/lib/python3.8/site-packages, outside of the buildroot, will not be invalidated.
client_1 | 18:22:57.97 [DEBUG] setting up service <pants.pantsd.service.scheduler_service.SchedulerService object at 0x7f37407fe130>
client_1 | 18:22:57.98 [DEBUG] setting up service <pants.pantsd.service.store_gc_service.StoreGCService object at 0x7f37407fe7c0>
client_1 | 18:22:57.98 [DEBUG] starting service <pants.pantsd.service.scheduler_service.SchedulerService object at 0x7f37407fe130>
client_1 | 18:22:57.98 [DEBUG] starting service <pants.pantsd.service.store_gc_service.StoreGCService object at 0x7f37407fe7c0>
client_1 | 18:22:57.98 [INFO] pantsd initialized.
client_1 | 18:22:58.07 [DEBUG] specs are: Specs(address_specs=AddressSpecs(literals=(), globs=(DescendantAddresses(directory='helloworld'),), filter_by_global_options=True), filesystem_specs=FilesystemSpecs(includes=(), ignores=()))
client_1 | 18:22:58.07 [DEBUG] changed_options are: ChangedOptions(since=None, diffspec=None, dependees=<DependeesOption.NONE: 'none'>)
client_1 | 18:22:58.07 [DEBUG] requesting <class 'pants.core.goals.test.Test'> to satisfy execution of `test` goal
client_1 | 18:22:58.07 [DEBUG] Launching 1 roots (poll=false).
client_1 | 18:22:58.08 [DEBUG] Starting: `test` goal
client_1 | 18:22:58.08 [DEBUG] Starting: Find targets from input specs