native: initial linking support for linux

[Spydr06]

This PR contains initial linking support for Linux using ld.lld. The generated object file now gets linked with some libraries (libc, pthread, etc.) and the C runtime objects. Currently, this isn't that useful since calling external functions is not implemented yet.

I don't know how many tests will pass... I only tested it on Linux x86_64, other OSes should be unchanged.

[Spydr06]

Seems like the native ci tests fail immediately... Is the linker installed on there?

[JalonSolov]

Ugh. We need our own linker. I'd prefer not to need one at all, but... 🤷‍♂️

[Spydr06]

Before implementing this I asked @medvednikov about this. He said it was ok to use an external Linker since writing a linker is a huge project on its own, we could reuse existing stuff and he thinks that native should only generate .o (or similar) files.

Edit: also, we can implement our own linker at any time in the future

[StunxFS]

Why not just call the method, if the check is done on it? 🤔

[Spydr06]

Because linking is only implemented for Linux. I don‘t accidentally want to call link() with another OS or else an error will be thrown. Once All OSes receive their linking functionality, I‘ll remove the first check. That‘s why there are // TEMPORARY comments.

[Spydr06]

As expected, all Linux ci tests fail. Maybe lld isn't installed on the systems

[JalonSolov]

As expected, all Linux ci tests fail. Maybe lld isn't installed on the systems

... and that's part of why I said we need our own linker. :-)

[Spydr06]

If you want to do this, go ahead. I think we should at least get the whole native backend working before we implement a custom linker.
We can always implement it later on.

[JalonSolov]

I agree, it's just going to be painful making sure everyone (including the CI runners) has the same linker installed.

[Spydr06]

I definitely see your point. We could also check for different linkers, just like the c compiler with cgen, the default gnu ld should be installed on every linux distro by default

[Spydr06]

@medvednikov what do you think about this?

[medvednikov]

@Spydr06 I still think what I said before.

lld can be installed separately for now, on ci boxes as well. Later V will quietly fetch it or distribute like it does for tcc.

New modern linkers are fast, no need to re-write them. It's the machine code generation that's slow.

[medvednikov]

FAIL  [2/2]  1519.784 ms vlib/v/gen/native/tests/native_test.v
FAIL  [ 1/13]  1124.511 ms '/home/runner/work/v/v/v' -o '/tmp/vtests/native/print.vv.exe' -b native '/home/runner/work/v/v/vlib/v/gen/native/tests/print.vv'
Summary for all V _test.v files: 1 failed, 1 passed, 2 total. Runtime: 2124 ms, on 1 job.

 FAIL  [ 2/13]     [7](https://github.com/vlang/v/runs/7652135624?check_suite_focus=true#step:5:8).25[8](https://github.com/vlang/v/runs/7652135624?check_suite_focus=true#step:5:9) ms '/home/runner/work/v/v/v' -o '/tmp/vtests/native/compare.vv.exe' -b native '/home/runner/work/v/v/vlib/v/gen/native/tests/compare.vv'
 FAIL  [ 3/13]    12.305 ms '/home/runner/work/v/v/v' -o '/tmp/vtests/native/assert.vv.exe' -b native '/home/runner/work/v/v/vlib/v/gen/native/tests/assert.vv'

[Spydr06]

Tests still fail even though lld is now installed...

[JalonSolov]

Question is: Why are they failing now? There's no message about ld.lld not being found.

[Spydr06]

I don't know it. On my local system everything works. Testing in a vanilla Ubuntu VM rn.

[JalonSolov]

Oof. Yeah, works for me too, on Manjaro:

$ ./v_lld test vlib/v/gen/native/tests
---- Testing... --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 OK     461.772 ms vlib/v/gen/native/tests/native_test.v
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Summary for all V _test.v files: 1 passed, 1 total. Runtime: 462 ms, on 1 job.

$

[Spydr06]

Yes, on the Ubuntu VM everything works as well

[JalonSolov]

V needs a "verbose test failure" switch (if it doesn't already have an undocumented one...)

[JalonSolov]

Question: Will use of the linker be optional?

I really, really like the 186 byte hello_world, as opposed to the 10k hello_world. :-\

[Spydr06]

I can implement a switch for that, should be easy. I think we should figure out with which libraries to link with in general. Currently, it just links with the stuff which cgen also links against. If there is no library to link with, we can just skip that step

[Spydr06]

maybe this will work
Edit: nope, still not working

[Spydr06]

---- Testing... ----------------------------------------------------------------
OK    [1/2]   617.174 ms vlib/v/gen/native/macho_test.v
 FAIL  [2/2]  3140.367 ms vlib/v/gen/native/tests/native_test.v
FAIL  [ 1/13]  2622.523 ms '/home/runner/work/v/v/v' -o '/tmp/vtests/native/forin.vv.exe' -b native '/home/runner/work/v/v/vlib/v/gen/native/tests/forin.vv' 2> '/home/runner/work/v/v/vlib/v/gen/native/tests/forin.vv.tmperr'
native error: ELF linking failed (ld.lld):
LLD 10.0.0 (compatible with GNU linkers)
ld.lld: error: cannot open /usr/lib/crt1.o: No such file or directory
ld.lld: error: cannot open /usr/lib/crti.o: No such file or directory
ld.lld: error: unable to find library -lc
ld.lld: error: unable to find library -lm
ld.lld: error: unable to find library -lpthread
ld.lld: error: cannot open /usr/lib/crtn.o: No such file or directory

Building V works nevertheless, so all those files should be there as they are necessary for V to compile

[Spydr06]

I really cannot think of any reason why this is happening, since the C compiler seems to work.
Maybe I am missing something, I don't really have that much experience with CI

[Wertzui123]

Before implementing this I asked @medvednikov about this. He said it was ok to use an external Linker since writing a linker is a huge project on its own, we could reuse existing stuff and he thinks that native should only generate .o (or similar) files.

Edit: also, we can implement our own linker at any time in the future

I think we should implement our own linker too since the easy cross-compilation is one of the huge advantages of the native backend.

[spytheman]

ld.lld -L/usr/lib/x86_64-linux-gnu -L/usr/lib64 -L/lib64 -L/usr/lib -L/lib -v -o /tmp/vtests/native/simple_fn_calls.vv.exe -m elf_x86_64 -dynamic-linker /lib64/ld-linux-x86-64.so.2 /usr/bin/../lib/gcc/x86_64-linux-gnu/11/../../../x86_64-linux-gnu/crt1.o /usr/bin/../lib/gcc/x86_64-linux-gnu/11/../../../x86_64-linux-gnu/crti.o -lc -lm -lpthread /usr/bin/../lib/gcc/x86_64-linux-gnu/11/../../../x86_64-linux-gnu/crtn.o /tmp/vtests/native/simple_fn_calls.vv.exe.o

^ This works on Ubuntu 20.04 (the paths are taken from clang-12 -v x.o -o ./x).

I am not sure that invoking the linker directly is a good idea at this stage (it will burden V with discovering what are the local conventions/paths, which is not trivial, and we do depend on a C compiler anyways).

It would be much simpler to rely on either gcc or clang for now (used just as frontends to the linkers).

[spytheman]

Before implementing this I asked @medvednikov about this. He said it was ok to use an external Linker since writing a linker is a huge project on its own, we could reuse existing stuff and he thinks that native should only generate .o (or similar) files.
Edit: also, we can implement our own linker at any time in the future

I think we should implement our own linker too since the easy cross-compilation is one of the huge advantages of the native backend.

In the long term, that is true, but as @Spydr06 said, it would be a huge project on its own, and it is out of scope for this PR.

[Spydr06]

Thank you very much, seems like it works now. I can fix the docker tests later myself I think

[medvednikov]

Great job as always @spytheman !

[spytheman]

If it's installed on the user's machine, we can use it, but I think the default ld is fast enough to rely on otherwise, since it's much more common

ok, I will make it default to ld then.

[Spydr06]

native-backend-ci passes now

[spytheman]

sorry, I will not change this PR anymore, to avoid merge conflicts

[Spydr06]

sorry, I will not change this PR anymore, to avoid merge conflicts

I have no problem with that, your help is much appreciated

[JalonSolov]

I'd prefer mold, too, except they don't have a Windows version yet. :-\

[Spydr06]

As I already said, we can check if mold is installed on the user's system and then use it. If not, we can fall back to ld, lld or similar

[Spydr06]

Seems like only alpine is the problem now, nice!

[medvednikov]

we can skip alpine for now

[spytheman]

It should be fine now.
(rebased on current master, because there was a breaking change in io)

[Spydr06]

Thank you all so much for your help, I really appreciate it.
I‘ll extend the functionality/portability of this soon.