Hi!
I've just packaged the latest muon edge version (commit cafeb37c) for Debian Experimental, to prepare for the next actual release.
Unfortunately though it seems that muon cannot bootstrap itself on the s390x architecture, since running the bootstrapped executable generated by bootstrap.sh leads to a segmentation fault just after printing the version banner. You can find the build log here; here's the relevant part:
+ c99 -g -O2 -Werror=implicit-function-declaration -Werror=array-bounds -Werror=clobbered -Werror=volatile-register-var -ffile-prefix-map=/<<PKGBUILDDIR>>=. -fstack-protector-strong -Wformat -Werror=format-security -DBOOTSTRAP_NO_SAMU -Wl,-z,relro -Wl,-z,now -Iinclude -I/usr/include/pkgconf -DBOOTSTRAP_HAVE_LIBPKGCONF src/amalgam.c -lpkgconf -o build/muon
In file included from src/amalgam.c:30:
In function ‘reopen_source’,
inlined from ‘reopen_source.constprop’ at src/error.c:349:1:
src/error.c:356:36: warning: argument 1 null where non-null expected [-Wnonnull]
356 | src->len = strlen(src->src);
| ^~~~~~~~~~~~~~~~
In file included from src/args.c:9,
from src/amalgam.c:12:
/usr/include/string.h: In function ‘reopen_source.constprop’:
/usr/include/string.h:407:15: note: in a call to function ‘strlen’ declared ‘nonnull’
407 | extern size_t strlen (const char *__s)
| ^~~~~~
build/muon setup -Dprefix=/usr -Dsamurai=disabled -Dbestline=enabled build
detected compiler gcc '13.3.0' (['cc']), linker: ld (gnu) (['ld']), static_linker: ar (gcc) (['ar'])
configuring 'muon', version: 0.2.0
make[1]: *** [debian/rules:12: override_dh_auto_build] Segmentation fault
The warning generated by GCC may be related, but I think it's not. In my (limited) testing it seems that reopen_source() doesn't get called when building muon. But I may be wrong!
Another thought is that s390x is big endian, so there may be some code which assumes little-endiannes and thus fails to work there.
What do you think may be causing the issue? If you have any though, I can ask for a s390x box to perform some tests!
Bye :)
muon is also segfaulting on all other Debian's big-endian architectures (ppc64 and sparc64), so it definitely seems that muon doesn't work on big-endian systems.
The stage 1 bootstrap muon binary doesn't segfault right away, for example it handles
muon version,muon check meson.buildandmuon fmt meson.build(with muon's own meson.build file). It seems that it crashes only when runningmuon setup.Here's a statically-linked s390x stage-1 muon binary with undefined behaviour sanitizer enabled, that you can test with qemu-user-s390x: https://andrea.pappacoda.it/muon
I've been able to more or less figure out what causes crashes: the if statement.
Here's a minimal example:
project('muon', 'c') compiler = meson.get_compiler('c') has_printf = compiler.has_function('printf') message(f'showing that has_function works: @has_printf@') if true message('inside if, I\'m about to crash') endif message('outside if')And here's what happens when running it with a stage 1 muon:
$ ./muon setup build detected compiler gcc '13.3.0' (['cc']), linker: ld (gnu) (['ld']), static_linker: ar (gcc) (['ar']) configuring 'muon', version: undefined c compiler: has function printf: YES message: showing that has_function works: true message: inside if, I'm about to crash qemu: uncaught target signal 11 (Segmentation fault) - core dumpedPretty odd!
After running the program under gdb-multiarch, I've been able to figure out the exact spot where the program segfaults: in src/lang/vm.c, at line 2261 (
wk->vm.ops.ops[wk->vm.code.e[cip]](wk))... Which doesn't seem really useful. Here's the state of thewkvariable right before crashing:Breakpoint 2, 0x000000000107bbaa in vm_execute_loop (wk=0x200007ff2d8) at src/lang/vm.c:2261 2261 wk->vm.ops.ops[wk->vm.code.e[cip]](wk); (gdb) print *wk $1 = {argv0 = 0x11cdf51 "/tmp/tmp.wRC0rQbFIN/unstable-s390x/root/muon-cafeb37c/muon", source_root = 0x11cddb0 "/tmp/tmp.wRC0rQbFIN/unstable-s390x/root/muon-cafeb37c", build_root = 0x11cdf15 "/tmp/tmp.wRC0rQbFIN/unstable-s390x/root/muon-cafeb37c/build", muon_private = 0x11cdf8c "/tmp/tmp.wRC0rQbFIN/unstable-s390x/root/muon-cafeb37c/build/.muon", original_commandline = {argc = 1, argv = 0x20000800ca8}, regenerate_deps = 18, host_machine = 17, binaries = 16, install = 19, install_scripts = 20, postconf_scripts = 21, subprojects = 22, global_args = 23, global_link_args = 24, dep_overrides_static = 25, dep_overrides_dynamic = 26, find_program_overrides = 27, global_opts = 28, compiler_check_cache = 29, dependency_handlers = 30, vm = {stack = {ba = {buckets = {len = 1, cap = 1, item_size = 16, e = 0x11a7690 ""}, item_size = 8, bucket_size = 128, len = 0, tail_bucket = 0}, page = 0x11a76b0, i = 0, bucket = 0}, call_stack = {len = 1, cap = 64, item_size = 32, e = 0x11a7ac0 ""}, locations = {len = 469, cap = 1024, item_size = 16, e = 0x11a9af0 ""}, code = {len = 1149, cap = 4096, item_size = 1, e = 0x11a82d0 ""}, src = {len = 4, cap = 64, item_size = 32, e = 0x11a92e0 ""}, ip = 1, nargs = 1, nkwargs = 0, scope_stack = 108, default_scope_stack = 4, ops = {ops = {0x0, 0x1076070 <vm_op_constant>, 0x10760d8 <vm_op_constant_list>, 0x10761d8 <vm_op_constant_dict>, 0x10763a0 <vm_op_constant_func>, 0x1076678 <vm_op_add>, 0x1077078 <vm_op_sub>, 0x1077318 <vm_op_mul>, 0x1077860 <vm_op_div>, 0x10775b8 <vm_op_mod>, 0x1078760 <vm_op_not>, 0x1078610 <vm_op_eq>, 0x10781e8 <vm_op_in>, 0x1077f68 <vm_op_gt>, 0x1077ce8 <vm_op_lt>, 0x10788f8 <vm_op_negate>, 0x1078ab0 <vm_op_stringify>, 0x1078c08 <vm_op_store>, 0x1076b18 <vm_op_add_store>, 0x1078dc8 <vm_op_load>, 0x1078ed0 <vm_op_try_load>, 0x107adc8 <vm_op_return>, 0x107adc8 <vm_op_return>, 0x1079848 <vm_op_call>, 0x1079900 <vm_op_call_method>, 0x1079d38 <vm_op_call_native>, 0x1078fc0 <vm_op_index>, 0x1079f20 <vm_op_iterator>, 0x107a4b8 <vm_op_iterator_next>, 0x107ad68 <vm_op_jmp>, 0x107ab80 <vm_op_jmp_if_true>, 0x107ac70 <vm_op_jmp_if_false>, 0x107aa58 <vm_op_jmp_if_disabler>, 0x107aaf8 <vm_op_jmp_if_disabler_keep>, 0x107a940 <vm_op_pop>, 0x107a978 <vm_op_dup>, 0x107a9d0 <vm_op_swap>, 0x107aef0 <vm_op_typecheck>, 0x0, 0x0}}, objects = {chrs = {buckets = {len = 1, cap = 1, item_size = 16, e = 0x11cdd60 ""}, item_size = 1, bucket_size = 4096, len = 3073, tail_bucket = 0}, objs = {buckets = {len = 1, cap = 1, item_size = 16, e = 0x11ced90 ""}, item_size = 8, bucket_size = 1024, len = 271, tail_bucket = 0}, dict_elems = {buckets = {len = 1, cap = 1, item_size = 16, e = 0x11d0dc0 ""}, item_size = 12, bucket_size = 1024, len = 46, tail_bucket = 0}, dict_hashes = { buckets = {len = 1, cap = 1, item_size = 16, e = 0x11d3df0 ""}, item_size = 152, bucket_size = 16, len = 1, tail_bucket = 0}, obj_aos = {{buckets = {len = 1, cap = 1, item_size = 16, e = 0x11d47a0 ""}, item_size = 8, bucket_size = 1024, len = 0, tail_bucket = 0}, {buckets = {len = 1, cap = 1, item_size = 16, e = 0x11d67d0 ""}, item_size = 16, bucket_size = 1024, len = 130, tail_bucket = 0}, {buckets = {len = 1, cap = 1, item_size = 16, e = 0x11da800 ""}, item_size = 20, bucket_size = 2048, len = 71, tail_bucket = 0}, {buckets = {len = 1, cap = 1, item_size = 16, e = 0x11e4830 ""}, item_size = 16, bucket_size = 512, len = 24, tail_bucket = 0}, {buckets = {len = 1, cap = 1, item_size = 16, e = 0x11e6860 ""}, item_size = 40, bucket_size = 4, len = 1, tail_bucket = 0}, {buckets = {len = 1, cap = 1, item_size = 16, e = 0x11e6930 ""}, item_size = 180, bucket_size = 16, len = 0, tail_bucket = 0}, {buckets = {len = 1, cap = 1, item_size = 16, e = 0x11e74a0 ""}, item_size = 36, bucket_size = 16, len = 0, tail_bucket = 0}, {buckets = { len = 1, cap = 1, item_size = 16, e = 0x11e7710 ""}, item_size = 8, bucket_size = 16, len = 0, tail_bucket = 0}, { buckets = {len = 1, cap = 1, item_size = 16, e = 0x11e77c0 ""}, item_size = 80, bucket_size = 16, len = 0, tail_bucket = 0}, {buckets = {len = 1, cap = 1, item_size = 16, e = 0x11e7cf0 ""}, item_size = 12, bucket_size = 32, len = 0, tail_bucket = 0}, { buckets = {len = 1, cap = 1, item_size = 16, e = 0x11e7ea0 ""}, item_size = 24, bucket_size = 32, len = 0, tail_bucket = 0}, {buckets = {len = 1, cap = 1, item_size = 16, e = 0x11e81d0 ""}, item_size = 16, bucket_size = 32, len = 0, tail_bucket = 0}, { buckets = {len = 1, cap = 1, item_size = 16, e = 0x11e8400 ""}, item_size = 4, bucket_size = 16, len = 0, tail_bucket = 0}, { buckets = {len = 1, cap = 1, item_size = 16, e = 0x11e8470 ""}, item_size = 48, bucket_size = 64, len = 0, tail_bucket = 0}, {buckets = {len = 1, cap = 1, item_size = 16, e = 0x11e90a0 ""}, item_size = 12, bucket_size = 16, len = 0, tail_bucket = 0}, { buckets = {len = 1, cap = 1, item_size = 16, e = 0x11e9190 ""}, item_size = 32, bucket_size = 128, len = 0, tail_bucket = 0}, {buckets = {len = 1, cap = 1, item_size = 16, e = 0x11ea1c0 ""}, item_size = 4, bucket_size = 4, len = 0, tail_bucket = 0}, {buckets = {len = 1, cap = 1, item_size = 16, e = 0x11ea200 ""}, item_size = 8, bucket_size = 16, len = 0, tail_bucket = 0}, {buckets = {len = 1, cap = 1, item_size = 16, e = 0x11ea2b0 ""}, item_size = 44, bucket_size = 32, len = 29, tail_bucket = 0}, {buckets = {len = 1, cap = 1, item_size = 16, e = 0x11ea860 ""}, item_size = 20, bucket_size = 16, len = 0, tail_bucket = 0}, {buckets = {len = 1, cap = 1, item_size = 16, e = 0x11ea9d0 ""}, item_size = 16, bucket_size = 16, len = 0, tail_bucket = 0}, {buckets = {len = 1, cap = 1, item_size = 16, e = 0x11eab00 ""}, item_size = 8, bucket_size = 4, len = 0, tail_bucket = 0}, {buckets = {len = 1, cap = 1, item_size = 16, e = 0x11eab50 ""}, item_size = 8, bucket_size = 4, len = 0, tail_bucket = 0}, {buckets = {len = 1, cap = 1, item_size = 16, e = 0x11eaba0 ""}, item_size = 8, bucket_size = 4, len = 0, tail_bucket = 0}, {buckets = { len = 1, cap = 1, item_size = 16, e = 0x11eabf0 ""}, item_size = 8, bucket_size = 4, len = 0, tail_bucket = 0}, { buckets = {len = 1, cap = 1, item_size = 16, e = 0x11eac40 ""}, item_size = 24, bucket_size = 32, len = 0, tail_bucket = 0}, {buckets = {len = 1, cap = 1, item_size = 16, e = 0x11eaf70 ""}, item_size = 3136, bucket_size = 32, len = 0, tail_bucket = 0}, {buckets = {len = 1, cap = 1, item_size = 16, e = 0x12037a0 ""}, item_size = 3136, bucket_size = 64, len = 0, tail_bucket = 0}, {buckets = {len = 4, cap = 4, item_size = 16, e = 0x1207580 ""}, item_size = 16, bucket_size = 32, len = 116, tail_bucket = 3}}, obj_hash = {meta = {len = 0, cap = 128, item_size = 1, e = 0x12039f0 '\200' <repeats 128 times>}, e = {len = 0, cap = 128, item_size = 16, e = 0x1203a80 ""}, keys = {len = 0, cap = 128, item_size = 4, e = 0x1204290 ""}, cap = 128, len = 0, load = 0, max_load = 64, capm = 127, keycmp = 0x1014970 <hash_keycmp_memcmp>, hash_func = 0x1014808 <fnv_1a_64>}, str_hash = {meta = {len = 0, cap = 256, item_size = 1, e = 0x1207bd0 "`\200K\200\200\200\f\200\200QF\200\200\200\200\200XX\200\200\200v\200\035\200;jJ\030\2005\022\200\200\021\200\200\200c*]\200\200\200\200\200\200\022\016\200\200\200g\200\200\200-Fr})bT6\200\200\177\200\200X%\200"}, e = {len = 0, cap = 256, item_size = 16, e = 0x1209090 ""}, keys = {len = 113, cap = 128, item_size = 16, e = 0x1204d40 ""}, cap = 256, len = 113, load = 113, max_load = 128, capm = 255, keycmp = 0x1014b68 <hash_keycmp_strcmp>, hash_func = 0x1014750 <fnv_1a_64_str>}, obj_clear_mark_set = false}, behavior = {assign_variable = 0x107b908 <vm_assign_variable>, unassign_variable = 0x107b880 <vm_unassign_variable>, push_local_scope = 0x107b7d8 <vm_push_local_scope>, pop_local_scope = 0x107b840 <vm_pop_local_scope>, scope_stack_dup = 0x107b760 <vm_scope_stack_dup>, get_variable = 0x107b650 <vm_get_variable>, eval_project_file = 0x10552e0 <eval_project_file>, native_func_dispatch = 0x107baa0 <vm_native_func_dispatch>, pop_args = 0x10736b8 <vm_pop_args>, func_lookup = 0x105ba70 <func_lookup>, execute_loop = 0x107bb20 <vm_execute_loop>}, compiler_state = {nodes = {buckets = { len = 1, cap = 1, item_size = 16, e = 0x11b5d30 ""}, item_size = 48, bucket_size = 2048, len = 47, tail_bucket = 0}, node_stack = {len = 0, cap = 4096, item_size = 8, e = 0x11adb00 ""}, loop_jmp_stack = {len = 0, cap = 64, item_size = 4, e = 0x11b5c20 ""}, if_jmp_stack = {len = 0, cap = 64, item_size = 4, e = 0x11b5b10 ""}, err = false}, dbg_state = {node = 0, last_line = 0, stepping = false, break_on_err = false, dump_signature = false, watched = 0, eval_trace = 0, eval_trace_subdir = true}, lang_mode = language_external, run = true, saw_disabler = false, in_analyzer = false, disable_fuzz_unsafe_functions = false, error = false}, stack = {mem = 0x1205550 "", len = 37, cap = 4096, name = 0x0, log = false, cb = 0x0, ctx = 0x0}, projects = {len = 1, cap = 16, item_size = 108, e = 0x1206560 ""}, option_overrides = {len = 0, cap = 32, item_size = 20, e = 0x1206c30 ""}, cur_project = 0} (gdb) stepi 0x0000000000000000 in ?? () (gdb) stepi Program received signal SIGSEGV, Segmentation fault. 0x0000000000000000 in ?? ()Edit: to be clear, I was debugging the previously mentioned minimal test file.
Here's why: for some reason,
wk->vm.ops.ops[wk->vm.code.e[cip]]is zero, and of course an attempt to call a function with an address of zero leads to a segfault. I have no idea how I can debug this further :/(gdb) print wk->vm.ops.ops[wk->vm.code.e[cip]] $8 = (vm_op_fn) 0x0
After a rather difficult bisecting session I've been almost able to figure out which commit introduced the regression: it's one of c10df431b05df3446927d77abc232e7a65b01725 and a0c8580983b173a9d2a9698aba67ab5aeea4be69, but I can't be sure since I cannot compile the latter.
Still, I am not sure what I've found is the exact same issue I've reported above, since here muon doesn't segfault, but aborts due to a failed assertion:
$ ./build/muon setup -Dsamurai=disabled -Dbestline=disabled build2 detected compiler gcc '13.3.0' (['cc']), linker: ld.bfd (['ld.bfd']), static_linker: ar (gcc) (['ar']) configuring 'muon', version: 0.2.0 muon: ../src/lang/vm.c:73: object_stack_pop_entry: Assertion `s->bucket' failed. (null):1:1: error encountered unhandled error qemu: uncaught target signal 6 (Aborted) - core dumped Aborted
The commit(s) above is the one which first broke muon on s390x (and big endian in general, probably), while the segfault specifically has been introduced in f267a9d070c090d9bd757d3687f8b3acd24e1176 (analyzer re-impl
#1).That's the best I can do, I believe :)
Wow, thanks so much! I'm on vacation all this week but if I find some time I might take a look at this. Based on your debugging I suspect the issue may be in the way muon is packing the bytecode. If
wk->vm.ops.ops[wk->vm.code.e[cip]]is 0 then something has gone horribly wrong because it means that the bytecodewk->vm.code.e[cip]is invalid, and likely reading out of bounds which could trigger lots of strange things. The reason the bytecode would be invalid is probably because the instruction pointer was incremented wrong :(
Hmm, I tried using the binary you supplied but it seems like it has been stripped so it won't be very useful. If you don't mind making one with debug symbols that would be helpful :)
Instead of giving you a binary, I'll try to explain how I managed to compile one myself.
I did all of this in Debian, but it should work on other distros too. Still, I recommend using Debian for the s390x chroot since it is one of the few with official s390x support. It is also fairly easy, since debootstrap is packaged by most distros.
First, setup the s390x environment
$ apt install debootstrap qemu-user-static gdb-multiarch $ # make sure you have binfmt support! # debootstrap --arch=s390x unstable unstable-s390x $ # You can also use chroot, you just need to manually set up /dev, /proc, etc. # systemd-nspawn --directory=unstable-s390x --suppress-sync=true $ # We are now inside the container # apt --update install --no-install-recommends gcc libc6-dev git ca-certificates # cd /opt # git clone https://git.sr.ht/~lattis/muon # cd muon # CFLAGS='-static -g3' ./bootstrap.sh buildNow, in another terminal session outside of the container, attach GDB to the foreign binary with the help of QEMU:
$ cd unstable-s390x/opt/muon # qemu-s390x-static -g 1234 build/muon setup buildFinally, in yet another terminal session, start GDB and attach it to the foreign binary
$ cd unstable-s390x/opt/muon $ gdb-multiarch (gdb) set architecture s390:64-bit (gdb) target remote localhost:1234And you can now start debugging!
the latest code has fixes for big-endian arch. Try to pull and check again, please
I can confirm that muon now works again on big-endian, thanks! I've only tested with qemu, so before closing this I'll update the Debian package to see if the build succeeds too on actual s390x hardware.
...I cannot actually push the update as is, since commit 2f2af259729e65e68241b80f0bc34b3c7ad1cc21 introduced some -Warray-bounds warnings, which are currently treated as errors.
Here's one error example:
In function ‘lookup_toolchain_arg_override’, inlined from ‘toolchain_linker_lib’ at ../src/compilers.c:1851:1: ../src/compilers.c:1749:20: warning: array subscript 27 is outside array bounds of ‘struct toolchain_arg_handler[19]’ [-Warray-bounds=] 1749 | && obj_dict_index_str(wk, overrides, toolchain_arg_handlers[component].handlers[arg].name, &override)) { | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ../src/compilers.c: In function ‘toolchain_linker_lib’: ../src/compilers.c:1673:37: note: at offset 432 into object ‘toolchain_linker_arg_handlers’ of size 304 1673 | static struct toolchain_arg_handler toolchain_linker_arg_handlers[] = { FOREACH_LINKER_ARG(TOOLCHAIN_ARG_MEMBER) }; | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~Running the tests with the address sanitizer doesn't reveal anything suspicious though, so this might be a false positive. Let me know!
maybe using valgrind gives another output that could help ?
I tried to repro this, but bookworm s390x (VM) gives me
$ CFLAGS='-static -g3' ./bootstrap.sh build + c99 -static -g3 -Iinclude -I/usr/include/pkgconf -DBOOTSTRAP_HAVE_LIBPKGCONF src/amalgam.c -lpkgconf -o build/muon $ ./build/muon setup -Dsamurai=disabled build2 detected compiler gcc '12.2.0' (['cc']), linker: ld (['ld']), static_linker: ar (['ar']) configuring 'muon', version: 0.2.0 c compiler: supports argument '-Wendif-labels': YES c compiler: supports argument '-Wimplicit-fallthrough=2': YES c compiler: supports argument '-Winit-self': YES c compiler: supports argument '-Wlogical-op': YES c compiler: supports argument '-Wmissing-include-dirs': YES c compiler: supports argument '-Wno-missing-braces': YES c compiler: supports argument '-Wno-missing-field-initializers': YES c compiler: supports argument '-Wno-unused-parameter': YES c compiler: supports argument '-Wold-style-definition': YES c compiler: supports argument '-Woverflow': YES c compiler: supports argument '-Wstrict-aliasing=2': YES c compiler: supports argument '-Wstrict-prototypes': YES c compiler: supports argument '-Wundef': YES c compiler: supports argument '-Wvla': YES c compiler: supports argument '-fstrict-aliasing': YES c compiler: supports argument '-std=c99': YES configuring '/tmp/muon/build2/src/version.c' found dependency libcurl version 7.88.1 found dependency libarchive version 3.6.2 found dependency libpkgconf version 1.8.1 [tinyjson] entering subproject 'tinyjson' [tinyjson] detected compiler gcc '12.2.0' (['cc']), linker: ld (['ld']), static_linker: ar (['ar']) [tinyjson] configuring 'tiny-json', version: undefined found dependency 'tinyjson' (declared dependency) version undefined static warn dependency ['tracy'] not found /tmp/muon/tests/project/meson.build:397:22: error cannot return the full_path() of an external program with multiple elements (have: ['/tmp/muon/build/muon', 'samu']) 397 | ninja = ninja.full_path() ^___________ /tmp/muon/tests/project/meson.build:397:22: error in method external_program.full_path 397 | ninja = ninja.full_path() ^___________ /tmp/muon/tests/meson.build:10:1: error in function subdir 10 | subdir('project') ^________________ /tmp/muon/meson.build:151:1: error in function subdir 151 | subdir('tests') ^______________on latest HEAD (788e2860fb58634e968de9571b1049881983fa81)
Ah, this is something I hadn't considered: passing
-Dsamurai=disabledto configure muon combined with muon currently being configured to have samurai enabled. I'll fix this soon. I think this isn't a s390x issue though.