...I cannot actually push the update as is, since commit 2f2af259729e65e68241b80f0bc34b3c7ad1cc21 introduced some -Warray-bounds warnings, which are currently treated as errors.

Here's one error example:

In function ‘lookup_toolchain_arg_override’,
    inlined from ‘toolchain_linker_lib’ at ../src/compilers.c:1851:1:
../src/compilers.c:1749:20: warning: array subscript 27 is outside array bounds of ‘struct toolchain_arg_handler[19]’ [-Warray-bounds=]
 1749 |                 && obj_dict_index_str(wk, overrides, toolchain_arg_handlers[component].handlers[arg].name, &override)) {
      |                    ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
../src/compilers.c: In function ‘toolchain_linker_lib’:
../src/compilers.c:1673:37: note: at offset 432 into object ‘toolchain_linker_arg_handlers’ of size 304
 1673 | static struct toolchain_arg_handler toolchain_linker_arg_handlers[] = { FOREACH_LINKER_ARG(TOOLCHAIN_ARG_MEMBER) };
      |                                     ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Running the tests with the address sanitizer doesn't reveal anything suspicious though, so this might be a false positive. Let me know!

I can confirm that muon now works again on big-endian, thanks! I've only tested with qemu, so before closing this I'll update the Debian package to see if the build succeeds too on actual s390x hardware.

Instead of giving you a binary, I'll try to explain how I managed to compile one myself.

I did all of this in Debian, but it should work on other distros too. Still, I recommend using Debian for the s390x chroot since it is one of the few with official s390x support. It is also fairly easy, since debootstrap is packaged by most distros.

First, setup the s390x environment

$ apt install debootstrap qemu-user-static gdb-multiarch
$ # make sure you have binfmt support!
# debootstrap --arch=s390x unstable unstable-s390x
$ # You can also use chroot, you just need to manually set up /dev, /proc, etc.
# systemd-nspawn --directory=unstable-s390x --suppress-sync=true
$ # We are now inside the container
# apt --update install --no-install-recommends gcc libc6-dev git ca-certificates
# cd /opt
# git clone https://git.sr.ht/~lattis/muon
# cd muon
# CFLAGS='-static -g3' ./bootstrap.sh build

Now, in another terminal session outside of the container, attach GDB to the foreign binary with the help of QEMU:

$ cd unstable-s390x/opt/muon
# qemu-s390x-static -g 1234 build/muon setup build

Finally, in yet another terminal session, start GDB and attach it to the foreign binary

$ cd unstable-s390x/opt/muon
$ gdb-multiarch
(gdb) set architecture s390:64-bit 
(gdb) target remote localhost:1234

And you can now start debugging!

The commit(s) above is the one which first broke muon on s390x (and big endian in general, probably), while the segfault specifically has been introduced in f267a9d070c090d9bd757d3687f8b3acd24e1176 (analyzer re-impl #1).

That's the best I can do, I believe :)

After a rather difficult bisecting session I've been almost able to figure out which commit introduced the regression: it's one of c10df431b05df3446927d77abc232e7a65b01725 and a0c8580983b173a9d2a9698aba67ab5aeea4be69, but I can't be sure since I cannot compile the latter.

Still, I am not sure what I've found is the exact same issue I've reported above, since here muon doesn't segfault, but aborts due to a failed assertion:

$ ./build/muon setup -Dsamurai=disabled -Dbestline=disabled build2
detected compiler gcc '13.3.0' (['cc']), linker: ld.bfd (['ld.bfd']), static_linker: ar (gcc) (['ar'])
configuring 'muon', version: 0.2.0
muon: ../src/lang/vm.c:73: object_stack_pop_entry: Assertion `s->bucket' failed.
(null):1:1: error encountered unhandled error
qemu: uncaught target signal 6 (Aborted) - core dumped
Aborted

Here's why: for some reason, wk->vm.ops.ops[wk->vm.code.e[cip]] is zero, and of course an attempt to call a function with an address of zero leads to a segfault. I have no idea how I can debug this further :/

(gdb) print wk->vm.ops.ops[wk->vm.code.e[cip]]
$8 = (vm_op_fn) 0x0

After running the program under gdb-multiarch, I've been able to figure out the exact spot where the program segfaults: in src/lang/vm.c, at line 2261 (wk->vm.ops.ops[wk->vm.code.e[cip]](wk))... Which doesn't seem really useful. Here's the state of the wk variable right before crashing:

Breakpoint 2, 0x000000000107bbaa in vm_execute_loop (wk=0x200007ff2d8) at src/lang/vm.c:2261
2261			wk->vm.ops.ops[wk->vm.code.e[cip]](wk);
(gdb) print *wk
$1 = {argv0 = 0x11cdf51 "/tmp/tmp.wRC0rQbFIN/unstable-s390x/root/muon-cafeb37c/muon", 
  source_root = 0x11cddb0 "/tmp/tmp.wRC0rQbFIN/unstable-s390x/root/muon-cafeb37c", 
  build_root = 0x11cdf15 "/tmp/tmp.wRC0rQbFIN/unstable-s390x/root/muon-cafeb37c/build", 
  muon_private = 0x11cdf8c "/tmp/tmp.wRC0rQbFIN/unstable-s390x/root/muon-cafeb37c/build/.muon", original_commandline = {argc = 1, 
    argv = 0x20000800ca8}, regenerate_deps = 18, host_machine = 17, binaries = 16, install = 19, install_scripts = 20, 
  postconf_scripts = 21, subprojects = 22, global_args = 23, global_link_args = 24, dep_overrides_static = 25, 
  dep_overrides_dynamic = 26, find_program_overrides = 27, global_opts = 28, compiler_check_cache = 29, dependency_handlers = 30, 
  vm = {stack = {ba = {buckets = {len = 1, cap = 1, item_size = 16, e = 0x11a7690 ""}, item_size = 8, bucket_size = 128, len = 0, 
        tail_bucket = 0}, page = 0x11a76b0, i = 0, bucket = 0}, call_stack = {len = 1, cap = 64, item_size = 32, e = 0x11a7ac0 ""}, 
    locations = {len = 469, cap = 1024, item_size = 16, e = 0x11a9af0 ""}, code = {len = 1149, cap = 4096, item_size = 1, 
      e = 0x11a82d0 ""}, src = {len = 4, cap = 64, item_size = 32, e = 0x11a92e0 ""}, ip = 1, nargs = 1, nkwargs = 0, 
    scope_stack = 108, default_scope_stack = 4, ops = {ops = {0x0, 0x1076070 <vm_op_constant>, 0x10760d8 <vm_op_constant_list>, 
        0x10761d8 <vm_op_constant_dict>, 0x10763a0 <vm_op_constant_func>, 0x1076678 <vm_op_add>, 0x1077078 <vm_op_sub>, 
        0x1077318 <vm_op_mul>, 0x1077860 <vm_op_div>, 0x10775b8 <vm_op_mod>, 0x1078760 <vm_op_not>, 0x1078610 <vm_op_eq>, 
        0x10781e8 <vm_op_in>, 0x1077f68 <vm_op_gt>, 0x1077ce8 <vm_op_lt>, 0x10788f8 <vm_op_negate>, 0x1078ab0 <vm_op_stringify>, 
        0x1078c08 <vm_op_store>, 0x1076b18 <vm_op_add_store>, 0x1078dc8 <vm_op_load>, 0x1078ed0 <vm_op_try_load>, 
        0x107adc8 <vm_op_return>, 0x107adc8 <vm_op_return>, 0x1079848 <vm_op_call>, 0x1079900 <vm_op_call_method>, 
        0x1079d38 <vm_op_call_native>, 0x1078fc0 <vm_op_index>, 0x1079f20 <vm_op_iterator>, 0x107a4b8 <vm_op_iterator_next>, 
        0x107ad68 <vm_op_jmp>, 0x107ab80 <vm_op_jmp_if_true>, 0x107ac70 <vm_op_jmp_if_false>, 0x107aa58 <vm_op_jmp_if_disabler>, 
        0x107aaf8 <vm_op_jmp_if_disabler_keep>, 0x107a940 <vm_op_pop>, 0x107a978 <vm_op_dup>, 0x107a9d0 <vm_op_swap>, 
        0x107aef0 <vm_op_typecheck>, 0x0, 0x0}}, objects = {chrs = {buckets = {len = 1, cap = 1, item_size = 16, e = 0x11cdd60 ""}, 
        item_size = 1, bucket_size = 4096, len = 3073, tail_bucket = 0}, objs = {buckets = {len = 1, cap = 1, item_size = 16, 
          e = 0x11ced90 ""}, item_size = 8, bucket_size = 1024, len = 271, tail_bucket = 0}, dict_elems = {buckets = {len = 1, 
          cap = 1, item_size = 16, e = 0x11d0dc0 ""}, item_size = 12, bucket_size = 1024, len = 46, tail_bucket = 0}, dict_hashes = {
        buckets = {len = 1, cap = 1, item_size = 16, e = 0x11d3df0 ""}, item_size = 152, bucket_size = 16, len = 1, tail_bucket = 0}, 
      obj_aos = {{buckets = {len = 1, cap = 1, item_size = 16, e = 0x11d47a0 ""}, item_size = 8, bucket_size = 1024, len = 0, 
          tail_bucket = 0}, {buckets = {len = 1, cap = 1, item_size = 16, e = 0x11d67d0 ""}, item_size = 16, bucket_size = 1024, 
          len = 130, tail_bucket = 0}, {buckets = {len = 1, cap = 1, item_size = 16, e = 0x11da800 ""}, item_size = 20, 
          bucket_size = 2048, len = 71, tail_bucket = 0}, {buckets = {len = 1, cap = 1, item_size = 16, e = 0x11e4830 ""}, 
          item_size = 16, bucket_size = 512, len = 24, tail_bucket = 0}, {buckets = {len = 1, cap = 1, item_size = 16, 
            e = 0x11e6860 ""}, item_size = 40, bucket_size = 4, len = 1, tail_bucket = 0}, {buckets = {len = 1, cap = 1, 
            item_size = 16, e = 0x11e6930 ""}, item_size = 180, bucket_size = 16, len = 0, tail_bucket = 0}, {buckets = {len = 1, 
            cap = 1, item_size = 16, e = 0x11e74a0 ""}, item_size = 36, bucket_size = 16, len = 0, tail_bucket = 0}, {buckets = {
            len = 1, cap = 1, item_size = 16, e = 0x11e7710 ""}, item_size = 8, bucket_size = 16, len = 0, tail_bucket = 0}, {
          buckets = {len = 1, cap = 1, item_size = 16, e = 0x11e77c0 ""}, item_size = 80, bucket_size = 16, len = 0, tail_bucket = 0}, 
        {buckets = {len = 1, cap = 1, item_size = 16, e = 0x11e7cf0 ""}, item_size = 12, bucket_size = 32, len = 0, tail_bucket = 0}, {
          buckets = {len = 1, cap = 1, item_size = 16, e = 0x11e7ea0 ""}, item_size = 24, bucket_size = 32, len = 0, tail_bucket = 0}, 
        {buckets = {len = 1, cap = 1, item_size = 16, e = 0x11e81d0 ""}, item_size = 16, bucket_size = 32, len = 0, tail_bucket = 0}, {
          buckets = {len = 1, cap = 1, item_size = 16, e = 0x11e8400 ""}, item_size = 4, bucket_size = 16, len = 0, tail_bucket = 0}, {
          buckets = {len = 1, cap = 1, item_size = 16, e = 0x11e8470 ""}, item_size = 48, bucket_size = 64, len = 0, tail_bucket = 0}, 
        {buckets = {len = 1, cap = 1, item_size = 16, e = 0x11e90a0 ""}, item_size = 12, bucket_size = 16, len = 0, tail_bucket = 0}, {
          buckets = {len = 1, cap = 1, item_size = 16, e = 0x11e9190 ""}, item_size = 32, bucket_size = 128, len = 0, 
          tail_bucket = 0}, {buckets = {len = 1, cap = 1, item_size = 16, e = 0x11ea1c0 ""}, item_size = 4, bucket_size = 4, len = 0, 
          tail_bucket = 0}, {buckets = {len = 1, cap = 1, item_size = 16, e = 0x11ea200 ""}, item_size = 8, bucket_size = 16, len = 0, 
          tail_bucket = 0}, {buckets = {len = 1, cap = 1, item_size = 16, e = 0x11ea2b0 ""}, item_size = 44, bucket_size = 32, 
          len = 29, tail_bucket = 0}, {buckets = {len = 1, cap = 1, item_size = 16, e = 0x11ea860 ""}, item_size = 20, 
          bucket_size = 16, len = 0, tail_bucket = 0}, {buckets = {len = 1, cap = 1, item_size = 16, e = 0x11ea9d0 ""}, 
          item_size = 16, bucket_size = 16, len = 0, tail_bucket = 0}, {buckets = {len = 1, cap = 1, item_size = 16, 
            e = 0x11eab00 ""}, item_size = 8, bucket_size = 4, len = 0, tail_bucket = 0}, {buckets = {len = 1, cap = 1, 
            item_size = 16, e = 0x11eab50 ""}, item_size = 8, bucket_size = 4, len = 0, tail_bucket = 0}, {buckets = {len = 1, 
            cap = 1, item_size = 16, e = 0x11eaba0 ""}, item_size = 8, bucket_size = 4, len = 0, tail_bucket = 0}, {buckets = {
            len = 1, cap = 1, item_size = 16, e = 0x11eabf0 ""}, item_size = 8, bucket_size = 4, len = 0, tail_bucket = 0}, {
          buckets = {len = 1, cap = 1, item_size = 16, e = 0x11eac40 ""}, item_size = 24, bucket_size = 32, len = 0, tail_bucket = 0}, 
        {buckets = {len = 1, cap = 1, item_size = 16, e = 0x11eaf70 ""}, item_size = 3136, bucket_size = 32, len = 0, 
          tail_bucket = 0}, {buckets = {len = 1, cap = 1, item_size = 16, e = 0x12037a0 ""}, item_size = 3136, bucket_size = 64, 
          len = 0, tail_bucket = 0}, {buckets = {len = 4, cap = 4, item_size = 16, e = 0x1207580 ""}, item_size = 16, 
          bucket_size = 32, len = 116, tail_bucket = 3}}, obj_hash = {meta = {len = 0, cap = 128, item_size = 1, 
          e = 0x12039f0 '\200' <repeats 128 times>}, e = {len = 0, cap = 128, item_size = 16, e = 0x1203a80 ""}, keys = {len = 0, 
          cap = 128, item_size = 4, e = 0x1204290 ""}, cap = 128, len = 0, load = 0, max_load = 64, capm = 127, 
        keycmp = 0x1014970 <hash_keycmp_memcmp>, hash_func = 0x1014808 <fnv_1a_64>}, str_hash = {meta = {len = 0, cap = 256, 
          item_size = 1, 
          e = 0x1207bd0 "`\200K\200\200\200\f\200\200QF\200\200\200\200\200XX\200\200\200v\200\035\200;jJ\030\2005\022\200\200\021\200\200\200c*]\200\200\200\200\200\200\022\016\200\200\200g\200\200\200-Fr})bT6\200\200\177\200\200X%\200"}, e = {len = 0, cap = 256, 
          item_size = 16, e = 0x1209090 ""}, keys = {len = 113, cap = 128, item_size = 16, e = 0x1204d40 ""}, cap = 256, len = 113, 
        load = 113, max_load = 128, capm = 255, keycmp = 0x1014b68 <hash_keycmp_strcmp>, hash_func = 0x1014750 <fnv_1a_64_str>}, 
      obj_clear_mark_set = false}, behavior = {assign_variable = 0x107b908 <vm_assign_variable>, 
      unassign_variable = 0x107b880 <vm_unassign_variable>, push_local_scope = 0x107b7d8 <vm_push_local_scope>, 
      pop_local_scope = 0x107b840 <vm_pop_local_scope>, scope_stack_dup = 0x107b760 <vm_scope_stack_dup>, 
      get_variable = 0x107b650 <vm_get_variable>, eval_project_file = 0x10552e0 <eval_project_file>, 
      native_func_dispatch = 0x107baa0 <vm_native_func_dispatch>, pop_args = 0x10736b8 <vm_pop_args>, 
      func_lookup = 0x105ba70 <func_lookup>, execute_loop = 0x107bb20 <vm_execute_loop>}, compiler_state = {nodes = {buckets = {
          len = 1, cap = 1, item_size = 16, e = 0x11b5d30 ""}, item_size = 48, bucket_size = 2048, len = 47, tail_bucket = 0}, 
      node_stack = {len = 0, cap = 4096, item_size = 8, e = 0x11adb00 ""}, loop_jmp_stack = {len = 0, cap = 64, item_size = 4, 
        e = 0x11b5c20 ""}, if_jmp_stack = {len = 0, cap = 64, item_size = 4, e = 0x11b5b10 ""}, err = false}, dbg_state = {node = 0, 
      last_line = 0, stepping = false, break_on_err = false, dump_signature = false, watched = 0, eval_trace = 0, 
      eval_trace_subdir = true}, lang_mode = language_external, run = true, saw_disabler = false, in_analyzer = false, 
    disable_fuzz_unsafe_functions = false, error = false}, stack = {mem = 0x1205550 "", len = 37, cap = 4096, name = 0x0, log = false, 
    cb = 0x0, ctx = 0x0}, projects = {len = 1, cap = 16, item_size = 108, e = 0x1206560 ""}, option_overrides = {len = 0, cap = 32, 
    item_size = 20, e = 0x1206c30 ""}, cur_project = 0}
(gdb) stepi
0x0000000000000000 in ?? ()
(gdb) stepi

Program received signal SIGSEGV, Segmentation fault.
0x0000000000000000 in ?? ()

Edit: to be clear, I was debugging the previously mentioned minimal test file.

I've been able to more or less figure out what causes crashes: the if statement.

Here's a minimal example:

project('muon', 'c')

compiler = meson.get_compiler('c')
has_printf = compiler.has_function('printf')
message(f'showing that has_function works: @has_printf@')

if true
  message('inside if, I\'m about to crash')
endif

message('outside if')

And here's what happens when running it with a stage 1 muon:

$ ./muon setup build
detected compiler gcc '13.3.0' (['cc']), linker: ld (gnu) (['ld']), static_linker: ar (gcc) (['ar'])
configuring 'muon', version: undefined
c compiler: has function printf: YES
message: showing that has_function works: true 
message: inside if, I'm about to crash 
qemu: uncaught target signal 11 (Segmentation fault) - core dumped

Pretty odd!

The stage 1 bootstrap muon binary doesn't segfault right away, for example it handles muon version, muon check meson.build and muon fmt meson.build (with muon's own meson.build file). It seems that it crashes only when running muon setup.

Here's a statically-linked s390x stage-1 muon binary with undefined behaviour sanitizer enabled, that you can test with qemu-user-s390x: https://andrea.pappacoda.it/muon

muon is also segfaulting on all other Debian's big-endian architectures (ppc64 and sparc64), so it definitely seems that muon doesn't work on big-endian systems.