basic test error

[FrancisLeeHa]

Hello. When I run mvx basic test on arm, I got this error on x86 kernel, It seems to be the filename or command error, but I don`t know how to fix it. Would you please help me, thanks!
I make this experiment on qume
popcorn@arm:~/mvx$ LD_PRELOAD=./loader.so ./basic

popcorn@x86:~/mvx$ [42121.379789] SYSC_open: filename open.c, ret 0
[42121.383889] SYSC_open: ret -2 (0xfffffffe)
[42121.387782] mvx_update_fd_vtab: Should not happen for fd id -2
[42121.407778] follower ret -9 != 18 from master
[42121.423786] [MVX Violation] kernel/popcorn/mvx_engine.c:328 mvx_follower_wait_exec
[42121.427780] mvx_update_fd_vtab: Should not happen for fd id -1
[42121.443900] follower ret -9 != 18 from master
[42121.455883] [MVX Violation] kernel/popcorn/mvx_engine.c:225 follower_writev
[42121.459780] follower ret -9 != 63 from master
[42121.463961] ------------[ cut here ]------------
[42121.464863] kernel BUG at fs/read_write.c:1007!
[42121.465599] invalid opcode: 0000 [#6] SMP
[42121.466357] Modules linked in: msg_socket
[42121.467057] CPU: 0 PID: 27729 Comm: open Tainted: G D 4.4.137-popcorn+ #1
[42121.467773] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.13.0-1ubuntu1 04/01/2014
[42121.467773] task: ffff8800bb78c580 ti: ffff8800b9134000 task.ti: ffff8800b9134000
[42121.467773] RIP: 0010:[] [] SyS_writev+0x1c7/0x220
[42121.467773] RSP: 0018:ffff8800b9137e90 EFLAGS: 00010202
[42121.467773] RAX: 000000000000007f RBX: 0000000000000001 RCX: 0000000000000000
[42121.467773] RDX: 000000000000007f RSI: 0000000000000000 RDI: ffff8800b9137f1f
[42121.467773] RBP: ffff8800bac55780 R08: 0000000000000004 R09: ffff8800b9137ea0
[42121.467773] R10: 0000000000000000 R11: 0000000000000000 R12: 000000000000007f
[42121.467773] R13: 000000000000007f R14: 0000000000000000 R15: 0000000000000000
[42121.467773] FS: 00007ffff7ffe288(0000) GS:ffff88013fc00000(0000) knlGS:0000000000000000
[42121.467773] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[42121.467773] CR2: 0000000000000003 CR3: 00000000bb74a000 CR4: 0000000000360670
[42121.467773] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[42121.467773] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[42121.467773] Stack:
[42121.467773] 000000000000003f ffffffff813fc25e 0000000000000000 0000000000000000
[42121.467773] 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[42121.467773] 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[42121.467773] Call Trace:
[42121.467773] [] ? __schedule+0x1ce/0x730
[42121.467773] [] ? entry_SYSCALL_64_fastpath+0x1e/0x8e
[42121.467773] Code: c3 76 2a 48 c1 e0 04 4d 89 e5 48 8d 7c 24 10 48 01 e8 48 83 78 08 7f 4c 0f 46 68 08 48 8b 30 44 89 ea e8 2d a1 13 00 85 c0 74 c6 <0f> 0b 48 89 ef e8 cf 94 ff ff 48 8b
[42121.467773] RIP [] SyS_writev+0x1c7/0x220
[42121.467773] RSP
[42121.467777] ---[ end trace 08c469fbdb08ec3f ]---

[xjtuwxg]

Hi, you need to build the basic on both x86 and arm nodes, using the same version of musl-gcc. Also, make sure to move the basic binary to the same location on each node.

[FrancisLeeHa]

Hi, you need to build the basic on both x86 and arm nodes, using the same version of musl-gcc. Also, make sure to move the basic binary to the same location on each node.

Thanks! But I just build the basic on both x86 and arm and using the same version of musl-gcc

popcorn@arm:~/mvx$ /usr/local/musl/bin/musl-gcc --version 
gcc (Debian/Linaro 4.9.2-10) 4.9.2
Copyright (C) 2014 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

popcorn@x86:~/mvx$  /usr/local/musl/bin/musl-gcc --version
gcc (Debian 4.9.2-10) 4.9.2
Copyright (C) 2014 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Also， I just put them to the same location on each node, but it still run with error.

• this is arm output

popcorn@arm:~/mvx$ pwd
/home/popcorn/mvx
popcorn@arm:~/mvx$ ls
Makefile  epoll    lighttpd       musl-1.2.0         open    tiny.c
basic     epoll.c  lighttpd.conf  musl-1.2.0.tar.gz  open.c
basic.c   info.h   loader.so      mvx_syscall.c      tiny
popcorn@arm:~/mvx$ LD_PRELOAD=./loader.so ./basic
Issue an MVX init syscall.
[ 1691.235831] 
[ 1691.235831] === HeterSec syscall ===
[ 1691.235977] Arg0: 0x1(1)
[ 1691.236098] [ 0] argv[0]: ./basic. arg_size 7
=== [MVX Unit Test] Testing 2 syscalls: ===
writev, read

Writting to stderr
[ 1691.249273] fd 2, count 20, ret 20
Input your name (type "quit" to exit): 
test
5 char(s) read, buf: test

^C
popcorn@arm:~/mvx$

• this is x86 output

popcorn@x86:~/mvx$ pwd
/home/popcorn/mvx
popcorn@x86:~/mvx$ ls
basic    epoll.c   lighttpd.conf  musl-1.2.0         open    tiny.c
basic.c  info.h    loader.so      musl-1.2.0.tar.gz  open.c
epoll    lighttpd  Makefile       mvx_syscall.c      tiny
popcorn@x86:~/mvx$ [ 1726.536003] follower ret -9 != 44 from master
[ 1726.537720] fd 2, count 20, ret -9
[ 1726.538878] [**MVX Violation**] kernel/popcorn/mvx_engine.c:254 follower_write
[ 1726.540680] follower ret -9 != 14 from master
[ 1726.543477] [**MVX Violation**] kernel/popcorn/mvx_engine.c:225 follower_writev
[ 1726.545314] follower ret -9 != 20 from master
[ 1726.548628] follower ret -9 != 40 from master
[ 1730.223155] [**MVX Violation**] kernel/popcorn/mvx_engine.c:225 follower_writev
[ 1730.225316] follower ret -9 != 5 from master
[ 1730.226863] ------------[ cut here ]------------
[ 1730.228359] kernel BUG at fs/read_write.c:1007!
[ 1730.229297] invalid opcode: 0000 [#2] SMP 
[ 1730.229297] Modules linked in: msg_socket
[ 1730.229297] CPU: 1 PID: 707 Comm: basic Tainted: G      D         4.4.137-popcorn+ #1
[ 1730.229297] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.13.0-1ubuntu1 04/01/2014
[ 1730.229297] task: ffff8800ba32c640 ti: ffff8800bb734000 task.ti: ffff8800bb734000
[ 1730.229297] RIP: 0010:[<ffffffff811616d7>]  [<ffffffff811616d7>] SyS_writev+0x1c7/0x220
[ 1730.229297] RSP: 0018:ffff8800bb737e90  EFLAGS: 00010202
[ 1730.229297] RAX: 000000000000007f RBX: 0000000000000001 RCX: 0000000000000000
[ 1730.229297] RDX: 000000000000007f RSI: 0000000000000000 RDI: ffff8800bb737f1f
[ 1730.229297] RBP: ffff8800b8815660 R08: 0000000000000007 R09: ffff8800bb737ea0
[ 1730.229297] R10: 0000000000000000 R11: 0000000000000000 R12: 000000000000007f
[ 1730.229297] R13: 000000000000007f R14: 0000000000000000 R15: 0000000000000000
[ 1730.229297] FS:  00007ffff7ffe288(0000) GS:ffff88013fd00000(0000) knlGS:0000000000000000
[ 1730.229297] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1730.229297] CR2: 0000000000000006 CR3: 00000000bb4ee000 CR4: 0000000000360670
[ 1730.229297] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1730.229297] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 1730.229297] Stack:
[ 1730.229297]  0000000000000005 ffffffff813fc25e 0000000000000000 0000000000000000
[ 1730.229297]  0000000000000000 0000000000000000 0000000000000000 0000000000000000
[ 1730.229297]  0000000000000000 0000000000000000 0000000000000000 0000000000000000
[ 1730.229297] Call Trace:
[ 1730.229297]  [<ffffffff813fc25e>] ? __schedule+0x1ce/0x730
[ 1730.229297]  [<ffffffff813ff90a>] ? entry_SYSCALL_64_fastpath+0x1e/0x8e
[ 1730.229297] Code: c3 76 2a 48 c1 e0 04 4d 89 e5 48 8d 7c 24 10 48 01 e8 48 83 78 08 7f 4c 0f 46 68 08 48 8b 30 44 89 ea e8 2d a1 13 00 85 c0 74 c6 <0f> 0b 48 89 ef e8 cf 94 ff ff 48 8b  
[ 1730.229297] RIP  [<ffffffff811616d7>] SyS_writev+0x1c7/0x220
[ 1730.229297]  RSP <ffff8800bb737e90>
[ 1730.254822] ---[ end trace fddde1203d3b9a64 ]---
popcorn@x86:~/mvx$

And if I run it separately, it will run correctly.

• this is arm

popcorn@arm:~/mvx$ ./basic 
=== [MVX Unit Test] Testing 2 syscalls: ===
writev, read

Writting to stderr
Input your name (type "quit" to exit): 
test
5 char(s) read, buf: test

Input your name (type "quit" to exit): 
quit
5 char(s) read, buf: quit

popcorn@arm:~/mvx$

• this is x86

 popcorn@x86:~/mvx$ ./basic 
=== [MVX Unit Test] Testing 2 syscalls: ===
writev, read

Writting to stderr
Input your name (type "quit" to exit): 
test
5 char(s) read, buf: test

Input your name (type "quit" to exit): 
exit
5 char(s) read, buf: exit

Input your name (type "quit" to exit): 
quit
5 char(s) read, buf: quit

popcorn@x86:~/mvx$

How to fix it? thanks

[xjtuwxg]

Hi, could you enable the debug information especially on the x86 side, and share us the log?
make menuconfig --> Popcorn Distributed Execution Support --> [*] MVX engine; [*] Syscall server; [*] Task migration

Also, could you share us the binaries you have (basic.x86 basic.arm)?

[FrancisLeeHa]

OK, I will share the binaries now and log later, I send binaries to your email xiaoguang@vt.edu, thanks!

[FrancisLeeHa]

I have the log now

• on arm

popcorn@arm:~/mvx$ LD_PRELOAD=./loader.so ./basic 
Issue an MVX init syscall.
[  333.494540] 
[  333.494540] === HeterSec syscall ===
[  333.494688] Arg0: 0x1(1)
[  333.495377] [ 0] argv[0]: ./basic. arg_size 7
[  333.495505] [MVX] mvx_prepare_init_req: Retrieved argc 1, cmdline size 8, cmdline ./basic, exe path /home/popcorn/mvx/basic
[  333.495690] [MVX] mvx_server_start_mvx: [268] exe_path /home/popcorn/mvx/basic. arg_start 0x7fffffffeed9, arg_end 0x7fffffffeee1
[  333.495871] [MVX] mvx_server_start_mvx: size 2070; dst nid 0
[  333.502394] [MVX] MVX master variant [PID:268] init ...
=== [MVX Unit Test] Testing 2 syscalls: ===
[  333.503957] [MVX] 
[  333.503957] [ 0] syscall 66: <SYS_writev>
[  333.504252] [MVX] master waits for follower return on syscall 66
[  333.534449] [MVX] --> handle_mvx_reply: ret 0x2c(44), follower syscall 20
[  333.534822] [MVX] 
[  333.534822] SYSC_writev: mvx [268]. fd 1, vlen 2. MVX master.
[  333.535102] [MVX] SYSC_writev: buf: === [MVX Unit Test] Testing 2
[  333.535220] [MVX] SYSC_writev: buf:  syscalls: ===
writev, read

[  333.535844] [MVX] 
[  333.535844] [ 1] syscall 66: <SYS_writev>
[  333.535996] [MVX] master waits for follower return on syscall 66
[  333.544207] [MVX] --> handle_mvx_reply: ret 0xe(14), follower syscall 1
[  333.544681] [MVX] 
[  333.544681] SYSC_writev: mvx [268]. fd 1, vlen 2. MVX master.
[  333.544997] [MVX] SYSC_writev: buf: writev, read
[  333.545157] [MVX] SYSC_writev: buf: 
[  333.545157] 
Writting to stderr
[  333.549648] fd 2, count 20, ret 20
[  333.549755] [MVX] 
[  333.549755] [ 2] syscall 64: <SYS_write>
[  333.549907] [MVX] master waits for follower return on syscall 64
[  333.553855] [MVX] --> handle_mvx_reply: ret 0x14(20), follower syscall 20
[  333.554736] [MVX] SYSC_write: mvx [268] Writting to stderr
[  333.554736] 
[  333.554870] [MVX] Not support SYSC_write yet.
Input your name (type "quit" to exit): 
[  333.555260] [MVX] 
[  333.555260] [ 3] syscall 66: <SYS_writev>
[  333.555389] [MVX] master waits for follower return on syscall 66
[  333.559381] [MVX] --> handle_mvx_reply: ret 0x28(40), follower syscall 20
[  333.559860] [MVX] 
[  333.559860] SYSC_writev: mvx [268]. fd 1, vlen 2. MVX master.
[  333.560175] [MVX] SYSC_writev: buf: Input your name (type "quit" to exit): 
[  333.560402] [MVX] SYSC_writev: buf: 
test[  345.775783] [MVX] 
[  345.775783] [ 4] syscall 63: <SYS_read>
[  345.776341] [MVX] master_read: fd 0, flag 2, len 5, retval 5.
[  345.776651] [MVX] master waits for follower return on syscall 63

[  345.788424] [MVX] --> handle_mvx_reply: ret 0x5(5), follower syscall 20
[  345.788887] [MVX post syscall] SYSC_read: ret 5
5 char(s) read, buf: test
[  345.790600] [MVX] 
[  345.790600] [ 5] syscall 66: <SYS_writev>
[  345.790888] [MVX] master waits for follower return on syscall 66

• on x86

popcorn@x86:~/mvx$ [  409.701426] [MVX] clone_mvx_thread: MVX helper thread [675] finished launching MVX process, bin path: /home/popcorn/mvx/basic.
[  409.704798] [MVX] argv[0] ./basic (7)
[  409.706156] [MVX] MVX follower variant [PID:676] init ...
[  409.708168] [MVX] --> handle_mvx_message: syscall #20, flag 2, len 0, retval 44 (master)
[  409.711781] [MVX] Cannot open STDIN file: -13.
[  409.712826] [MVX] [676] init_mvx_follower_process: Finished the Follower MVX process initialization.
[  409.716864] [MVX] mvx_process_fork: Finished MVX process setup. ret 0
[  409.731424] [MVX] 
[  409.731825] [MVX] Follower waits msg on
[  409.731825] [ 0] syscall: <SYS_writev>
[  409.733575] [MVX] syscall 20: <SYS_writev>. flag 2. master msg syscall 20
[  409.734889] follower ret -9 != 44 from master
[  409.735805] [MVX] 
[  409.735805] SYSC_writev: mvx [676]. fd 1, vlen 2. MVX follower.
[  409.737515] [MVX] SYSC_writev: buf: === [MVX Unit Test] Testing 2
[  409.738398] [MVX] SYSC_writev: buf:  syscalls: ===
[  409.739157] fd 2, count 20, ret -9
[  409.739841] [MVX] 
[  409.740064] [MVX] --> handle_mvx_message: syscall #20, flag 2, len 0, retval 14 (master)
[  409.741413] [MVX] Follower waits msg on
[  409.741413] [ 1] syscall: <SYS_write>
[  409.742974] [**MVX Violation**] kernel/popcorn/mvx_engine.c:254 follower_write
[  409.744118] [MVX] syscall 1: <SYS_write>. flag 2. master msg syscall 20
[  409.745140] follower ret -9 != 14 from master
[  409.745935] [MVX] SYSC_write: mvx [676] Writting to stderr
[  409.745935] 
[  409.746657] [MVX] Not support SYSC_write yet.
[  409.747053] [MVX] 
[  409.747245] [MVX] Follower waits msg on
[  409.747245] [ 2] syscall: <SYS_writev>
[  409.753264] [MVX] --> handle_mvx_message: syscall #1, flag 2, len 0, retval 20 (master)
[  409.754293] [**MVX Violation**] kernel/popcorn/mvx_engine.c:225 follower_writev
[  409.754957] [MVX] syscall 20: <SYS_writev>. flag 2. master msg syscall 1
[  409.755587] follower ret -9 != 20 from master
[  409.756159] [MVX] 
[  409.756159] SYSC_writev: mvx [676]. fd 1, vlen 2. MVX follower.
[  409.757033] [MVX] SYSC_writev: buf: writev, read
[  409.757033] 
[  409.757033] Input your name (type "quit" to exit): 
[  409.758033] [MVX] SYSC_writev: buf: 
[  409.758333] [MVX] 
[  409.758547] [MVX] Follower waits msg on
[  409.758547] [ 3] syscall: <SYS_writev>
[  409.759359] [MVX] --> handle_mvx_message: syscall #20, flag 2, len 0, retval 40 (master)
[  409.760348] [MVX] syscall 20: <SYS_writev>. flag 2. master msg syscall 20
[  409.760989] follower ret -9 != 40 from master
[  409.761409] [MVX] 
[  409.761409] SYSC_writev: mvx [676]. fd 1, vlen 2. MVX follower.
[  409.762224] [MVX] SYSC_writev: buf: your name (type "quit" to exit): 
[  409.762799] [MVX] SYSC_writev: buf: 
[  409.763099] [MVX] 
[  409.763309] [MVX] Follower waits msg on
[  409.763309] [ 4] syscall: <SYS_writev>
[  421.981646] [MVX] --> handle_mvx_message: syscall #0, flag 2, len 5, retval 5 (master)
[  421.983961] [MVX] --> buf: test
[  421.983961] 
[  421.985666] [**MVX Violation**] kernel/popcorn/mvx_engine.c:225 follower_writev
[  421.987810] [MVX] syscall 20: <SYS_writev>. flag 2. master msg syscall 0
[  421.989817] follower ret -9 != 5 from master
[  421.991080] [MVX] 
[  421.991080] SYSC_writev: mvx [676]. fd 1, vlen 1. MVX follower.
[  421.994065] ------------[ cut here ]------------
[  421.995408] kernel BUG at fs/read_write.c:1007!
[  421.996813] invalid opcode: 0000 [#1] SMP 
[  421.997596] Modules linked in: msg_socket
[  421.997596] CPU: 1 PID: 676 Comm: basic Not tainted 4.4.137-popcorn+ #3
[  421.997596] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.13.0-1ubuntu1 04/01/2014
[  421.997596] task: ffff88013ad36a80 ti: ffff88013aeb8000 task.ti: ffff88013aeb8000
[  421.997596] RIP: 0010:[<ffffffff8124f04d>]  [<ffffffff8124f04d>] SyS_writev.cold.4+0xc8/0xca
[  421.997596] RSP: 0018:ffff88013aebbe90  EFLAGS: 00010202
[  421.997596] RAX: 000000000000007f RBX: ffff8800bad66820 RCX: 0000000000000000
[  421.997596] RDX: 000000000000007f RSI: 0000000000000000 RDI: ffff88013aebbf1f
[  421.997596] RBP: 0000000000000001 R08: 0000000000000007 R09: ffff88013aebbea0
[  421.997596] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
[  421.997596] R13: 000000000000007f R14: 0000000000000000 R15: 0000000000000000
[  421.997596] FS:  00007ffff7ffe288(0000) GS:ffff88013fd00000(0000) knlGS:0000000000000000
[  421.997596] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  421.997596] CR2: 0000000000000006 CR3: 000000013ad52000 CR4: 0000000000360670
[  421.997596] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[  421.997596] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[  421.997596] Stack:
[  421.997596]  0000000000000005 ffffffff813fcb3e 0000000000000000 0000000000000000
[  421.997596]  0000000000000000 0000000000000000 0000000000000000 0000000000000000
[  421.997596]  0000000000000000 0000000000000000 0000000000000000 0000000000000000
[  421.997596] Call Trace:
[  421.997596]  [<ffffffff813fcb3e>] ? __schedule+0x1ce/0x730
[  421.997596]  [<ffffffff8140020a>] ? entry_SYSCALL_64_fastpath+0x1e/0x8e
[  421.997596] Code: 83 c4 01 48 c7 c6 00 91 41 81 48 c7 c7 6b df 57 81 e8 48 3c ed ff 49 63 c4 48 39 c5 77 ab 48 89 df e8 e8 c1 f0 ff e9 95 2c f1 ff <0f> 0b 48 8b 35 3a 5c 3f 00 48 c7 c7  
[  421.997596] RIP  [<ffffffff8124f04d>] SyS_writev.cold.4+0xc8/0xca
[  421.997596]  RSP <ffff88013aebbe90>
[  422.020294] ---[ end trace 900084b06eee6e9e ]---
[  422.021356] [MVX] --> handle_mvx_message: syscall #20, flag 2, len 0, retval 26 (master)

[FrancisLeeHa]

What a magic! It can run basic mvx after I try it again and again

• on arm

popcorn@arm:~/mvx$ LD_PRELOAD=./loader.so ./basic 
Issue an MVX init syscall.
[  703.469623] 
[  703.469623] === HeterSec syscall ===
[  703.469755] Arg0: 0x1(1)
[  703.469879] [ 0] argv[0]: ./basic. arg_size 7
[  703.469953] [MVX] mvx_prepare_init_req: Retrieved argc 1, cmdline size 8, cmdline ./basic, exe path /home/popcorn/mvx/basic
[  703.470102] [MVX] mvx_server_start_mvx: [271] exe_path /home/popcorn/mvx/basic. arg_start 0x7fffffffeed9, arg_end 0x7fffffffeee1
[  703.470250] [MVX] mvx_server_start_mvx: size 2070; dst nid 0
[  703.470668] [MVX] MVX master variant [PID:271] init ...
=== [MVX Unit Test] Testing 2 syscalls: ===
[  703.470917] [MVX] 
[  703.470917] [ 0] syscall 66: <SYS_writev>
[  703.471039] [MVX] master waits for follower return on syscall 66
[  703.480244] [MVX] --> handle_mvx_reply: ret 0x2c(44), follower syscall 20
[  703.480478] [MVX] 
[  703.480478] SYSC_writev: mvx [271]. fd 1, vlen 2. MVX master.
[  703.480608] [MVX] SYSC_writev: buf: === [MVX Unit Test] Testing 2
[  703.480690] [MVX] SYSC_writev: buf:  syscalls: ===
writev, read

[  703.480910] [MVX] 
[  703.480910] [ 1] syscall 66: <SYS_writev>
[  703.481033] [MVX] master waits for follower return on syscall 66
[  703.488854] [MVX] --> handle_mvx_reply: ret 0xe(14), follower syscall 20
[  703.489027] [MVX] 
[  703.489027] SYSC_writev: mvx [271]. fd 1, vlen 2. MVX master.
[  703.489141] [MVX] SYSC_writev: buf: writev, read
[  703.489204] [MVX] SYSC_writev: buf: 
[  703.489204] 
Writting to stderr
[  703.491273] fd 2, count 20, ret 20
[  703.491370] [MVX] 
[  703.491370] [ 2] syscall 64: <SYS_write>
[  703.491513] [MVX] master waits for follower return on syscall 64
[  703.497649] [MVX] --> handle_mvx_reply: ret 0x14(20), follower syscall 20
[  703.497854] [MVX] SYSC_write: mvx [271] Writting to stderr
[  703.497854] 
[  703.497963] [MVX] Not support SYSC_write yet.
Input your name (type "quit" to exit): 
[  703.498176] [MVX] 
[  703.498176] [ 3] syscall 66: <SYS_writev>
[  703.498296] [MVX] master waits for follower return on syscall 66
[  703.498920] [MVX] --> handle_mvx_reply: ret 0x28(40), follower syscall 20
[  703.499064] [MVX] 
[  703.499064] SYSC_writev: mvx [271]. fd 1, vlen 2. MVX master.
[  703.499174] [MVX] SYSC_writev: buf: Input your name (type "quit" to exit): 
[  703.499265] [MVX] SYSC_writev: buf: 
test
[  705.966089] [MVX] 
[  705.966089] [ 4] syscall 63: <SYS_read>
[  705.966393] [MVX] master_read: fd 0, flag 2, len 5, retval 5.
[  705.966660] [MVX] master waits for follower return on syscall 63
[  705.978652] [MVX] --> handle_mvx_reply: ret 0x5(5), follower syscall 20
[  705.978995] [MVX post syscall] SYSC_read: ret 5
5 char(s) read, buf: test
[  705.979420] [MVX] 
[  705.979420] [ 5] syscall 66: <SYS_writev>
[  705.979638] [MVX] master waits for follower return on syscall 66
[  705.985690] [MVX] --> handle_mvx_reply: ret 0x1a(26), follower syscall 20
[  705.985970] [MVX] 
[  705.985970] SYSC_writev: mvx [271]. fd 1, vlen 2. MVX master.
[  705.986148] [MVX] SYSC_writev: buf: 5 char(s) read, buf: 
[  705.986252] [MVX] SYSC_writev: buf: test

[  705.986482] [MVX] 
[  705.986482] [ 6] syscall 66: <SYS_writev>
[  705.986645] [MVX] master waits for follower return on syscall 66
[  705.987503] [MVX] --> handle_mvx_reply: ret 0x1(1), follower syscall 20
[  705.987703] [MVX] 
[  705.987703] SYSC_writev: mvx [271]. fd 1, vlen 2. MVX master.
[  705.987853] [MVX] SYSC_writev: buf: 
[  705.987926] [MVX] SYSC_writev: buf: 
Input your name (type "quit" to exit): 
[  705.988159] [MVX] 
[  705.988159] [ 7] syscall 66: <SYS_writev>
[  705.988298] [MVX] master waits for follower return on syscall 66
[  705.988877] [MVX] --> handle_mvx_reply: ret 0x28(40), follower syscall 20
[  705.989071] [MVX] 
[  705.989071] SYSC_writev: mvx [271]. fd 1, vlen 2. MVX master.
[  705.989220] [MVX] SYSC_writev: buf: Input your name (type "quit" to exit): 
[  705.989535] [MVX] SYSC_writev: buf: 
haha[  709.093173] [MVX] 
[  709.093173] [ 8] syscall 63: <SYS_read>
[  709.093767] [MVX] master_read: fd 0, flag 2, len 5, retval 5.
[  709.094028] [MVX] master waits for follower return on syscall 63

[  709.107712] [MVX] --> handle_mvx_reply: ret 0x5(5), follower syscall 20
[  709.108136] [MVX post syscall] SYSC_read: ret 5
5 char(s) read, buf: haha
[  709.108657] [MVX] 
[  709.108657] [ 9] syscall 66: <SYS_writev>
[  709.108929] [MVX] master waits for follower return on syscall 66
[  709.115427] [MVX] --> handle_mvx_reply: ret 0x1a(26), follower syscall 20
[  709.115854] [MVX] 
[  709.115854] SYSC_writev: mvx [271]. fd 1, vlen 2. MVX master.
[  709.116140] [MVX] SYSC_writev: buf: 5 char(s) read, buf: 
[  709.116301] [MVX] SYSC_writev: buf: haha

[  709.116659] [MVX] 
[  709.116659] [10] syscall 66: <SYS_writev>
[  709.116917] [MVX] master waits for follower return on syscall 66
[  709.118215] [MVX] --> handle_mvx_reply: ret 0x1(1), follower syscall 20
[  709.118457] [MVX] 
[  709.118457] SYSC_writev: mvx [271]. fd 1, vlen 2. MVX master.
[  709.118606] [MVX] SYSC_writev: buf: 
[  709.118672] [MVX] SYSC_writev: buf: 
Input your name (type "quit" to exit): 
[  709.118925] [MVX] 
[  709.118925] [11] syscall 66: <SYS_writev>
[  709.119066] [MVX] master waits for follower return on syscall 66
[  709.120015] [MVX] --> handle_mvx_reply: ret 0x28(40), follower syscall 20
[  709.120268] [MVX] 
[  709.120268] SYSC_writev: mvx [271]. fd 1, vlen 2. MVX master.
[  709.120420] [MVX] SYSC_writev: buf: Input your name (type "quit" to exit): 
[  709.120529] [MVX] SYSC_writev: buf: 

• on x86

popcorn@x86:~/mvx$ [  779.711388] [MVX] --> handle_mvx_message: syscall #20, flag 2, len 0, retval 44 (master)
[  779.713785] [MVX] syscall 20: <SYS_writev>. flag 2. master msg syscall 20
[  779.715727] follower ret -9 != 44 from master
[  779.717734] [MVX] 
[  779.717734] SYSC_writev: mvx [680]. fd 1, vlen 0. MVX follower.
[  779.719528] [MVX] 
[  779.720091] [MVX] Follower waits msg on
[  779.720091] [37] syscall: <SYS_writev>
[  779.721750] [MVX] --> handle_mvx_message: syscall #20, flag 2, len 0, retval 14 (master)
[  779.722992] [MVX] syscall 20: <SYS_writev>. flag 2. master msg syscall 20
[  779.724122] follower ret -9 != 14 from master
[  779.725067] [MVX] clone_mvx_thread: MVX helper thread [681] finished launching MVX process, bin path: /home/popcorn/mvx/basic.
[  779.726910] [MVX] 
[  779.726910] SYSC_writev: mvx [680]. fd 1, vlen 0. MVX follower.
[  779.728407] [MVX] 
[  779.728715] [MVX] Follower waits msg on
[  779.728715] [38] syscall: <SYS_writev>
[  779.729567] [MVX] argv[0] ./basic (7)
[  779.730114] [MVX] MVX follower variant [PID:682] init ...
[  779.731659] [MVX] --> handle_mvx_message: syscall #1, flag 2, len 0, retval 20 (master)
[  779.732242] [**MVX Violation**] kernel/popcorn/mvx_engine.c:225 follower_writev
[  779.732243] [MVX] syscall 20: <SYS_writev>. flag 2. master msg syscall 1
[  779.732244] follower ret -9 != 20 from master
[  779.732249] [MVX] 
[  779.732249] SYSC_writev: mvx [678]. fd 1, vlen 1. MVX follower.
[  779.732310] ------------[ cut here ]------------
[  779.732311] kernel BUG at fs/read_write.c:1007!
[  779.732317] invalid opcode: 0000 [#2] SMP 
[  779.732323] Modules linked in: msg_socket
[  779.732332] CPU: 1 PID: 678 Comm: basic Tainted: G      D         4.4.137-popcorn+ #3
[  779.732333] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.13.0-1ubuntu1 04/01/2014
[  779.732345] task: ffff8800bb4f4b00 ti: ffff8800bb734000 task.ti: ffff8800bb734000
[  779.732386] RIP: 0010:[<ffffffff8124f04d>]  [<ffffffff8124f04d>] SyS_writev.cold.4+0xc8/0xca
[  779.732390] RSP: 0018:ffff8800bb737e90  EFLAGS: 00010202
[  779.732391] RAX: 000000000000007f RBX: ffff8800b94e6400 RCX: 0000000000000000
[  779.732391] RDX: 000000000000007f RSI: 0000000000000000 RDI: ffff8800bb737f1f
[  779.732392] RBP: 0000000000000001 R08: 0000000000000006 R09: ffff8800bb737ea0
[  779.732392] R10: ffffffff817d4d5c R11: 00000000000002c9 R12: 0000000000000000
[  779.732393] R13: 000000000000007f R14: 0000000000000000 R15: 0000000000000000
[  779.732394] FS:  00007ffff7ffe288(0000) GS:ffff88013fd00000(0000) knlGS:0000000000000000
[  779.732394] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  779.732395] CR2: 0000000000000005 CR3: 00000000bac7c000 CR4: 0000000000360670
[  779.732398] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[  779.732398] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[  779.732399] Stack:
[  779.732400]  0000000000000014 ffff8800bb4f4b60 0000000000000000 0000000000000000
[  779.732401]  0000000000000000 0000000000000000 0000000000000000 0000000000000000
[  779.732402]  0000000000000000 0000000000000000 0000000000000000 0000000000000000
[  779.732407] Call Trace:
[  779.732431]  [<ffffffff8140020a>] ? entry_SYSCALL_64_fastpath+0x1e/0x8e
[  779.732448] Code: 83 c4 01 48 c7 c6 00 91 41 81 48 c7 c7 6b df 57 81 e8 48 3c ed ff 49 63 c4 48 39 c5 77 ab 48 89 df e8 e8 c1 f0 ff e9 95 2c f1 ff <0f> 0b 48 8b 35 3a 5c 3f 00 48 c7 c7  
[  779.732450] RIP  [<ffffffff8124f04d>] SyS_writev.cold.4+0xc8/0xca
[  779.732451]  RSP <ffff8800bb737e90>
[  779.733824] ---[ end trace 900084b06eee6e9f ]---
[  779.738344] [MVX] --> handle_mvx_message: syscall #20, flag 2, len 0, retval 40 (master)
[  779.738349] [MVX] syscall 20: <SYS_writev>. flag 2. master msg syscall 20
[  779.738349] follower ret -9 != 40 from master
[  779.738352] [MVX] 
[  779.738352] SYSC_writev: mvx [680]. fd 1, vlen 0. MVX follower.
[  779.738391] [MVX] 
[  779.738391] [MVX] Follower waits msg on
[  779.738391] [39] syscall: <SYS_writev>
[  779.767430] [MVX] Cannot open STDIN file: -13.
[  779.767885] [MVX] [682] init_mvx_follower_process: Finished the Follower MVX process initialization.
[  779.769193] [MVX] 
[  779.769410] [MVX] Follower waits msg on
[  779.769410] [ 0] syscall: <SYS_writev>
[  779.769700] [MVX] mvx_process_fork: Finished MVX process setup. ret 0
[  782.207582] [MVX] --> handle_mvx_message: syscall #0, flag 2, len 5, retval 5 (master)
[  782.209919] [MVX] --> buf: test
[  782.209919] 
[  782.211279] [**MVX Violation**] kernel/popcorn/mvx_engine.c:225 follower_writev
[  782.213400] [MVX] syscall 20: <SYS_writev>. flag 2. master msg syscall 0
[  782.215317] follower ret -9 != 5 from master
[  782.216293] [MVX] 
[  782.216293] SYSC_writev: mvx [680]. fd 1, vlen 0. MVX follower.
[  782.217921] [MVX] 
[  782.218357] [MVX] Follower waits msg on
[  782.218357] [ 1] syscall: <SYS_writev>
[  782.220281] [MVX] --> handle_mvx_message: syscall #20, flag 2, len 0, retval 26 (master)
[  782.221882] [MVX] syscall 20: <SYS_writev>. flag 2. master msg syscall 20
[  782.223813] follower ret -9 != 26 from master
[  782.225106] [MVX] 
[  782.225106] SYSC_writev: mvx [682]. fd 1, vlen 2. MVX follower.
[  782.227123] [MVX] --> handle_mvx_message: syscall #20, flag 2, len 0, retval 1 (master)
[  782.227128] [MVX] syscall 20: <SYS_writev>. flag 2. master msg syscall 20
[  782.227128] follower ret -9 != 1 from master
[  782.227131] [MVX] 
[  782.227131] SYSC_writev: mvx [680]. fd 1, vlen 0. MVX follower.
[  782.227243] [MVX] 
[  782.227243] [MVX] Follower waits msg on
[  782.227243] [ 2] syscall: <SYS_writev>
[  782.228595] [MVX] --> handle_mvx_message: syscall #20, flag 2, len 0, retval 40 (master)
[  782.228598] [MVX] syscall 20: <SYS_writev>. flag 2. master msg syscall 20
[  782.228599] follower ret -9 != 40 from master
[  782.228600] [MVX] 
[  782.228600] SYSC_writev: mvx [680]. fd 1, vlen 0. MVX follower.
[  782.228623] [MVX] 
[  782.228643] [MVX] Follower waits msg on
[  782.228643] [ 3] syscall: <SYS_writev>
[  782.241296] [MVX] SYSC_writev: buf: === [MVX Unit Test] Testing 2
[  782.242036] [MVX] SYSC_writev: buf:  syscalls: ===
[  782.242669] [MVX] 
[  782.242931] [MVX] Follower waits msg on
[  782.242931] [ 4] syscall: <SYS_writev>
[  785.335182] [MVX] --> handle_mvx_message: syscall #0, flag 2, len 5, retval 5 (master)
[  785.337514] [MVX] --> buf: haha
[  785.337514] 
[  785.338886] [**MVX Violation**] kernel/popcorn/mvx_engine.c:225 follower_writev
[  785.341006] [MVX] syscall 20: <SYS_writev>. flag 2. master msg syscall 0
[  785.342947] follower ret -9 != 5 from master
[  785.344809] [MVX] 
[  785.344809] SYSC_writev: mvx [680]. fd 1, vlen 0. MVX follower.
[  785.347299] [MVX] 
[  785.347607] [MVX] Follower waits msg on
[  785.347607] [ 5] syscall: <SYS_writev>
[  785.350055] [MVX] --> handle_mvx_message: syscall #20, flag 2, len 0, retval 26 (master)
[  785.351163] [MVX] syscall 20: <SYS_writev>. flag 2. master msg syscall 20
[  785.353500] follower ret -9 != 26 from master
[  785.355014] [MVX] 
[  785.355014] SYSC_writev: mvx [682]. fd 1, vlen 2. MVX follower.
[  785.357740] [MVX] SYSC_writev: buf: g 2[  785.358102] [MVX] --> handle_mvx_message: syscall #20, flag 2, len 0, retval 1 (master)
[  785.358107] [MVX] syscall 20: <SYS_writev>. flag 2. master msg syscall 20
[  785.358108] follower ret -9 != 1 from master
[  785.358112] [MVX] 
[  785.358112] SYSC_writev: mvx [680]. fd 1, vlen 0. MVX follower.
[  785.358221] [MVX] 
[  785.358222] [MVX] Follower waits msg on
[  785.358222] [ 6] syscall: <SYS_writev>
[  785.359913] [MVX] --> handle_mvx_message: syscall #20, flag 2, len 0, retval 40 (master)
[  785.359918] [MVX] syscall 20: <SYS_writev>. flag 2. master msg syscall 20
[  785.359918] follower ret -9 != 40 from master
[  785.359921] [MVX] 
[  785.359921] SYSC_writev: mvx [680]. fd 1, vlen 0. MVX follower.
[  785.360055] [MVX] 
[  785.360056] [MVX] Follower waits msg on
[  785.360056] [ 7] syscall: <SYS_writev>

[  785.371539] [MVX] SYSC_writev: buf:  syscalls: ===
[  785.372105] [MVX] 
[  785.372342] [MVX] Follower waits msg on
[  785.372342] [ 8] syscall: <SYS_writev>

But it still have kernel bug on x86
[ 779.732311] kernel BUG at fs/read_write.c:1007!
What`s problem?

[xjtuwxg]

Hey, thanks for the log. This is actually a bug/enhancement. The problem is caused by the hardcoded pseudo STDIN/STDOUT file locations in kernel/popcorn/mvx_server.c:

 35 /* Pseudo file locations for follower fd 0, 1 and 2. */                         
 36 #define MVX_STDIN_LOC           "/dev/pts/0"
 37 #define MVX_STDOUT_LOC          "/dev/pts/0"

So, if you use ssh to connect to the VMs, and do it on the ssh client it should work.

popcorn@x86:~/mvx$ [ 409.701426] [MVX] clone_mvx_thread: MVX helper thread [675] finished launching MVX process, bin path: /home/popcorn/mvx/basic.
[ 409.704798] [MVX] argv[0] ./basic (7)
[ 409.706156] [MVX] MVX follower variant [PID:676] init ...
[ 409.708168] [MVX] --> handle_mvx_message: syscall #20, flag 2, len 0, retval 44 (master)
[ 409.711781] [MVX] Cannot open STDIN file: -13.

[FrancisLeeHa]

Thanks!
When I use the command create the VMs, I just get into the terminals that I can`t exit (if quit by force command, VMs will be shutdown too). The TTY is ttyS0, which is not pts/0

popcorn@x86:~$ ps
  PID TTY          TIME CMD
  618 ttyS0    00:00:00 bash
  663 ttyS0    00:00:00 ps

I just keep this two terminal and use ssh connect the VMs, now the tty change to pts/0

popcorn@arm:~/mvx$ ps
  PID TTY          TIME CMD
  232 pts/0    00:00:00 bash
  252 pts/0    00:00:00 ps

And the mvx demo run correctly.

Why can`t I inter pts/0 TTY just when I create the VMs?
And how can I create the VMs running on background? So that I will ssh to login instead of using this ttyS0 TTY