Realized service health was not showing up in dashboard… checked the monit service and it seems like broken somehow.
[root@node01 apnscp]# systemctl status monit
● monit.service - Pro-active monitoring utility for unix systems
Loaded: loaded (/usr/lib/systemd/system/monit.service; enabled; vendor preset: disabled)
Active: failed (Result: core-dump) since Thu 2025-02-27 00:34:42 CET; 1s ago
Process: 3014093 ExecStart=/usr/bin/monit -I (code=dumped, signal=ABRT)
Main PID: 3014093 (code=dumped, signal=ABRT)
Feb 27 00:34:41 node01.com monit[3014093]: 'node01.com' Monit 5.34.5 started
Feb 27 00:34:41 node01.com monit[3014093]: 'node01.com' Monit 5.34.5 started
Feb 27 00:34:41 node01.com monit[3014093]: Aborting queued event '/var/spool/monit/1737938548_55d21c4a6ff0' - service dovecot not found in monit configuration
Feb 27 00:34:41 node01.com monit[3014093]: Aborting queued event '/var/spool/monit/1737938548_55d21c4a6ff0' - service dovecot not found in monit configuration
Feb 27 00:34:41 node01.com monit[3014093]: 'haproxy' service restarted 5 times within 5 cycles(s) - unmonitor
Feb 27 00:34:41 node01.com monit[3014093]: 'haproxy' service restarted 5 times within 5 cycles(s) - unmonitor
Feb 27 00:34:41 node01.com monit[3014093]: monit: src/util.c:449: Util_replaceString: Assertion `new' failed.
Feb 27 00:34:42 node01.com systemd-coredump[3014256]: Process 3014093 (monit) of user 0 dumped core.
Feb 27 00:34:42 node01.com systemd[1]: monit.service: Main process exited, code=dumped, status=6/ABRT
Feb 27 00:34:42 node01.com systemd[1]: monit.service: Failed with result 'core-dump'.
What does dmesg report? I’ve got a preliminary release of 5.34.5 that fixes 2 bugs (#1120, #1121 pertaining to network recovery on firewall-cmd --reload if duplicate rules). Does this crash happen immediately upon systemctl restart monit? What if you run upcp -sb software/haproxy software/argos?
Yes it crashed immediately after restarting service. But this fixed it: upcp -sb software/haproxy software/argos
I will still post the log and result prior the fix for better troubleshooting for you.
monit -I result:
[root@node01 ~]# monit -I
Starting Monit 5.34.5 daemon with http interface at [localhost]:2812
Monit will delay for 240 seconds on first start after reboot ...
'node01.com' Monit 5.34.5 started
Aborting queued event '/var/spool/monit/1737938548_55d21c4a6ff0' - service dovecot not found in monit configuration
'haproxy' service restarted 5 times within 5 cycles(s) - unmonitor
monit: src/util.c:449: Util_replaceString: Assertion `new' failed.
Aborted (core dumped)
dmesg log (capped):
[ 4.619661] amdgpu 0000:0e:00.0: amdgpu: ring gfx_0.0.0 uses VM inv eng 0 on hub 0
[ 4.619717] amdgpu 0000:0e:00.0: amdgpu: ring comp_1.0.0 uses VM inv eng 1 on hub 0
[ 4.619772] amdgpu 0000:0e:00.0: amdgpu: ring comp_1.1.0 uses VM inv eng 4 on hub 0
[ 4.619825] amdgpu 0000:0e:00.0: amdgpu: ring comp_1.2.0 uses VM inv eng 5 on hub 0
[ 4.619878] amdgpu 0000:0e:00.0: amdgpu: ring comp_1.3.0 uses VM inv eng 6 on hub 0
[ 4.619930] amdgpu 0000:0e:00.0: amdgpu: ring comp_1.0.1 uses VM inv eng 7 on hub 0
[ 4.619981] amdgpu 0000:0e:00.0: amdgpu: ring comp_1.1.1 uses VM inv eng 8 on hub 0
[ 4.620037] amdgpu 0000:0e:00.0: amdgpu: ring comp_1.2.1 uses VM inv eng 9 on hub 0
[ 4.620087] amdgpu 0000:0e:00.0: amdgpu: ring comp_1.3.1 uses VM inv eng 10 on hub 0
[ 4.620137] amdgpu 0000:0e:00.0: amdgpu: ring kiq_2.1.0 uses VM inv eng 11 on hub 0
[ 4.620187] amdgpu 0000:0e:00.0: amdgpu: ring sdma0 uses VM inv eng 12 on hub 0
[ 4.620236] amdgpu 0000:0e:00.0: amdgpu: ring vcn_dec_0 uses VM inv eng 0 on hub 1
[ 4.620286] amdgpu 0000:0e:00.0: amdgpu: ring vcn_enc_0.0 uses VM inv eng 1 on hub 1
[ 4.620336] amdgpu 0000:0e:00.0: amdgpu: ring vcn_enc_0.1 uses VM inv eng 4 on hub 1
[ 4.620386] amdgpu 0000:0e:00.0: amdgpu: ring jpeg_dec uses VM inv eng 5 on hub 1
[ 4.620870] [drm] Initialized amdgpu 3.52.0 20150101 for 0000:0e:00.0 on minor 1
[ 4.623774] amdgpu 0000:0e:00.0: [drm] Cannot find any crtc or sizes
[ 4.623842] [drm] DSC precompute is not needed.
[ 5.119849] IPv6: ADDRCONF(NETDEV_UP): enp11s0u9u3c2: link is not ready
[ 5.150080] IPv6: ADDRCONF(NETDEV_UP): enp1s0f0: link is not ready
[ 7.716446] mlx5_core 0000:01:00.0 enp1s0f0: Link down
[ 7.719304] IPv6: ADDRCONF(NETDEV_UP): enp1s0f0: link is not ready
[ 7.720760] IPv6: ADDRCONF(NETDEV_UP): enp1s0f1: link is not ready
[ 8.192870] mlx5_core 0000:01:00.1 enp1s0f1: Link up
[ 8.195511] IPv6: ADDRCONF(NETDEV_UP): enp1s0f1: link is not ready
[ 8.206606] IPv6: ADDRCONF(NETDEV_CHANGE): enp1s0f1: link becomes ready
[ 8.207523] IPv6: ADDRCONF(NETDEV_UP): enp5s0: link is not ready
[ 8.242426] IPv6: ADDRCONF(NETDEV_UP): enp5s0: link is not ready
[ 8.244890] IPv6: ADDRCONF(NETDEV_UP): enp6s0: link is not ready
[ 8.278082] IPv6: ADDRCONF(NETDEV_UP): enp6s0: link is not ready
[ 11.845934] igb 0000:05:00.0 enp5s0: igb: enp5s0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
[ 11.951873] IPv6: ADDRCONF(NETDEV_CHANGE): enp5s0: link becomes ready
[ 245.459382] nf_conntrack: default automatic helper assignment has been turned off for security reasons and CT-based firewall rule not found. Use the iptables CT targ et to attach helpers instead.
[134661.012531] bash (3417931): drop_caches: 1
[320031.692265] md: data-check of RAID array md2
[320034.077481] md: md2: data-check done.
[320091.798138] md: data-check of RAID array md1
[320097.810407] md: delaying data-check of md0 until md1 has finished (they share one or more physical units)
[324945.175016] md: md1: data-check done.
[324945.175961] md: data-check of RAID array md0
[324950.223158] md: md0: data-check done.
[924848.736163] md: data-check of RAID array md2
[924851.275030] md: md2: data-check done.
[924908.838083] md: data-check of RAID array md1
[924914.849457] md: delaying data-check of md0 until md1 has finished (they share one or more physical units)
[929785.111145] md: md1: data-check done.
[929785.112297] md: data-check of RAID array md0
[929790.199659] md: md0: data-check done.
[1036611.625763] php-cgi2[1820849]: segfault at 7fbca7947e08 ip 00007fbcbe731ebb sp 00007ffc24163580 error 4 in ld-2.28.so[7fbcbe729000+2f000]
[1036611.626248] Code: 00 00 00 83 e0 f7 41 88 87 1c 03 00 00 49 8b 87 10 01 00 00 48 85 c0 0f 84 92 00 00 00 f6 05 cb 6c 22 00 02 0f 85 9d 00 00 00 <4c> 8b 70 08 49 8b 87 20 01 00 00 49 8b 37 48 8b 50 08 4c 01 f6 48
[1412440.755235] TCP: request_sock_TCP: Possible SYN flooding on port [::]:80. Sending cookies.
[1440030.582570] TCP: request_sock_TCP: Possible SYN flooding on port 0.0.0.0:21. Sending cookies.
[1529665.070690] md: data-check of RAID array md2
[1529667.615293] md: md2: data-check done.
[1529725.166318] md: data-check of RAID array md1
[1529731.176688] md: delaying data-check of md0 until md1 has finished (they share one or more physical units)
[1534601.376500] md: md1: data-check done.
[1534601.377567] md: data-check of RAID array md0
[1534606.441725] md: md0: data-check done.
[2037759.954497] overlayfs: upperdir is in-use as upperdir/workdir of another mount, accessing files from both mounts will result in undefined behavior.
[2037759.954743] overlayfs: workdir is in-use as upperdir/workdir of another mount, accessing files from both mounts will result in undefined behavior.
[2134481.532077] md: data-check of RAID array md2
[2134484.172733] md: md2: data-check done.
[2134541.634789] md: data-check of RAID array md1
[2134547.646129] md: delaying data-check of md0 until md1 has finished (they share one or more physical units)
[2139433.700036] md: md1: data-check done.
[2139433.701144] md: data-check of RAID array md0
[2139438.749733] md: md0: data-check done.
[2317169.986741] httpd[220508]: segfault at 7f07f8006140 ip 00007f2a9cf329c7 sp 00007f2a91f39ea0 error 4 in libc-2.28.so[7f2a9ce97000+1cd000]
[2317169.987055] Code: 01 48 85 ed 74 65 64 48 c7 00 00 00 00 00 48 8d 9d 80 00 00 00 4c 8d a5 80 02 00 00 eb 17 0f 1f 44 00 00 48 89 f8 48 c1 e8 0c <48> 33 07 48 89 03 e8 9e fe ff ff 48 8b 3b 48 85 ff 74 16 40 f6 c7
[2386546.886694] httpd[2371644]: segfault at 7f07f8034300 ip 00007f2a9cf329c7 sp 00007f2a90736ea0 error 4 in libc-2.28.so[7f2a9ce97000+1cd000]
[2386546.887082] Code: 01 48 85 ed 74 65 64 48 c7 00 00 00 00 00 48 8d 9d 80 00 00 00 4c 8d a5 80 02 00 00 eb 17 0f 1f 44 00 00 48 89 f8 48 c1 e8 0c <48> 33 07 48 89 03 e8 9e fe ff ff 48 8b 3b 48 85 ff 74 16 40 f6 c7
[2739297.339816] md: data-check of RAID array md2
[2739299.845376] md: md2: data-check done.
[2739357.440357] md: data-check of RAID array md1
[2739363.451310] md: delaying data-check of md0 until md1 has finished (they share one or more physical units)
[2744236.217538] md: md1: data-check done.
[2744236.219090] md: data-check of RAID array md0
[2744241.313669] md: md0: data-check done.
[2748450.676460] httpd[3897596]: segfault at 7f07ac01fef0 ip 00007f2a9cf329c7 sp 00007f29c37fdea0 error 4 in libc-2.28.so[7f2a9ce97000+1cd000]
[2748450.676811] Code: 01 48 85 ed 74 65 64 48 c7 00 00 00 00 00 48 8d 9d 80 00 00 00 4c 8d a5 80 02 00 00 eb 17 0f 1f 44 00 00 48 89 f8 48 c1 e8 0c <48> 33 07 48 89 03 e8 9e fe ff ff 48 8b 3b 48 85 ff 74 16 40 f6 c7
[3008908.456789] httpd[1082273]: segfault at 7f07ec00e720 ip 00007f2a9cf329c7 sp 00007f2a8f633ea0 error 4 in libc-2.28.so[7f2a9ce97000+1cd000]
[3008908.457164] Code: 01 48 85 ed 74 65 64 48 c7 00 00 00 00 00 48 8d 9d 80 00 00 00 4c 8d a5 80 02 00 00 eb 17 0f 1f 44 00 00 48 89 f8 48 c1 e8 0c <48> 33 07 48 89 03 e8 9e fe ff ff 48 8b 3b 48 85 ff 74 16 40 f6 c7
[3180899.722156] httpd[4049309]: segfault at 7f07cc002d80 ip 00007f2a9cf329c7 sp 00007f2a5fff6ea0 error 4 in libc-2.28.so (deleted)[7f2a9ce97000+1cd000]
[3180899.722486] Code: 01 48 85 ed 74 65 64 48 c7 00 00 00 00 00 48 8d 9d 80 00 00 00 4c 8d a5 80 02 00 00 eb 17 0f 1f 44 00 00 48 89 f8 48 c1 e8 0c <48> 33 07 48 89 03 e8 9e fe ff ff 48 8b 3b 48 85 ff 74 16 40 f6 c7
[3344114.325038] md: data-check of RAID array md2
[3344116.854410] md: md2: data-check done.
[3344174.422124] md: data-check of RAID array md1
[3344180.433537] md: delaying data-check of md0 until md1 has finished (they share one or more physical units)
[3349049.926710] md: md1: data-check done.
[3349049.928074] md: data-check of RAID array md0
[3349054.984318] md: md0: data-check done.
[3520160.355779] httpd[174441]: segfault at 7f07cc01aa90 ip 00007f2a9cf329c7 sp 00007f29db7fdea0 error 4 in libc-2.28.so (deleted)[7f2a9ce97000+1cd000]
[3520160.356133] Code: 01 48 85 ed 74 65 64 48 c7 00 00 00 00 00 48 8d 9d 80 00 00 00 4c 8d a5 80 02 00 00 eb 17 0f 1f 44 00 00 48 89 f8 48 c1 e8 0c <48> 33 07 48 89 03 e8 9e fe ff ff 48 8b 3b 48 85 ff 74 16 40 f6 c7
[3611601.263808] httpd[98410]: segfault at 7f07f80183c0 ip 00007f2a9cf329c7 sp 00007f2a91637ea0 error 4 in libc-2.28.so (deleted)[7f2a9ce97000+1cd000]
[3611601.264128] Code: 01 48 85 ed 74 65 64 48 c7 00 00 00 00 00 48 8d 9d 80 00 00 00 4c 8d a5 80 02 00 00 eb 17 0f 1f 44 00 00 48 89 f8 48 c1 e8 0c <48> 33 07 48 89 03 e8 9e fe ff ff 48 8b 3b 48 85 ff 74 16 40 f6 c7
[3773651.498823] httpd[1860709]: segfault at 7f07d41d0d80 ip 00007f2a9cf32326 sp 00007f29daffbfb0 error 4 in libc-2.28.so (deleted)[7f2a9ce97000+1cd000]
[3773651.499193] Code: 0f b7 0e 66 85 c9 0f 84 46 ff ff ff 48 8d 3c d8 48 8b 97 80 00 00 00 f6 c2 0f 0f 85 d4 01 00 00 48 89 d0 83 e9 01 48 c1 e8 0c <48> 33 02 48 89 87 80 00 00 00 48 89 d0 66 89 0e 48 c7 42 08 00 00