Modify

Opened 5 years ago

Closed 5 years ago

#114 closed defect (fixed)

Crashes related to number of TLS relays

Reported by: brandon.owen@… Owned by: brandon.owen@…
Priority: normal Component: Professional Caster
Version: Keywords:
Cc:

Description

I've recently started testing the TLS relays and (for my server) the caster crashes fairly consistently on start up when over 80 TLS relays are configured. I can have the same number of relays without the -s flag and the software works fine.

The stack traces which the caster reports seem to be inconsistent, as are the kernal logs in /var/log/messages. But typically I see the following kernal logs:

Jun  4 06:08:01 ip-10-0-3-88 casterwatch[6194]: Starting Ntrip caster service
Jun  4 06:08:01 ip-10-0-3-88 kernel: [24243.912185] ntripdaemon[6258]: segfault at a0 ip 00007ff1c2135725 sp 00007ff1b31369e0 error 4
Jun  4 06:08:01 ip-10-0-3-88 kernel: [24243.912267] ntripdaemon[6219]: segfault at a0 ip 00007ff1c2135725 sp 00007ff1b3ab6a00 error 4
Jun  4 06:08:01 ip-10-0-3-88 kernel: [24243.912271] ntripdaemon[6221]: segfault at a0 ip 00007ff1c2135725 sp 00007ff1b3a36a00 error 4
Jun  4 06:08:01 ip-10-0-3-88 kernel: [24243.912326] ntripdaemon[6339]: segfault at a0 ip 00007ff1c2135725 sp 00007ff1b1cf6a00 error 4
Jun  4 06:08:01 ip-10-0-3-88 kernel: [24243.912446] ntripdaemon[6331]: segfault at a0 ip 00007ff1c2135725 sp 00007ff1b1ef69b0 error 4
Jun  4 06:08:01 ip-10-0-3-88 kernel: [24243.912583]  in libcrypto.so.1.0.2k[7ff1c2012000+232000]
Jun  4 06:08:01 ip-10-0-3-88 kernel: [24243.912645] ntripdaemon[6341]: segfault at a0 ip 00007ff1c2135725 sp 00007ff1b1c76a00 error 4
Jun  4 06:08:01 ip-10-0-3-88 kernel: [24243.915475] ntripdaemon[6288]: segfault at a0 ip 00007ff1c2135725 sp 00007ff1b29b6a00 error 4
Jun  4 06:08:01 ip-10-0-3-88 kernel: [24243.915544] ntripdaemon[6322]: segfault at a0 ip 00007ff1c2135725 sp 00007ff1b2136a00 error 4
Jun  4 06:08:01 ip-10-0-3-88 kernel: [24243.915547] ntripdaemon[6231]: segfault at a0 ip 00007ff1c2135725 sp 00007ff1b37b60b0 error 4
Jun  4 06:08:01 ip-10-0-3-88 kernel: [24243.919532] ntripdaemon[6245]: segfault at a0 ip 00007ff1c2135725 sp 00007ff1b3436a00 error 4
Jun  4 06:08:01 ip-10-0-3-88 kernel: [24243.926471]  in libcrypto.so.1.0.2k[7ff1c2012000+232000]
Jun  4 06:08:01 ip-10-0-3-88 kernel: [24243.926486]  in libcrypto.so.1.0.2k[7ff1c2012000+232000]
Jun  4 06:08:01 ip-10-0-3-88 kernel: [24243.926496]  in libcrypto.so.1.0.2k[7ff1c2012000+232000]
Jun  4 06:08:01 ip-10-0-3-88 kernel: [24243.926521]  in libcrypto.so.1.0.2k[7ff1c2012000+232000]
Jun  4 06:08:01 ip-10-0-3-88 kernel: [24243.926641]  in libcrypto.so.1.0.2k[7ff1c2012000+232000]
Jun  4 06:08:01 ip-10-0-3-88 kernel: [24243.926645]  in libcrypto.so.1.0.2k[7ff1c2012000+232000]
Jun  4 06:08:01 ip-10-0-3-88 kernel: [24243.926680]  in libcrypto.so.1.0.2k[7ff1c2012000+232000]
Jun  4 06:08:01 ip-10-0-3-88 kernel: [24243.933729]  in libcrypto.so.1.0.2k[7ff1c20
12000+232000]
Jun  4 06:08:01 ip-10-0-3-88 kernel: [24243.933787]  in libcrypto.so.1.0.2k[7ff1c2012000+232000]

With accompanying output from the ntripcaster daemon:

NtripCaster Version 2.0.36 Initializing...
NtripCaster comes with NO WARRANTY, to the extent permitted by law.
You may redistribute copies of NtripCaster under the terms of the
GNU General Public License.
For more information about these matters, see the file named COPYING.
Starting thread engine...
*** Error in `/opt/ntripcaster/sbin/ntripdaemon': realloc(): invalid next size: 0x00007f32b8002200 ***
*** Error in `/opt/ntripcaster/sbin/ntripdaemon': realloc(): invalid next size: 0x00007f32b8002200 ***
*** Error in `/opt/ntripcaster/sbin/ntripdaemon': realloc(): invalid next size: 0x00007f32b8002200 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x7f644)[0x7f32fc190644]
/lib64/libc.so.6(+0x83c48)[0x7f32fc194c48]
/lib64/libc.so.6(realloc+0x1d2)[0x7f32fc1968a2]
/lib64/libcrypto.so.10(CRYPTO_realloc+0x49)[0x7f32fc768529]
/lib64/libcrypto.so.10(lh_insert+0x101)[0x7f32fc81dad1]
/lib64/libcrypto.so.10(+0x12640e)[0x7f32fc82040e]
/lib64/libcrypto.so.10(ERR_get_state+0xde)[0x7f32fc820ebe]
/lib64/libcrypto.so.10(ERR_clear_error+0x15)[0x7f32fc821045]
/lib64/libssl.so.10(ssl23_connect+0x45)[0x7f32fcb884a5]
/opt/ntripcaster/sbin/ntripdaemon[0x426f67]
/opt/ntripcaster/sbin/ntripdaemon[0x41b47b]
/opt/ntripcaster/sbin/ntripdaemon[0x41b8e4]
/opt/ntripcaster/sbin/ntripdaemon[0x41b9e6]
/lib64/libpthread.so.0(+0x7de5)[0x7f32fc4e5de5]
/lib64/libc.so.6(clone+0x6d)[0x7f32fc20ef1d]
======= Memory map: ========
======= Backtrace: =========
======= Backtrace: =========
/lib64/libc.so.6(+0x7f644)[0x7f32fc190644]
/lib64/libc.so.6(+0x7f644)[0x7f32fc190644]
/lib64/libc.so.6(+0x83c48)[0x7f32fc194c48]
/lib64/libc.so.6(+0x83c48)[0x7f32fc194c48]
/lib64/libc.so.6(realloc+0x1d2)[0x7f32fc1968a2]
/lib64/libc.so.6(realloc+0x1d2)[0x7f32fc1968a2]
/lib64/libcrypto.so.10(CRYPTO_realloc+0x49)[0x7f32fc768529]
/lib64/libcrypto.so.10(CRYPTO_realloc+0x49)[0x7f32fc768529]
/lib64/libcrypto.so.10(lh_insert+0x101)[0x7f32fc81dad1]
/lib64/libcrypto.so.10(lh_insert+0x101)[0x7f32fc81dad1]
/lib64/libcrypto.so.10(+0x12640e)[0x7f32fc82040e]
/lib64/libcrypto.so.10(+0x12640e)[0x7f32fc82040e]
/lib64/libcrypto.so.10(ERR_get_state+0xde)[0x7f32fc820ebe]
/lib64/libcrypto.so.10(ERR_get_state+0xde)[0x7f32fc820ebe]
/lib64/libcrypto.so.10(ERR_clear_error+0x15)[0x7f32fc821045]
/lib64/libssl.so.10(ssl23_connect+0x45)[0x7f32fcb884a5]
/opt/ntripcaster/sbin/ntripdaemon[0x426f67]
/opt/ntripcaster/sbin/ntripdaemon[0x41b47b]
/opt/ntripcaster/sbin/ntripdaemon[0x41b8e4]
/opt/ntripcaster/sbin/ntripdaemon[0x41b9e6]
/lib64/libpthread.so.0(+0x7de5)[0x7f32fc4e5de5]
/lib64/libc.so.6(clone+0x6d)[0x7f32fc20ef1d]
======= Memory map: ========
/lib64/libcrypto.so.10(ERR_clear_error+0x15)[0x7f32fc821045]
/lib64/libssl.so.10(ssl23_connect+0x45)[0x7f32fcb884a5]
/opt/ntripcaster/sbin/ntripdaemon[0x426f67]
/opt/ntripcaster/sbin/ntripdaemon[0x41b47b]
/opt/ntripcaster/sbin/ntripdaemon[0x41b8e4]
/opt/ntripcaster/sbin/ntripdaemon[0x41b9e6]
/lib64/libpthread.so.0(+0x7de5)[0x7f32fc4e5de5]
/lib64/libc.so.6(clone+0x6d)[0x7f32fc20ef1d]
======= Memory map: ========
*** Error in `/opt/ntripcaster/sbin/ntripdaemon': realloc(): invalid next size: 0x00007f32b8002200 ***
00400000-00442000 r-xp 00000000 103:01 282416                            /opt/ntripcaster/sbin/ntripdaemon
00641000-00645000 rw-p 00041000 103:01 282416                            /opt/ntripcaster/sbin/ntripdaemon
00645000-00646000 rw-p 00000000 00:00 0 
01b18000-01cc5000 rw-p 00000000 00:00 0                                  [heap]
01cc5000-01d03000 rw-p 00000000 00:00 0                                  [heap]
7f327bd6a000-7f327bd80000 r-xp 00000000 103:01 734                       /lib64/libgcc_s-7-20170915.so.1
7f327bd80000-7f327bf7f000 ---p 00016000 103:01 734                       /lib64/libgcc_s-7-20170915.so.1
7f327bf7f000-7f327bf80000 rw-p 00015000 103:01 734                       /lib64/libgcc_s-7-20170915.so.1
7f327bf80000-7f327bf81000 ---p 00000000 00:00 0 
7f327bf81000-7f327bfc0000 rw-p 00000000 00:00 0 
7f327bfc0000-7f327bfc1000 ---p 00000000 00:00 0 
7f327bfc1000-7f327c000000 rw-p 00000000 00:00 0 
7f327c000000-7f327c064000 rw-p 00000000 00:00 0 
7f327c064000-7f3280000000 ---p 00000000 00:00 0 
00400000-00442000 r-xp 00000000 103:01 282416                            /opt/ntripcaster/sbin/ntripdaemon
00641000-00645000 rw-p 00041000 103:01 282416                            /opt/ntripcaster/sbin/ntripdaemon
00645000-00646000 rw-p 00000000 00:00 0 
01b18000-01cc5000 rw-p 00000000 00:00 0                                  [heap]
01cc5000-01d03000 rw-p 00000000 00:00 0                                  [heap]
7f327bd6a000-7f327bd80000 r-xp 00000000 103:01 734                       /lib64/libgcc_s-7-20170915.so.1
7f327bd80000-7f327bf7f000 ---p 00016000 103:01 734                       /lib64/libgcc_s-7-20170915.so.1
7f327bf7f000-7f327bf80000 rw-p 00015000 103:01 734                       /lib64/libgcc_s-7-20170915.so.1
7f327bf80000-7f327bf81000 ---p 00000000 00:00 0 
7f327bf81000-7f327bfc0000 rw-p 00000000 00:00 0 
7f327bfc0000-7f327bfc1000 ---p 00000000 00:00 0 
7f327bfc1000-7f327c000000 rw-p 00000000 00:00 0 
7f327c000000-7f327c065000 rw-p 00000000 00:00 0 
7f327c065000-7f3280000000 ---p 00000000 00:00 0 
00400000-00442000 r-xp 00000000 103:01 282416                            /opt/ntripcaster/sbin/ntripdaemon
00641000-00645000 rw-p 00041000 103:01 282416                            /opt/ntripcaster/sbin/ntripdaemon
00645000-00646000 rw-p 00000000 00:00 0 
01b18000-01cc5000 rw-p 00000000 00:00 0                                  [heap]
01cc5000-01d03000 rw-p 00000000 00:00 0                                  [heap]
7f327bd6a000-7f327bd80000 r-xp 00000000 103:01 734                       /lib64/libgcc_s-7-20170915.so.1
7f327bd80000-7f327bf7f000 ---p 00016000 103:01 734                       /lib64/libgcc_s-7-20170915.so.1
7f327bf7f000-7f327bf80000 rw-p 00015000 103:01 734                       /lib64/libgcc_s-7-20170915.so.1
7f327bf80000-7f327bf81000 ---p 00000000 00:00 0 
7f327bf81000-7f327bfc0000 rw-p 00000000 00:00 0 
7f327bfc0000-7f327bfc1000 ---p 00000000 00:00 0 
7f327bfc1000-7f327c000000 rw-p 00000000 00:00 0 
7f327c000000-7f327c065000 rw-p 00000000 00:00 0 
7f327c065000-7f3280000000 ---p 00000000 00:00 0 
7f3280000000-7f328003d000 rw-p 00000000 00:00 0 
7f328003d000-7f3284000000 ---p 00000000 00:00 0 
7f3284000000-7f3284054000 rw-p 00000000 00:00 0 
7f3284054000-7f3288000000 ---p 00000000 00:00 0 
7f3288000000-7f3288056000 rw-p 00000000 00:00 0 
7f3288056000-7f328c000000 ---p 00000000 00:00 0 
7f328c000000-7f328c03d000 rw-p 00000000 00:00 0 
7f328c03d000-7f3290000000 ---p 00000000 00:00 0 
7f3290000000-7f3290059000 rw-p 00000000 00:00 0 
7f3290059000-7f3294000000 ---p 00000000 00:00 0 
7f3294000000-7f3294036000 rw-p 00000000 00:00 0 
7f3294036000-7f3298000000 ---p 00000000 00:00 0 
7f3298000000-7f3298056000 rw-p 00000000 00:00 0 
7f3298056000-7f329c000000 ---p 00000000 00:00 0 
7f329c000000-7f329c054000 rw-p 00000000 00:00 0 
7f329c054000-7f32a0000000 ---p 00000000 00:00 0 
7f32a0000000-7f32a0050000 rw-p 00000000 00:00 0 
7f32a0050000-7f32a4000000 ---p 00000000 00:00 0 
7f32a4000000-7f32a4048000 rw-p 00000000 00:00 0 
7f32a4048000-7f32a8000000 ---p 00000000 00:00 0 
7f32a8000000-7f32a8059000 rw-p 00000000 00:07f3280000000-7f328003d000 rw-p 00000000 00:00 0 
7f328003d000-7f3284000000 ---p 00000000 00:00 0 
7f3284000000-7f3284054000 rw-p 00000000 00:00 0 
7f3284054000-7f3288000000 ---p 00000000 00:00 0 
7f3288000000-7f3288056000 rw-p 00000000 00:00 0 
7f3288056000-7f328c000000 ---p 00000000 00:00 0 
7f328c000000-7f328c03d000 rw-p 00000000 00:00 0 
7f328c03d000-7f3290000000 ---p 00000000 00:00 0 
7f3290000000-7f3290059000 rw-p 00000000 00:00 0 
7f3290059000-7f3294000000 ---p 00000000 00:00 0 
7f3294000000-7f3294036000 rw-p 00000000 00:00 0 
7f3294036000-7f3298000000 ---p 00000000 00:00 0 
7f3298000000-7f3298056000 rw-p 00000000 00:00 0 
7f3298056000-7f329c000000 ---p 00000000 00:00 0 
7f329c000000-7f329c054000 rw-p 00000000 00:00 0 
7f329c054000-7f32a0000000 ---p 00000000 00:00 0 
7f32a0000000-7f32a0050000 rw-p 00000000 00:00 0 
7f32a0050000-7f32a4000000 ---p 00000000 00:00 0 
7f32a4000000-7f32a4048000 rw-p 00000000 00:00 0 
7f32a4048000-7f32a8000000 ---p 00000000 00:00 0 
7f32a8000000-7f32a8059000 rw-p 00000000 00:07f3280000000-7f328003d000 rw-p 00000000 00:00 0 
7f328003d000-7f3284000000 ---p 00000000 00:00 0 
7f3284000000-7f3284054000 rw-p 00000000 00:00 0 
7f3284054000-7f3288000000 ---p 00000000 00:00 0 
7f3288000000-7f3288056000 rw-p 00000000 00:00 0 
7f3288056000-7f328c000000 ---p 00000000 00:00 0 
7f328c000000-7f328c03d000 rw-p 00000000 00:00 0 
7f328c03d000-7f3290000000 ---p 00000000 00:00 0 
7f3290000000-7f3290059000 rw-p 00000000 00:00 0 
7f3290059000-7f3294000000 ---p 00000000 00:00 0 
7f3294000000-7f3294036000 rw-p 00000000 00:00 0 
7f3294036000-7f3298000000 ---p 00000000 00:00 0 
7f3298000000-7f3298056000 rw-p 00000000 00:00 0 
7f3298056000-7f329c000000 ---p 00000000 00:00 0 
7f329c000000-7f329c054000 rw-p 00000000 00:00 0 
7f329c054000-7f32a0000000 ---p 00000000 00:00 0 
7f32a0000000-7f32a0050000 rw-p 00000000 00:00 0 
7f32a0050000-7f32a4000000 ---p 00000000 00:00 0 
7f32a4000000-7f32a4048000 rw-p 00000000 00:00 0 
7f32a4048000-7f32a8000000 ---p 00000000 00:00 0 
7f32a8000000-7f32a8059000 rw-p 00000000 00:00 0 
7f32a8059000-7f32ac000000 ---p 00000000 00:00 0 
7f32ac000000-7f32ac03e000 rw-p 00000000 00:00 0 
7f32ac03e000-7f32b0000000 ---p 00000000 00:00 0 
7f32b0000000-7f32b0058000 rw-p 00000000 00:00 0 
7f32b0058000-7f32b4000000 ---p 00000000 00:00 0 
7f32b4000000-7f32b4048000 rw-p 00000000 00:00 0 
7f32b4048000-7f32b8000000 ---p 00000000 00:00 0 
7f32b8000000-7f32b8041000 rw-p 00000000 00:00 0 
7f32b8041000-7f32bc000000 ---p 00000000 00:00 0 
7f32bc000000-7f32bc04d000 rw-p 00000000 00:00 0 
7f32bc04d000-7f32c0000000 ---p 00000000 00:00 0 
7f32c0000000-7f32c0054000 rw-p 00000000 00:00 0 
7f32c0054000-7f32c4000000 ---p 00000000 00:00 0 
7f32c4000000-7f32c4066000 rw-p 00000000 00:00 0 
7f32c4066000-7f32c8000000 ---p 00000000 00:00 0 
7f32c8000000-7f32c8064000 rw-p 00000000 00:00 0 
7f32c8064000-7f32cc000000 ---p 00000000 00:00 0 
7f32cc000000-7f32cc05b000 rw-p 00000000 00:00 0 
7f32cc05b000-7f32d0000000 ---p 00000000 00:00 0 
7f32d0000000-7f32d0043000 rw-p 00000000 00:00 0 
7f32d0043000-7f32d4000000 ---p 00000000======= Backtrace: =========
/lib64/libc.so.6(+0x7f644)[0x7f32fc190644]
/lib64/libc.so.6(+0x83c48)[0x7f32fc194c48]
/lib64/libc.so.6(realloc+0x1d2)[0x7f32fc1968a2]
/lib64/libcrypto.so.10(CRYPTO_realloc+0x49)[0x7f32fc768529]
/lib64/libcrypto.so.10(lh_insert+0x101)[0x7f32fc81dad1]
/lib64/libcrypto.so.10(+0x12640e)[0x7f32fc82040e]
/lib64/libcrypto.so.10(ERR_get_state+0xde)[0x7f32fc820ebe]
/lib64/libcrypto.so.10(ERR_clear_error+0x15)[0x7f32fc821045]
/lib64/libssl.so.10(ssl23_connect+0x45)[0x7f32fcb884a5]
/opt/ntripcaster/sbin/ntripdaemon[0x426f67]
/opt/ntripcaster/sbin/ntripdaemon[0x41b47b]
/opt/ntripcaster/sbin/ntripdaemon[0x41b8e4]
/opt/ntripcaster/sbin/ntripdaemon[0x41b9e6]
/lib64/libpthread.so.0(+0x7de5)[0x7f32fc4e5de5]
0 0 
7f32a805a000-7f32ac000000 ---p 00000000 00:00 0 
7f32ac000000-7f32ac03f000 rw-p 00000000 00:00 0 
7f32ac03f000-7f32b0000000 ---p 00000000 00:00 0 
7f32b0000000-7f32b0058000 rw-p 00000000 00:00 0 
7f32b0058000-7f32b4000000 ---p 00000000 00:00 0 
7f32b4000000-7f32b4048000 rw-p 00000000 00:00 0 
7f32b4048000-7f32b8000000 ---p 00000000 00:00 0 
7f32b8000000-7f32b8041000 rw-p 00000000 00:00 0 
7f32b8041000-7f32bc000000 ---p 00000000 00:00 0 
7f32bc000000-7f32bc04d000 rw-p 00000000 00:00 0 
7f32bc04d000-7f32c0000000 ---p 00000000 00:00 0 
7f32c0000000-7f32c0054000 rw-p 00000000 00:00 0 
7f32c0054000-7f32c4000000 ---p 00000000 00:00 0 
7f32c4000000-7f32c4066000 rw-p 00000000 00:00 0 
7f32c4066000-7f32c8000000 ---p 00000000 00:00 0 
7f32c8000000-7f32c8064000 rw-p 00000000 00:00 0 
7f32c8064000-7f32cc000000 ---p 00000000 00:00 0 
7f32cc000000-7f32cc05b000 rw-p 00000000 00:00 0 
7f32cc05b000-7f32d0000000 ---p 00000000 00:00 0 
7f32d0000000-7f32d0043000 rw-p 00000000 00:00 0 
7f32d0043000-7f32d4000000 ---p 00000000/lib64/libc.so.6(clone+0x6d)[0x7f32fc20ef1d]
======= Memory map: ========
0 0 
7f32a805a000-7f32ac000000 ---p 00000000 00:00 0 
7f32ac000000-7f32ac03f000 rw-p 00000000 00:00 0 
7f32ac03f000-7f32b0000000 ---p 00000000 00:00 0 
7f32b0000000-7f32b0058000 rw-p 00000000 00:00 0 
7f32b0058000-7f32b4000000 ---p 00000000 00:00 0 
7f32b4000000-7f32b4048000 rw-p 00000000 00:00 0 
7f32b4048000-7f32b8000000 ---p 00000000 00:00 0 
7f32b8000000-7f32b8041000 rw-p 00000000 00:00 0 
7f32b8041000-7f32bc000000 ---p 00000000 00:00 0 
7f32bc000000-7f32bc04d000 rw-p 00000000 00:00 0 
7f32bc04d000-7f32c0000000 ---p 00000000 00:00 0 
7f32c0000000-7f32c0054000 rw-p 00000000 00:00 0 
7f32c0054000-7f32c4000000 ---p 00000000 00:00 0 
7f32c4000000-7f32c4066000 rw-p 00000000 00:00 0 
7f32c4066000-7f32c8000000 ---p 00000000 00:00 0 
7f32c8000000-7f32c8064000 rw-p 00000000 00:00 0 
7f32c8064000-7f32cc000000 ---p 00000000 00:00 0 
7f32cc000000-7f32cc05b000 rw-p 00000000 00:00 0 
7f32cc05b000-7f32d0000000 ---p 00000000 00:00 0 
7f32d0000000-7f32d0043000 rw-p 00000000 00:00 0 
7f32d0043000-7f32d4000000 ---p 00000000 00:00 0 
7f32d4000000-7f32d4067000 rw-p 00000000 00:00 0 
7f32d4067000-7f32d8000000 ---p 00000000 00:00 0 
7f32d8000000-7f32d8053000 rw-p 00000000 00:00 0 
7f32d8053000-7f32dc000000 ---p 00000000 00:00 0 
7f32dc000000-7f32dc048000 rw-p 00000000 00:00 0 
7f32dc048000-7f32e0000000 ---p 00000000 00:00 0 
7f32e0000000-7f32e005a000 rw-p 00000000 00:00 0 
7f32e005a000-7f32e4000000 ---p 00000000 00:00 0 
7f32e4000000-7f32e403f000 rw-p 00000000 00:00 0 
7f32e403f000-7f32e8000000 ---p 00000000 00:00 0 
7f32e8000000-7f32e8039000 rw-p 00000000 00:00 0 
7f32e8039000-7f32ec000000 ---p 00000000 00:00 0 
7f32ec000000-7f32ec03e000 rw-p 00000000 00:00 0 
7f32ec03e000-7f32f0000000 ---p 00000000 00:00 0 
7f32f0000000-7f32f0044000 rw-p 00000000 00:00 0 
7f32f0044000-7f32f4000000 ---p 00000000 00:00 0 
7f32f4000000-7f32f40ee000 rw-p 00000000 00:00 0 
7f32f40ee000-7f32f8000000 ---p 00000000 00:00 0 
7f32f803b000-7f32f803c000 ---p 00000000 00:00 0 
7f32f803c000-7f32f807b000 rw-p 00000000 00:00 0 
7f32f807b000-7f32f807c000 ---p 000 00:00 0 
7f32d4000000-7f32d4067000 rw-p 00000000 00:00 0 
7f32d4067000-7f32d8000000 ---p 00000000 00:00 0 
7f32d8000000-7f32d8053000 rw-p 00000000 00:00 0 
7f32d8053000-7f32dc000000 ---p 00000000 00:00 0 
7f32dc000000-7f32dc048000 rw-p 00000000 00:00 0 
7f32dc048000-7f32e0000000 ---p 00000000 00:00 0 
7f32e0000000-7f32e005a000 rw-p 00000000 00:00 0 
7f32e005a000-7f32e4000000 ---p 00000000 00:00 0 
7f32e4000000-7f32e403f000 rw-p 00000000 00:00 0 
7f32e403f000-7f32e8000000 ---p 00000000 00:00 0 
7f32e8000000-7f32e8039000 rw-p 00000000 00:00 0 
7f32e8039000-7f32ec000000 ---p 00000000 00:00 0 
7f32ec000000-7f32ec03e000 rw-p 00000000 00:00 0 
7f32ec03e000-7f32f0000000 ---p 00000000 00:00 0 
7f32f0000000-7f32f0044000 rw-p 00000000 00:00 0 
7f32f0044000-7f32f4000000 ---p 00000000 00:00 0 
7f32f4000000-7f32f40ee000 rw-p 00000000 00:00 0 
7f32f40ee000-7f32f8000000 ---p 00000000 00:00 0 
7f32f803b000-7f32f803c000 ---p 00000000 00:00 0 
7f32f803c000-7f32f807b000 rw-p 00000000 00:00 0 
7f32f807b000-7f32f807c000 ---p 00000400000-00442000 r-xp 00000000 103:01 282416                            /opt/ntripcaster/sbin/ntripdaemon
00641000-00645000 rw-p 00041000 103:01 282416                            /opt/ntripcaster/sbin/ntripdaemon
00645000-00646000 rw-p 00000000 00:00 0 
01b18000-01cc5000 rw-p 00000000 00:00 0                                  [heap]
01cc5000-01d03000 rw-p 00000000 00:00 0                                  [heap]
7f327bd6a000-7f327bd80000 r-xp 00000000 103:01 734                       /lib64/libgcc_s-7-20170915.so.1
7f327bd80000-7f327bf7f000 ---p 00016000 103:01 734                       /lib64/libgcc_s-7-20170915.so.1
7f327bf7f000-7f327bf80000 rw-p 00015000 103:01 734                       /lib64/libgcc_s-7-20170915.so.1
7f327bf80000-7f327bf81000 ---p 00000000 00:00 0 
7f327bf81000-7f327bfc0000 rw-p 00000000 00:00 0 
7f327bfc0000-7f327bfc1000 ---p 00000000 00:00 0 
7f327bfc1000-7f327c000000 rw-p 00000000 00:00 0 
7f327c000000-7f327c066000 rw-p 00000000 00:00 0 
7f327c066000-7f3280000000 ---p 00000000 00:00 0 
 00:00 0 
7f32d4000000-7f32d4067000 rw-p 00000000 00:00 0 
7f32d4067000-7f32d8000000 ---p 00000000 00:00 0 
7f32d8000000-7f32d8053000 rw-p 00000000 00:00 0 
7f32d8053000-7f32dc000000 ---p 00000000 00:00 0 
7f32dc000000-7f32dc048000 rw-p 00000000 00:00 0 
7f32dc048000-7f32e0000000 ---p 00000000 00:00 0 
7f32e0000000-7f32e005a000 rw-p 00000000 00:00 0 
7f32e005a000-7f32e4000000 ---p 00000000 00:00 0 
7f32e4000000-7f32e403f000 rw-p 00000000 00:00 0 
7f32e403f000-7f32e8000000 ---p 00000000 00:00 0 
7f32e8000000-7f32e8039000 rw-p 00000000 00:00 0 
7f32e8039000-7f32ec000000 ---p 00000000 00:00 0 
7f32ec000000-7f32ec03e000 rw-p 00000000 00:00 0 
7f32ec03e000-7f32f0000000 ---p 00000000 00:00 0 
7f32f0000000-7f32f0044000 rw-p 00000000 00:00 0 
7f32f0044000-7f32f4000000 ---p 00000000 00:00 0 
7f32f4000000-7f32f40ee000 rw-p 00000000 00:00 0 
7f32f40ee000-7f32f8000000 ---p 00000000 00:00 0 
7f32f803b000-7f32f803c000 ---p 00000000 00:00 0 
7f32f803c000-7f32f807b000 rw-p 00000000 00:00 0 
7f32f807b000-7f32f807c000 ---p 000^C
[ec2-user@auscors-one ~]$ 
[ec2-user@auscors-one ~]$ 
[ec2-user@auscors-one ~]$ 
[ec2-user@auscors-one ~]$ [04/Jun/2019:06:29:26] WARNING: Unrecognized group [] specified for mount [default]
NtripCaster Version 2.0.36 Initializing...
NtripCaster comes with NO WARRANTY, to the extent permitted by law.
You may redistribute copies of NtripCaster under the terms of the
GNU General Public License.
For more information about these matters, see the file named COPYING.
Starting thread engine...
*** Error in `/opt/ntripcaster/sbin/ntripdaemon': realloc(): invalid next size: 0x00007f1c98022fd0 ***
*** Error in `/opt/ntripcaster/sbin/ntripdaemon': realloc(): invalid next size: 0x00007f1c98022fd0 ***
*** Error in `/opt/ntripcaster/sbin/ntripdaemon': realloc(): invalid next size: 0x00007f1c98022fd0 ***
*** Error in `/opt/ntripcaster/sbin/ntripdaemon': realloc(): invalid next size: 0x00007f1c98022fd0 ***
*** Error in `/opt/ntripcaster/sbin/ntripdaemon': realloc(): invalid next size: 0x00007f1c98022fd0 ***
*** Error in `/opt/ntripcaster/sbin/ntripdaemon': realloc(): invalid next size: 0x00007f1c98022fd0 ***
*** Error in `/opt/ntripcaster/sbin/ntripdaemon': realloc(): invalid next size: 0x00007f1c98022fd0 ***
*** Error in `/opt/ntripcaster/sbin/ntripdaemon': realloc(): invalid next size: 0x00007f1c98022fd0 ***
*** Error in `/opt/ntripcaster/sbin/ntripdaemon': realloc(): invalid next size: 0x00007f1c98022fd0 ***
*** Error in `/opt/ntripcaster/sbin/ntripdaemon': realloc(): invalid next size: 0x00007f1c98022fd0 ***
*** Error in `/opt/ntripcaster/sbin/ntripdaemon': realloc(): invalid next size: 0x00007f1c98022fd0 ***
*** Error in `/opt/ntripcaster/sbin/ntripdaemon': realloc(): invalid next size: 0x00007f1c98022fd0 ***
*** Error in `/opt/ntripcaster/sbin/ntripdaemon': realloc(): invalid next size: 0x00007f1c98022fd0 ***
*** Error in `/opt/ntripcaster/sbin/ntripdaemon': realloc(): invalid next size: 0x00007f1c98022fd0 ***
*** Error in `/opt/ntripcaster/sbin/ntripdaemon': realloc(): invalid next size: 0x00007f1c98022fd0 ***
*** Error in `/opt/ntripcaster/sbin/ntripdaemon': realloc(): invalid next size: 0x00007f1c98022fd0 ***
*** Error in `/opt/ntripcaster/sbin/ntripdaemon': realloc(): invalid next size: 0x00007f1c98022fd0 ***
*** Error in `/opt/ntripcaster/sbin/ntripdaemon': realloc(): invalid next size: 0x00007f1c98022fd0 ***
[04/Jun/2019:06:29:51] WARNING: Unrecognized group [] specified for mount [default]
NtripCaster Version 2.0.36 Initializing...
NtripCaster comes with NO WARRANTY, to the extent permitted by law.
You may redistribute copies of NtripCaster under the terms of the
GNU General Public License.
For more information about these matters, see the file named COPYING.
Starting thread engine...
*** Error in `/opt/ntripcaster/sbin/ntripdaemon': realloc(): invalid next size: 0x00007f442c004520 ***
*** Error in `/opt/ntripcaster/sbin/ntripdaemon': realloc(): invalid next size: 0x00007f442c004520 ***
*** Error in `/opt/ntripcaster/sbin/ntripdaemon': realloc(): invalid next size: 0x00007f442c004520 ***
*** Error in `/opt/ntripcaster/sbin/ntripdaemon': realloc(): invalid next size: 0x00007f442c004520 ***
*** Error in `/opt/ntripcaster/sbin/ntripdaemon': realloc(): invalid next size: 0x00007f442c004520 ***
*** Error in `/opt/ntripcaster/sbin/ntripdaemon': realloc(): invalid next size: 0x00007f442c004520 ***
*** Error in `/opt/ntripcaster/sbin/ntripdaemon': realloc(): invalid next size: 0x00007f442c004520 ***
*** Error in `/opt/ntripcaster/sbin/ntripdaemon': realloc(): invalid next size: 0x00007f442c004520 ***
*** Error in `/opt/ntripcaster/sbin/ntripdaemon': realloc(): invalid next size: 0x00007f442c004520 ***
*** Error in `/opt/ntripcaster/sbin/ntripdaemon': realloc(): invalid next size: 0x00007f442c004520 ***
*** Error in `/opt/ntripcaster/sbin/ntripdaemon': realloc(): invalid next size: 0x00007f442c004520 ***
*** Error in `/opt/ntripcaster/sbin/ntripdaemon': realloc(): invalid next size: 0x00007f442c004520 ***
*** Error in `/opt/ntripcaster/sbin/ntripdaemon': realloc(): invalid next size: 0x00007f442c004520 ***
*** Error in `/opt/ntripcaster/sbin/ntripdaemon': realloc(): invalid next size: 0x00007f442c004520 ***

Sometimes accompanied by a full stack trace ("Backtrace"). Sometimes with no logs from the caster. But always at least one segfault in kernal logs.

I found this similar error in the icecast git repo which was resolved but no info was added: https://github.com/karlheyes/icecast-kh/issues/142

Have you experienced the same limitation?

Attachments (1)

lock.patch (1.2 KB ) - added by stoecker 5 years ago.

Download all attachments as: .zip

Change History (11)

comment:1 by stoecker, 5 years ago

Is it possible to use openssl >= 1.1.0 instead of 1.0.2?

comment:2 by stoecker, 5 years ago

Component: OtherProfessional Caster
Owner: changed from team to stoecker

by stoecker, 5 years ago

Attachment: lock.patch added

comment:3 by stoecker, 5 years ago

Owner: changed from stoecker to brandon.owen@…
Status: newneedinfo

Can you please try if attached patch fixes this also for your 1.0.2k version?

comment:4 by anonymous, 5 years ago

I'll need to do some more testing, but using version 1.1.0 seems to have resolved the issue. Thanks very much. I'll follow up today. I don't think I'll have time to test the patch on the older version.

Last edited 5 years ago by anonymous (previous) (diff)

comment:5 by anonymous, 5 years ago

The caster has been running well for a little while so seems fine to close this job.

I also just found out another issue which is that I can't create a loop of alias'. I.e. if you have a source called NAME0 and NAME1, and you have alias /NAME0 /NAME1 and alias /NAME1 /NAME0 you get consistently reproducible segfaults (even if the sources are not running). It makes sense, but it just means I have to do a bit more work in transitioning sources to use new mountpoint names. So I'm not going to bother creating another issue for this, unless you think it matters.

comment:6 by stoecker, 5 years ago

So I'm not going to bother creating another issue for this, unless you think it matters.

Please open an issue for this with an easy description how to reach the segfault. This setup should not work, but it should not segfault either (rather issue a warning).

comment:7 by neumaier, 5 years ago

Hello,
I'd just like to tell you that I tried the new version together with openssl 1.0.1f and was able to successfully setup 151 TLS relays.
Regards, Peter Neumaier

comment:8 by anonymous, 5 years ago

Interesting, is that including your patch? If not then maybe it's specifically an issue with 1.0.3.

comment:9 by anonymous, 5 years ago

I tried the caster software including the new patch. I didn't try that without the patch, so unfortunately I can't say whether it's running because of the patch or because I used version 1.0.1f?

in reply to:  9 comment:10 by stoecker, 5 years ago

Resolution: fixed
Status: needinfoclosed

Replying to anonym:

I tried the caster software including the new patch. I didn't try that without the patch, so unfortunately I can't say whether it's running because of the patch or because I used version 1.0.1f?

There should be no difference between 1.0.1 and 1.0.2. The patch is for any version <1.1.

Modify Ticket

Change Properties
Action
as closed The owner will remain brandon.owen@….
The resolution will be deleted. Next status will be 'reopened'.

Add Comment


E-mail address and name can be saved in the Preferences .
 
Note: See TracTickets for help on using tickets.