Data loss at beginning of the day

[droma]

We're using bnc 2.12.2 to retrieve data from different casters (about ~250 different mountpoints) at a sample rate of 10 seconds. Every day at midnight we loss all the mountpoints data. It looks like there is some kind of problem with the time reference used by bnc in this case and therefore it discards all date with "Old epoch" messages (find attached full log).

In case it is useful for your, there seem to be two things also happening at the day change: configuration read (which is always done, even if the onTheFlyInterval is not set) and rinex files creation. Perhaps some of this tasks is blocking the others, which adds a delay bigger than the sample rate?

Thanks in advance,
David Roma

[droma]

BNC log

[stuerze]

Dear David,

would it be possible to get access to one of your 10 samples per seconds data stream for test purposes?

Thanks in advance, Andrea.

[droma]

Dear Andrea,

may I send you the bnc configuration file through some private channel (mail)? Since most of them require password access I don't want to leave the file here. Or do you prefer that I post here the mountpoints of euref and other public casters which doesn't require password?

Regards,
David

[stuerze]

Dear David,

I was a little bit irritated from your message, to retrieve data at a rate of 10 seconds. The data that are available from our EUREF-Caster www.euref-ip.net are coming in 1 samples per seconds and therefore I need no account from you ;).
Nevertheless it would be easier to have a configuration file from you, to understand, what your aim and your problem could be. You can send it to andrea.stuerze@….

Some remarks to your findings: RINEX file creation is done if it is configured only.

Best regards, Andrea

[stuerze]

Last but not least: using the RINEX file generation option, you are able to check, whether there is really data loss at the beginning of the new day. We are using BNC to generate a lot of RINEX files and cannot detect such bahavior.
The message "Epoch thrown away" means, that the epoch was already transmitted, hence it is not lost.

[droma]

Dear Andrea,

I sent you the configuration file via e-mail. Please let me know if there is something wrong. One short, unrelated probably, question, what should we use inside the configuration file: ignoreSslErrors or sslIgnoreErrors? From the bnc help I think it should be sslIgnoreErrors but the example configuration 07_FeedEngine uses ignoreSslErrors.

To clarify how we are using bnc: we only use it to acquire data from the different mountpoints and write it to a file (outFile of the bnc configuration). The file shows that there is a lot of missing data at the start of the data. If you want I can provide you also with one of them.

The RINEX file don't seem to have this data missing. I will check afterwards in deep and report back to you, but at a first glance it looks like it was only not written to the outFile. Is this possible?

Regards,
David

[stuerze]

David, the correct key is sslIgnoreErrors. The example configuration contains an earlier Version. I'm sorry for that.

Best regards, Andrea

[stuerze]

If your problem occurs only in connection with the synchronized feed engine output, you should check the latency of all of your streams and compare it with the time given in 'wait for full obs epoch' (key outWait). Everything what is later than this will be dropped:

[droma]

Dear Andrea,

is there any option in bnc to print latency statistics? I don't think this issue is related with the latency, since first of all it always happens at the beginning of the day and I will expect to have a much more random behaviour and not systemtical fall to zero mountpoints on the start of the day. Second, we didn't had the problem with the previous versions of bnc.

Regarding the outWait parameter, can it be set to a higher value than the outSampl? Is it acting like a fixed delay then until which bnc will be buffering the data?

Regards,
David

[stuerze]

Here you will find some information about BNC's latency check:

​http://software.rtcm-ntrip.org/export/HEAD/ntrip/branches/BNC_2.12/src/bnchelp.html#misc
​http://software.rtcm-ntrip.org/export/HEAD/ntrip/branches/BNC_2.12/src/bnchelp.html#miscperf

and here some information about the parameter Wait for Full Obs Epoch:

​http://software.rtcm-ntrip.org/export/HEAD/ntrip/branches/BNC_2.12/src/bnchelp.html#syncwait

[stuerze]

Dear David,

last night I checked about 50 of your streams using BNC version 2.12.3 (available at ​https://igs.bkg.bund.de/ntrip/bnc) and I cannot detect "zero mountpoints" this morning.

Meanwhile I remember, that there was a bug fixed: Now a check regarding wrong observation epochs is done during latency check as well to prevent erroneous latencies. Maybe, that helps in your case as well. Please give it a try.

BTW, I'm not sure even it is a really good idea to write ascii data from about 300 streams into a file. There is also the possibility to use the local IP port or RINEX files.

Best regards, Andrea.

[droma]

Dear Andrea,

I updated to BNC version 2.12.3 as you suggested. I also disabled the rinex file creation. With both changes, there is still a small fall at the beginning of the day but it's in the range of the falls that sometimes happen during a normal operation. So if you want you can mark as closed the bug.

FYI, I tried to disable the rinex file creation without restarting bnc (version 2.12.2), leaving the rnxPath parameter blank in the configuration file and waiting for the configuration reload. The result was unexpected for me. Instead of stopping the rinex files generation, it started creating them in the directory where bnc was been executed.

One last question regarding your comment of using the outPort instead of outFile. Which are the advantages? Last time I checked bnc was writing about 50 kb/s so I wouldn't expect any I/O problem there, specially if the write operation is async.

Thanks,
David

[stuerze]

Dear David,

nice to hear, that the already solved bug also helps in your case.
Disabling the RINEX file generation should be independent of that.

Nevertheless, RINEX high-rate files from IGS, MGEX anf EUREF stations are already available within the respective data bases. For example

• ​ftp://igs.bkg.bund.de/IGS/highrate/2016/ or
• ​ftp://igs.bkg.bund.de/IGS/highrate_v3/2016/ or
• ​ftp://igs.bkg.bund.de/MGEX/highrate_v3/2016

The port usage was only a hint how to feed an application more directly.

I tested to delete the RINEX path within the configuration file, which is running in no window mode. And I cannot confirm your observation. To store the files within the directory of a running BNC you have to use './'. But maybe the following hints help to explain the behaviour in your environment: https://software.rtcm-ntrip.org/export/HEAD/ntrip/branches/BNC_2.12/src/bnchelp.html#introConf

Best regards, Andrea.