Data browser with archive appliance data source doesn't show all available data

[MichaelRitzert]

I frequently notice that the databrowser widget embedded in an OPI fails to display all archived data. It's always the time just before switching to live data where data is missing.

I never notice this when I right-click the widget and open in the full data browser using the same .plt file. See attached screenshots of roughly the same time period. The OPI has just been opened, i.e. live data is only at the very right of the display.

The archiver we use is the archiver appliance.

[kasemir]

Check the data browser preference setting to "Automatically refresh historic data when the live sample buffer is full",

# Automatically refresh history data when the liver buffer is full
# This will prevent the horizontal lines in the shown data when the buffer
# is too small to cover the selected time range 
org.csstudio.trends.databrowser2/automatic_history_refresh=false

The data gap with the horizontal line between end of historic samples fetched from archive and the live sample buffer is created because at start of the data browser, it fetches archived data until "now". Then it adds live samples to the live data buffer. That live data buffer, however, has a limited size because if not you would eventually run out of memory.

When you look at the data browser widget inside a long running *.opi, that's likely to happen.
When you then start the normal data browser, it basically starts over with newly fetched archived data.

Fix 1)
Configure the live data buffer such that it holds enough samples for the width of the plot.
This can be configured for each trace.
Upside: You'll have the fill resolution live data for the trace.
Downside: You need to figure out how many samples you need, and you may use more memory.

Fix 2)
Enable that automatic_history_refresh option which will re-read historic data.
Upside: Easier to configure, just turn it on, no thinking required.
Downside: Archive access is usually slow, and you tend to get lower resolution data (MDEL vs. ADEL).

[kasemir]

Other idea:
Increase the log level so you see the archive data requests.
Is the widget-version not requesting the full range of samples from the archive?

From the context menu, try to open the sample view to check if
a) There is indeed a gap in the samples
b) The archived data is all there, but it's not displayed??

[MichaelRitzert]

It's not in long-running OPIs. I can open CSS and it happens right away. E.g. CSS is running for 1 minute, but the gap is 15 minutes. I still tried
org.csstudio.trends.databrowser2/automatic_history_refresh=true
but that didn't change anything.

I just had an occurrence of the problem in the full databrowser view. So it DOES happen, but much less frequently than in the widget. I checked, and it's missing the samples in the sample view also.

The log output says that one full hour of data is requested:
Apr 26, 2016 8:38:04 PM org.epics.archiverappliance.retrieval.client.RawDataRetrieval getDataForPV
INFO: URL to fetch data is http://archive:17668/retrieval/data/getData.raw?pv=ncount%28DESY2%3ACURR%3Acur%29&from=2016-04-26T19%3A38%3A04.936%2B02%3A00&to=2016-04-26T20%3A38%3A04.936%2B02%3A00
Apr 26, 2016 8:38:04 PM org.epics.archiverappliance.retrieval.client.RawDataRetrieval getDataForPV
INFO: URL to fetch data is http://archive:17668/retrieval/data/getData.raw?pv=DESY2%3ACURR%3Acur&from=2016-04-26T19%3A38%3A04.936%2B02%3A00&to=2016-04-26T20%3A38%3A04.936%2B02%3A00

The output from the archiver is fine. I downloaded the above URL with wget (replaced raw with csv), plotted with gnuplot, and it's perfect. Actually, I can just reopen the plot and there's a fair chance that it's fine the next time.

[kasemir]

.. it's missing the samples in the sample view also ..
That's a useful clue.

We've had problems like #1443 where the plot showed a large gap because it got confused by some specific sequence of samples that included NaN.
But in your case it seems like the data has the gap, and the plot just reflects that.

So then the problem could be

1. http://archive:17668/retrieval/data/getData.raw doesn't always return all the data. Often, but not always.
2. org.epics.archiverappliance.retrieval.client.RawDataRetrieval somehow drops samples.
3. org.csstudio.trends.databrowser2.archive.ArchiveFetchJob fails to completely merge the received samples.

I'm afraid you'll need to insert log/print statements in for example RawDataRetrieval to see what it gets.
The ArchiveFetchJob will log "Ended ... with 2134234 samples in .. seconds" for FINE log level, maybe that also helps to see if the number of received suddenly gets smaller when you see the gap.

[MichaelRitzert]

I just ran the HTTP request (with raw output) 100 times, and it always came back with the same data (checked with sha256sum). While taking the 100 samples, I did see problems with the GUI, so it's also not about the load on the server. That makes 1) unlikely.

Do you know a fast way to increase the debug level for this logger only?

To add more debug statements, I have to change my workflow to compile everything locally instead of against the online p2s. I can have a look into that the day after tomorrow.

[kasemir]

If your CSS product includes the logging.ui, you should have a menu entry CSS, Debugging, Logging Configuration where you can configure individual loggers.

[MichaelRitzert]

OK, here's the relevant output. I happened to get a plot with hardly any data in it, and the output is
2016-04-27 09:43:36.863 FINE [Thread 77] org.csstudio.trends.databrowser2.archive.ArchiveFetchJob$WorkerThread (run) - Ended Read data: MARCO:TempAfterDet:TEMP:curr, 2016-04-27 08:43:36.819 +0200 - 2016-04-27 09:43:36.819 +0200 with 9 samples in 0.042 seconds
Normally, it's around 901 samples.

[MichaelRitzert]

And I just acquired a simultaneous tcpdump of the traffic with the archiver and log output. The request was

GET /retrieval/data/getData.raw?pv=optimized_800%28MARCO%3ATempBeforeDet%3ATEMP%3Acurr%29&from=2016-04-27T08%3A52%3A36.850%2B02%3A00&to=2016-04-27T09%3A52%3A36.850%2B02%3A00 HTTP/1.1
(which is different from the one in the log message above), the log says

2016-04-27 09:52:37.480 FINE [Thread 74] org.csstudio.trends.databrowser2.archive.ArchiveFetchJob$WorkerThread (run) - Ended Read data: MARCO:TempBeforeDet:TEMP:curr, 2016-04-27 08:52:36.850 +0200 - 2016-04-27 09:52:36.850 +0200 with 167 samples in 0.619 seconds

but the tcpdump looks like way more samples were sent. I look at the dump with wireshark and without decoding the PBF, I see several hundred instances of a very similar pattern.

[kasemir]

From your log info I'm pretty sure the problem is in the second step:

1. http://archive:17668/retrieval/data/getData.raw doesn't always return all the data. Often, but not always.
2. org.epics.archiverappliance.retrieval.client.RawDataRetrieval somehow drops samples.
3. org.csstudio.trends.databrowser2.archive.ArchiveFetchJob fails to completely merge the received samples.

The ArchiveFetchJob does 'merge' data that it might receive from several archive data sources, plus it then later on ignores samples time stamped after the oldest 'live' sample.
But that count in the log message is simply the sum of all the received samples, before merging etc.

So "ArchiveFetchJob$WorkerThread (run) - Ended ...with 9 samples" should really mean that the archiverappliance.retrieval.client code only provided 9 samples. If the basic size of the network data returned by the getData.raw servlet appears to be much larger, the error must be in the archiverappliance.retrieval.client code.

I have no idea about the archive appliance, and didn't work on the archiverappliance.retrieval.client code. The network call with "pv=optimized_800.." looks like it's fetching optimized data, which in principle is good.
Maybe try configuring the channel to use original samples, and see if the problem only happens when the archiverappliance.retrieval.client code tries to decode optimized data, and not when it decodes raw data.

[MichaelRitzert]

I just took the .plot file and replaced all OPTIMIZED with RAW.
The problem still occurs.

[kasemir]

@jbobnar Jaka, do you have suggestions how to tell if the problem is in the pbrawclient org.epics.archiverappliance.retrieval.client) or the org.csstudio.archive.reader.appliance code?
Looks like one of them is returning only a fraction of the samples.

[jbobnar]

I've never seen this. I tried to reproduce it (on master branch), but it always worked OK. The fact that it always works in DataBrowser but not in OPIs is puzzling. If the problem was in the appliance code, you would probably see it in data browser at least from time to time.

Are you using a single server or a cluster with load balancer? Are all of your server instances of the same version? What about CSS version? Your previous comment (#1769 (comment)) suggests that your appliance supports "optimized" retrieval. However, in #1769 (comment) I see that CSS made a call using "ncount" operator, which suggests that your CSS might not be up to date (this was implemented about 9 months ago and is available on 4.2.x and master branches).

Can you check the log files on the server? Look for the file /retrieval/logs/localhost-2016...txt. It should contain all GET requests that the clients have made. The last number in each log entry is the number of bytes in the response of the appliance. Do you see any logs with significantly lower sizes? That would suggest that the appliance is sending back less points (only 9 samples).