Goodness of fit

[erex]

This may be difficult to achieve. Models are fitted and estimates produced for each simulation replicate, but there is no assessment of model fit. Likelihoods and AIC scores are produced, but absolute fit is not assessed. In my particular application, I'd be interested to know how often density estimates are being derived from models that do not fit the data.

I suppose this would mean sending each replicate data set and fitted model off to gof_ds and harvesting statistics from there.

[LHMarshall]

Given that the Chi-square is better assessed by eye over a number of options for break points for the bins and that it is largely unfeasible to run a bootstrap within a simulation I imagine the best option for this would be to add an option for Cramer-von Mises p-values to be collected and stored?

[lenthomas]

Yes I agree ks or CvM better for continuous data (are stuck with chi-sq for discrete data).

[erex]

Would be happy if CvM could be included as an option. You're right, performing KS test (that requires bootstrap) would not be a great idea.

[lenthomas]

I don't think a KS test requires a bootstrap (see, e.g., ks.test in R). The CvM required a simulation study to produce a lookup table of critical values in the old FORTRAN code, but I think the p-value is now derived some other way in the GoF outputs in mrds. CvM is supposidly more powerful than KS I think, so possibly preferable. In my experience they rarely differ in terms of inference made on GoF for fitted detection functions.

[erex]

Yes, KS requires a bootstrap. See this line from Miller et al. (2019) Distance sampling in R.

Cramér-von Mises is generally more powerful than the Kolmogorov-Smirnov test, which uses the largest difference between a point on the Q-Q plot and the line y = x (which is also available in Distance, though is not produced by default as it requires computationally demanding bootstraps)

This arose from the reviews of this manuscript, when author noted something because parameters were being estimated, so standard K-S test is not appropriate. I've forgotten the details of that manuscript review comments.

I agree that P-values for CvM are now exact values from the mrds package rather than the table lookup that came from DistWin. Still would like CvM coming from dsims if possible.

[erex]

Supplement from comments within gof_ds in the Distance package

#' Note that a bootstrap procedure is required for the Kolmogorov-Smirnov test
#' to ensure that the p-values from the procedure are correct as the we are
#' comparing the cumulative distribution function (CDF) and empirical
#' distribution function (EDF) and we have estimated the parameters of the
#' detection function.