root-project / root Public

In #7616, it was not taken care
that the `RooNLLVar` constructor interface changes were backwards
compatible, since the affected constructors were considered as internal.

However, this caused problems for users nonetheless, hence the interface
was changed to be backwards compatible.

Closing #8976.

The global observables need to be treated consistently in the constraint
sum and the actual likelihood term, even though it would be unusual that
global observables appear in the likelihood outside of the constraints.

The `GlobalObservablesSource` command argument can take one of two
values:

  * `data`: to take the values from the dataset, falling back to the pdf
    value if a given global observable is not available.  If no
    `GlobalObservables` or `GlobalObservablesTag` command argument is
    given, the set of global observables will be automatically defined
    to be the set stored in the data.

  * `model`: to take all values from the pdf and completely ignore the
    set of global observables stored in the data (not even using it to
    automatically define the set of global observables if the
    `GlobalObservables` or `GlobalObservablesTag` command arguments are
    not given).

The default option is `data`. The `model` option basically means to
ignore the new feature of global observables stored in the dataset.

A unit test to implement that the `model` source works is also
implemented in this commit.

Since the parameters of the RooConstraintSum need to be syncronized with
the dataset now when global observables are used from the dataset, the
`RooAbsReal::setData()` function needs to be overridden.

Adapt `RooConstraintSum::createConstraintTerm` to consider when the
dataset stores global observables.

The RooFit behavior changes in two ways:

  * If no global observables are specified with `GlobalObservables` in
    the `RooAbsPdf::fitTo()` command arguments, the set of global
    observables is automatically detected from the dataset and the
    values from the dataset are used.
  * If the global observables are specified with `GlobalObservables`,
    their values are taken from the dataset if they are stored in it,
    otherwise they are taken from the model as before. It's also
    supported that only a subset of the specified global observables is
    stored in the data.

This is to conveniently define the global observables stored in the
`RooDataSet` or `RooDataHist` in the constructor.

This commit doesn't change any fitting behavior. It only adds the
possibility to attach global observable snapshots to the RooAbsData.

The `RooAbsPdf::fitTo()` function creates the likelihood function
itself, using the `RooAbsPdf::createNLL()` function. But if one has
already created the NLL manually, it is not possible to easily reproduce
the minimizer configuration that `RooAbsPdf::fitTo()` uses.

This commit refactors `RooAbsPdf::fitTo()` such that the minimizer part
is put into a new function called `RooAbsPdf::minimizeNLL()`. This is
useful for unit tests and debuggings where one wants to do something
with the likelihood function before proceeding with the minimization.

To be linked from the ROOT manual in the "First Steps" section.

RooUnbinnedL contained an old version of the "scalar" calculation method from RooNLLVar. This code switches RooUnbinnedL to instead use the scalar compuation method from RooNLLVar by default. To do this, the computation was factored out into a static member function inside RooNLLVar, so that RooUnbinnedL can reach it too and we avoid code duplication.

In addition, RooUnbinnedL can now also be set into batch-evaluation
mode. This too uses the existing batch calculation methods inside
RooNLLVar, which also was factored out into a static member function.
testLikelihoodSerial also contains a test that covers the batch mode
now.

Based on code review by @hageboeck in PR #8700.

We previously used a set of custom Kahan summation routines, but now switched from those and from a few manual Kahan summation lines of code strewn around to ROOT::Math::KahanSum.

In addition, we refactored evaluatePartition and friends to return KahanSum objects instead of returning a single double result value and storing the carry separately in the classes. This simplifies the code, and also made it easier to retain the OffsettingMode::Legacy option, which was built for numerical compatibility with the old test statistics classes.

buildLikelihood is a likelihood builder that takes a pdf, dataset and optional parameters, analyzes all input, and returns the most suitable likelihood class object.

The builder is now also used in testLikelihoodSerial.

We removed one of the tests, because it highlights an oversight in
RooNLLVar, and should hence be tested there. The oversight is that
RooAbsPdf::createNLL only activates binned mode for simultaneous PDFs,
not for single PDFs, even if you explicitly set the BinnedLikelihood
attribute.

Based on review by @hageboeck in PR #8700.

Based on review by @hageboeck and @guitargeek in PR #8700.

- Cleaned remaining TODOs, commented out code, etc.
- Replaced TIterator and RooFIter with range-based for-loops.
- Include cleanup.
- Replace raw with unique_ptrs where possible, also thereby fixing many memory leaks in tests.
- Remove unused functions (processEmptyDataSets)
- Use auto.
- Break circular dependency between RooMinimizer & MinuitFcnGrad
  * Also fixes some minor ensuing include errors and reorders RooMinimizer.cxx includes.
- Some minor automatic style fixes and corrects to debugging/warning prints in RooAbsMinimizerFcn.
- Fixes RooFormulaVar failure in RooRealL test, thanks to @guitargeek.

This commit introduces a major refactoring of the RooAbs(Opt)TestStatistic-RooNLLVar inheritance tree into:
1. statistics-based classes on the one hand;
2. calculation/evaluation/optimization based classes on the other hand.

The likelihood is the central unit on the statistics side. The RooAbsL class is implemented for four kinds of likelihoods: binned, unbinned, "subsidiary" (an optimization for numerical stability that gathers components like global observables) and "sum" (over multiple components of the other types). These classes provide ways to compute their components in parallelizable chunks that can be used by the calculator classes as they see fit. On top of the likelihood classes, we also provide for convenience a set of likelihood builders.

The calculator "Wrapper" classes are abstract interfaces. These can be implemented for different kinds of algorithms, or with different kinds of optimization "back-ends" in mind. In an upcoming PR, we will introduce the fork-based multi-processing implementation based on RooFit::MultiProcess. Other possible implementations could use the GPU or external tools like TensorFlow.

The coupling of all these classes to RooMinimizer is made via the MinuitFcnGrad class, which owns the Wrappers that calculate the likelihood components.

* [io] Accelerate TDirectoryFile Find/Get functions

Fix issue #8046 by implementing the fix from Axel for other similar functions in TDirectoryFile

Co-authored-by: Nicolas Morange <[email protected]>
Co-authored-by: Philippe Canal <[email protected]>

Patch proposed by Timur

Default-constructed RResultHandles might serve as placeholders in
certain cases, the same way that a default constructor for
RResultPtrs is sometimes useful e.g. as an array element.

This means that RResultHandle can now be null, so some sanity
checks are needed in its methods.

* Add RNN operator

* Removed files from PR

* Add missing default constructor

* Add tests for RNN

* Fix standard library headers handling

* Add missing header for RNN

* Small fixes

* Fix RNN with different sequence lengths

* Add documentation

This function allows users to route clang diagnostics to the ROOT error handler.
Closes issue ROOT-7587.

These changes enable the replacement of the default DiagnosticConsumer provided
by CIFactory.  Concretely, two member functions have been added to the
Interpreter class:

- replaceDiagnosticConsumer(): replaces the default CIFactory-provided DiagnosticConsumer.

- hasReplacedDiagnosticConsumer(): returns whether the default diagnostic
consumer has been replaced.

To have better diagnostics, `TRint::ProcessLineNr()` prepends a
`#line 1 "ROOT_prompt_xxx"` PP directive to each input line.  However, this
causes problems if the previous line is continued with '\', e.g.
```
root [0] #define m(x) printf("%s", \
root (cont'ed, cancel with .@) [1]x);
ROOT_prompt_0:2:2: error: '#' is not followed by a macro parameter
 ^
ROOT_prompt_0:3:1: error: use of undeclared identifier 'x'
x);
^
```
This patch fixes issue #8762.

Replace implementation of `InputValidator::validate()` by simpler, more
maintainable code that also fixes ROOT-9202.

The previous implementation was unable to properly handle line continuation
after ',' or '\'.  Specifically, it skipped over non-punctuation tokens,
entering continuation mode even if ',' or '\' were not the last tokens in the
input, e.g. `int a, b` or 'int a \      b'.

Fixes ROOT-9202.

…Line()`

This commit includes the following changes to MetaLexer:
- Update `MetaLexer::Lex()` to return `/*` (tok::l_comment), `*/` (tok::r_comment),
and `<` (tok::less) as tokens.

- Added `MetaLexer::ReadToEndOfLine()` function: consume input until `\r` or `\n`.

- Added `MetaLexer::RAII`, a RAII object that saves/restores the current position.

Co-authored-by: Axel Naumann <[email protected]>

This test needs a char** which is a bit tricky to create without stepping into compiler errors or warnings.
Currently it is created on the heap but improperly deallocated, so switch to a stack variable to avoid the need for memory management.

#8971)

* Make sure the pythonisation doc shows up at the end of the classes documentation

* Too many lines were commented out `makeinput.sh`

* Do changes only related to the python Frame placement

* Create the .pyzdoc at the right place.

* Update documentation/doxygen/extract_docstrings.py

add `_out`

Co-authored-by: Enric Tejedor <[email protected]>

* Add comment

Co-authored-by: Enric Tejedor <[email protected]>