Sorry, the docs are out-of-date here. Supporting both versions of cats was the initial plan, but the TextCategorizer only supports whole document cats at this point.

The idea of supporting subdocument cats could make sense for training with the current implementation, but an evaluation that uses this format gets very complicated: do you evaluate every possible span in an eval document? What kind of summary score would you provide for a whole document after evaluating every single span?

Your task is reasonable, but unfortunately not a perfect match for either the current TextCategorizer or EntityRecgonizer. The EntityRecognizer probably gets closer because it handles spans, but it's better at shorter proper name-like spans than longer text spans.

See #3961 for related discussion. Suggestions/help are very welcome!

Thank you for the reply and the link to the related discussion.

I'm new to spaCy, ML, NLP, etc and rather looking around and trying different things to figure out how it works and whether it is possible to solve the problem I have using spaCy and NLP. It is very possible that the approach I've chosen is not right.

What I'd like to achieve is to find ingredient sentences in the recipe and then in each sentence/span to find ingredient name, amount and unit of measurement. For ingredient name, amount and unit of measurement, I want to use NER to identify ingredient name, amount and unit of measurement and for ingredient sentences/lines I wanted to use sentence categorization.

If categorization doesn't work for sentences yet it seems like I can create two models. First will use NER to label ingredient sentences/spans and second will be used to identify ingredient name, amount and unit of measurement in these sentences. Please kindly advise if I'm on the right track or I have to choose another approach. Or maybe I've to use totally different tools for that :)