Feature Request: User-specified Sections

[BrianPugh]

Motivation: BrianPugh/cyclopts#477

Basically, I don't want docstring_parser to handle "Examples" sections. I haven't looked to hard at the code base, so I'm not sure where the best place to change is. Ideally I'd want this available through the common parse function.

I'm willing to help out on a PR if someone (@mauvilsa ?) could give me pointers/recommendations.

[mauvilsa]

It would have been nicer if you explained here, instead of just adding links and have to go through two conversations to figure out what you want. Anyway, I went through them. And I don't understand the reason why docstring_parser should change. When you parse the docstring, the examples can be accessed through the examples attribute. You are free to use it however you see fit. Furthermore, docstring_parser is used in many other projects which can potentially break. Suggesting that examples should be handled differently is already a no go. Maybe you have in mind something that wouldn't break other projects. But since you haven't explained, then no way to know.

[BrianPugh]

Why the harsh tone?

Some parsers, like the GoogleParser and NumpydocParser, take in sections that specifies which sections to parse from the docstring. If not set, they default to their respective DEFAULT_SECTIONS. Some parsers, like epydoc and rest, do not have this feature (I'm also not familiar with these documentation styles). Despite GoogleParser and NumypdocParser both taking in sections, they actually use independent Section classes.

The main function, parse, does not provide a mechanism to specify sections, and docstring_parser as a whole does not provide a mechanism to identify the style of a supplied docstring. Between the two (and the separate Section classes), there's no real way of taking advantage of this feature.

It might be reasonable to add a new parameter sections: Sequence | None = None to compose that gets passed along to GoogleParser and NumpydocParser, but we would need to unify the specification a little bit between the two parsers. They both should probably just have sections in their constructors be a Sequence[Literal['Arguments', 'Parameters', ...]], and then we can make a mapping between the strings and their instantiated Section objects. This can be implemented in a backwards compatible way.

Because of the bad ergonomics of specifying a 10-or-20-long array of strings, it would also be good to add an exclude_sections option.

I opened this issue to get guidance on how we would want to go about implementing this, because there are many ways to go about it. I did not intend for the initial post to come across as lazy or lacking in information.

[mauvilsa]

Why the harsh tone?

Sorry about that. It was not my intention to sound harsh. It was just my honest opinion about the short initial description.

From your last post, I would say that you are more familiar with the code base than me. I can guide. But maybe more from the perspective of the feature making sense and being useful for a broad audience.

You say you want to add a sections parameter to parse(). And you already mention issues, like sections not in all parsers, which would make it a bit weird to have such a parameter. But it is still not clear to me why this specific change. As I mentioned before, the examples can be accessed through the examples attribute. Why isn't this useful for you? Or why can't that be improved so that it becomes more useful?

On the other hand, it is not clear to me what exactly this means:

I don't want docstring_parser to handle "Examples" sections

If a section is not handled, then what is the behavior? If I go to the google parser and comment-out the examples section, then the content of the examples seems to either be 1) just lost if it appears after some other section, or 2) be part of the long description if before any other section.

I think it would be good for you to explain what is the actual problem you want to solve, to then figure out what could be the simplest solution. Instead of immediately thinking about how to implement a sections parameter which already looks to be a complex feature.

[danielgafni]

@BrianPugh do you think we could revisit this one? I would love to see examples in Cyclopt's --help and generated Mkdocs output

It seems like - according to @mauvilsa - you should still be able to access the .examples attribute?

[BrianPugh]

I think I might be able to resolve the upstream cyclopts issue by using the Docstring.meta field. We can keep that discussion over in the Cyclopts repo. Regardless, the use of the sections parameter in some of these parsing classes is a bit odd given that they cannot be controlled unless the caller knows which type of docstring they are parsing.

[mauvilsa]

I still don't understand the underlying issue. If the objective is getting access to examples in the docstring, why is the solution in cyclopts using Docstring.meta and not use the examples attribute? Is it that the examples attribute is not working correctly or not useful in cyclopts case?

Also I still don't understand why the original proposal to solve the issue was docstring-parser not handling example sections. To me it makes more sense that docstring-parser always extracts everything, and whoever ran the extraction is free to use whatever they need.

[BrianPugh]

If the objective is getting access to examples in the docstring, why is the solution in cyclopts using Docstring.meta and not use the examples attribute?

Using Docstring.meta addresses 2 issues:

1. Other sections (e.g. a References section) do not have direct attributes in the returned Docstring object.
2. We need to preserver order of the sections. If a user has a References section, then an Examples, we want to preserve that when displaying it back to them.

Is it that the examples attribute is not working correctly or not useful in cyclopts case?

It's kind of working, but it's a bit unintuitive to me (or i might have some formatting wrong in the docstring). Probably not related to the present Github Issue, but presenting for completeness:

from docstring_parser import parse

docstring = """\
Short description.

Notes
-----
Some notes here [1].

References
----------
[1] A reference.

Examples
--------
To print the number 1:

>>> print(1)
1

To print the number 2:

>>> print(2)
2
"""

parsed = parse(docstring)

for i, item in enumerate(parsed.examples):
    print(f"  [{i}] description={item.description!r}     snippet={item.snippet!r}")


#  [0] description='To print the number 1:\n'     snippet=None
#  [1] description='1\n\nTo print the number 2:\n'     snippet='>>> print(1)'
#  [2] description='2'     snippet='>>> print(2)'

Also I still don't understand why the original proposal to solve the issue was docstring-parser not handling example sections.

The end goal in Cyclopts' use-case is to present all of the not-related-to-the-function-signature sections as if they were part of with parsed.long_description. Namely, this is including sections like Notes, References, Examples, etc. It seems like there's 2 ways to go about this:

1. Tell docstring_parser to not parse these sections so that they just end up as part of long_description. The signatures of NumpydocParser and GoogleParser suggests that this feature was intended at one point.
2. Reconstruct the description using the parsed.metas attribute. This has some additional niceties like applying different formatting for different section types.

I am currently pursuing (2).

[mauvilsa]

Thanks for the explanation. I think it is now clear for me. Option 2 does sound better to me. Not sure if there could be improvements in docstring-parser to make reconstruction easier.