Add option to include line number on <testcase> file path

[KieranP]

Use case: I am using CircleCI to run my Rspec tests. I want to split my tests at the individual test level, rather than at file level. To do this with Rspec, you can reference the line number when running tests, e.g. rspec spec/models/user_spec.rb:32. You can get a list of all specs available using:

rspec --order=defined --format=json --dry-run spec/**/*_spec.rb --out=specs.json
cat specs.json | jq ".examples.[].id"

This rspec junit formatter currently doesn't include line numbers in the results. So CircleCI is unable to do anything more granular than file level splitting. If we could toggle the file attribute of testcase results to include line numbers, then we could use something like this:

rspec --order=defined --format=json --dry-run spec/**/*_spec.rb --out=specs.json
cat specs.json | jq ".examples.[].id" | circleci tests run --command="xargs bundle exec rspec --format progress --format RspecJunitFormatter -o ~/rspec/rspec.xml" --verbose --split-by=timings

circleci tests run would get the file names with line numbers, match it against past results using the junit testcast file attribute, and be able to better split at the individual test level rather than just file.

Is this possible currently? If not, would this be considered?

[sj26]

This isn’t currently possible, sorry. This has come up a few times. Because it isn’t part of the junit spec - it breaks some consumers - I don’t think we should add it to the gem.

I’d advise using the json if you need more than junit provides. Perhaps there’s a way to use it with circleci? If not, there probably should be.

[KieranP]

@sj26 Ok, thank you. I will see what CircleCI thinks, but it is unlikely they will support a custom json format such as that which rspec outputs :-( So looks like we'll need to roll our own system to store the json output, and then analyze it on future runs.