[camera_avfoundation] handle interruptions and use single offset

[misos1]

Handle interruptions of capture sessions. Stop receiving and writing video and audio samples during such interruptions to avoid synchronization issues between video and audio.

Handle asynchronous error notifications posted about capture sessions. This can happen for example when there is an incoming phone call and then the camera starts recording with audio. The camera controller enters an error state and can be recovered by disposing and recreating.

Use single offset for both video and audio as separate offsets can cause unsynchronized video with audio. Seems audio timestamps tend to be ignored except for the first sample (probably by the video encoder, maybe it is inherent to the video format used) so prefer audio samples for calculating offset to avoid desynchronization. Avoid errors when trying to append samples with timestamp less or equal to the previous sample by not appending them. Fix a bug when after adjusting offset the last sample time was not updated which could cause adding past offsets multiple times (this could happen for example if videoIsDisconnected is set to true too early after it was previously set to true).

• fixes [camera_avfoundation] Separate time offsets for video and audio can cause shifted audio flutter#149978
• fixes [camera]When is recording interrupted by alarm or phone call video is not synchronized with audio flutter#151253

Pre-Review Checklist

• I read the Contributor Guide and followed the process outlined there for submitting PRs.
• I read the Tree Hygiene page, which explains my responsibilities.
• I read and followed the relevant style guides and ran the auto-formatter.
• I signed the CLA.
• The title of the PR starts with the name of the package surrounded by square brackets, e.g. [shared_preferences]
• I linked to at least one issue that this PR fixes in the description above.
• I updated pubspec.yaml with an appropriate new version according to the pub versioning philosophy, or I have commented below to indicate which version change exemption this PR falls under¹.
• I updated CHANGELOG.md to add a description of the change, following repository CHANGELOG style, or I have commented below to indicate which CHANGELOG exemption this PR falls under¹.
• I updated/added any relevant documentation (doc comments with ///).
• I added new tests to check the change I am making, or I have commented below to indicate which test exemption this PR falls under¹.
• All existing and new tests are passing.

If you need help, consider asking for advice on the #hackers-new channel on Discord.

Footnotes

1. Regular contributors who have demonstrated familiarity with the repository guidelines only need to comment if the PR is not auto-exempted by repo tooling. ↩ ↩² ↩³

[misos1]

Mac_arm64 ios_platform_tests_shard_2 master
Testing failed:
	Type 'AVCaptureSession' has no member 'wasInterruptedNotification'
	Command SwiftCompile failed with a nonzero exit code
	Testing cancelled because the build failed.

I am not sure what this should mean. AVCaptureSession.wasInterruptedNotification should be there since ios 4 (I managed to run tests successfully locally on real device and also on simulator).

[hellohuanlin]

nit: can infer .zero
nit2: split into multiple lines

[hellohuanlin]

... to fix a bug where audio and video are out of sync after interrupted by phone cals.

[hellohuanlin]

when will run time error happen?

[misos1]

As is in the description "for example when there is an incoming phone call and the camera starts recording with audio". When startVideoRecording is called with enabled audio while there is an incoming call (does not need to be answered in my case, ringing is enough). Audio session then stops running so no more audio can be recorded after this happens until the session is started again or FLTCam recreated (seems the only possibility to recover from the dart side for now). Seems no reasonable detection of this state is possible from the dart side for now (sending error to dart side on this error notification enables such detection).

[hellohuanlin]

can you put some comments there?

[hellohuanlin]

this is moved to top right?

[misos1]

Yes.

[hellohuanlin]

is lastAppendedVideoSampleTime the same as lastSampleEndTime?

Can you add some doc on these 4 properties?

[misos1]

lastAppendedVideoSampleTime is related to this text from the description "Avoid errors when trying to append samples with timestamp less or equal to the previous sample.". appendPixelBuffer throws an error if a sample with specified sample time (PresentationTime) is appended after another sample with higher or equal time was already appended before (and maybe this should be end time rather than start time too).

[hellohuanlin]

am i reading it right that

(1) recordingTimeOffset covers videoTimeOffset and audioTimeOffset and

(2) lastSampleEndTime covers lastVideoSampleTime and lastAudioSampleTime?

[misos1]

Yes, lastAudioSampleTime was actually the end time of a sample so I named it accordingly. And lastVideoSampleTime was practically end time too, according to logic for offset calculation both before and after this PR, it just happens that video samples tend to not have duration which can be interpreted as zero duration in this case so end time is same as start time. For that logic to work right this needs to be end time of sample, not start time.

[hellohuanlin]

we didn't have this output. can this be just a local variable?

[misos1]

That output is either audio or video output depending on whether audio or video output should be used for offset adjusting.

[hellohuanlin]

is lastAppendedVideoSampleTime the same as lastSampleEndTime?

Can you add some doc on these 4 properties?

[stuartmorgan-g]

From triage: what's the status of this PR? It's not clear to me if it's waiting for updates, or waiting for re-review.

[misos1]

It is waiting for updates.

[misos1]

Test testImageStreamEventFormat is failing locally with XCTAssertEqual(camera.isStreamingImages, true) XCTAssertEqual failed: ("false") is not equal to ("true"). Seems waitForQueueRoundTrip is "waiting" for the wrong queue as isStreamingImages is changed on captureSessionQueue. But even with the right queue it cannot wait on something which was not already dispatched at the time waitForQueueRoundTrip was called, and the block where that variable changes may not be. There should be a better way to wait for some variable to change in xc test.

[hellohuanlin]

can you add comment on what "adjusting of recording offset" means and why we need to adjust the offset?

[misos1]

It should have been "adjusting of recording time offset" -> recordingTimeOffset which as mentioned is sum of all pauses in recording so adjusting it of course means that if there is another pause then add duration of that pause into that offset. So after that offset is applied then we get continuous recording.

[hellohuanlin]

can you add comment on why using the audio output as the outputForOffsetAdjusting

[misos1]

This is because seems that audio samples (except the first) cannot be really moved in time meaning that audioWriterInput.append accepts sample with presentation time but seems ignores that time (except for the first) and just puts next sample at time of end of previous sample so they are always continuous. I suspect that is because these audio samples are stored in a video file in a single big chunk with just single presentation time. So if audio times cannot really be adjusted then video times need to be flexed according to audio times. I cannot be really sure that this is how it works on every platform and device so in case individual audio samples can be adjusted somewhere then it should be irrelevant whether audio or video is used for such adjusting. Thus audio output is preferred for adjusting.

[hellohuanlin]

i may need some help understanding this change. Can you walk me thought this change?

[misos1]

That check was moved elsewhere (line 308).

[hellohuanlin]

@misos1 are you still working on updating this PR? Thanks

[misos1]

@hellohuanlin yes

[hellohuanlin]

FYI this likely conflict with the ongoing swift migration, and you may have to migrate this PR too.

[hellohuanlin]

@misos1 from triage: thank you soooooooo much for working on this. we are making this a draft. Please re-open when it's ready for another review.

[misos1]

we are making this a draft. Please re-open when it's ready for another review.

Funny, I have been resolving conflicts of this for almost a year now.

[gemini-code-assist]

Code Review

This pull request introduces significant improvements to the camera_avfoundation package for handling capture session interruptions and synchronizing audio/video timestamps. It refactors the timestamp offset logic to use a single offset, preventing desynchronization issues after pauses or interruptions. It also adds handling for asynchronous error notifications from the capture session. The changes are well-tested with new unit tests covering the new offset logic and interruption handling. Additionally, a fix is included in the camera package to correctly handle camera error events at the controller level.

My review identifies a potential memory leak due to un-removed notification observers and suggests an improvement for error message formatting.

[gemini-code-assist]

Using String(describing:) on an optional error will include "Optional(...)" in the error message, which is not user-friendly. It's better to unwrap the error and use its localizedDescription, providing a fallback message if the error is nil.

Suggested change

	reportErrorMessage(
	"\(String(describing: notification.userInfo?[AVCaptureSessionErrorKey] as? Error))")
	let errorMessage = (notification.userInfo?[AVCaptureSessionErrorKey] as? Error)?.localizedDescription ?? "An unknown capture session runtime error occurred."
	reportErrorMessage(errorMessage)

[misos1]

@hellohuanlin I saw that "Optional(...)" string but I do not care a lot. To me a lot of error messages here look not very user-friendly. To me these errors from flutter or flutter plugins are more like debug messages for me. I would rather display something else to the actual user in a lot of cases. So should I proceed?

[hellohuanlin]

Gemini probably assumed this message is displayed to real users. But we actual send it to developers, so unlocalized message should be fine.

[stuartmorgan-g]

@LouiseHsu Is this on your radar for secondary review?

[LouiseHsu]

LGTM! Thank you so much for spending the time to resolve all those conflicts

[stuartmorgan-g]

It looks like this has the necessary approvals; let's get it landed before anything else can conflict with it :)