Skip to content

Fix Mask/Transitions for Global Timeline Effects #675

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
jonoomph merged 2 commits into from
May 20, 2021
Merged

Fix Mask/Transitions for Global Timeline Effects #675

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
441cb18
Refactor of global timeline effects, to address a regression with glo…
jonoomph May 18, 2021
a505f87
Refactoring timeline_frame_number out of apply_layer() and TimelineIn…
jonoomph May 20, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
36 changes: 33 additions & 3 deletions src/Clip.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -335,15 +335,36 @@ std::shared_ptr<Frame> Clip::GetFrame(int64_t frame_number)

// Get the original frame and pass it to GetFrame overload
std::shared_ptr<Frame> original_frame = GetOrCreateFrame(frame_number);
return GetFrame(original_frame, frame_number);
return GetFrame(original_frame, frame_number, NULL);
}
else
// Throw error if reader not initialized
throw ReaderClosed("No Reader has been initialized for this Clip. Call Reader(*reader) before calling this method.");
}

// Use an existing openshot::Frame object and draw this Clip's frame onto it
// Create an openshot::Frame object for a specific frame number of this reader.
std::shared_ptr<Frame> Clip::GetFrame(std::shared_ptr<openshot::Frame> background_frame, int64_t frame_number)
{
// Check for open reader (or throw exception)
if (!is_open)
throw ReaderClosed("The Clip is closed. Call Open() before calling this method.");

if (reader)
{
// Adjust out of bounds frame number
frame_number = adjust_frame_number_minimum(frame_number);

// Get the original frame and pass it to GetFrame overload
std::shared_ptr<Frame> original_frame = GetOrCreateFrame(frame_number);
return GetFrame(original_frame, frame_number, NULL);
}
else
// Throw error if reader not initialized
throw ReaderClosed("No Reader has been initialized for this Clip. Call Reader(*reader) before calling this method.");
}

// Use an existing openshot::Frame object and draw this Clip's frame onto it
std::shared_ptr<Frame> Clip::GetFrame(std::shared_ptr<openshot::Frame> background_frame, int64_t frame_number, openshot::TimelineInfoStruct* options)
{
// Check for open reader (or throw exception)
if (!is_open)
Expand All @@ -367,9 +388,18 @@ std::shared_ptr<Frame> Clip::GetFrame(std::shared_ptr<openshot::Frame> backgroun
// TODO: Handle variable # of samples, since this resamples audio for different speeds (only when time curve is set)
get_time_mapped_frame(original_frame, new_frame_number);

// Apply effects to the frame (if any)
// Apply local effects to the frame (if any)
apply_effects(original_frame);

// Apply global timeline effects (i.e. transitions & masks... if any)
if (timeline != NULL && options != NULL) {
if (options->is_top_clip) {
// Apply global timeline effects (only to top clip... if overlapping, pass in timeline frame number)
Timeline* timeline_instance = (Timeline*) timeline;
original_frame = timeline_instance->apply_effects(original_frame, background_frame->number, Layer());
}
}

// Apply keyframe / transforms
apply_keyframes(original_frame, background_frame->GetImage());

Expand Down
29 changes: 21 additions & 8 deletions src/Clip.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -164,7 +164,6 @@ namespace openshot {
void reverse_buffer(juce::AudioSampleBuffer* buffer);



public:
openshot::GravityType gravity; ///< The gravity of a clip determines where it snaps to its parent
openshot::ScaleType scale; ///< The scale determines how a clip should be resized to fit its parent
Expand Down Expand Up @@ -214,25 +213,39 @@ namespace openshot {
/// Look up an effect by ID
openshot::EffectBase* GetEffect(const std::string& id);

/// @brief Get an openshot::Frame object for a specific frame number of this timeline. The image size and number
/// @brief Get an openshot::Frame object for a specific frame number of this clip. The image size and number
/// of samples match the source reader.
///
/// @returns A new openshot::Frame object
/// @param frame_number The frame number (starting at 1) of the clip or effect on the timeline.
/// @param frame_number The frame number (starting at 1) of the clip
std::shared_ptr<openshot::Frame> GetFrame(int64_t frame_number) override;

/// @brief Get an openshot::Frame object for a specific frame number of this timeline. The image size and number
/// of samples can be customized to match the Timeline, or any custom output. Extra samples will be moved to the
/// next Frame. Missing samples will be moved from the next Frame.
/// @brief Get an openshot::Frame object for a specific frame number of this clip. The image size and number
/// of samples match the background_frame passed in and the timeline (if available).
///
/// A new openshot::Frame objects is returned, based on a copy from the source image, with all keyframes and clip effects
/// rendered.
/// rendered/rasterized.
///
/// @returns The modified openshot::Frame object
/// @param background_frame The frame object to use as a background canvas (i.e. an existing Timeline openshot::Frame instance)
/// @param frame_number The frame number (starting at 1) of the clip or effect on the timeline.
/// @param frame_number The frame number (starting at 1) of the clip. The image size and number
/// of samples match the background_frame passed in and the timeline (if available)
std::shared_ptr<openshot::Frame> GetFrame(std::shared_ptr<openshot::Frame> background_frame, int64_t frame_number);

/// @brief Get an openshot::Frame object for a specific frame number of this clip. The image size and number
/// of samples match the background_frame passed in and the timeline (if available).
///
/// A new openshot::Frame objects is returned, based on a copy from the source image, with all keyframes and clip effects
/// rendered/rasterized.
///
/// @returns The modified openshot::Frame object
/// @param background_frame The frame object to use as a background canvas (i.e. an existing Timeline openshot::Frame instance)
/// @param frame_number The frame number (starting at 1) of the clip on the timeline. The image size and number
/// of samples match the timeline.
/// @param options The openshot::TimelineInfoStruct pointer, with more details about this specific timeline clip,
/// such as, if it's a top clip. This info is used to apply global transitions and masks, if needed.
std::shared_ptr<openshot::Frame> GetFrame(std::shared_ptr<openshot::Frame> background_frame, int64_t frame_number, openshot::TimelineInfoStruct* options);

/// Open the internal reader
void Open() override;

Expand Down
29 changes: 14 additions & 15 deletions src/Timeline.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -444,7 +444,7 @@ std::shared_ptr<Frame> Timeline::apply_effects(std::shared_ptr<Frame> frame, int
}

// Get or generate a blank frame
std::shared_ptr<Frame> Timeline::GetOrCreateFrame(std::shared_ptr<Frame> background_frame, Clip* clip, int64_t number)
std::shared_ptr<Frame> Timeline::GetOrCreateFrame(std::shared_ptr<Frame> background_frame, Clip* clip, int64_t number, openshot::TimelineInfoStruct* options)
{
std::shared_ptr<Frame> new_frame;

Expand All @@ -456,7 +456,7 @@ std::shared_ptr<Frame> Timeline::GetOrCreateFrame(std::shared_ptr<Frame> backgro
ZmqLogger::Instance()->AppendDebugMethod("Timeline::GetOrCreateFrame (from reader)", "number", number, "samples_in_frame", samples_in_frame);

// Attempt to get a frame (but this could fail if a reader has just been closed)
new_frame = std::shared_ptr<Frame>(clip->GetFrame(background_frame, number));
new_frame = std::shared_ptr<Frame>(clip->GetFrame(background_frame, number, options));

// Return real frame
return new_frame;
Expand All @@ -475,29 +475,28 @@ std::shared_ptr<Frame> Timeline::GetOrCreateFrame(std::shared_ptr<Frame> backgro
}

// Process a new layer of video or audio
void Timeline::add_layer(std::shared_ptr<Frame> new_frame, Clip* source_clip, int64_t clip_frame_number, int64_t timeline_frame_number, bool is_top_clip, float max_volume)
void Timeline::add_layer(std::shared_ptr<Frame> new_frame, Clip* source_clip, int64_t clip_frame_number, bool is_top_clip, float max_volume)
{
// Get the clip's frame, composited on top of the current timeline frame
// Create timeline options (with details about this current frame request)
TimelineInfoStruct* options = new TimelineInfoStruct();
options->is_top_clip = is_top_clip;

// Get the clip's frame, composited on top of the current timeline frame
std::shared_ptr<Frame> source_frame;
source_frame = GetOrCreateFrame(new_frame, source_clip, clip_frame_number);
source_frame = GetOrCreateFrame(new_frame, source_clip, clip_frame_number, options);
delete options;

// No frame found... so bail
if (!source_frame)
return;

// Debug output
ZmqLogger::Instance()->AppendDebugMethod("Timeline::add_layer", "new_frame->number", new_frame->number, "clip_frame_number", clip_frame_number, "timeline_frame_number", timeline_frame_number);

/* Apply effects to the source frame (if any). If multiple clips are overlapping, only process the
* effects on the top clip. */
if (is_top_clip) {
source_frame = apply_effects(source_frame, timeline_frame_number, source_clip->Layer());
}
ZmqLogger::Instance()->AppendDebugMethod("Timeline::add_layer", "new_frame->number", new_frame->number, "clip_frame_number", clip_frame_number);

/* COPY AUDIO - with correct volume */
if (source_clip->Reader()->info.has_audio) {
// Debug output
ZmqLogger::Instance()->AppendDebugMethod("Timeline::add_layer (Copy Audio)", "source_clip->Reader()->info.has_audio", source_clip->Reader()->info.has_audio, "source_frame->GetAudioChannelsCount()", source_frame->GetAudioChannelsCount(), "info.channels", info.channels, "clip_frame_number", clip_frame_number, "timeline_frame_number", timeline_frame_number);
ZmqLogger::Instance()->AppendDebugMethod("Timeline::add_layer (Copy Audio)", "source_clip->Reader()->info.has_audio", source_clip->Reader()->info.has_audio, "source_frame->GetAudioChannelsCount()", source_frame->GetAudioChannelsCount(), "info.channels", info.channels, "clip_frame_number", clip_frame_number);

if (source_frame->GetAudioChannelsCount() == info.channels && source_clip->has_audio.GetInt(clip_frame_number) != 0)
for (int channel = 0; channel < source_frame->GetAudioChannelsCount(); channel++)
Expand Down Expand Up @@ -550,7 +549,7 @@ void Timeline::add_layer(std::shared_ptr<Frame> new_frame, Clip* source_clip, in
}
else
// Debug output
ZmqLogger::Instance()->AppendDebugMethod("Timeline::add_layer (No Audio Copied - Wrong # of Channels)", "source_clip->Reader()->info.has_audio", source_clip->Reader()->info.has_audio, "source_frame->GetAudioChannelsCount()", source_frame->GetAudioChannelsCount(), "info.channels", info.channels, "clip_frame_number", clip_frame_number, "timeline_frame_number", timeline_frame_number);
ZmqLogger::Instance()->AppendDebugMethod("Timeline::add_layer (No Audio Copied - Wrong # of Channels)", "source_clip->Reader()->info.has_audio", source_clip->Reader()->info.has_audio, "source_frame->GetAudioChannelsCount()", source_frame->GetAudioChannelsCount(), "info.channels", info.channels, "clip_frame_number", clip_frame_number);
}

// Debug output
Expand Down Expand Up @@ -753,7 +752,7 @@ std::shared_ptr<Frame> Timeline::GetFrame(int64_t requested_frame)
ZmqLogger::Instance()->AppendDebugMethod("Timeline::GetFrame (Calculate clip's frame #)", "clip->Position()", clip->Position(), "clip->Start()", clip->Start(), "info.fps.ToFloat()", info.fps.ToFloat(), "clip_frame_number", clip_frame_number);

// Add clip's frame as layer
add_layer(new_frame, clip, clip_frame_number, requested_frame, is_top_clip, max_volume);
add_layer(new_frame, clip, clip_frame_number, is_top_clip, max_volume);

} else {
// Debug output
Expand Down
10 changes: 5 additions & 5 deletions src/Timeline.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -175,7 +175,7 @@ namespace openshot {
int max_concurrent_frames; ///< Max concurrent frames to process at one time

/// Process a new layer of video or audio
void add_layer(std::shared_ptr<openshot::Frame> new_frame, openshot::Clip* source_clip, int64_t clip_frame_number, int64_t timeline_frame_number, bool is_top_clip, float max_volume);
void add_layer(std::shared_ptr<openshot::Frame> new_frame, openshot::Clip* source_clip, int64_t clip_frame_number, bool is_top_clip, float max_volume);

/// Apply a FrameMapper to a clip which matches the settings of this timeline
void apply_mapper_to_clip(openshot::Clip* clip);
Expand All @@ -198,10 +198,7 @@ namespace openshot {
std::vector<openshot::Clip*> find_intersecting_clips(int64_t requested_frame, int number_of_frames, bool include);

/// Get a clip's frame or generate a blank frame
std::shared_ptr<openshot::Frame> GetOrCreateFrame(std::shared_ptr<Frame> background_frame, openshot::Clip* clip, int64_t number);

/// Apply effects to the source frame (if any)
std::shared_ptr<openshot::Frame> apply_effects(std::shared_ptr<openshot::Frame> frame, int64_t timeline_frame_number, int layer);
std::shared_ptr<openshot::Frame> GetOrCreateFrame(std::shared_ptr<Frame> background_frame, openshot::Clip* clip, int64_t number, openshot::TimelineInfoStruct* options);

/// Compare 2 floating point numbers for equality
bool isEqual(double a, double b);
Expand Down Expand Up @@ -249,6 +246,9 @@ namespace openshot {
/// @param effect Add an effect to the timeline. An effect can modify the audio or video of an openshot::Frame.
void AddEffect(openshot::EffectBase* effect);

/// Apply global/timeline effects to the source frame (if any)
std::shared_ptr<openshot::Frame> apply_effects(std::shared_ptr<openshot::Frame> frame, int64_t timeline_frame_number, int layer);

/// Apply the timeline's framerate and samplerate to all clips
void ApplyMapperToClips();

Expand Down
15 changes: 15 additions & 0 deletions src/TimelineBase.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,8 +31,23 @@
#ifndef OPENSHOT_TIMELINE_BASE_H
#define OPENSHOT_TIMELINE_BASE_H

#include <cstdint>


namespace openshot {
/**
* @brief This struct contains info about the current Timeline clip instance
*
* When the Timeline requests an openshot::Frame instance from a Clip, it passes
* this struct along, with some additional details from the Timeline, such as if this clip is
* above or below overlapping clips, etc... This info can help determine if a Clip should apply
* global effects from the Timeline, such as a global Transition/Mask effect.
*/
struct TimelineInfoStruct
{
bool is_top_clip; ///< Is clip on top (if overlapping another clip)
};

/**
* @brief This class represents a timeline (used for building generic timeline implementations)
*/
Expand Down