Detection Algorithms

scenedetect.detectors Module

This module contains the following scene detection algorithms:

  • ContentDetector:

    Detects shot changes by considering pixel changes in the HSV colorspace.

  • ThresholdDetector:

    Detects transitions below a set pixel intensity (cuts or fades to black).

  • AdaptiveDetector:

    Two-pass version of ContentDetector that handles fast camera movement better in some cases.

Detection algorithms are created by implementing the SceneDetector interface. Detectors are typically attached to a SceneManager when processing videos, however they can also be used to process frames directly.

ContentDetector

ContentDetector compares the difference in content between adjacent frames against a set threshold/score, which if exceeded, triggers a scene cut.

This detector is available from the command-line as the detect-content command.

class scenedetect.detectors.content_detector.ContentDetector(threshold=27.0, min_scene_len=15, weights=Components(delta_hue=1.0, delta_sat=1.0, delta_lum=1.0, delta_edges=0.0), luma_only=False, kernel_size=None)

Detects fast cuts using changes in colour and intensity between frames.

Since the difference between frames is used, unlike the ThresholdDetector, only fast cuts are detected with this method. To detect slow fades between content scenes still using HSV information, use the DissolveDetector.

Parameters:

  • threshold (float) – Threshold the average change in pixel intensity must exceed to trigger a cut.

  • min_scene_len (int) – Once a cut is detected, this many frames must pass before a new one can be added to the scene list.

  • weights (ContentDetector.Components) – Weight to place on each component when calculating frame score (content_val in a statsfile, the value threshold is compared against).

  • luma_only (bool) – If True, only considers changes in the luminance channel of the video. Equivalent to specifying weights as ContentDetector.LUMA_ONLY. Overrides weights if both are set.

  • kernel_size (int | None) – Size of kernel for expanding detected edges. Must be odd integer greater than or equal to 3. If None, automatically set using video resolution.

class Components(delta_hue=1.0, delta_sat=1.0, delta_lum=1.0, delta_edges=0.0)

Components that make up a frame’s score, and their default values.

Create new instance of Components(delta_hue, delta_sat, delta_lum, delta_edges)

Parameters:

  • delta_hue (float) –

  • delta_sat (float) –

  • delta_lum (float) –

  • delta_edges (float) –

delta_edges: float

Difference between calculated edges of adjacent frames.

Edge differences are typically larger than the other components, so the detection threshold may need to be adjusted accordingly.

delta_hue: float

Difference between pixel hue values of adjacent frames.

delta_lum: float

Difference between pixel luma (brightness) values of adjacent frames.

delta_sat: float

Difference between pixel saturation values of adjacent frames.

DEFAULT_COMPONENT_WEIGHTS = Components(delta_hue=1.0, delta_sat=1.0, delta_lum=1.0, delta_edges=0.0)

Default component weights. Actual default values are specified in Components to allow adding new components without breaking existing usage.

FRAME_SCORE_KEY = 'content_val'

Key in statsfile representing the final frame score after weighed by specified components.

LUMA_ONLY_WEIGHTS = Components(delta_hue=0.0, delta_sat=0.0, delta_lum=1.0, delta_edges=0.0)

Component weights to use if luma_only is set.

METRIC_KEYS = ['content_val', 'delta_hue', 'delta_sat', 'delta_lum', 'delta_edges']

All statsfile keys this detector produces.

get_metrics()

Get Metrics: Get a list of all metric names/keys used by the detector.

Returns:

List of strings of frame metric key names that will be used by the detector when a StatsManager is passed to process_frame.

is_processing_required(frame_num)

[DEPRECATED] DO NOT USE

Test if all calculations for a given frame are already done.

Returns:

False if the SceneDetector has assigned _metric_keys, and the stats_manager property is set to a valid StatsManager object containing the required frame metrics/calculations for the given frame - thus, not needing the frame to perform scene detection.

True otherwise (i.e. the frame_img passed to process_frame is required to be passed to process_frame for the given frame_num).

process_frame(frame_num, frame_img)

Similar to ThresholdDetector, but using the HSV colour space DIFFERENCE instead of single-frame RGB/grayscale intensity (thus cannot detect slow fades with this method).

Parameters:

  • frame_num (int) – Frame number of frame that is being passed.

  • frame_img (ndarray) – Decoded frame image (numpy.ndarray) to perform scene detection on. Can be None only if the self.is_processing_required() method (inhereted from the base SceneDetector class) returns True.

Returns:

List of frames where scene cuts have been detected. There may be 0 or more frames in the list, and not necessarily the same as frame_num.

Return type:

List[int]

scenedetect.detectors.content_detector.estimated_kernel_size(frame_width, frame_height)

Estimate kernel size based on video resolution.

Parameters:

  • frame_width (int) –

  • frame_height (int) –

Return type:

int

scenedetect.detectors.content_detector.mean_pixel_distance(left, right)

Return the mean average distance in pixel values between left and right. Both left and `right should be 2 dimensional 8-bit images of the same shape.

Parameters:

  • left (ndarray) –

  • right (ndarray) –

Return type:

float

AdaptiveDetector

AdaptiveDetector compares the difference in content between adjacent frames similar to ContentDetector except the threshold isn’t fixed, but is a rolling average of adjacent frame changes. This can help mitigate false detections in situations such as fast camera motions.

This detector is available from the command-line as the detect-adaptive command.

class scenedetect.detectors.adaptive_detector.AdaptiveDetector(adaptive_threshold=3.0, min_scene_len=15, window_width=2, min_content_val=15.0, weights=Components(delta_hue=1.0, delta_sat=1.0, delta_lum=1.0, delta_edges=0.0), luma_only=False, kernel_size=None, video_manager=None, min_delta_hsv=None)

Two-pass detector that calculates frame scores with ContentDetector, and then applies a rolling average when processing the result that can help mitigate false detections in situations such as camera movement.

Parameters:

  • adaptive_threshold (float) – Threshold (float) that score ratio must exceed to trigger a new scene (see frame metric adaptive_ratio in stats file).

  • min_scene_len (int) – Minimum length of any scene.

  • window_width (int) – Size of window (number of frames) before and after each frame to average together in order to detect deviations from the mean. Must be at least 1.

  • min_content_val (float) – Minimum threshold (float) that the content_val must exceed in order to register as a new scene. This is calculated the same way that detect-content calculates frame score based on weights/luma_only/kernel_size.

  • weights (Components) – Weight to place on each component when calculating frame score (content_val in a statsfile, the value threshold is compared against). If omitted, the default ContentDetector weights are used.

  • luma_only (bool) – If True, only considers changes in the luminance channel of the video. Equivalent to specifying weights as ContentDetector.LUMA_ONLY. Overrides weights if both are set.

  • kernel_size (int | None) – Size of kernel to use for post edge detection filtering. If None, automatically set based on video resolution.

  • video_manager – [DEPRECATED] DO NOT USE. For backwards compatibility only.

  • min_delta_hsv (float | None) – [DEPRECATED] DO NOT USE. Use min_content_val instead.

ADAPTIVE_RATIO_KEY_TEMPLATE = 'adaptive_ratio{luma_only} (w={window_width})'
property event_buffer_length: int

Number of frames any detected cuts will be behind the current frame due to buffering.

get_content_val(frame_num)

Returns the average content change for a frame.

Parameters:

frame_num (int) –

Return type:

float | None

get_metrics()

Combines base ContentDetector metric keys with the AdaptiveDetector one.

Return type:

List[str]

post_process(_unused_frame_num)

Not required for AdaptiveDetector.

Parameters:

_unused_frame_num (int) –

process_frame(frame_num, frame_img)

Similar to ThresholdDetector, but using the HSV colour space DIFFERENCE instead of single-frame RGB/grayscale intensity (thus cannot detect slow fades with this method).

Parameters:

  • frame_num (int) – Frame number of frame that is being passed.

  • frame_img (ndarray | None) – Decoded frame image (numpy.ndarray) to perform scene detection on. Can be None only if the self.is_processing_required() method (inhereted from the base SceneDetector class) returns True.

Returns:

Empty list

Return type:

List[int]

stats_manager_required()

Not required for AdaptiveDetector.

Return type:

bool

ThresholdDetector

ThresholdDetector uses a set intensity as a threshold to detect cuts, which are triggered when the average pixel intensity exceeds or falls below this threshold.

This detector is available from the command-line as the detect-threshold command.

class scenedetect.detectors.threshold_detector.ThresholdDetector(threshold=12, min_scene_len=15, fade_bias=0.0, add_final_scene=False, block_size=None)

Detects fast cuts/slow fades in from and out to a given threshold level.

Detects both fast cuts and slow fades so long as an appropriate threshold is chosen (especially taking into account the minimum grey/black level).

Parameters:

  • threshold (float) – 8-bit intensity value that each pixel value (R, G, and B) must be <= to in order to trigger a fade in/out.

  • min_scene_len (int) – FrameTimecode object or integer greater than 0 of the minimum length, in frames, of a scene (or subsequent scene cut).

  • fade_bias (float) – Float between -1.0 and +1.0 representing the percentage of timecode skew for the start of a scene (-1.0 causing a cut at the fade-to-black, 0.0 in the middle, and +1.0 causing the cut to be right at the position where the threshold is passed).

  • add_final_scene (bool) – Boolean indicating if the video ends on a fade-out to generate an additional scene at this timecode.

  • block_size – [DEPRECATED] DO NOT USE. For backwards compatibility.

THRESHOLD_VALUE_KEY = 'delta_rgb'
get_metrics()

Get Metrics: Get a list of all metric names/keys used by the detector.

Returns:

List of strings of frame metric key names that will be used by the detector when a StatsManager is passed to process_frame.

Return type:

List[str]

post_process(frame_num)

Writes a final scene cut if the last detected fade was a fade-out.

Only writes the scene cut if add_final_scene is true, and the last fade that was detected was a fade-out. There is no bias applied to this cut (since there is no corresponding fade-in) so it will be located at the exact frame where the fade-out crossed the detection threshold.

Parameters:

frame_num (int) –

process_frame(frame_num, frame_img)
Parameters:

  • frame_num (int) – Frame number of frame that is being passed.

  • frame_img (numpy.ndarray or None) – Decoded frame image (numpy.ndarray) to perform scene detection with. Can be None only if the self.is_processing_required() method (inhereted from the base SceneDetector class) returns True.

Returns:

List of frames where scene cuts have been detected. There may be 0 or more frames in the list, and not necessarily the same as frame_num.

Return type:

List[int]

scenedetect.detectors.threshold_detector.compute_frame_average(frame)

Computes the average pixel value/intensity for all pixels in a frame.

The value is computed by adding up the 8-bit R, G, and B values for each pixel, and dividing by the number of pixels multiplied by 3.

Parameters:

frame (ndarray) – Frame representing the RGB pixels to average.

Returns:

Average pixel intensity across all 3 channels of frame

Return type:

float