src.clean package

Submodules

src.clean.clean_utils module

Utility classes, functions, and variables used in the data cleaning process

class src.clean.clean_utils.HidePrints

Bases: object

Helper class that prevents a function from printing to stdout when used as a context manager

class src.clean.clean_utils.ItemMaker(item: dict, **kwargs)

Bases: object

Makes a single item in the corpus by downloading from YouTube, splitting audio channels, and separating

_download_audio_excerpt_from_youtube() → None

Downloads an item in the corpus from a YouTube link

_get_valid_links(bad_pattern: str = '"playabilityStatus":{"status":"ERROR"') → list

Returns a list of valid YouTube links from the Corpus JSON

_logger_wrapper(msg) → None

Simple wrapper that logs a given message and indexes it for later access

_split_left_right_audio_channels(timeout: int = 10) → None

Splits audio into left and right channels for independent processing

abs_tol = 0.05

finalize_output(include_log: bool = False) → None

Finalizes the output by cleaning up leftover files and setting any final attributes

fmt = 'wav'

get_item() → None

Tries to find a corpus item locally, and downloads it from the internet if not present

instrs = ['piano', 'bass', 'drums']

separate_audio() → None

Checks whether to separate audio, gets requred commands, opens subprocessses, then cleans up

ydl_opts = {'extract_audio': True, 'format': 'wav/bestaudio[ext=wav]/best', 'logger': <class 'src.clean.clean_utils.YtDlpFakeLogger'>, 'overwrites': True, 'postprocessors': [{'key': 'FFmpegExtractAudio', 'preferredcodec': 'wav'}], 'quiet': True}

class src.clean.clean_utils.YtDlpFakeLogger

Bases: object

Fake logging class passed to yt-dlp instances to disable overly-verbose logging and unnecessary warnings

debug(msg=None)

error(msg=None)

warning(msg=None)

class src.clean.clean_utils._DemucsMaker(**kwargs)

Bases: ItemMaker

Internal class called in ItemMaker for separating audio using Facebook’s HTDemucs model

_download_audio_excerpt_from_youtube() → None

Downloads an item in the corpus from a YouTube link

_get_valid_links(bad_pattern: str = '"playabilityStatus":{"status":"ERROR"') → list

Returns a list of valid YouTube links from the Corpus JSON

_logger_wrapper(msg) → None

Simple wrapper that logs a given message and indexes it for later access

_split_left_right_audio_channels(timeout: int = 10) → None

Splits audio into left and right channels for independent processing

abs_tol = 0.05

cleanup_post_separation() → None

Cleans up after demucs by removing unnecessary files and moving file locations

finalize_output(include_log: bool = False) → None

Finalizes the output by cleaning up leftover files and setting any final attributes

fmt = 'wav'

fname: str

get_cmd(in_file: str | None = None) → list

Gets the required command for running demucs as a subprocess

get_item() → None

Tries to find a corpus item locally, and downloads it from the internet if not present

get_lr_audio: bool

in_file: str

instrs = ['piano', 'bass', 'drums']

run_separation(cmd) → None

Conducts separation in Demucs by opening a new subprocess

separate_audio() → None

Checks whether to separate audio, gets requred commands, opens subprocessses, then cleans up

use_demucs: bool

use_mvsep: bool

use_spleeter: bool

ydl_opts = {'extract_audio': True, 'format': 'wav/bestaudio[ext=wav]/best', 'logger': <class 'src.clean.clean_utils.YtDlpFakeLogger'>, 'overwrites': True, 'postprocessors': [{'key': 'FFmpegExtractAudio', 'preferredcodec': 'wav'}], 'quiet': True}

class src.clean.clean_utils._MVSEPMaker(**kwargs)

Bases: ItemMaker

LOAD_KWS = {'dtype': <class 'numpy.float64'>, 'offset': 0, 'res_type': 'soxr_vhq', 'sr': 44100}

MVSEP_KWARGS = {'chunk_size': 500000, 'cpu': False, 'large_gpu': False, 'only_vocals': False, 'overlap_large': 0.4, 'overlap_small': 0.3, 'single_onnx': False, 'use_kim_model_1': False}

MVSEP_LOCATION = 'c:\\python projects\\jazz-corpus-analysis/MVSEP-MDX23-music-separation-model'

_download_audio_excerpt_from_youtube() → None

Downloads an item in the corpus from a YouTube link

_get_valid_links(bad_pattern: str = '"playabilityStatus":{"status":"ERROR"') → list

Returns a list of valid YouTube links from the Corpus JSON

_logger_wrapper(msg) → None

Simple wrapper that logs a given message and indexes it for later access

_split_left_right_audio_channels(timeout: int = 10) → None

Splits audio into left and right channels for independent processing

abs_tol = 0.05

align_audio_signals(files_to_pad: list[str])

For a list of demixes stems files_to_pad, align these with their raw audio file

static calculate_best_shift(raw_env: array, proc_env: array, start: float = 5.0, duration: float = 10.0, max_shift: float = 2.0, step: int = 10) → int

Calculate the number of samples required to shift proc_env to maximise r vs proc_env

cleanup_post_separation(new_dirpath: str | None = None) → None

Cleans up after running MVSEP by renaming files and removing any unnecessary files

finalize_output(include_log: bool = False) → None

Finalizes the output by cleaning up leftover files and setting any final attributes

fmt = 'wav'

fname: str

get_cmd(in_file: str | None = None) → list

Gets the required command for running MVSEP as a subprocess

get_files_to_keep(root_fpath: str)

Returns a generator of files to keep from the separation

get_item() → None

Tries to find a corpus item locally, and downloads it from the internet if not present

get_lr_audio: bool

in_file: str

instrs = ['piano', 'bass', 'drums']

rename_piano_file(root_fpath: str) → None

Tries to rename the other file to piano, if it can be found

run_separation(cmd: list) → None

Runs separation in MVSEP

separate_audio() → None

Checks whether to separate audio, gets requred commands, opens subprocessses, then cleans up

static shift_audio_signal(audio: array, n_samples: int, shape_to_match: int) → array

Shift a signal audio by samples n_samples and pad to the same shape as shape_to_match

use_demucs: bool

use_mvsep: bool

use_spleeter: bool

ydl_opts = {'extract_audio': True, 'format': 'wav/bestaudio[ext=wav]/best', 'logger': <class 'src.clean.clean_utils.YtDlpFakeLogger'>, 'overwrites': True, 'postprocessors': [{'key': 'FFmpegExtractAudio', 'preferredcodec': 'wav'}], 'quiet': True}

class src.clean.clean_utils._SpleeterMaker(**kwargs)

Bases: ItemMaker

Internal class called in ItemMaker for separating audio using Deezer’s Spleeter model

_download_audio_excerpt_from_youtube() → None

Downloads an item in the corpus from a YouTube link

_get_valid_links(bad_pattern: str = '"playabilityStatus":{"status":"ERROR"') → list

Returns a list of valid YouTube links from the Corpus JSON

_logger_wrapper(msg) → None

Simple wrapper that logs a given message and indexes it for later access

_split_left_right_audio_channels(timeout: int = 10) → None

Splits audio into left and right channels for independent processing

abs_tol = 0.05

cleanup_post_separation() → None

Cleans up after spleeter by removing unnecessary files and unused files

finalize_output(include_log: bool = False) → None

Finalizes the output by cleaning up leftover files and setting any final attributes

fmt = 'wav'

fname: str

get_cmd(in_file: str | None = None) → list

Gets the required command for running Spleeter as a subprocess

get_item() → None

Tries to find a corpus item locally, and downloads it from the internet if not present

get_lr_audio: bool

in_file: str

instrs = ['piano', 'bass', 'drums']

run_separation(cmd: list, good_pattern: str = 'written succesfully') → None

Conducts separation in Spleeter by opening a new subprocess

separate_audio() → None

Checks whether to separate audio, gets requred commands, opens subprocessses, then cleans up

use_demucs: bool

use_mvsep: bool

use_spleeter: bool

ydl_opts = {'extract_audio': True, 'format': 'wav/bestaudio[ext=wav]/best', 'logger': <class 'src.clean.clean_utils.YtDlpFakeLogger'>, 'overwrites': True, 'postprocessors': [{'key': 'FFmpegExtractAudio', 'preferredcodec': 'wav'}], 'quiet': True}

src.clean.clean_utils.return_timestamp(timestamp: str = 'start') → int

Returns a formatted timestamp from a JSON element

src.clean.make_dataset module

Generates the final dataset of separated recordings from the items in the corpus

Module contents