maestral.sync

This module contains the main syncing functionality.

class maestral.sync.Conflict(value)[source]

Bases: enum.Enum

Enumeration of sync conflict types

RemoteNewer = 'remote newer'
Conflict = 'conflict'
Identical = 'identical'
LocalNewerOrIdentical = 'local newer or identical'
class maestral.sync.SyncDirection(value)[source]

Bases: enum.Enum

Enumeration of sync directions

Up = 'up'
Down = 'down'
class maestral.sync.FSEventHandler(file_event_types=('created', 'deleted', 'modified', 'moved'), dir_event_types=('created', 'deleted', 'moved'))[source]

Bases: watchdog.events.FileSystemEventHandler

A local file event handler

Handles captured file events and adds them to SyncEngine’s file event queue to be uploaded by upload_worker(). This acts as a translation layer between watchdog.Observer and SyncEngine.

White lists of event types to handle are supplied as file_event_types and dir_event_types. This is for forward compatibility as additional event types may be added to watchdog in the future.

Parameters
  • file_event_types (Tuple[str, ...]) – Types of file events to handle. This acts as a whitelist.

  • dir_event_types (Tuple[str, ...]) – Types of folder events to handle. This acts as a whitelist.

Variables

ignore_timeout (float) – Timeout in seconds after which filters for ignored events will expire.

Return type

None

local_file_event_queue: Queue[FileSystemEvent]
property enabled: bool

Whether queuing of events is enabled.

enable()[source]

Turn on queueing of events.

Return type

None

disable()[source]

Turn off queueing of new events and remove all events from queue.

Return type

None

ignore(*events, recursive=True)[source]

A context manager to ignore local file events

Once a matching event has been registered, further matching events will no longer be ignored unless recursive is True. If no matching event has occurred before leaving the context, the event will be ignored for ignore_timeout sec after leaving then context and then discarded. This accounts for possible delays in the emission of local file system events.

This context manager is used to filter out file system events caused by maestral itself, for instance during a download or when moving a conflict.

Example

Prevent triggereing a sync event when creating a local file:

>>> from watchdog.events import FileCreatedEvent
>>> from maestral.main import Maestral
>>> m = Maestral()
>>> with m.sync.fs_events.ignore(FileCreatedEvent('path')):
...     open('path').close()
Parameters
  • events (watchdog.events.FileSystemEvent) – Local events to ignore.

  • recursive (bool) – If True, all child events of a directory event will be ignored as well. This parameter will be ignored for file events.

Return type

Iterator[None]

expire_ignored_events()[source]

Removes all expired ignore entries.

Return type

None

on_any_event(event)[source]

Checks if the system file event should be ignored. If not, adds it to the queue for events to upload. If syncing is paused or stopped, all events will be ignored.

Parameters

event (watchdog.events.FileSystemEvent) – Watchdog file event.

Return type

None

queue_event(event)[source]

Queues an individual file system event. Notifies / wakes up all threads that are waiting with wait_for_event().

Parameters

event (watchdog.events.FileSystemEvent) – File system event to queue.

Return type

None

wait_for_event(timeout=40)[source]

Blocks until an event is available in the queue or a timeout occurs, whichever comes first. You can use with method to wait for file system events in another thread.

Note

If there are multiple threads waiting for events, all of them will be notified. If one of those threads starts getting events from local_file_event_queue, other threads may find that the queue is empty despite being woken. You should therefore be prepared to handle an empty queue even if this method returns True.

Parameters

timeout (float) – Maximum time to block in seconds.

Returns

True if an event is available, False if the call returns due to a timeout.

Return type

bool

class maestral.sync.SyncEngine(client)[source]

Bases: object

Class that handles syncing with Dropbox

Provides methods to wait for local or remote changes and sync them, including conflict resolution and updates to our index.

Parameters

client (maestral.client.DropboxClient) – Dropbox API client instance.

sync_errors: Set[maestral.errors.SyncError]
syncing: Dict[str, maestral.database.SyncEvent]
reload_cached_config()[source]

Reloads all config and state values that are otherwise cached by this class for faster access. Call this method if config or state values where modified directly instead of using SyncEngine APIs.

Return type

None

property dropbox_path: str

Path to local Dropbox folder, as loaded from the config file. Before changing dropbox_path, make sure that syncing is paused. Move the dropbox folder to the new location before resuming the sync. Changes are saved to the config file.

property database_path: str

Path SQLite database.

property file_cache_path: str

Path to cache folder for temporary files (read only). The cache folder ‘.maestral.cache’ is located inside the local Dropbox folder to prevent file transfer between different partitions or drives during sync.

property excluded_items: List[str]

List of all files and folders excluded from sync. Changes are saved to the config file. If a parent folder is excluded, its children will automatically be removed from the list. If only children are given but not the parent folder, any new items added to the parent will be synced. Change this property before downloading newly included items or deleting excluded items.

static clean_excluded_items_list(folder_list)[source]

Removes all duplicates and children of excluded items from the excluded items list.

Parameters

folder_list (List[str]) – Dropbox paths to exclude.

Returns

Cleaned up items.

Return type

List[str]

property max_cpu_percent: float

Maximum CPU usage for parallel downloads or uploads in percent of the total available CPU time per core. Individual workers in a thread pool will pause until the usage drops below this value. Tasks in the main thread such as indexing file changes may still use more CPU time. Setting this to 200% means that two full logical CPU core can be used.

property remote_cursor: str

Cursor from last sync with remote Dropbox. The value is updated and saved to the config file on every successful download of remote changes.

property local_cursor: float

Time stamp from last sync with remote Dropbox. The value is updated and saved to the config file on every successful upload of local changes.

property last_change: float

The time stamp of the last file change or 0.0 if there are no file changes in our history.

property last_reindex: float

Time stamp of last full indexing. This is used to determine when the next full indexing should take place.

property history: List[maestral.database.SyncEvent]

A list of the last SyncEvents in our history. History will be kept for the interval specified by the config value keep_history (defaults to two weeks) but at most 1,000 events will be kept.

clear_sync_history()[source]

Clears the sync history.

Return type

None

reset_sync_state()[source]

Resets all saved sync state. Settings are not affected.

Return type

None

get_index()[source]

Returns a copy of the local index of synced files and folders.

Returns

List of index entries.

Return type

List[maestral.database.IndexEntry]

iter_index()[source]

Returns an iterator over the local index of synced files and folders.

Returns

Iterator over index entries.

Return type

Iterator[maestral.database.IndexEntry]

index_count()[source]

Returns the number if items in our index without loading any items.

Returns

Number of index entries.

Return type

int

get_local_rev(dbx_path_lower)[source]

Gets revision number of local file.

Parameters

dbx_path_lower (str) – Normalized lower case Dropbox path.

Returns

Revision number as str or None if no local revision number has been saved.

Return type

Optional[str]

get_last_sync(dbx_path_lower)[source]

Returns the timestamp of last sync for an individual path.

Parameters

dbx_path_lower (str) – Normalized lower case Dropbox path.

Returns

Time of last sync.

Return type

float

get_index_entry(dbx_path_lower)[source]

Gets the index entry for the given Dropbox path.

Parameters

dbx_path_lower (str) – Normalized lower case Dropbox path.

Returns

Index entry or None if no entry exists for the given path.

Return type

Optional[maestral.database.IndexEntry]

get_local_hash(local_path)[source]

Computes content hash of a local file.

Parameters

local_path (str) – Absolute path on local drive.

Returns

Content hash to compare with Dropbox’s content hash, or ‘folder’ if the path points to a directory. None if there is nothing at the path.

Return type

Optional[str]

clear_hash_cache()[source]

Clears the sync history.

Return type

None

update_index_from_sync_event(event)[source]

Updates the local index from a SyncEvent.

Parameters

event (maestral.database.SyncEvent) – SyncEvent from download.

Return type

None

update_index_from_dbx_metadata(md, client=None)[source]

Updates the local index from Dropbox metadata.

Parameters
Return type

None

remove_node_from_index(dbx_path_lower)[source]

Removes any local index entries for the given path and all its children.

Parameters

dbx_path_lower (str) – Normalized lower case Dropbox path.

Return type

None

clear_index()[source]

Clears the revision index.

Return type

None

property mignore_path: str

Path to mignore file on local drive (read only).

property mignore_rules: pathspec.pathspec.PathSpec

List of mignore rules following git wildmatch syntax (read only).

load_mignore_file()[source]

Loads rules from mignore file. No rules are loaded if the file does not exist or cannot be read.

Returns

PathSpec instance with ignore patterns.

Return type

None

ensure_dropbox_folder_present()[source]

Checks if the Dropbox folder still exists where we expect it to be.

Raises

NoDropboxDirError – When local Dropbox directory does not exist.

Return type

None

ensure_cache_dir_present()[source]

Checks for or creates a directory at file_cache_path.

Raises

CacheDirError – When local cache directory cannot be created.

Return type

None

clean_cache_dir(raise_error=True)[source]

Removes all items in the cache directory.

Parameters

raise_error (bool) – Whether errors should raised or only logged.

Return type

None

correct_case(dbx_path, client=None)[source]

Converts a Dropbox path with correctly cased basename to a fully cased path. This is useful because the Dropbox API guarantees the correct casing for the basename only. In practice, casing of parent directories is often incorrect. This method retrieves the correct casing of of all ancestors in the path, either from our cache, our database, or from Dropbox servers.

Performance may vary significantly with the number of parent folders and the method used to resolve the casing of all parent directory names:

  1. If the parent directory is already in our cache, performance is O(1).

  2. If the parent directory is already in our sync index, performance is slower because it requires a sqlite query but still O(1).

  3. If the parent directory is unknown to us, its metadata (including the correct casing of directory’s basename) is queried from Dropbox. This is used to construct a correctly cased path by calling correct_case() again. At best, performance will be of O(2) if the parent directory is known to us, at worst if will be of order O(n) involving queries to Dropbox servers for each parent directory.

When calling correct_case() repeatedly for paths from the same tree, it is therefore best to do so in hierarchical order.

Parameters
Returns

Correctly cased Dropbox path.

Return type

str

to_dbx_path(local_path)[source]

Converts a local path to a path relative to the Dropbox folder. Casing of the given local_path will be preserved.

Parameters

local_path (str) – Absolute path on local drive.

Returns

Relative path with respect to Dropbox folder.

Raises

ValueError – When the path lies outside of the local Dropbox folder.

Return type

str

to_dbx_path_lower(local_path)[source]

Converts a local path to a path relative to the Dropbox folder. The path will be normalized as on Dropbox servers (lower case and some additional normalisations).

Parameters

local_path (str) – Absolute path on local drive.

Returns

Relative path with respect to Dropbox folder.

Raises

ValueError – When the path lies outside of the local Dropbox folder.

Return type

str

to_local_path_from_cased(dbx_path_cased)[source]

Converts a correctly cased Dropbox path to the corresponding local path. This is more efficient than to_local_path() which accepts uncased paths.

Parameters

dbx_path_cased (str) – Path relative to Dropbox folder, correctly cased.

Returns

Corresponding local path on drive.

Return type

str

to_local_path(dbx_path, client=None)[source]

Converts a Dropbox path to the corresponding local path. Only the basename must be correctly cased, as guaranteed by the Dropbox API for the display_path attribute of file or folder metadata.

This method slower than to_local_path_from_cased().

Parameters
  • dbx_path (str) – Path relative to Dropbox folder, must be correctly cased in its basename.

  • client (Optional[maestral.client.DropboxClient]) – Client instance to use. If not given, use the instance provided in the constructor.

Returns

Corresponding local path on drive.

Return type

str

has_sync_errors()[source]

Returns True in case of sync errors, False otherwise.

Return type

bool

clear_sync_error(event)[source]

Clears all sync errors after a successful sync event.

Parameters

event (maestral.database.SyncEvent) – Successfully applied sync event.

Return type

None

clear_sync_errors()[source]

Clears all sync errors.

Return type

None

static is_excluded(path)[source]

Checks if a file is excluded from sync. Certain file names are always excluded from syncing, following the Dropbox support article:

https://help.dropbox.com/installs-integrations/sync-uploads/files-not-syncing

This includes file system files such as ‘desktop.ini’ and ‘.DS_Store’ and some temporary files as well as caches used by Dropbox or Maestral. is_excluded accepts both local and Dropbox paths.

Parameters

path (str) – Can be an absolute path, a path relative to the Dropbox folder or just a file name. Does not need to be normalized.

Returns

Whether the path is excluded from syncing.

Return type

bool

is_excluded_by_user(dbx_path_lower)[source]

Check if file has been excluded through “selective sync” by the user.

Parameters

dbx_path_lower (str) – Normalised lower case Dropbox path.

Returns

Whether the path is excluded from download syncing by the user.

Return type

bool

is_mignore(event)[source]

Check if local file change has been excluded by an mignore pattern.

Parameters

event (maestral.database.SyncEvent) – SyncEvent for local file event.

Returns

Whether the path is excluded from upload syncing by the user.

Return type

bool

cancel_sync()[source]

Raises a maestral.errors.CancelledError in all sync threads and waits for them to shut down.

Return type

None

busy()[source]

Checks if we are currently syncing.

Returns

True if sync_lock cannot be acquired, False otherwise.

Return type

bool

upload_local_changes_while_inactive()[source]

Collects changes while sync has not been running and uploads them to Dropbox. Call this method when resuming sync.

Return type

None

wait_for_local_changes(timeout=40)[source]

Blocks until local changes are available.

Parameters

timeout (float) – Maximum time in seconds to wait.

Returns

True if changes are available, False otherwise.

Return type

bool

upload_sync_cycle()[source]

Performs a full upload sync cycle by calling in order:

Handles updating the local cursor for you. If monitoring for local file events was interrupted, call upload_local_changes_while_inactive() instead.

list_local_changes(delay=1)[source]

Waits for local file changes. Returns a list of local changes with at most one entry per path.

Parameters

delay (float) – Delay in sec to wait for subsequent changes before returning.

Returns

(list of sync times events, time_stamp)

Return type

Tuple[List[maestral.database.SyncEvent], float]

apply_local_changes(sync_events)[source]

Applies locally detected changes to the remote Dropbox. Changes which should be ignored (mignore or always ignored files) are skipped.

Parameters

sync_events (List[maestral.database.SyncEvent]) – List of local file system events.

Return type

List[maestral.database.SyncEvent]

get_remote_item(dbx_path, client=None)[source]

Downloads a remote file or folder and updates its local rev. If the remote item does not exist, any corresponding local items will be deleted. If dbx_path refers to a folder, the download will be handled by _get_remote_folder(). If it refers to a single file, the download will be performed by _create_local_entry().

This method can be used to fetch individual items outside of the regular sync cycle, for instance when including a previously excluded file or folder.

Parameters
  • dbx_path (str) – Path relative to Dropbox folder.

  • client (Optional[maestral.client.DropboxClient]) – Client instance to use. If not given, use the instance provided in the constructor.

Returns

Whether download was successful.

Return type

bool

wait_for_remote_changes(last_cursor, timeout=40, client=None)[source]

Blocks until changes to the remote Dropbox are available.

Parameters
  • last_cursor (str) – Cursor form last sync.

  • timeout (int) – Timeout in seconds before returning even if there are no changes. Dropbox adds random jitter of up to 90 sec to this value.

  • client (Optional[maestral.client.DropboxClient]) – Client instance to use. If not given, use the instance provided in the constructor.

Returns

True if changes are available, False otherwise.

Return type

bool

download_sync_cycle(client=None)[source]

Performs a full download sync cycle by calling in order:

Handles updating the remote cursor and resuming interrupted syncs for you. Calling this method will perform a full indexing if this is the first download.

Parameters

client (Optional[maestral.client.DropboxClient]) – Client instance to use. If not given, use the instance provided in the constructor.

Return type

None

list_remote_changes_iterator(last_cursor, client=None)[source]

Get remote changes since the last download sync, as specified by last_cursor. If the last_cursor is from paginating through a previous set of changes, continue where we left off. If last_cursor is an emtpy string, tart a full indexing of the Dropbox folder.

Parameters
  • last_cursor (str) – Cursor from last download sync.

  • client (Optional[maestral.client.DropboxClient]) – Client instance to use. If not given, use the instance provided in the constructor.

Returns

Iterator yielding tuples with remote changes and corresponding cursor.

Return type

Iterator[Tuple[List[maestral.database.SyncEvent], str]]

apply_remote_changes(sync_events)[source]

Applies remote changes to local folder. Call this on the result of list_remote_changes(). The saved cursor is updated after a set of changes has been successfully applied. Entries in the local index are created after successful completion.

Parameters

sync_events (List[maestral.database.SyncEvent]) – List of remote changes.

Returns

List of changes that were made to local files and bool indicating if all download syncs were successful.

Return type

List[maestral.database.SyncEvent]

notify_user(sync_events, client=None)[source]

Shows a desktop notification for the given file changes.

Parameters
Return type

None

rescan(local_path)[source]

Forces a rescan of a local path: schedules created events for every folder, modified events for every file and deleted events for every deleted item (compared to our index).

Parameters

local_path (str) – Path to rescan.

Return type

None