maestral.sync
This module contains the main syncing functionality.
Module Contents
- class maestral.sync.FSEventHandler(file_event_types=(EVENT_TYPE_CREATED, EVENT_TYPE_DELETED, EVENT_TYPE_MODIFIED, EVENT_TYPE_MOVED), dir_event_types=(EVENT_TYPE_CREATED, EVENT_TYPE_DELETED, EVENT_TYPE_MOVED))[source]
Bases:
watchdog.events.FileSystemEventHandler
A local file event handler
Handles captured file events and adds them to
SyncEngine
’s file event queue to be uploaded byupload_worker()
. This acts as a translation layer betweenwatchdog.Observer
andSyncEngine
.White lists of event types to handle are supplied as
file_event_types
anddir_event_types
. This is for forward compatibility as additional event types may be added to watchdog in the future.- Parameters
- Variables
ignore_timeout (float) – Timeout in seconds after which filters for ignored events will expire.
- disable(self)[source]
Turn off queueing of new events and remove all events from queue.
- Return type
None
- ignore(self, *events, recursive=True)[source]
A context manager to ignore local file events
Once a matching event has been registered, further matching events will no longer be ignored unless
recursive
isTrue
. If no matching event has occurred before leaving the context, the event will be ignored forignore_timeout
sec after leaving then context and then discarded. This accounts for possible delays in the emission of local file system events.This context manager is used to filter out file system events caused by maestral itself, for instance during a download or when moving a conflict.
- Example
Prevent triggereing a sync event when creating a local file:
>>> from watchdog.events import FileCreatedEvent >>> from maestral.main import Maestral >>> m = Maestral() >>> with m.sync.fs_events.ignore(FileCreatedEvent('path')): ... open('path').close()
- Parameters
events (watchdog.events.FileSystemEvent) – Local events to ignore.
recursive (bool) – If
True
, all child events of a directory event will be ignored as well. This parameter will be ignored for file events.
- Return type
Iterator[None]
- on_any_event(self, event)[source]
Checks if the system file event should be ignored. If not, adds it to the queue for events to upload. If syncing is paused or stopped, all events will be ignored.
- Parameters
event (watchdog.events.FileSystemEvent) – Watchdog file event.
- Return type
None
- queue_event(self, event)[source]
Queues an individual file system event. Notifies / wakes up all threads that are waiting with
wait_for_event()
.- Parameters
event (watchdog.events.FileSystemEvent) – File system event to queue.
- Return type
None
- wait_for_event(self, timeout=40)[source]
Blocks until an event is available in the queue or a timeout occurs, whichever comes first. You can use with method to wait for file system events in another thread.
Note
If there are multiple threads waiting for events, all of them will be notified. If one of those threads starts getting events from
local_file_event_queue
, other threads may find that the queue is empty despite being woken. You should therefore be prepared to handle an empty queue even if this method returnsTrue
.
- class maestral.sync.SyncEngine(client)[source]
Class that handles syncing with Dropbox
Provides methods to wait for local or remote changes and sync them, including conflict resolution and updates to our index.
- Parameters
client (maestral.client.DropboxClient) – Dropbox API client instance.
- reload_cached_config(self)[source]
Reloads all config and state values that are otherwise cached by this class for faster access. Call this method if config or state values where modified directly instead of using
SyncEngine
APIs.- Return type
None
- property dropbox_path(self)[source]
Path to local Dropbox folder, as loaded from the config file. Before changing
dropbox_path
, make sure that syncing is paused. Move the dropbox folder to the new location before resuming the sync. Changes are saved to the config file.- Return type
- property is_fs_case_sensitive(self)[source]
Whether the local Dropbox directory lies on a case-sensitive file system.
- Return type
- property file_cache_path(self)[source]
Path to cache folder for temporary files (read only). The cache folder ‘.maestral.cache’ is located inside the local Dropbox folder to prevent file transfer between different partitions or drives during sync.
- Return type
- property excluded_items(self)[source]
List of all files and folders excluded from sync. Changes are saved to the config file. If a parent folder is excluded, its children will automatically be removed from the list. If only children are given but not the parent folder, any new items added to the parent will be synced. Change this property before downloading newly included items or deleting excluded items.
- static clean_excluded_items_list(folder_list)[source]
Removes all duplicates and children of excluded items from the excluded items list. Normalises all paths to lower case.
- property max_cpu_percent(self)[source]
Maximum CPU usage for parallel downloads or uploads in percent of the total available CPU time per core. Individual workers in a thread pool will pause until the usage drops below this value. Tasks in the main thread such as indexing file changes may still use more CPU time. Setting this to 200% means that two full logical CPU core can be used.
- Return type
- property remote_cursor(self)[source]
Cursor from last sync with remote Dropbox. The value is updated and saved to the config file on every successful download of remote changes.
- Return type
- property local_cursor(self)[source]
Time stamp from last sync with remote Dropbox. The value is updated and saved to the config file on every successful upload of local changes.
- Return type
- property last_change(self)[source]
The time stamp of the last file change or 0.0 if there are no file changes in our history.
- Return type
- property last_reindex(self)[source]
Time stamp of last full indexing. This is used to determine when the next full indexing should take place.
- Return type
- property history(self)[source]
A list of the last SyncEvents in our history. History will be kept for the interval specified by the config value
keep_history
(defaults to two weeks) but at most 1,000 events will be kept.- Return type
- reset_sync_state(self)[source]
Resets all saved sync state. Settings are not affected.
- Return type
None
- sync_errors_for_path(self, dbx_path_lower, direction=None)[source]
Returns a list of all sync errors for the given path and its children.
- Parameters
dbx_path_lower (str) – Normalised Dropbox path.
direction (SyncDirection | None) – Direction to filter sync errors. If not given, both upload and download errors will be returned.
- Returns
List of sync errors.
- Return type
- clear_sync_errors_for_path(self, dbx_path_lower, recursive=False)[source]
Clear all sync errors for a path after a successful sync event.
- clear_sync_errors_from_event(self, event)[source]
Clears sync errors corresponding to a sync event.
- Parameters
event (maestral.models.SyncEvent) –
- Return type
None
- get_index(self)[source]
Returns a copy of the local index of synced files and folders.
- Returns
List of index entries.
- Return type
- get_index_entry(self, dbx_path_lower)[source]
Gets the index entry for the given Dropbox path.
- Parameters
dbx_path_lower (str) – Normalized lower case Dropbox path.
- Returns
Index entry or
None
if no entry exists for the given path.- Return type
IndexEntry | None
- iter_index(self)[source]
Returns an iterator over the local index of synced files and folders.
- Returns
Iterator over index entries.
- Return type
Iterator[maestral.models.IndexEntry]
- index_count(self)[source]
Returns the number of items in our index without loading any items.
- Returns
Number of index entries.
- Return type
- get_last_sync(self, dbx_path_lower)[source]
Returns the timestamp of last sync for an individual path.
- update_index_from_sync_event(self, event)[source]
Updates the local index from a SyncEvent.
- Parameters
event (maestral.models.SyncEvent) – SyncEvent from download.
- Return type
None
- update_index_from_dbx_metadata(self, md, client=None)[source]
Updates the local index from Dropbox metadata.
- Parameters
md (maestral.core.Metadata) – Dropbox metadata.
client (DropboxClient | None) – DropboxClient instance to use. If not given, use the global instance.
- Return type
None
- remove_node_from_index(self, dbx_path_lower)[source]
Removes any local index entries for the given path and all its children.
- Parameters
dbx_path_lower (str) – Normalized lower case Dropbox path.
- Return type
None
- property mignore_rules(self)[source]
List of mignore rules following git wildmatch syntax (read only).
- Return type
pathspec.PathSpec
- load_mignore_file(self)[source]
Loads rules from mignore file. No rules are loaded if the file does not exist or cannot be read.
- Returns
PathSpec instance with ignore patterns.
- Return type
None
- ensure_dropbox_folder_present(self)[source]
Checks if the Dropbox folder still exists where we expect it to be.
- Raises
NoDropboxDirError – When local Dropbox directory does not exist.
- Return type
None
- ensure_cache_dir_present(self)[source]
Checks for or creates a directory at
file_cache_path
.- Raises
CacheDirError – When local cache directory cannot be created.
- Return type
None
- clean_cache_dir(self, raise_error=True)[source]
Removes all items in the cache directory.
- Parameters
raise_error (bool) – Whether errors should be raised or only logged.
- Return type
None
- correct_case(self, dbx_path, client=None)[source]
Converts a Dropbox path with correctly cased basename to a fully cased path. This is useful because the Dropbox API guarantees the correct casing for the basename only. In practice, casing of parent directories is often incorrect. This method retrieves the correct casing of all ancestors in the path, either from our cache, our database, or from Dropbox servers.
Performance may vary significantly with the number of parent folders and the method used to resolve the casing of all parent directory names:
If the parent directory is already in our cache, performance is O(1).
If the parent directory is already in our sync index, performance is slower because it requires a sqlite query but still O(1).
If the parent directory is unknown to us, its metadata (including the correct casing of directory’s basename) is queried from Dropbox. This is used to construct a correctly cased path by calling
correct_case()
again. At best, performance will be of O(2) if the parent directory is known to us, at worst it will be of order O(n) involving queries to Dropbox servers for each parent directory.
When calling
correct_case()
repeatedly for paths from the same tree, it is therefore best to do so in hierarchical order.- Parameters
dbx_path (str) – Dropbox path with correctly cased basename, as provided by
dropbox.files.Metadata.path_display
ordropbox.files.Metadata.name
.client (DropboxClient | None) – Client instance to use. If not given, use the instance provided in the constructor.
- Returns
Correctly cased Dropbox path.
- Return type
- to_dbx_path(self, local_path)[source]
Converts a local path to a path relative to the Dropbox folder. Casing of the given
local_path
will be preserved.- Parameters
local_path (str) – Absolute path on local drive.
- Returns
Relative path with respect to Dropbox folder.
- Raises
ValueError – When the path lies outside the local Dropbox folder.
- Return type
- to_dbx_path_lower(self, local_path)[source]
Converts a local path to a path relative to the Dropbox folder. The path will be normalized as on Dropbox servers (lower case and some additional normalisations).
- Parameters
local_path (str) – Absolute path on local drive.
- Returns
Relative path with respect to Dropbox folder.
- Raises
ValueError – When the path lies outside the local Dropbox folder.
- Return type
- to_local_path_from_cased(self, dbx_path_cased)[source]
Converts a correctly cased Dropbox path to the corresponding local path. This is more efficient than
to_local_path()
which accepts uncased paths.
- to_local_path(self, dbx_path, client=None)[source]
Converts a Dropbox path to the corresponding local path. Only the basename must be correctly cased, as guaranteed by the Dropbox API for the
display_path
attribute of file or folder metadata.This method slower than
to_local_path_from_cased()
.- Parameters
dbx_path (str) – Path relative to Dropbox folder, must be correctly cased in its basename.
client (DropboxClient | None) – Client instance to use. If not given, use the instance provided in the constructor.
- Returns
Corresponding local path on drive.
- Return type
- is_excluded(self, path)[source]
Checks if a file is excluded from sync. Certain file names are always excluded from syncing, following the Dropbox support article:
https://help.dropbox.com/installs-integrations/sync-uploads/files-not-syncing
This includes file system files such as ‘desktop.ini’ and ‘.DS_Store’ and some temporary files as well as caches used by Dropbox or Maestral. is_excluded accepts both local and Dropbox paths.
- is_excluded_by_user(self, dbx_path_lower)[source]
Check if file has been excluded through “selective sync” by the user.
- is_mignore(self, event)[source]
Check if local file change has been excluded by a mignore pattern.
- Parameters
event (maestral.models.SyncEvent) – SyncEvent for local file event.
- Returns
Whether the path is excluded from upload syncing by the user.
- Return type
- cancel_sync(self)[source]
Raises a
maestral.exceptions.CancelledError
in all sync threads and waits for them to shut down.- Return type
None
- busy(self)[source]
Checks if we are currently syncing.
- Returns
True
ifsync_lock
cannot be acquired,False
otherwise.- Return type
- upload_local_changes_while_inactive(self)[source]
Collects changes while sync has not been running and uploads them to Dropbox. Call this method when resuming sync.
- Return type
None
- upload_sync_cycle(self)[source]
Performs a full upload sync cycle by calling in order:
Handles updating the local cursor for you. If monitoring for local file events was interrupted, call
upload_local_changes_while_inactive()
instead.
- list_local_changes(self, delay=1)[source]
Returns a list of local changes with at most one entry per path.
- Parameters
delay (float) – Delay in sec to wait for subsequent changes before returning.
- Returns
(list of sync times events, time_stamp)
- Return type
- apply_local_changes(self, sync_events)[source]
Applies locally detected changes to the remote Dropbox. Changes which should be ignored (mignore or always ignored files) are skipped.
- Parameters
sync_events (list[maestral.models.SyncEvent]) – List of local file system events.
- Return type
- get_remote_item(self, dbx_path, client=None)[source]
Downloads a remote file or folder and updates its local rev. If the remote item does not exist, any corresponding local items will be deleted. If
dbx_path
refers to a folder, the download will be handled by_get_remote_folder()
. If it refers to a single file, the download will be performed by_create_local_entry()
.This method can be used to fetch individual items outside the regular sync cycle, for instance when including a previously excluded file or folder.
- Parameters
dbx_path (str) – Path relative to Dropbox folder.
client (DropboxClient | None) – Client instance to use. If not given, use the instance provided in the constructor.
- Returns
Whether download was successful.
- Return type
- wait_for_remote_changes(self, last_cursor, timeout=40, client=None)[source]
Blocks until changes to the remote Dropbox are available.
- Parameters
last_cursor (str) – Cursor form last sync.
timeout (int) – Timeout in seconds before returning even if there are no changes. Dropbox adds random jitter of up to 90 sec to this value.
client (DropboxClient | None) – Client instance to use. If not given, use the instance provided in the constructor.
- Returns
True
if changes are available,False
otherwise.- Return type
- download_sync_cycle(self, client=None)[source]
Performs a full download sync cycle by calling in order:
Handles updating the remote cursor and resuming interrupted syncs for you. Calling this method will perform a full indexing if this is the first download.
- Parameters
client (DropboxClient | None) – Client instance to use. If not given, use the instance provided in the constructor.
- Return type
None
- list_remote_changes_iterator(self, last_cursor, client=None)[source]
Get remote changes since the last download sync, as specified by
last_cursor
. If thelast_cursor
is from paginating through a previous set of changes, continue where we left off. Iflast_cursor
is an empty string, perform a full indexing of the Dropbox folder.- Parameters
last_cursor (str) – Cursor from last download sync.
client (DropboxClient | None) – Client instance to use. If not given, use the instance provided in the constructor.
- Returns
Iterator yielding tuples with remote changes and corresponding cursor.
- Return type
Iterator[tuple[list[maestral.models.SyncEvent], str]]
- apply_remote_changes(self, sync_events)[source]
Applies remote changes to local folder. Call this on the result of
list_remote_changes()
. The saved cursor is updated after a set of changes has been successfully applied. Entries in the local index are created after successful completion.- Parameters
sync_events (list[maestral.models.SyncEvent]) – List of remote changes.
- Returns
List of changes that were made to local files and bool indicating if all download syncs were successful.
- Return type
- notify_user(self, sync_events, client=None)[source]
Shows a desktop notification for the given file changes.
- Parameters
sync_events (list[maestral.models.SyncEvent]) – List of SyncEvents from download sync.
client (DropboxClient | None) – Client instance to use. If not given, use the instance provided in the constructor.
- Return type
None