maestral.sync

This module contains the main syncing functionality.

Module Contents

Classes

SyncEvent

Represents a file or folder change in the sync queue

HashCacheEntry

Represents an entry in our cache of content hashes

IndexEntry

Represents an entry in our local sync index

SyncDirection

Enumeration of sync directions

SyncStatus

Enumeration of sync status

ItemType

Enumeration of SyncEvent types

ChangeType

Enumeration of SyncEvent change types

Conflict

Enumeration of sync conflict types

FSEventHandler

A local file event handler

PersistentStateMutableSet

Wraps a list in our state file as a MutableSet

SyncEngine

Class that handles syncing with Dropbox

SyncMonitor

Class to manage sync threads

Functions

download_worker(sync: SyncEngine, syncing: Event, running: Event, connected: Event) → None

Worker to sync changes of remote Dropbox with local folder.

download_worker_added_item(sync: SyncEngine, syncing: Event, running: Event, connected: Event, added_item_queue: Queue[str]) → None

Worker to download items which have been newly included in sync.

upload_worker(sync: SyncEngine, syncing: Event, running: Event, connected: Event) → None

Worker to sync local changes to remote Dropbox.

startup_worker(sync: SyncEngine, syncing: Event, running: Event, connected: Event, startup: Event, paused_by_user: Event) → None

Worker to sync local changes to remote Dropbox.

class maestral.sync.Conflict[source]

Bases: enum.Enum

Enumeration of sync conflict types

RemoteNewer = remote newer[source]
Conflict = conflict[source]
Identical = identical[source]
LocalNewerOrIdentical = local newer or identical[source]
class maestral.sync.FSEventHandler(syncing: Event, startup: Event)[source]

Bases: watchdog.events.FileSystemEventHandler

A local file event handler

Handles captured file events and adds them to SyncEngine’s file event queue to be uploaded by upload_worker(). This acts as a translation layer between watchdog.Observer and SyncEngine.

Parameters
  • syncing – Set when syncing is running.

  • startup – Set when startup is running.

Variables

ignore_timeout (float) – Timeout in seconds after which ignored paths will be discarded.

local_file_event_queue[source]
ignore(self, *events: FileSystemEvent, recursive: bool = True) → Iterator[None][source]

A context manager to ignore local file events

Once a matching event has been registered, further matching events will no longer be ignored unless recursive is True. If no matching event has occurred before leaving the context, the event will be ignored for ignore_timeout sec after leaving then context and then discarded. This accounts for possible delays in the emission of local file system events.

This context manager is used to filter out file system events caused by maestral itself, for instance during a download or when moving a conflict.

Parameters
  • events – Local events to ignore.

  • recursive – If True, all child events of a directory event will be ignored as well. This parameter will be ignored for file events.

expire_ignored_events(self)None[source]

Removes all expired ignore entries.

on_any_event(self, event: FileSystemEvent)None[source]

Checks if the system file event should be ignored. If not, adds it to the queue for events to upload. If syncing is paused or stopped, all events will be ignored.

Parameters

event – Watchdog file event.

class maestral.sync.PersistentStateMutableSet(config_name: str, section: str, option: str)[source]

Bases: collections.abc.MutableSet

Wraps a list in our state file as a MutableSet

Parameters
  • config_name – Name of config (determines name of state file).

  • section – Section name in state file.

  • option – Option name in state file.

add(self, entry: Any)None[source]

Add an element.

discard(self, entry: Any)None[source]

Remove an element. Do not raise an exception if absent.

update(self, *others: Any)None[source]
difference_update(self, *others: Any)None[source]
clear(self)None[source]

Clears all elements.

class maestral.sync.SyncEngine(client: DropboxClient, fs_events_handler: FSEventHandler)[source]

Class that handles syncing with Dropbox

Provides methods to wait for local or remote changes and sync them, including conflict resolution and updates to our index.

Parameters
  • client – Dropbox API client instance.

  • fs_events_handler – File system event handler to inform us of local events.

sync_errors :Set[SyncError][source]
syncing :List[SyncEvent][source]
property dropbox_path(self)str[source]

Path to local Dropbox folder, as loaded from the config file. Before changing dropbox_path, make sure that syncing is paused. Move the dropbox folder to the new location before resuming the sync. Changes are saved to the config file.

property database_path(self)str[source]

Path SQLite database.

property file_cache_path(self)str[source]

Path to cache folder for temporary files (read only). The cache folder ‘.maestral.cache’ is located inside the local Dropbox folder to prevent file transfer between different partitions or drives during sync.

property excluded_items(self) → List[str][source]

List of all files and folders excluded from sync. Changes are saved to the config file. If a parent folder is excluded, its children will automatically be removed from the list. If only children are given but not the parent folder, any new items added to the parent will be synced. Change this property before downloading newly included items or deleting excluded items.

static clean_excluded_items_list(folder_list: List[str]) → List[str][source]

Removes all duplicates and children of excluded items from the excluded items list.

Parameters

folder_list – Dropbox paths to exclude.

Returns

Cleaned up items.

property max_cpu_percent(self)float[source]

Maximum CPU usage for parallel downloads or uploads in percent of the total available CPU time per core. Individual workers in a thread pool will pause until the usage drops below this value. Tasks in the main thread such as indexing file changes may still use more CPU time. Setting this to 200% means that two full logical CPU core can be used.

property remote_cursor(self)str[source]

Cursor from last sync with remote Dropbox. The value is updated and saved to the config file on every successful download of remote changes.

property local_cursor(self)float[source]

Time stamp from last sync with remote Dropbox. The value is updated and saved to the config file on every successful upload of local changes.

property last_change(self)float[source]

The time stamp of the last file change or 0.0 if there are no file changes in our history.

property last_reindex(self)float[source]

Time stamp of last full indexing. This is used to determine when the next full indexing should take place.

property history(self) → List[SyncEvent][source]

A list of the last SyncEvents in our history. History will be kept for the interval specified by the config value keep_history (defaults to two weeks) but at most 1,000 events will be kept.

clear_sync_history(self)None[source]

Clears the sync history.

get_index(self) → List[IndexEntry][source]

Returns a copy of the local index of synced files and folders.

Returns

List of index entries.

iter_index(self) → Iterator[IndexEntry][source]

Returns an iterator over the local index of synced files and folders.

Returns

Iterator over index entries.

get_local_rev(self, dbx_path: str) → Optional[str][source]

Gets revision number of local file.

Parameters

dbx_path – Dropbox path.

Returns

Revision number as str or None if no local revision number has been saved.

get_last_sync(self, dbx_path: str)float[source]

Returns the timestamp of last sync for an individual path.

Parameters

dbx_path – Dropbox path.

Returns

Time of last sync.

get_index_entry(self, dbx_path: str) → Optional[IndexEntry][source]

Gets the index entry for the given Dropbox path.

Parameters

dbx_path – Dropbox path.

Returns

Index entry or None if no entry exists for the given path.

get_local_hash(self, local_path: str) → Optional[str][source]

Computes content hash of a local file.

Parameters

local_path – Absolute path on local drive.

Returns

Content hash to compare with Dropbox’s content hash, or ‘folder’ if the path points to a directory. None if there is nothing at the path.

save_local_hash(self, local_path: str, hash_str: Optional[str], mtime: Optional[float])None[source]

Save the content hash for a file in our cache.

Parameters
  • local_path – Absolute path on local drive.

  • hash_str – Hash string to save. If None, the existing cache entry will be deleted.

  • mtime – Mtime of the file when the hash was computed.

clear_hash_cache(self)None[source]

Clears the sync history.

update_index_from_sync_event(self, event: SyncEvent)None[source]

Updates the local index from a SyncEvent.

Parameters

event – SyncEvent from download.

update_index_from_dbx_metadata(self, md: Metadata)None[source]

Updates the local index from Dropbox metadata.

Parameters

md – Dropbox metadata.

remove_node_from_index(self, dbx_path: str)None[source]

Removes any local index entries for the given path and all its children.

Parameters

dbx_path – Dropbox path.

clear_index(self)None[source]

Clears the revision index.

property mignore_path(self)str[source]

Path to mignore file on local drive (read only).

property mignore_rules(self) → pathspec.PathSpec[source]

List of mignore rules following git wildmatch syntax (read only).

property is_case_sensitive(self)bool[source]

Returns True if the local Dropbox folder is located on a partition with a case-sensitive file system, False otherwise.

ensure_dropbox_folder_present(self)None[source]

Checks if the Dropbox folder still exists where we expect it to be.

Raises

DropboxDeletedError – When localal Dropbox directory does not exist.

clean_cache_dir(self)None[source]

Removes all items in the cache directory.

correct_case(self, dbx_path: str)str[source]

Converts a Dropbox path with correctly cased basename to a fully cased path. This is because Dropbox metadata guarantees the correct casing for the basename only. In practice, casing of parent directories is often incorrect. This is done by retrieving the correct casing of the dirname, either from our cache, our database or from Dropbox servers.

Performance may vary significantly with the number of parent folders:

  1. If the parent directory is already in our cache, performance is O(1).

  2. If the parent directory is already in our sync index, performance is O(1) but slower than the first case because it requires a SQLAlchemy query.

  3. If the parent directory is unknown to us, its metadata (including the correct casing of directory’s basename) is queried from Dropbox. This is used to construct a correctly cased path by calling correct_case() again. At best, performance will be of O(2) if the parent directory is known to us, at worst if will be of order O(n) involving queries to Dropbox servers for each parent directory.

When running correct_case() on a large tree of paths, it is therefore best to do so in hierarchical order.

Parameters

dbx_path – Dropbox path with correctly cased basename, as provided by dropbox.files.Metadata.path_display or dropbox.files.Metadata.name.

Returns

Correctly cased Dropbox path.

to_dbx_path(self, local_path: str)str[source]

Converts a local path to a path relative to the Dropbox folder. Casing of the given local_path will be preserved.

Parameters

local_path – Absolute path on local drive.

Returns

Relative path with respect to Dropbox folder.

Raises

ValueError – When the path lies outside of the local Dropbox folder.

to_local_path_from_cased(self, dbx_path_cased: str)str[source]

Converts a correctly cased Dropbox path to the corresponding local path. This is more efficient than to_local_path() which accepts uncased paths.

Parameters

dbx_path_cased – Path relative to Dropbox folder, correctly cased.

Returns

Corresponding local path on drive.

to_local_path(self, dbx_path: str)str[source]

Converts a Dropbox path to the corresponding local path. Only the basename must be correctly cased. This is slower than to_local_path_from_cased().

Parameters

dbx_path – Path relative to Dropbox folder, must be correctly cased in its basename.

Returns

Corresponding local path on drive.

has_sync_errors(self)bool[source]

Returns True in case of sync errors, False otherwise.

clear_sync_error(self, local_path: Optional[str] = None, dbx_path: Optional[str] = None)None[source]

Clears all sync errors for local_path or dbx_path.

Parameters
  • local_path – Absolute path on local drive.

  • dbx_path – Path relative to Dropbox folder.

clear_sync_errors(self)None[source]

Clears all sync errors.

static is_excluded(path)bool[source]

Checks if a file is excluded from sync. Certain file names are always excluded from syncing, following the Dropbox support article:

https://help.dropbox.com/installs-integrations/sync-uploads/files-not-syncing

This includes file system files such as ‘desktop.ini’ and ‘.DS_Store’ and some temporary files as well as caches used by Dropbox or Maestral. is_excluded accepts both local and Dropbox paths.

Parameters

path – Path of item. Can be both a local or Dropbox paths.

Returns

Whether the path is excluded from syncing.

is_excluded_by_user(self, dbx_path: str)bool[source]

Check if file has been excluded through “selective sync” by the user.

Parameters

dbx_path – Path relative to Dropbox folder.

Returns

Whether the path is excluded from download syncing by the user.

is_mignore(self, event: SyncEvent)bool[source]

Check if local file change has been excluded by an mignore pattern.

Parameters

event – SyncEvent for local file event.

Returns

Whether the path is excluded from upload syncing by the user.

busy(self)bool[source]

Checks if we are currently syncing.

Returns

True if sync_lock cannot be acquired, False otherwise.

free_memory(self)None[source]

Frees memory by resetting our database session and the requests session, clearing out case-conversion cache and clearing all expired event ignores and.

upload_local_changes_while_inactive(self)None[source]

Collects changes while sync has not been running and uploads them to Dropbox. Call this method when resuming sync.

wait_for_local_changes(self, timeout: float = 40, delay: float = 1) → Tuple[List[SyncEvent], float][source]

Waits for local file changes. Returns a list of local changes with at most one entry per path.

Parameters
  • timeout – If no changes are detected within timeout (sec), an empty list is returned.

  • delay – Delay in sec to wait for subsequent changes before returning.

Returns

(list of sync times events, time_stamp)

apply_local_changes(self, sync_events: List[SyncEvent], local_cursor: float) → List[SyncEvent][source]

Applies locally detected changes to the remote Dropbox. Changes which should be ignored (mignore or always ignored files) are skipped.

Parameters
  • sync_events – List of local file system events.

  • local_cursor – Time stamp of last event in events.

get_remote_folder(self, dbx_path: str = '/')bool[source]

Gets all files/folders from Dropbox and writes them to the local folder dropbox_path. Call this method on first run of the Maestral. Indexing and downloading may take several minutes, depending on the size of the user’s Dropbox folder.

Parameters

dbx_path – Path relative to Dropbox folder. Defaults to root (‘/’).

Returns

Whether download was successful.

get_remote_item(self, dbx_path: str)bool[source]

Downloads a remote file or folder and updates its local rev. If the remote item does not exist, any corresponding local items will be deleted. If dbx_path refers to a folder, the download will be handled by get_remote_folder(). If it refers to a single file, the download will be performed by _create_local_entry().

This method can be used to fetch individual items outside of the regular sync cycle, for instance when including a previously excluded file or folder.

Parameters

dbx_path – Path relative to Dropbox folder.

Returns

Whether download was successful.

wait_for_remote_changes(self, last_cursor: str, timeout: int = 40, delay: float = 2)bool[source]

Blocks until changes to the remote Dropbox are available.

Parameters
  • last_cursor – Cursor form last sync.

  • timeout – Timeout in seconds before returning even if there are no changes. Dropbox adds random jitter of up to 90 sec to this value.

  • delay – Delay in sec to wait for subsequent changes that may be duplicates. This delay is typically only necessary folders are shared / un-shared with other Dropbox accounts.

list_remote_changes(self, last_cursor: str) → Tuple[List[SyncEvent], str][source]

Lists remote changes since the last download sync.

Parameters

last_cursor – Cursor from last download sync.

Returns

Tuple with remote changes and corresponding cursor

list_remote_changes_iterator(self, last_cursor: str) → Iterator[Tuple[List[SyncEvent], Optional[str]]][source]

Lists remote changes since the last download sync. Works the same as list_remote_changes() but returns an iterator over remote changes. Only the last result will have a valid cursor which is not None.

Parameters

last_cursor – Cursor from last download sync.

Returns

Iterator yielding tuples with remote changes and corresponding cursor.

apply_remote_changes(self, sync_events: List[SyncEvent], cursor: Optional[str]) → List[SyncEvent][source]

Applies remote changes to local folder. Call this on the result of list_remote_changes(). The saved cursor is updated after a set of changes has been successfully applied. Entries in the local index are created after successful completion.

Parameters
  • sync_events – List of remote changes.

  • cursor – Remote cursor corresponding to changes. Take care to only pass cursors which represent the state of the entire Dropbox. Pass None instead if you are only downloading a subset of changes.

Returns

List of changes that were made to local files and bool indicating if all download syncs were successful.

notify_user(self, sync_events: List[SyncEvent])None[source]

Shows a desktop notification for the given file changes.

Parameters

sync_events – List of SyncEvents from download sync.

rescan(self, local_path: str)None[source]

Forces a rescan of a local path: schedules created events for every folder, modified events for every file and deleted events for every deleted item (compared to our index).

Parameters

local_path – Path to rescan.

maestral.sync.download_worker(sync: SyncEngine, syncing: Event, running: Event, connected: Event)None[source]

Worker to sync changes of remote Dropbox with local folder.

Parameters
  • sync – Instance of SyncEngine.

  • syncing – Event that indicates if workers are running or paused.

  • running – Event to shutdown local file event handler and worker threads.

  • connected – Event that indicates if we can connect to Dropbox.

maestral.sync.download_worker_added_item(sync: SyncEngine, syncing: Event, running: Event, connected: Event, added_item_queue: Queue[str])None[source]

Worker to download items which have been newly included in sync.

Parameters
  • sync – Instance of SyncEngine.

  • syncing – Event that indicates if workers are running or paused.

  • running – Event to shutdown local file event handler and worker threads.

  • connected – Event that indicates if we can connect to Dropbox.

  • added_item_queue – Queue with newly added items to download. Entries are Dropbox paths.

maestral.sync.upload_worker(sync: SyncEngine, syncing: Event, running: Event, connected: Event)None[source]

Worker to sync local changes to remote Dropbox.

Parameters
  • sync – Instance of SyncEngine.

  • syncing – Event that indicates if workers are running or paused.

  • running – Event to shutdown local file event handler and worker threads.

  • connected – Event that indicates if we can connect to Dropbox.

maestral.sync.startup_worker(sync: SyncEngine, syncing: Event, running: Event, connected: Event, startup: Event, paused_by_user: Event)None[source]

Worker to sync local changes to remote Dropbox.

Parameters
  • sync – Instance of SyncEngine.

  • syncing – Event that indicates if workers are running or paused.

  • running – Event to shutdown local file event handler and worker threads.

  • connected – Event that indicates if we can connect to Dropbox.

  • startup – Set when we should run startup routines.

  • paused_by_user – Set when syncing has been paused by the user.

class maestral.sync.SyncMonitor(client: DropboxClient)[source]

Class to manage sync threads

Parameters

client – The Dropbox API client, a wrapper around the Dropbox Python SDK.

added_item_queue[source]

Queue of dropbox paths which have been newly included in syncing.

property reindex_interval(self)float[source]

Interval in sec for period reindexing. Changes will be saved to state file.

property activity(self) → List[SyncEvent][source]

Returns a list all items queued for or currently syncing.

property history(self) → List[SyncEvent][source]

A list of the last SyncEvents in our history. History will be kept for the interval specified by the config value``keep_history`` (defaults to two weeks) but at most 1,000 events will kept.

property idle_time(self)float[source]

Returns the idle time in seconds since the last file change or since startup if there haven’t been any changes in our current session.

start(self)None[source]

Creates observer threads and starts syncing.

pause(self)None[source]

Pauses syncing.

resume(self)None[source]

Checks for changes while idle and starts syncing.

stop(self)None[source]

Stops syncing and destroys worker threads.

connection_monitor(self)None[source]

Monitors the connection to Dropbox servers. Pauses syncing when the connection is lost and resumes syncing when reconnected and syncing has not been paused by the user.

on_connect(self)None[source]

Callback to run when we have lost the connection to Dropbox

on_disconnect(self)None[source]

Callback to run when we have reestablished the connection to Dropbox

reset_sync_state(self)None[source]

Resets all saved sync state. Settings are not affected.

rebuild_index(self)None[source]

Rebuilds the rev file by comparing remote with local files and updating rev numbers from the Dropbox server. Files are compared by their content hashes and conflicting copies are created if the contents differ. File changes during the rebuild process will be queued and uploaded once rebuilding has completed.

Rebuilding will be performed asynchronously.