maestral.sync ============= .. py:module:: maestral.sync .. autoapi-nested-parse:: This module contains the main syncing functionality. Module Contents --------------- .. py:class:: Conflict Bases: :py:obj:`enum.Enum` Enumeration of sync conflict types .. py:attribute:: RemoteNewer :value: 'remote newer' .. py:attribute:: Conflict :value: 'conflict' .. py:attribute:: Identical :value: 'identical' .. py:attribute:: LocalNewerOrIdentical :value: 'local newer or identical' .. py:class:: FSEventHandler(file_event_types = (EVENT_TYPE_CREATED, EVENT_TYPE_DELETED, EVENT_TYPE_MODIFIED, EVENT_TYPE_MOVED), dir_event_types = (EVENT_TYPE_CREATED, EVENT_TYPE_DELETED, EVENT_TYPE_MOVED)) Bases: :py:obj:`watchdog.events.FileSystemEventHandler` A local file event handler Handles captured file events and adds them to :class:`SyncEngine`'s file event queue to be uploaded by :meth:`upload_worker`. This acts as a translation layer between :class:`watchdog.Observer` and :class:`SyncEngine`. White lists of event types to handle are supplied as ``file_event_types`` and ``dir_event_types``. This is for forward compatibility as additional event types may be added to watchdog in the future. :param file_event_types: Types of file events to handle. Acts as an allow list. :param dir_event_types: Types of folder events to handle. Acts as an allow list. :cvar float ignore_timeout: Timeout in seconds after which filters for ignored events will expire. .. py:attribute:: local_file_event_queue :type: queue.Queue[watchdog.events.FileSystemEvent] .. py:attribute:: has_events .. py:attribute:: file_event_types .. py:attribute:: dir_event_types .. py:attribute:: ignore_timeout :value: 2.0 .. py:property:: enabled :type: bool Whether queuing of events is enabled. .. py:method:: enable() Turn on queueing of events. .. py:method:: disable() Turn off queueing of new events and remove all events from queue. .. py:method:: ignore(*events, recursive = True) A context manager to ignore local file events Once a matching event has been registered, further matching events will no longer be ignored unless ``recursive`` is ``True``. If no matching event has occurred before leaving the context, the event will be ignored for :attr:`ignore_timeout` sec after leaving then context and then discarded. This accounts for possible delays in the emission of local file system events. This context manager is used to filter out file system events caused by maestral itself, for instance during a download or when moving a conflict. :Example: Prevent triggereing a sync event when creating a local file: >>> from watchdog.events import FileCreatedEvent >>> from maestral.main import Maestral >>> m = Maestral() >>> with m.sync.fs_events.ignore(FileCreatedEvent('path')): ... open('path').close() :param events: Local events to ignore. :param recursive: If ``True``, all child events of a directory event will be ignored as well. This parameter will be ignored for file events. .. py:method:: expire_ignored_events() Removes all expired ignore entries. .. py:method:: on_any_event(event) Checks if the system file event should be ignored. If not, adds it to the queue for events to upload. If syncing is paused or stopped, all events will be ignored. :param event: Watchdog file event. .. py:method:: queue_event(event) Queues an individual file system event. Notifies / wakes up all threads that are waiting with :meth:`wait_for_event`. :param event: File system event to queue. .. py:method:: wait_for_event(timeout = 40) Blocks until an event is available in the queue or a timeout occurs, whichever comes first. You can use with method to wait for file system events in another thread. .. note:: If there are multiple threads waiting for events, all of them will be notified. If one of those threads starts getting events from :attr:`local_file_event_queue`, other threads may find that the queue is empty despite being woken. You should therefore be prepared to handle an empty queue even if this method returns ``True``. :param timeout: Maximum time to block in seconds. :returns: ``True`` if an event is available, ``False`` if the call returns due to a timeout. .. py:class:: ActivityNode(name, sync_events = (), parent = None) A node in a sparse tree to represent syncing activity. Each node represents an item in the local Dropbox folder. Apart from the root node, items will only be present if they or any of their children have any sync activity. :attr children: All children with sync activity. Leaf nodes must represent items that are being uploaded, downloaded, or have a sync error. :attr sync_events: All SyncEvents of this node and its children. .. py:attribute:: name .. py:attribute:: parent :value: None .. py:attribute:: children :type: dict[str, ActivityNode] .. py:attribute:: sync_events .. py:class:: ActivityTree Bases: :py:obj:`ActivityNode` The root node in a sync activity tree. Represents Dropbox root. .. py:method:: add(event) .. py:method:: remove(event) .. py:method:: discard(event) .. py:method:: has_path(dbx_path) .. py:method:: get_node(dbx_path) .. py:class:: SyncEngine(client, desktop_notifier = None) Class that handles syncing with Dropbox Provides methods to wait for local or remote changes and sync them, including conflict resolution and updates to our index. :param client: Dropbox API client instance. :param desktop_notifier: Desktop notifier instance to use for user notifications. .. py:attribute:: client .. py:attribute:: config_name .. py:attribute:: fs_events .. py:attribute:: desktop_notifier :value: None .. py:attribute:: sync_lock .. py:attribute:: activity .. py:method:: reload_cached_config() Reloads all config and state values that are otherwise cached by this class for faster access. Call this method if config or state values where modified directly instead of using :class:`SyncEngine` APIs. .. py:property:: dropbox_path :type: str Path to local Dropbox folder, as loaded from the config file. Before changing :attr:`dropbox_path`, make sure that syncing is paused. Move the dropbox folder to the new location before resuming the sync. Changes are saved to the config file. .. py:property:: is_fs_case_sensitive :type: bool Whether the local Dropbox directory lies on a case-sensitive file system. .. py:property:: database_path :type: str Path SQLite database. .. py:property:: file_cache_path :type: str Path to cache folder for temporary files (read only). The cache folder '.maestral.cache' is located inside the local Dropbox folder to prevent file transfer between different partitions or drives during sync. .. py:property:: excluded_items :type: set[str] Set of all files and folders excluded from sync. Changes are saved to the config file. If a parent folder is excluded, its children will automatically be removed from the list. If only children are given but not the parent folder, any new items added to the parent will be synced. Change this property *before* downloading newly included items or deleting excluded items. .. py:method:: clean_excluded_items_list(dbx_paths) :staticmethod: Removes all duplicates and children of excluded items from the excluded items list. Normalises all paths to lower case. :param dbx_paths: Dropbox paths to exclude. :returns: Cleaned up items. .. py:property:: max_cpu_percent :type: float Maximum CPU usage for parallel downloads or uploads in percent of the total available CPU time per core. Individual workers in a thread pool will pause until the usage drops below this value. Tasks in the main thread such as indexing file changes may still use more CPU time. Setting this to 200% means that two full logical CPU core can be used. .. py:property:: remote_cursor :type: str Cursor from last sync with remote Dropbox. The value is updated and saved to the config file on every successful download of remote changes. .. py:property:: local_cursor :type: float Time stamp from last sync with remote Dropbox. The value is updated and saved to the config file on every successful upload of local changes. .. py:property:: last_change :type: float The time stamp of the last file change or 0.0 if there are no file changes in our history. .. py:property:: last_reindex :type: float Time stamp of last full indexing. This is used to determine when the next full indexing should take place. .. py:method:: get_history(dbx_path = None) A list of the last SyncEvents in our history. History will be kept for the interval specified by the config value ``keep_history`` (defaults to two weeks) but at most 1,000 events will be kept. .. py:method:: reset_sync_state() Resets all saved sync state. Settings are not affected. .. py:property:: sync_errors :type: list[maestral.models.SyncErrorEntry] Returns a list of all sync errors. .. py:property:: upload_errors :type: list[maestral.models.SyncErrorEntry] Returns a list of all upload errors. .. py:property:: download_errors :type: list[maestral.models.SyncErrorEntry] Returns a list of all download errors. .. py:method:: has_sync_errors() Returns ``True`` in case of sync errors, ``False`` otherwise. .. py:method:: sync_errors_for_path(dbx_path_lower, direction = None) Returns a list of all sync errors for the given path and its children. :param dbx_path_lower: Normalised Dropbox path. :param direction: Direction to filter sync errors. If not given, both upload and download errors will be returned. :returns: List of sync errors. .. py:method:: clear_sync_errors_for_path(dbx_path_lower, recursive = False) Clear all sync errors for a path after a successful sync event. :param dbx_path_lower: Normalised Dropbox path to clear. :param recursive: Whether to clear sync errors for children of the given path. .. py:method:: clear_sync_errors_from_event(event) Clears sync errors corresponding to a sync event. .. py:method:: get_index() Returns a copy of the local index of synced files and folders. :returns: List of index entries. .. py:method:: get_index_entry(dbx_path_lower) Gets the index entry for the given Dropbox path. :param dbx_path_lower: Normalized lower case Dropbox path. :returns: Index entry or ``None`` if no entry exists for the given path. .. py:method:: get_index_entry_for_local_path(local_path) Gets the index entry for the given local path. Ensures that the index entry has the correct casing but ignore unicode normalisation differences. Dropbox always normalizes to composed (NFC) on upload, even in the display_path, so we cannot distinguish. :param local_path: Local path as returned by file system APIs. :returns: Index entry or ``None`` if no entry exists for the given path. .. py:method:: iter_index() Returns an iterator over the local index of synced files and folders. :returns: Iterator over index entries. .. py:method:: index_count() Returns the number of items in our index without loading any items. :returns: Number of index entries. .. py:method:: get_local_rev(dbx_path_lower) Gets revision number of local file. :param dbx_path_lower: Normalized lower case Dropbox path. :returns: Revision number as str or ``None`` if no local revision number has been saved. .. py:method:: get_last_sync(dbx_path_lower) Returns the timestamp of last sync for an individual path. :param dbx_path_lower: Normalized lower case Dropbox path. :returns: Time of last sync. .. py:method:: update_index_from_sync_event(event) Updates the local index from a SyncEvent. :param event: SyncEvent from download. .. py:method:: update_index_from_dbx_metadata(md) Updates the local index from Dropbox metadata. :param md: Dropbox metadata. .. py:method:: remove_node_from_index(dbx_path_lower) Removes any local index entries for the given path and all its children. :param dbx_path_lower: Normalized lower case Dropbox path. .. py:method:: get_local_hash(local_path) Computes content hash of a local file. :param local_path: Absolute path on local drive. :returns: Content hash to compare with Dropbox's content hash, or 'folder' if the path points to a directory. ``None`` if there is nothing at the path. .. py:property:: mignore_path :type: str Path to mignore file on local drive (read only). .. py:property:: mignore_rules :type: pathspec.PathSpec List of mignore rules following git wildmatch syntax (read only). .. py:method:: load_mignore_file() Loads rules from mignore file. No rules are loaded if the file does not exist or cannot be read. :returns: PathSpec instance with ignore patterns. .. py:method:: ensure_dropbox_folder_present() Checks if the Dropbox folder still exists where we expect it to be. :raises NoDropboxDirError: When local Dropbox directory does not exist. .. py:method:: ensure_cache_dir_present() Checks for or creates a directory at :attr:`file_cache_path`. :raises CacheDirError: When local cache directory cannot be created. .. py:method:: clean_cache_dir(raise_error = True) Removes all items in the cache directory. :param raise_error: Whether errors should be raised or only logged. .. py:method:: correct_case(dbx_path_basename_cased) Converts a Dropbox path with correctly cased basename to a fully cased path. This is useful because the Dropbox API guarantees the correct casing for the basename only. In practice, casing of parent directories is often incorrect. This method retrieves the correct casing of all ancestors in the path, either from our cache, our database, or from Dropbox servers. Performance may vary significantly with the number of parent folders and the method used to resolve the casing of all parent directory names: 1) If the parent directory is already in our cache, performance is O(1). 2) If the parent directory is already in our sync index, performance is slower because it requires a sqlite query but still O(1). 3) If the parent directory is unknown to us, its metadata (including the correct casing of the directory's basename) is queried from Dropbox. This is used to construct a correctly cased path by calling :meth:`correct_case` again. At best, performance will be of O(2) if the parent directory is known to us, at worst it will be of order O(N) involving queries to Dropbox servers for each of the N parent directories. When calling :meth:`correct_case` repeatedly for paths from the same tree, it is therefore best to do so in top-down traversal. :param dbx_path_basename_cased: Dropbox path with correctly cased basename, as provided by :attr:`dropbox.files.Metadata.path_display` or :attr:`dropbox.files.Metadata.name`. :returns: Correctly cased Dropbox path. .. py:method:: to_dbx_path(local_path) Converts a local path to a path relative to the Dropbox folder. Casing of the given ``local_path`` will be preserved. :param local_path: Absolute path on local drive. :returns: Relative path with respect to Dropbox folder. :raises ValueError: When the path lies outside the local Dropbox folder. .. py:method:: to_dbx_path_lower(local_path) Converts a local path to a path relative to the Dropbox folder. The path will be normalized as on Dropbox servers (lower case and some additional normalisations). :param local_path: Absolute path on local drive. :returns: Relative path with respect to Dropbox folder. :raises ValueError: When the path lies outside the local Dropbox folder. .. py:method:: to_local_path_from_cased(dbx_path_cased) Converts a correctly cased Dropbox path to the corresponding local path. This is more efficient than :meth:`to_local_path` which accepts uncased paths. :param dbx_path_cased: Path relative to Dropbox folder, correctly cased. :returns: Corresponding local path on drive. .. py:method:: to_local_path(dbx_path) Converts a Dropbox path to the corresponding local path. Only the basename must be correctly cased, as guaranteed by the Dropbox API for the ``display_path`` attribute of file or folder metadata. This method slower than :meth:`to_local_path_from_cased`. :param dbx_path: Path relative to Dropbox folder, must be correctly cased in its basename. :returns: Corresponding local path on drive. .. py:method:: is_excluded(path) Checks if a file is excluded from sync. Certain file names are always excluded from syncing, following the Dropbox support article: https://help.dropbox.com/installs-integrations/sync-uploads/files-not-syncing This includes file system files such as 'desktop.ini' and '.DS_Store' and some temporary files as well as caches used by Dropbox or Maestral. `is_excluded` accepts both local and Dropbox paths. :param path: Can be an absolute path, a path relative to the Dropbox folder or just a file name. Does not need to be normalized. :returns: Whether the path is excluded from syncing. .. py:method:: is_excluded_by_user(dbx_path_lower) Check if file has been excluded through "selective sync" by the user. :param dbx_path_lower: Normalised lower case Dropbox path. :returns: Whether the path is excluded from download syncing by the user. .. py:method:: is_mignore(event) Check if local file change has been excluded by a mignore pattern. :param event: SyncEvent for local file event. :returns: Whether the path is excluded from upload syncing by the user. .. py:method:: cancel_sync() Raises a :exc:`maestral.exceptions.CancelledError` in all sync threads and waits for them to shut down. .. py:method:: busy() Checks if we are currently syncing. :returns: ``True`` if :attr:`sync_lock` cannot be acquired, ``False`` otherwise. .. py:method:: upload_local_changes_while_inactive() Collects changes while sync has not been running and uploads them to Dropbox. Call this method when resuming sync. .. py:method:: wait_for_local_changes(timeout = 40) Blocks until local changes are available. :param timeout: Maximum time in seconds to wait. :returns: ``True`` if changes are available, ``False`` otherwise. .. py:method:: upload_sync_cycle() Performs a full upload sync cycle by calling in order: 1) :meth:`list_local_changes` 2) :meth:`apply_local_changes` Handles updating the local cursor for you. If monitoring for local file events was interrupted, call :meth:`upload_local_changes_while_inactive` instead. .. py:method:: list_local_changes(delay = 1) Returns a list of local changes with at most one entry per path. :param delay: Delay in sec to wait for subsequent changes before returning. :returns: (list of sync times events, time_stamp) .. py:method:: apply_local_changes(sync_events) Applies locally detected changes to the remote Dropbox. Changes which should be ignored (mignore or always ignored files) are skipped. :param sync_events: List of local file system events. .. py:method:: get_remote_item(dbx_path) Downloads a remote file or folder and updates its local rev. If the remote item does not exist, any corresponding local items will be deleted. If ``dbx_path`` refers to a folder, the download will be handled by :meth:`_get_remote_folder`. If it refers to a single file, the download will be performed by :meth:`_create_local_entry`. This method can be used to fetch individual items outside the regular sync cycle, for instance when including a previously excluded file or folder. :param dbx_path: Path relative to Dropbox folder. :returns: Whether download was successful. .. py:method:: wait_for_remote_changes(last_cursor, timeout = 40) Blocks until changes to the remote Dropbox are available. :param last_cursor: Cursor form last sync. :param timeout: Timeout in seconds before returning even if there are no changes. Dropbox adds random jitter of up to 90 sec to this value. :returns: ``True`` if changes are available, ``False`` otherwise. .. py:method:: download_sync_cycle() Performs a full download sync cycle by calling in order: 1) :meth:`list_remote_changes_iterator` 2) :meth:`apply_remote_changes` Handles updating the remote cursor and resuming interrupted syncs for you. Calling this method will perform a full indexing if this is the first download. .. py:method:: list_remote_changes_iterator(last_cursor) Get remote changes since the last download sync, as specified by ``last_cursor``. If the ``last_cursor`` is from paginating through a previous set of changes, continue where we left off. If ``last_cursor`` is an empty string, perform a full indexing of the Dropbox folder. :param last_cursor: Cursor from last download sync. :returns: Iterator yielding tuples with remote changes and corresponding cursor. .. py:method:: apply_remote_changes(sync_events) Applies remote changes to local folder. Call this on the result of :meth:`list_remote_changes`. The saved cursor is updated after a set of changes has been successfully applied. Entries in the local index are created after successful completion. :param sync_events: List of remote changes. :returns: List of changes that were made to local files and bool indicating if all download syncs were successful. .. py:method:: notify_user(sync_events) Shows a desktop notification for the given file changes. :param sync_events: List of SyncEvents from download sync. .. py:method:: rescan(local_path) Forces a rescan of a local path: schedules created events for every folder, modified events for every file and deleted events for every deleted item (compared to our index). :param local_path: Path to rescan. .. py:class:: pf_repr(obj) Class that wraps an object and creates a pretty formatted representation for it. This can be used to get pretty formatting in log messages while deferring the actual formatting until the message is created. :param obj: Object to wrap. .. py:attribute:: obj