Encoding (#35)

* Remove trailing garbage from output lines

* Add an on_error hook that allows to ignore handler exceptions

Update #30

* Prefer https links in comments and docs

* Cleanup

* Start using utf-8

* Update docs

* Honor server time offset and send OPTS UTF8

* Fix json.dumps for Py2

* Fix for Py2

* Encode on py27

* black

* Use unicode internally

* Back to storing native strings internally

* Fix for Py2

* Update docs

* Fix metadata keys for Py2

* Prepare 3.0.0.a1

.pydevproject

@@ -1,10 +1,8 @@
 <?xml version="1.0" encoding="UTF-8" standalone="no"?>
-<?eclipse-pydev version="1.0"?>
-
-<pydev_project>
+<?eclipse-pydev version="1.0"?><pydev_project>
 <pydev_pathproperty name="org.python.pydev.PROJECT_SOURCE_PATH">
 <path>/pyftpsync</path>
 </pydev_pathproperty>
 <pydev_property name="org.python.pydev.PYTHON_PROJECT_VERSION">python 2.7</pydev_property>
-<pydev_property name="org.python.pydev.PYTHON_PROJECT_INTERPRETER">Default</pydev_property>
+<pydev_property name="org.python.pydev.PYTHON_PROJECT_INTERPRETER">C:\Users\Martin\.virtualenvs\pyftpsync-f-IJ7A-L\Scripts\python.exe</pydev_property>
 </pydev_project>

.settings/org.eclipse.core.resources.prefs

@@ -1,12 +1,15 @@
 eclipse.preferences.version=1
+encoding//build/lib/ftpsync/ftp_target.py=utf-8
 encoding//docs/conf.py=utf-8
+encoding//ftpsync/cli_common.py=utf-8
 encoding//ftpsync/compat.py=iso-8859-1
-encoding//ftpsync/ftp_target.py=iso-8859-1
-encoding//ftpsync/metadata.py=iso-8859-1
-encoding//ftpsync/pyftpsync.py=iso-8859-1
-encoding//ftpsync/resources.py=iso-8859-1
-encoding//ftpsync/scan_command.py=iso-8859-1
-encoding//ftpsync/synchronizers.py=iso-8859-1
-encoding//ftpsync/targets.py=iso-8859-1
-encoding//ftpsync/util.py=iso-8859-1
+encoding//ftpsync/ftp_target.py=utf-8
+encoding//ftpsync/metadata.py=utf-8
+encoding//ftpsync/pyftpsync.py=utf-8
+encoding//ftpsync/resources.py=utf-8
+encoding//ftpsync/scan_command.py=utf-8
+encoding//ftpsync/synchronizers.py=utf-8
+encoding//ftpsync/targets.py=utf-8
+encoding//ftpsync/util.py=utf-8
 encoding//src/service.py=iso-8859-1
+encoding//test/debug_adhoc.py=utf-8

CHANGES.md

@@ -1,6 +1,23 @@
-2.1.1 (unreleased)
+3.0.0 (unreleased)
 ------------------
-- 
+- This release addresses some known encoding-related issues:
+  - The internal path format are now native strings (i.e. unicode on Python 3 
+    or UTF-8 bytes on Python 2)
+  - FTP targets are now assumed to support UTF-8.
+    **TODO:** pass `--remote-encoding` to override this.
+  - #30: Fallback to CP-1252 encoding when FTP server returns non-UTF-8
+  - Local filesystem targets now consider the OS encoding.
+  - Modified format of `.pyftpsync-meta.json`: File names are now stored as UTF-8
+    (was the unmodified binary format of the target platform before).
+  - See also the 'encoding' section in the 
+    [spec](https://github.com/mar10/pyftpsync/blob/master/docs/sphinx/pyftpsync-spec.pdf).
+
+- Remove trailing garbage from output lines
+
+**Breaking Changes:**
+  - Modified format of `.pyftpsync-meta.json`.
+    Pass `--migrate` option to convert from a prvious version (note that this
+    cannot be undone)
 
 2.1.0 (2018-08-25)
 ------------------

Pipfile

@@ -8,6 +8,7 @@ flake8-bugbear = "~=18.2"
 flake8-quotes = "~=1.0"
 black = "*"
 isort = "*"
+pylint = "*"
 
 [packages]
 colorama = "==0.3.9"

README.md

@@ -2,13 +2,14 @@
 [![Build Status](https://travis-ci.org/mar10/pyftpsync.svg?branch=master)](https://travis-ci.org/mar10/pyftpsync)
 [![Latest Version](https://img.shields.io/pypi/v/pyftpsync.svg)](https://pypi.python.org/pypi/pyftpsync/)
 [![License](https://img.shields.io/pypi/l/pyftpsync.svg)](https://github.com/mar10/pyftpsync/blob/master/LICENSE.txt)
-[![Documentation Status](https://readthedocs.org/projects/pyftpsync/badge/?version=latest)](http://pyftpsync.readthedocs.io/)
+[![Documentation Status](https://readthedocs.org/projects/pyftpsync/badge/?version=latest)](https://pyftpsync.readthedocs.io/)
 [![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/ambv/black)
 
 > Synchronize directories using FTP(S) or file system access.
 
 [ ![sample](teaser.png?raw=true) ](https://github.com/mar10/pyftpsync "Live demo")
 
+
 ## Summary
 
 Synchronize directories using FTP(S) or file system access.
@@ -25,15 +26,18 @@ Make sure to adjust your scripts accordingly after update.
 
 ## Quickstart
 
-[Python](http://www.python.org/download/Python) 2.7+ or 3.4+ is required,
+[Python](https://www.python.org/download/Python) 2.7+ or 3.4+ is required,
 [pip](http://www.pip-installer.org/) recommended:
 
 ```bash
 $ pip install pyftpsync --upgrade
 $ pyftpsync --help
 ```
 
+**Note:** Windows users may prefer the 
+[MSI Installer](https://github.com/mar10/pyftpsync/releases/latest).
+
 
 ## Documentation
 
-[Read the Docs](http://pyftpsync.readthedocs.io/) for details.
+[Read the Docs](https://pyftpsync.readthedocs.io/) for details.

docs/sphinx/index.rst

@@ -45,10 +45,11 @@ Features
   * Recursive synchronisation of folders on file system and/or FTP targets.
   * Upload, download, and bi-directional synchronization mode.
   * Configurable conflict resolution strategies.
-  * Unlike naive implementations, pyftpsync maintains additional meta data to detect
-    conflicts and decide wether to replicate deletions or additions.
-  * Unlike more complex implementations, pyftpsync does not require a database or
-    a service running on the targets.
+  * Unlike naive implementations, pyftpsync maintains additional meta data to
+    detect conflicts and decide whether to replicate a missing file as deletion
+    or addition.
+  * Unlike more complex implementations, pyftpsync does not require a database
+    or a service running on the targets.
   * Optional FTPS (TLS) support.
   * Architecture is open to add other target types.
 
@@ -62,7 +63,7 @@ Features
 
 .. note:: Known Limitations
 
-  * The FTP server must support the `MLSD command <http://tools.ietf.org/html/rfc3659>`_.
+  * The FTP server must support the `MLSD command <https://tools.ietf.org/html/rfc3659>`_.
   * pyftpsync uses file size and modification dates to detect file changes.
     This is efficient, but not as robust as CRC checksums could be.
   * pyftpsync tries to detect conflicts (i.e. simultaneous modifications of
@@ -107,5 +108,5 @@ be installed using `pip <http://www.pip-installer.org/>`_::
    :target: https://github.com/mar10/pyftpsync/blob/master/LICENSE.txt
 
 .. |rtd_badge| image:: https://readthedocs.org/projects/pyftpsync/badge/?version=latest
-   :target: http://pyftpsync.readthedocs.io/
+   :target: https://pyftpsync.readthedocs.io/
    :alt: Documentation Status

docs/sphinx/make.bat

@@ -65,7 +65,7 @@ if errorlevel 9009 (
 	echo.may add the Sphinx directory to PATH.
 	echo.
 	echo.If you don't have Sphinx installed, grab it from
-	echo.http://sphinx-doc.org/
+	echo.https://sphinx-doc.org/
 	exit /b 1
 )
 

ftpsync/__init__.py

@@ -13,4 +13,4 @@
     Developmental release (to mark 3.0.0 as 'used'. Don't publish this):
         '3.0.0.dev1'
 """
-__version__ = "2.1.0.dev1"
+__version__ = "3.0.0.a1"

ftpsync/cli_common.py

@@ -1,7 +1,7 @@
 # -*- coding: utf-8 -*-
 """
 (c) 2012-2019 Martin Wendt; see https://github.com/mar10/pyftpsync
-Licensed under the MIT license: http://www.opensource.org/licenses/mit-license.php
+Licensed under the MIT license: https://www.opensource.org/licenses/mit-license.php
 """
 from __future__ import print_function
 

ftpsync/compat.py

@@ -1,7 +1,7 @@
 # -*- coding: utf-8 -*-
 """
 (c) 2012-2019 Martin Wendt; see https://github.com/mar10/pyftpsync
-Licensed under the MIT license: http://www.opensource.org/licenses/mit-license.php
+Licensed under the MIT license: https://www.opensource.org/licenses/mit-license.php
 """
 # flake8: noqa
 
@@ -75,7 +75,7 @@ def is_unicode(s):
         """Return True for unicode strings, i.e. for unicode on Py2 and str on Py3."""
         return isinstance(s, unicode)
 
-    def to_bytes(s, encoding="utf8"):
+    def to_bytes(s, encoding="utf-8"):
         """Convert unicode (text strings) to binary data, i.e. str on Py2 and bytes on Py3."""
         if type(s) is unicode:
             s = s.encode(encoding)
@@ -86,7 +86,7 @@ def to_bytes(s, encoding="utf8"):
     to_native = to_bytes
     """Convert data to native str type, i.e. bytestring on Py2 and unicode on Py3."""
 
-    def to_unicode(s, encoding="utf8"):
+    def to_unicode(s, encoding="utf-8"):
         """Convert data to unicode text, i.e. unicode on Py2 and str on Py3."""
         if type(s) is not unicode:
             s = unicode(s, encoding)
@@ -115,13 +115,13 @@ def is_unicode(s):
         """Return True for unicode strings, i.e. for unicode on Py2 and str on Py3."""
         return isinstance(s, str)
 
-    def to_bytes(s, encoding="utf8"):
+    def to_bytes(s, encoding="utf-8"):
         """Convert a text string (unicode) to bytestring, i.e. str on Py2 and bytes on Py3."""
         if type(s) is not bytes:
             s = bytes(s, encoding)
         return s
 
-    def to_native(s, encoding="utf8"):
+    def to_native(s, encoding="utf-8"):
         """Convert data to native str type, i.e. bytestring on Py2 and unicode on Py3."""
         if type(s) is bytes:
             s = str(s, encoding)