Python 3 support for Dragonfly + NatLink

[bijanbina]

First of all thanks for all help of the community, dragonfly is one of the best open-source projects I've seen.

I'm trying to use dragonfly with DNS but I've been lost. In the Installation section in the document, it has been mentioned to install

Python, v2.7 or later – available from the Python dowloads page.

Does this support python3? I think the doc is a bit unclear about that. Also, I have seen many typos in the document (like download in the above sentence). I would like to take the time and fix them. Can I have access?

I tried python2.7.18 but in the pip installation regex installation failed. I think because python 2.7 has reached its end of life.

[LexiconCode]

This only really affects DNS/DPI as they are required to use Python 32-bit 2.7.x due to its dependence on Natlink. Natlink is working towards Python 3 support. Dragonfly supports Python 3 64 bit in all other engines.

Microsoft has removed the following Compilers for Python 2.7 as it is no longer supported. It is no longer available to download officially from Microsoft. regex cannot compile error: Microsoft Visual C++ 9.0 under Python 2.7.x.

Work around with pre-compiled package pip install https://download.lfd.uci.edu/pythonlibs/w4tscw6k/cp27/regex-2020.1.8-cp27-cp27m-win32.whl

[drmfinlay]

Hi @bijanbina,

I'm glad you have been helped out by the community around Dragonfly and similar projects. We do our best, however I think it is an exaggeration to say it is one of the best open-source projects out there.

As LexiconCode said, Python 2.7 is only required for using Dragonfly with DNS. I can say that the experimental Natlink support for Python 3 seems to work well with Dragonfly. You can find out more about that in the development channel for natlink. Whichever major version is used, Natlink also requires 32-bit Python.

I would welcome improvements to Dragonfly's documentation. It can be difficult to keep it orderly and up-to-date. Changes are typically submitted through GitHub pull requests, rather than direct branch access.

[bijanbina]

@LexiconCode Thanks for your help

Look like wxPython needed to be installed too. Thanks to your tip found the package here and installed it with

pip install https://download.lfd.uci.edu/pythonlibs/w4tscw6k/cp27/wxPython-3.0.2.0-cp27-none-win32.whl

Unfortunately, it didn't help me get past of the following error when executing start_configurenatlink.py

Try to run configurenatlink.py, the Natlink Config GUI, in Elevated mode...

Unable to run the GUI configuration program of NatLink/Unimacro/Vocola
because module wx was not found.  This probably
means that wxPython is not installed correct:

Look like the only option to use Dragonfly with DNS is to upgrade to python3

[drmfinlay]

@bijanbina Maybe try the wxPython3.0-win32-3.0.2.0-py27.exe file available here.

[LexiconCode]

@bijanbina connect with me over Gitter and I'm willing to help you out through some remote support using team viewer quick support

[bijanbina]

Cheers!
Finally able to run Natlink. @Danesprite , Thanks to your help now the wxPython error is gone.
@LexiconCode, Thanks for your kind support. So for the time being Natlink only supports python2.
For any other user, as LexiconCode said here, python3 version is unstable.

[drmfinlay]

Glad we could help!

Stable Python 3 support for Natlink is getting there. It is a difficult process, unfortunately. I'll give this issue a more appropriate name and leave it open for now.

I'll also update the documentation to mention this.

[LexiconCode]

Unfortunately the direct links to the regex package above expire. For an end-user not too big of an issue but it does add extra steps. The following:

• Due to Python 2.7 a precompiled regex package needs to be installed.
  • Search for and download regex file ending in cp27-cp27m-win32.whl
  • Command prompt example pip install regex-2020.1.8-cp27-cp27m-win32.whl from the download location.

This is more complicated which I haven't quite figured out for continuous integration needing to obtain the regex package.

[drmfinlay]

@LexiconCode
I have also had trouble with this package in other scenarios using Python 3. I think the best thing to do here is to remove the regex package as a dependency and change the relevant code under dragonfly.accessibility to use the built-in re module instead. I will do this in the next (patch) version.

[drmfinlay]

I have added the following paragraph in the relevant documentation sections:

Python version 2.7 (32-bit) is required if using the Natlink engine back-end, at least for the moment. Support for this version is not maintained for the other engine back-ends and will be dropped completely in the first MAJOR release following stable Natlink support for Python 3.

Does this look good/reasonable?

[LexiconCode]

Unfortunately the direct links to the regex package above expire. For an end-user not too big of an issue but it does add extra steps. The following:

* Due to Python 2.7 a precompiled [regex](https://www.lfd.uci.edu/~gohlke/pythonlibs/#regex) package needs to be installed.
  
  * Search for and download `regex` file ending in `cp27-cp27m-win32.whl`
  * Command prompt example `pip install regex-2020.1.8-cp27-cp27m-win32.whl` from the download location.

This is more complicated which I haven't quite figured out for continuous integration needing to obtain the regex package.

Install Microsoft Visual C++ Compiler for Python 2.7 from web.archive.org mirror

[drmfinlay]

I have also had trouble with this package in other scenarios using Python 3. I think the best thing to do here is to remove the regex package as a dependency and change the relevant code under dragonfly.accessibility to use the built-in re module instead. I will do this in the next (patch) version.

Both myself and Aaron have found this rather difficult. I'm afraid this dependency will have to stay in for the moment, unfortunately. I will make it an optional "extra" dependency for the functionality under dragonfly.accessibility in version 1.0.0 (see issue #238).

The other troubles with this package I referred to above are edge cases arising from the decision of the regex package maintainers to bundle their internal Python extension module, i.e. a Python module written in C, with the Python code that uses it. This is not a problem unless you use both 32-bit and 64-bit builds of a particular version of Python, in my case Python 3.7.

[LexiconCode]

Here is some beta instructions to install Natlink on python 3 32-bit with PIP.

These instructions work for any natlink based project like dragonfly or caster.

Preinstall requirements

• DPI 15 or derivative of the same version
• Python 3.8.X 32 bit is required.
• Make sure any previous versions of Natlink are unregistered and uninstalled. (Dragon must be close during that process)
• Delete caster settings in user directory
• (optionally) If you have tried the the Natlink test package and/or main Natlink repository make sure there's no remnants derivatives of Natlink in the site packages of Python.

Natlink Install Instructions

1. close Dragon
2. open cmd/power shell as administrator
3. pip install natlink
4. natlinkconfig_cli # should auto setup and register itself.
5. type n C:\Backup\Library\Documents\Caster' to set the natlink user directory. Modify to wherever
6. restart Dragon

Extra commands if needed

• type u for to see commandline options
• type r or R to register/unregister natlink
• type e or E - enable/disable Natlink
• type n or N to set/clear the natlink user directory

Info

• If natlink is properly registered natlink.pyd file path location stored C:\Users\Your-User\.natlink or DICTATIONTOOLBOXUSER\.natlink environment variable`.

Troubleshooting Please post here if you have any issues.
On non-administrator accounts:

• You may need to manually set or delete natlink.pyd as administrator
• Running terminal as administrator changes the user account causing a mismatch between user directories between administrator/non-administrator. This impacts where your settings are stored for natlink.
  • Fix:- Create an OS environment variable DICTATIONTOOLBOXUSER pointing to a directory to store .natlink. After upgrading to natlink 5.1.3 and run natlinkconfig_cli. if needed "r" re-register.

[drmfinlay]

Thank you for these instructions, Aaron!

[drmfinlay]

I'd like to say here that I have changed my mind on dropping Python 2.7 support for Dragonfly in the future. Since this is an old, basically complete library that has always worked with Python 2 and I'm familiar enough with supporting both major versions, I'm happy to continue support for both 2.7 and 3.x.

This would be for the core functionality (i.e. non-optional features), the Natlink, SAPI 5 and Sphinx engine implementations, and the dragonfly.accessibility sub-package. The exceptions to this are as follows:

1. The Kaldi engine back-end requires Python 3, unless @daanzu wishes otherwise.
2. The Talon integration, after 326 is merged, will require Python 3.

I'll document these exceptions somewhere visible and, perhaps, include a succinct explanation as to why, excluding the above exceptions, we are ignoring the Python 2 EOL.

[drmfinlay]

I'm not sure why I closed this issue. @LexiconCode Am I right in thinking there is still no stable version of Natlink for Python 3?

[LexiconCode]

I would say the current release that's out there is usable 14 through 15. Many people have used natlink installer without issue.

The only aspect I would consider unstable would be DNS 13. Which my PR seems to resolve despite it being workaround.

If all goes to plan the next release for include DNS 16 support and the patch for DNS 13

Setup process can lead to some edge cases. Therefore, I'm working on redoing the setup process without the installer.

[drmfinlay]

Thank you for the quick reply. Good to hear about the fixes for DNS 13 and 16. I need to know what link to use in Dragonfly's documentation, however. Should it be this one, the GitHub releases page or something else?

[LexiconCode]

Thank you for the quick reply. Good to hear about the fixes for DNS 13 and 16. I need to know what link to use in Dragonfly's documentation, however. Should it be this one, the GitHub releases page or something else?

The documentation needs to be updated on the master branch. However, I can confidently say we should use GitHub Natlink documentation for installation.

The original website has some good information but it should be updated, distilled and brought into the repository. Which of course could be referenced by the website as needed.

[drmfinlay]

All right then. So the Natlink repository's readme file, once it is updated? For the time being, this page seems clearer to me, although there is obviously something wrong with the Markdown links. I should be able to help fix that, if you like.

As stated in earlier comments, I would like to keep Python 2.7 support around for Dragonfly. I'll retain the link to the old install instructions and add a new link for Python 3 users.

[LexiconCode]

Yeah that would be helpful fixing the links. Sounds like a plan!