Windows install requires additional software

[whtowbin]

Hi Patricio,
It seems there is a problem installing MC3 on Windows via pip or Conda. It gives an error about needing a C++ compiler. It might be helpful to have a note for Windows users in the getting started section about how to resolve this.

The error is this:

  building 'mc3.lib.timeavg' extension
  error: Microsoft Visual C++ 14.0 or greater is required. Get it with "Microsoft C++ Build Tools": https://visualstudio.microsoft.com/visual-cpp-build-tools/

This can be easily rectified by installing the latest version from this page and then retrying the pip or Conda install. I found this by looking at answers here.

Thanks for making such a great package,
Henry

[norberttoth398]

This error can persist even after installing the proposed software. I found the following error message:

error: command 'C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Tools\MSVC\14.35.32215\bin\HostX86\x64\cl.exe' failed with exit code 2
[end of output]

This error may be unrelated to the above but regardless, there now seems to be an issue with building mc3 wheel from the source files. Any help would be much appreciated.

[sarahshi]

Hi Patricio,

Hope you are well. I was wondering if you had any insight on how we can potentially help with getting a Windows wheel built for mc3? We are preparing to submit the paper now and some Windows users have already reported this issue to us, preventing them from using the package and prompting potential rewrite. We previously had the solution of installing a Windows Subsystem for Linux (WSL), which worked for a test user, but later stopped working. If it is possible -- not sure if the barrier to entry is too high -- we would be very happy to contribute to this effort. Thank you!

Best,
Sarah

[pcubillos]

Hi, yeah I don't think I can be of much help here. I haven't owned a Windows machine in years so I'm not familiar with how to do the proper packaging for that nor I have the means to implement/test. I think somewhere in the docs it says it's only tested for unix and OSX.
Certainly, if you find a workaround, I'd be happy to add that to the documentation and/or code base.

[sarahshi]

Hi Patricio,

@whtowbin and I may have found a solution. cibuildwheel seems to offer a solution to this, all done within GitHub Actions. I have locally forked a version that is nearly working -- unsurprisingly perhaps, the OSX and Linux wheels are being built. The primary issue seems to lie within the different compilers of using the GCC vs. MVCC (Microsoft Visual C++ Compiler). MVCC is far less permissive with type safety than GCC. I locally have a forked version of mc3 with some added cibuildwheel that is currently working for Mac/Linux. However, MVCC is raising these following errors — see in actions here:

src_c/include/stats.h(98): error C2036: 'void *': unknown size
src_c/include/stats.h(99): error C2036: 'void *': unknown size
src_c/include/stats.h(102): error C2036: 'void *': unknown size
src_c/include/stats.h(103): error C2036: 'void *': unknown size
src_c/include/stats.h(105): error C2036: 'void *': unknown size
src_c/_binarray.c(57): error C2036: 'void *': unknown size
src_c/_binarray.c(59): error C2036: 'void *': unknown size
src_c/_binarray.c(60): error C2036: 'void *': unknown size
src_c/_binarray.c(70): error C2036: 'void *': unknown size
src_c/_binarray.c(71): error C2036: 'void *': unknown size
src_c/_binarray.c(73): error C2036: 'void *': unknown size
src_c/_binarray.c(74): error C2036: 'void *': unknown size
src_c/_binarray.c(76): error C2036: 'void *': unknown size
src_c/_binarray.c(77): error C2036: 'void *': unknown size

It looks like all of these lines contain INDd. If these void pointers are cast to pointers of the correct type before performing any arithmetic, we should be able to get a Windows wheel going!

Best,
Sarah and @whtowbin

[whtowbin]

Hi @pcubillos ,
@sarahshi fixed many of the issues with the Windows C compiler by modifying the definitions of INDd and INDi. She also fixed some a minor problem with the definition of M_Pi since Windows doesn't define it by default. Most of the other commits she has made are to set up the build process for cibuildwheel.
The wheel seems to build properly, but testing fails due to an issue with how Multiprocessing handles the chains. Apparently Windows only allows for "Spawing" new Python interpreters and not "Forking" them. I'm not sure if there is an easy solution since this will affect how data is passed from one chain to another. Are there any shared variables that the chains need to have access to or can they all append their data after they each run? It seems like you brought this up in a previous issue.
I am not super familiar with how multiprocessing works but please let me and @sarahshi if there is a simple fix we can assist with.
Thanks,
Henry

[pcubillos]

Hi Henry, Sarah,
This is really great! Sorry I haven't had much time to look over the edits lately. I will check it later today. Thanks

[pcubillos]

Hi, sorry I didn't had time to go over this in detail (and I'll be out of the office for a week). But I'd be excited to add your edits into the code base once the we make sure it works well on all machines.
Have you figured out the issue with multiprocessing? The if/else you added for fork vs spawn sounds like the right way to solve it. The reason I enforced forking was to allow the code to work in interactive runs in OS X, so if spawning allows the code to work on Windows, then that should be the way to go when running on Win, I'd say.

[sarahshi]

Hi Patricio,

I am currently in the field so not able to do more testing right now. The if/else statement does not quite solve things yet, as forking/spawning changes how pytest works. I might need your guidance on how to best modify this bit of code. I do not entirely know what information is needed for the pytests. You can see this error in the latest actions in my fork of mc3.

Best,
Sarah

[sarahshi]

Hi Patricio,

Hope you're well. Have you had a chance to look through the modifications? I still need a bit of your guidance on how to modify forking/spawning for pytest. I would be happy to hop on a Zoom sometime next week if that suits you.

Best,
Sarah

[norberttoth398]

Hi all

Using @sarahshi 's fork I was able to install mc3 and run it on my windows machine.

@pcubillos could the main repo be updated to reflect this?

Thanks
Norbert

[sarahshi]

Hi All,

Scratch that, it looks like spawning on Windows still doesn't get around some of the issues. @norberttoth398 found different results between the Linux and Windows runs. Patricio, can you please offer some clarity on what information from different chains is required to be passed around? I assume it may be best to just enforce spawn across all and figure out how to save or log the information that is actually required.

Cheers,
Sarah

[pcubillos]

Hi all, sorry that I haven't been able to keep up with this. I'm right in the middle of (JWST) proposals season. @sarahshi : yes, we can schedule a meeting, please email me at pcubillos at fulbrightmail dot org to arrange things

[pcubillos]

Hi Sarah, Henry,
I've been getting up to speed with this lately. I got over the error C2036: 'void *': unknown size with the build commands you suggested (and also following the edits you made in ind.h, thanks!).

Now I'm facing a different error here below, it doesn't sound like the one you had, right?

Details

  
  + pip install virtualenv -c 'C:\hostedtoolcache\windows\Python\3.10.11\x64\Lib\site-packages\cibuildwheel\resources\constraints-python310.txt'
  Collecting virtualenv
    Downloading virtualenv-20.25.1-py3-none-any.whl.metadata (4.4 kB)
  Collecting distlib<1,>=0.3.7 (from virtualenv)
    Downloading distlib-0.3.8-py2.py3-none-any.whl.metadata (5.1 kB)
  Collecting filelock<4,>=3.12.2 (from virtualenv)
    Downloading filelock-3.13.1-py3-none-any.whl.metadata (2.8 kB)
  Collecting platformdirs<5,>=3.9.1 (from virtualenv)
    Using cached platformdirs-4.2.0-py3-none-any.whl.metadata (11 kB)
  Downloading virtualenv-20.25.1-py3-none-any.whl (3.8 MB)
     ---------------------------------------- 3.8/3.8 MB 34.3 MB/s eta 0:00:00
  Downloading distlib-0.3.8-py2.py3-none-any.whl (468 kB)
     --------------------------------------- 468.9/468.9 kB 14.8 MB/s eta 0:00:00
  Downloading filelock-3.13.1-py3-none-any.whl (11 kB)
  Using cached platformdirs-4.2.0-py3-none-any.whl (17 kB)
  Installing collected packages: distlib, platformdirs, filelock, virtualenv
  Successfully installed distlib-0.3.8 filelock-3.13.1 platformdirs-4.2.0 virtualenv-20.25.1
  + python -m virtualenv --no-download 'C:\Users\runneradmin\AppData\Local\Temp\cibw-run-mwn9orwe\cp310-win32\venv-test'
  created virtual environment CPython3.10.11.final.0-32 in 1051ms
    creator CPython3Windows(dest=C:\Users\runneradmin\AppData\Local\Temp\cibw-run-mwn9orwe\cp310-win32\venv-test, clear=False, no_vcs_ignore=False, global=False)
    seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=C:\Users\runneradmin\AppData\Local\pypa\virtualenv)
      added seed packages: pip==24.0, setuptools==69.1.0, wheel==0.42.0
    activators BashActivator,BatchActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator
  + where python
  C:\Users\runneradmin\AppData\Local\Temp\cibw-run-mwn9orwe\cp310-win32\venv-test\Scripts\python.exe
  C:\Users\runneradmin\AppData\Local\Temp\cibw-run-mwn9orwe\cp310-win32\build\venv\Scripts\python.exe
  C:\hostedtoolcache\windows\Python\3.10.11\x64\python.exe
  C:\hostedtoolcache\windows\Python\3.9.13\x64\python.exe
  + pip install 'C:\Users\runneradmin\AppData\Local\Temp\cibw-run-mwn9orwe\cp310-win32\repaired_wheel\mc3-3.1.3-cp310-cp310-win32.whl'
  Processing c:\users\runneradmin\appdata\local\temp\cibw-run-mwn9orwe\cp310-win32\repaired_wheel\mc3-3.1.3-cp310-cp310-win32.whl
  Collecting numpy>=1.19.5 (from mc3==3.1.3)
    Using cached numpy-1.26.4-cp310-cp310-win32.whl.metadata (61 kB)
  Collecting scipy>=1.5.4 (from mc3==3.1.3)
    Downloading scipy-1.13.0.tar.gz (57.2 MB)
       ---------------------------------------- 57.2/57.2 MB 3.7 MB/s eta 0:00:00
    Installing build dependencies: started
    Installing build dependencies: finished with status 'done'
    Getting requirements to build wheel: started
    Getting requirements to build wheel: finished with status 'done'
    Installing backend dependencies: started
    Installing backend dependencies: finished with status 'done'
    Preparing metadata (pyproject.toml): started
    Preparing metadata (pyproject.toml): finished with status 'error'
    error: subprocess-exited-with-error
    
    Preparing metadata (pyproject.toml) did not run successfully.
    exit code: 1
    
    [22 lines of output]
    + meson setup C:\Users\runneradmin\AppData\Local\Temp\pip-install-r63inwms\scipy_a36527f3a56348b6a1d252b7ee532fde C:\Users\runneradmin\AppData\Local\Temp\pip-install-r63inwms\scipy_a36527f3a56348b6a1d252b7ee532fde\.mesonpy-g37647tf -Dbuildtype=release -Db_ndebug=if-release -Db_vscrt=md --native-file=C:\Users\runneradmin\AppData\Local\Temp\pip-install-r63inwms\scipy_a36527f3a56348b6a1d252b7ee532fde\.mesonpy-g37647tf\meson-python-native-file.ini
    The Meson build system
    Version: 1.4.0
    Source dir: C:\Users\runneradmin\AppData\Local\Temp\pip-install-r63inwms\scipy_a36527f3a56348b6a1d252b7ee532fde
    Build dir: C:\Users\runneradmin\AppData\Local\Temp\pip-install-r63inwms\scipy_a36527f3a56348b6a1d252b7ee532fde\.mesonpy-g37647tf
    Build type: native build
    Project name: scipy
    Project version: 1.13.0
    C compiler for the host machine: gcc (gcc 12.2.0 "gcc (x86_64-posix-seh-rev2, Built by MinGW-W64 project) 12.2.0")
    C linker for the host machine: gcc ld.bfd 2.39
    C++ compiler for the host machine: c++ (gcc 12.2.0 "c++ (x86_64-posix-seh-rev2, Built by MinGW-W64 project) 12.2.0")
    C++ linker for the host machine: c++ ld.bfd 2.39
    Cython compiler for the host machine: cython (cython 3.0.10)
    Host machine cpu family: x86_64
    Host machine cpu: x86_64
    Program python found: YES (C:\Users\runneradmin\AppData\Local\Temp\cibw-run-mwn9orwe\cp310-win32\venv-test\Scripts\python.exe)
    Need python for x86_64, but found x86
    Run-time dependency python found: NO (tried sysconfig)
    
    ..\meson.build:22:14: ERROR: Python dependency not found
    
    A full log can be found at C:\Users\runneradmin\AppData\Local\Temp\pip-install-r63inwms\scipy_a36527f3a56348b6a1d252b7ee532fde\.mesonpy-g37647tf\meson-logs\meson-log.txt
    [end of output]
    
    note: This error originates from a subprocess, and is likely not a problem with pip.
  error: metadata-generation-failed
  
  Encountered error while generating package metadata.
  
  See above for output.
  
  note: This is an issue with the package mentioned above, not pip.
  hint: See above for details.
Error: Command ['pip', 'install', 'C:\\Users\\runneradmin\\AppData\\Local\\Temp\\cibw-run-mwn9orwe\\cp310-win32\\repaired_wheel\\mc3-3.1.3-cp310-cp310-win32.whl'] failed with code 1. None

[pcubillos]

Hi,
A quick update. Last month I spent some time checking the windows installation:
https://github.com/pcubillos/mc3/tree/windows
Apparently, I managed to correctly install the code and launch the automated tests, but the tests involving multiprocessing are failing :(
So, some progress, but I'm still not clear what can be the issue...

[sarahshi]

Hi Patricio,

Apologies for the delay in response. Thanks for your work on this! I think this is related to what I found during my tests. multiprocessing in Windows is limited by the ability only to spawn new interpreters, whereas Linux/OS allow for forking. I quote Henry's response from earlier. I was wondering how the information passed across the chains is actually used? Are there some simplifications that can be made towards the same end goal?

Best,
Sarah

The wheel seems to build properly, but testing fails due to an issue with how Multiprocessing handles the chains. Apparently Windows only allows for "Spawing" new Python interpreters and not "Forking" them. I'm not sure if there is an easy solution since this will affect how data is passed from one chain to another. Are there any shared variables that the chains need to have access to or can they all append their data after they each run? It seems like you brought this up in a previous issue.