macOS Terminal issues & troubleshooting

ongoing note

04 Sep 2022

Need to figure out the difference betweens .bash_profile & .zprofile, both in user root folder, and their interaction with VS Code terminal.

.bash_profile: 

# Setting PATH for Python 3.8
PATH="/Library/Python/3.8/bin:${PATH}"
export PATH

.zprofile: 

PATH="/Library/Python3.9/bin:${PATH}"
export PATH

eval "$(/opt/homebrew/bin/brew shellenv)"

How to Add to the Shell Path in macOS Big Sur and Catalina using Terminal
https://wpbeaches.com/how-to-add-to-the-shell-path-in-macos-using-terminal/

Created manually a .zshrc file with: 

export PATH=$PATH:/Users/nic/Library/Python/3.8/bin

14 Sep 2022

started having problems again, whereby all my scripts return ModuleNotFoundError: No module named 'xxxx' with every library that is installed in the project's venv.
The script runs with venv activated ((venv)...), confirmed with which python which accurately returns the path of Python in venv.

When trying to (re)install those libraries, with venv still active, they get installed in root Python with following warning: 

WARNING: The scripts f2py, f2py3 and f2py3.9 are installed in '/Users/xxxxxx/Library/Python/3.9/bin' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.

Possibly since update to macOS Monterey 12.6 (which included some Command Line tools for XCode). 

ls -l /usr/bin/python*

is supposed to list all python instances on system, and it lists only one for me: 

-rwxr-xr-x  76 root  wheel  167120 24 Aug 10:59 /usr/bin/python3

actually: this is just a search in usr/bin and other paths are not included.

.bash_profile, .zprofile & .zshrc are the same as above.

🤔

Check what shell I am using: 

echo $SHELL

returns /bin/zsh when run in VS Code. Same from stand-alone terminal window (clue in title bar!).
So I assume Bash is not used, so everything should run with 3.9 now.
My ZSH points to Python 3.9 in .zprofile.

trying:

echo $PATH

returns (edited for readability): 

/opt/homebrew/bin:
/opt/homebrew/sbin:
/Library/Python3.9/bin:
/usr/local/bin:
/usr/bin:
/bin:
/usr/sbin:
/sbin:
/usr/local/MacGPG2/bin:
/Library/Apple/usr/bin:
/Users/xxxx/Library/Python/3.8/bin

notes:

  • there is no path /Library/Python3.9/bin
  • the last path (/Users/..) seems to be the one used by VS Code/Terminal.
  • though need to add missing path: /Users/xxxx/Library/Python/3.9/bin

one by one:

  • /opt/homebrew/bin: no Python binary in folder
  • /opt/homebrew/sbin: no Python binary in folder
  • /Library/Python3.9/bin: folder Python3.9 does not exist
  • /usr/local/bin: /usr/local/bin/python found +++
  • /usr/bin: /usr/bin/python3 found +++
  • /bin: no Python binary in folder
  • /usr/sbin: no Python binary in folder
  • /sbin: no Python binary in folder
  • /usr/local/MacGPG2/bin: no Python binary in folder
  • /Library/Apple/usr/bin: no Python binary in folder
  • /Users/xxxx/Library/Python/3.8/bin: no Python binary in folder 🤔
  • not returned with echo $PATH: /Users/xxxx/Library/Python/3.9/bin also no Python binary in folder 🤔

checking the symlink of a Python binary in one of my project's venv, I see the original to be:
/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/bin/python3.9
which is not in path list!? 🤔
Installed with the macOS update today?

it seems that all the symlinks to Python in the virtual environments of all of my VS Code projects now point to the binary above, and none of the projects work anymore ModuleNotFoundError.

Asked for help on Stackoverflow:

terminal - All Python scripts stopped working: path changed in all projects after macOS Monterey 12.6 update - Stack Overflow
stackoverflow
https://stackoverflow/questions/73722967/all-python-scripts-stopped-working-path-changed-in-all-projects-after-macos-mon

15 Sep 2022

got pointed to: https://stackoverflow/questions/58562928/how-do-i-update-a-python-virtual-environment-with-venv-to-use-a-newer-version 

xxx@yyy ~ % cd Python/homee
xxx@yyy homee % . venv/bin/activate
(venv) xxx@yyy homee % python -V
Python 3.9.6
(venv) xxx@yyy homee % which python
/Users/xxx/Python/homee/venv/bin/python
(venv) xxx@yyy homee %

already on latest Python version.

run this to see all paths being searched by python: 

import sys

for path in sys.path:
    print(path)

prints: 

/Users/xxxx/Python/homee
/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python39.zip
/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9
/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/lib-dynload
/Users/xxxx/Python/indeXee
/Users/xxxx/Python/Scrapee

last 2 are added in-script in boilerplate.
first path is my project, so venv

started separate note on Python paths for clarity: How to find all paths being searched by Python

tried renaming /Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/bin to /bin_old
result: 

(venv) xxx@yyy homee % /Users/xxxx/Python/homee/venv/bin/python /Users/xxxx/Python/homee/test.py
zsh: no such file or directory: /Users/xxxx/Python/homee/venv/bin/python

reverted. 

xxx@yyy % which python 
/usr/local/bin/python
xxx@yyy % which python3
/usr/bin/python3

/usr/local/bin/python = symlink to /usr/bin/python3
/usr/bin/python3 = executable

let's reboot and try to change the symlink in one venv to /usr/bin/python3 to test.

Rebooted and everything seems to work again without changing anything.
So 🤗 but also 🤯 - still not clear about what did not work & resolution path for next time 😕.
Anyway, back to work....

Arrgh.. rejoiced too soon.
Seems scripts using built-in packages (or packages available at global level?) work, but still ModuleNotFoundError for most.

Note: site-packages in venv are installed under /lib/python3.8 though symlink to `/Library/Developer/CommandLineTools/... is to Python 3.9.
This is probably the issue? 

xxx@yyy ~ % which python
/usr/local/bin/python
xxx@yyy ~ % where python
/usr/local/bin/python
xxx@yyy ~ % python -V
xcode-select: Failed to locate 'python', requesting installation of command line developer tools.
xxx@yyy ~ % cd /usr/bin/python
cd: no such file or directory: /usr/bin/python
xxx@yyy ~ % cd usr/bin
cd: no such file or directory: usr/bin
xxx@yyy ~ % sudo cd usr/bin
Password:
/usr/bin/cd: line 4: cd: usr/bin: No such file or directory
xxx@yyy ~ %

Try symlink update.
Can't find a quick/clean way.

Re-installing venv from scratch results in:

  • symlink still to the /Library/Developer/.. executable
  • /lib has only python3.9 vs old venv had only python3.8

Trying to install Python 3.10 (latest stable release) from https://www.python.org/downloads/release/python-3107/
Adds an app Python 3.10 under /Applications
No change in paths: 

Paths:
/Users/xxxx/Python/homee
/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python39.zip
/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9
/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/lib-dynload
/Users/xxxx/Python/homee/venv/lib/python3.9/site-packages
/Users/xxxx/Python/indeXee
/Users/xxxx/Python/Scrapee

Python lib:
/Users/xxxx/Python/homee/venv/lib/python3.9/site-packages

rebooting. 

(venv) xxx@yyy homee % which python
/Users/xxxx/Python/homee/venv/bin/python
(venv) xxx@yyy homee % where python
/Users/xxxx/Python/homee/venv/bin/python
/usr/local/bin/python
(venv) xxx@yyy homee % which pip
/Users/xxxx/Python/homee/venv/bin/pip
(venv) xxx@yyy homee % which pip3 
/Users/xxxx/Python/homee/venv/bin/pip3
(venv) xxx@yyy homee % where pip
/Users/xxxx/Python/homee/venv/bin/pip
/Users/xxxx/Library/Python/3.8/bin/pip
/Users/xxxx/Library/Python/3.8/bin/pip
(venv) xxx@yyy homee % where pip3
/Users/xxxx/Python/homee/venv/bin/pip3
/Library/Frameworks/Python.framework/Versions/3.10/bin/pip3
/usr/local/bin/pip3
/usr/bin/pip3
/Library/Frameworks/Python.framework/Versions/3.10/bin/pip3
/Users/xxxx/Library/Python/3.8/bin/pip3
/Users/xxxx/Library/Python/3.8/bin/pip3
(venv) xxx@yyy homee %

notes:

  • why 2x /Users/xxxx/Library/Python/3.8/bin/pip in where pip?
  • new Python install seems to be at /Library/Frameworks/Python.framework/Versions/3.10/bin/pip3

test installing new venv now (old venv renamed as venv_220915 for archive, not deleted). 

Paths:
/Users/xxxx/Python/homee
/Library/Frameworks/Python.framework/Versions/3.10/lib/python310.zip
/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10
/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/lib-dynload
/Users/xxxx/Python/homee/venv/lib/python3.10/site-packages
/Users/xxxx/Python/indeXee
/Users/xxxx/Python/Scrapee

Python lib:
/Users/xxxx/Python/homee/venv/lib/python3.10/site-packages
  • /Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/bin seems to have disappeared on its own? 🤔 
(venv) xxx@yyy homee % where pip
/Users/xxxx/Python/homee/venv/bin/pip
/Users/xxxx/Library/Python/3.8/bin/pip
/Users/xxxx/Library/Python/3.8/bin/pip
(venv) xxx@yyy homee % where pip3
/Users/xxxx/Python/homee/venv/bin/pip3
/Library/Frameworks/Python.framework/Versions/3.10/bin/pip3
/usr/local/bin/pip3
/usr/bin/pip3
/Library/Frameworks/Python.framework/Versions/3.10/bin/pip3
/Users/xxxx/Library/Python/3.8/bin/pip3
/Users/xxxx/Library/Python/3.8/bin/pip3
(venv) xxx@yyy homee % where python
/Users/xxxx/Python/homee/venv/bin/python
/usr/local/bin/python
(venv) xxx@yyy homee % where python3
/Users/xxxx/Python/homee/venv/bin/python3
/Library/Frameworks/Python.framework/Versions/3.10/bin/python3
/usr/local/bin/python3
/usr/bin/python3
/Library/Frameworks/Python.framework/Versions/3.10/bin/python3

Python in new venv now symlinked to /Library/Frameworks/Python.framework/Versions/3.10/bin/python3.10
and venv/lib has now only /python3.10

Let's reinstall everything and use this as default one moving forward. 🤞🏼

For each project:

  • rename current /venv with /venv.yymmdd (archive)
  • python3 -m venv venv
  • close and restart VS Code
  • ensure python & pip are correct (where/which)
  • install packages with requirements.txt where available, else manually
  • once everything is working again and on path, pip3 freeze > requirements.txt to backup

Certificate issues with some scripts. Solved & details here:

How to solve urlopen error SSL: CERTIFICATE_VERIFY_FAILED

links

social