Python updates and OpenCV

I recently updated my laptop to run Python 3.7.0. I forgot that updates knocks out all the Python virtual environments that expect Python 3.6. I’ve worked my way through the virtual environments, removed them, recreated them and restarted them.

> rm -rf venv

> python3 -m venv venv

> source venv/bin/activate

I noticed that I don’t have a requirements.txt file to make it simple to reload Python modules. I’ve been creating that file as needed. I also noticed I was having trouble with loading the modules needed for an OpenCV project.

i installed OpenCV with homebrew and I knew it was outdated, so I upgraded it (and other dependent files) with brew. I reloaded all the related Python modules, but still could not get the sample file to run.

> brew outdated

> brew upgrade opencv

Instructions here show full steps for installing OpenCV. Note step 6 where a symlink is set up to point from the openCV libraries installed by home-brew and pointing back to the virtual environment

> cd venv/lib/python3.7/site-packages/

> ln -s /usr/local/opt/opencv3/lib/python3.7/site-packages/cv2.cpython-37m-darwin.so cv2.so

Note that file paths now show 3.7. If and when Python 3.8 or Python 4 show up, the paths will probably need to be updated to show the Python version.

Advertisements

installation of OpenCV, finally

I mentioned in another post about my plans for WorldWarIICasualtyProject.org. In short, the U.S. National Archives has scanned pages out of books that list American casualties that took place in World War II. I was curious to find out more, but discovered that those records did not exist in searchable form. I thought it would be interesting to figure out how to scan them (gif files, really!) and read the data as OCR.

I decided to go ahead on my initial plan: use a Python OCR module to read the scan. However … the Python module I tracked down (pytesseract) also required PIL (another module, part of the dependencies) and strongly suggested I install the python science packages. I figured I would need them at some point, so I installed numpy, scipy, matplotlib, scikit-image, scikit-learn, ipython, and pandas. ( https://www.learnopencv.com/install-opencv3-on-macos/)

At this point, I paused. I found several pages that suggested OpenCV be installed with Homebrew. That’s not a big deal because I use Homebrew for python 2/3. It gets confusing here. At one time, OpenCV was kept in a specialized area named “homebrew/science” but was moved to “homebrew/core”. I’m told “homebrew/science” is empty, so there should be no reason to link to it. We’ll see.

Note: use ‘> brew tap’ to list all taps connected for homebrew

Also note: opencv3 does not exist anymore. I think it has been renamed to opencv. Opencv2 has been renamed ‘opencv@2’. … So confusing …

Then there’s the question of linking OpenCV to “… Homebrew Python’s site-packages directory”. What? See https://www.learnopencv.com/install-opencv3-on-macos/

I’m sticking with these instructions: https://robferguson.org/blog/2017/10/06/how-to-install-opencv-and-python-using-homebrew-on-macos-sierra/ except for the part where I tap into homebrew/science (it doesn’t exist any more) and I install opencv3. (It’s been renamed to opencv).

2018-02-21 update

I installed OpenCV through homebrew. Lots of dependencies were installed. Interestingly enough, I can see opencv through the default homebrew python3 install, but not in virtual environment I created for custom work. In other words:

> python3
>>> import cv2
>>> cv2.__version__
‘3.4.0’

However, when I go to the virtual environment set up for ww2cp, I don’t see it.

> source ww2venv/bin/activate
(ww2venv) > python3
>>> import cv2
Traceback (most recent call last):
File “<stdin>”, line 1, in <module>
ModuleNotFoundError: No module named ‘cv2’

So, following the instructions here: (https://robferguson.org/blog/2017/10/06/how-to-install-opencv-and-python-using-homebrew-on-macos-sierra/), I set up a symbolic link between homebrew’s openCV install and the site-packages inside the ww2 venv folder.

Now it works!

PHP weirdness

I use the stock PHP version installed for macOS Sierra. In my case, it’s PHP 5.6.30. It runs well on my local web apps and it’s useful for my development environment.

I’m not going to stick with it forever. The vagrant environment in Laravel (Homestead) uses PHP 7. The vagrant machine I’ve seen recommended for development comes with PHP 7. I’ll move to it eventually.

During the spring, I was trying to clean up my brew environment to allow for a cleaner install. I paid attention to the warnings, but I accidentally ran “brew upgrade” without any modifiers. I ended up installing a ton of stuff that I don’t use regularly, including PHP 7. “brew services” shows that it’s not running, so that’s OK, but still … I was not paying attention and now I have PHP 7.

I use composer to keep all my PHP packages and modules updated and tracked. I added a new module to a composer.json file, so I needed to update it. I ran composer and I got the following message:

“Your requirements could not be resolved to an installable set of packages.

Problem 1

– This package requires php ^5.5.38 but your PHP version (7.0.15) does not satisfy that requirement.”

It turns out that “which php” points to /usr/local/bin/php, which is the home brew install. When I check $PATH, it turns out that /usr/local/bin appears before /usr/bin, where the stock install of PHP 5 is located. That’s why PHP 7 is the PHP appearing in the command line.

Solutions?

1. Change the $PATH order

No! This is a bad idea. By changing this, a lot of the home brew installs will use the wrong version. It seems too much effort to fix one item.

2. Update Apache to use PHP 7.

I have to do it anyway, just not now.

3. Change the php requirement in composer.json

I could change the requirement to “>=5.6.30” instead of “^5.5.38”. The ^ requires that PHP stay within PHP 5. ‘>=‘ only requires that the PHP version be greater than or equal to that number. It’s a band-aid, but it gets the updates running for tonight.

 

 

So I decided to install Java …

I’m not a Java coder. I work primarily in PHP, with Python on the side. I’ve been rewriting an old PHP project to conform with modern standards, including testing. I decided to use Codeception for PHP testing, mainly because it looked like PHPUnit was included and unit/integration/acceptance testing looked easy (if you follow the example given in the website). I had used Gherkin to write BDD tests, so I was happy to see it with PHP. I also saw auto-testing with PhpBrowser and … Selenium!

I had used Selenium for testing several years ago. I remembered that all I had to do was fire up the Selenium server and run the tests to catch any Javascript browser issues. I remember that being very helpful, so I downloaded the Selenium server and tried to fire it up. … Wait a minute. I don’t have java on this machine any more? The OS upgrades probably turned it to toast. OK. I’ll install that first.

I spent a few days thinking about how to install it. Since I use brew to install other command-line applications, I wondered how brew would handle a JDK installation. It turns out that my Google search: “jdk homebrew” came up with several web pages that did not fully work. I had to put together instructions from these pages.

For starters, I ran into a command that mentioned casks. Eventually, after another Google search (“homebrew install cask”), I discovered that “cask” is a way of managing graphical applications through brew. Anyway, after some review, I finally have Java (1.8.0_131) running on my El Cap box. Yay, me! You’re welcome to try these or to review the pages linked above to figure out yourself. It looks like it will work either way.

brew update
> brew install caskroom/cask/brew-cask  …  (probably did nothing)
> brew tap caskroom/cask
> brew cask install java

Note that I did not choose to install “jenv”, which creates virtual environments similar to “virtualenv” or “venv” (?) with Python. Somewhere in one of these pages, there was a note that I needed to install Java 7 first. I never found a reason why it was needed, so I skipped it. We’ll see if I do need it.

I should have known that.

Recently, I upgraded my laptop to run macOS Sierra. Along with the OS updates, I also used brew to update python3. I was not paying attention and ended up upgrading PostgreSQL to the newest version, 9.6.2. I was unhappy that I was not paying attention, but I was glad that upgrades to PostgreSQL and MongoDB were also done with the upgrade to Python 3.

I use brew to start and stop PostgreSQL as needed, so I started it to check to see how it ran after the update. ‘psql’ gave me a version number, but I could not get it started to show me the schemas or tables from previous work. I also saw the following error message:

psql: could not connect to server: no such file or directory
Is the server running locally and accepting
connections on Unix domain socket “/tmp/.s.PGSQL.5432”?

Very strange error message, so I started doing Google searches to see what could be wrong. There some dead ends involving checking for a PID file and checking permissions in /usr/local/var/postgres. Eventually, I ran into what looks like the answer in two related locations: a blog post describing upgrades to PostgreSQL and another one describing the upgrade process using brew. Apparently, I had two versions of PostgreSQL on my machine.

> brew info postgresql

confirmed it. Both pages described what I have to do, so I’ll do it. I should have known I needed to do this. Now I know.

Catching up with Python

I always forget that major software upgrades in OS X reset permissions in /usr/local. When I checked my brew install, I saw that message. I played around with write permissions only, but in the end, did what brew recommended and reset the permissions as described. That allowed me to update brew and move on to the next task: updating Python3.

I was not as far back as I expected. I was sitting on Python 3.4. The upgrades went smoothly and I’m back to Python 3.5.1. I thought I had modules installed, but I’m not getting any list of local modules installed. It’s very possible, since I never did much with on the desktop. That will be the next task.

odd home-brew issue with permissions,

I was going to update files installed through home brew and I noticed an odd error message:

Error: unable to unlink old ‘share/man/man1/brew.1’ (Permission denied)

or something like that. That’s never happened before. /usr/local should not have a permissions problem, unless MySQL is involved somehow. (B’). I run ‘brew doctor’ on the command line and it suggests permission fixes for /usr/local/bin/, /usr/local/share, /usr/local/share/man and /usr/local/share/man/man (?). Why did this happen?

I know that “rootless” access is being enabled in OS X 10.11, El Capitan. The idea is this “System Integrity Protection” will make it difficult for malware to install itself in /System, /bin, /sbin, and /usr. However, /usr/local was supposed to be left alone. Also, I’m running Yosemite, not El Cap. Even so, I need to run my updates, so I run the following in Terminal

sudo chown -R “$USER”:admin /usr/local/bin

sudo chown -R “$USER”:admin /usr/local/share

When I run “brew update” again, my system is ready for brewing.

From what I can tell, it looks like folder permissions may be reset during each system software update. I can’t confirm it, but it makes sense. If I had problems after a system update, I would run the Disk Utility and have it repair permissions. Usually, whatever problem I had would go away. It looks like OS X wants to be helpful, but also interferes with the way I have things set up. I’ll make it work.