Python updates and OpenCV

I recently updated my laptop to run Python 3.7.0. I forgot that updates knocks out all the Python virtual environments that expect Python 3.6. I’ve worked my way through the virtual environments, removed them, recreated them and restarted them.

> rm -rf venv

> python3 -m venv venv

> source venv/bin/activate

I noticed that I don’t have a requirements.txt file to make it simple to reload Python modules. I’ve been creating that file as needed. I also noticed I was having trouble with loading the modules needed for an OpenCV project.

i installed OpenCV with homebrew and I knew it was outdated, so I upgraded it (and other dependent files) with brew. I reloaded all the related Python modules, but still could not get the sample file to run.

> brew outdated

> brew upgrade opencv

Instructions here show full steps for installing OpenCV. Note step 6 where a symlink is set up to point from the openCV libraries installed by home-brew and pointing back to the virtual environment

> cd venv/lib/python3.7/site-packages/

> ln -s /usr/local/opt/opencv3/lib/python3.7/site-packages/cv2.cpython-37m-darwin.so cv2.so

Note that file paths now show 3.7. If and when Python 3.8 or Python 4 show up, the paths will probably need to be updated to show the Python version.

Advertisements

installation of OpenCV, finally

I mentioned in another post about my plans for WorldWarIICasualtyProject.org. In short, the U.S. National Archives has scanned pages out of books that list American casualties that took place in World War II. I was curious to find out more, but discovered that those records did not exist in searchable form. I thought it would be interesting to figure out how to scan them (gif files, really!) and read the data as OCR.

I decided to go ahead on my initial plan: use a Python OCR module to read the scan. However … the Python module I tracked down (pytesseract) also required PIL (another module, part of the dependencies) and strongly suggested I install the python science packages. I figured I would need them at some point, so I installed numpy, scipy, matplotlib, scikit-image, scikit-learn, ipython, and pandas. ( https://www.learnopencv.com/install-opencv3-on-macos/)

At this point, I paused. I found several pages that suggested OpenCV be installed with Homebrew. That’s not a big deal because I use Homebrew for python 2/3. It gets confusing here. At one time, OpenCV was kept in a specialized area named “homebrew/science” but was moved to “homebrew/core”. I’m told “homebrew/science” is empty, so there should be no reason to link to it. We’ll see.

Note: use ‘> brew tap’ to list all taps connected for homebrew

Also note: opencv3 does not exist anymore. I think it has been renamed to opencv. Opencv2 has been renamed ‘opencv@2’. … So confusing …

Then there’s the question of linking OpenCV to “… Homebrew Python’s site-packages directory”. What? See https://www.learnopencv.com/install-opencv3-on-macos/

I’m sticking with these instructions: https://robferguson.org/blog/2017/10/06/how-to-install-opencv-and-python-using-homebrew-on-macos-sierra/ except for the part where I tap into homebrew/science (it doesn’t exist any more) and I install opencv3. (It’s been renamed to opencv).

2018-02-21 update

I installed OpenCV through homebrew. Lots of dependencies were installed. Interestingly enough, I can see opencv through the default homebrew python3 install, but not in virtual environment I created for custom work. In other words:

> python3
>>> import cv2
>>> cv2.__version__
‘3.4.0’

However, when I go to the virtual environment set up for ww2cp, I don’t see it.

> source ww2venv/bin/activate
(ww2venv) > python3
>>> import cv2
Traceback (most recent call last):
File “<stdin>”, line 1, in <module>
ModuleNotFoundError: No module named ‘cv2’

So, following the instructions here: (https://robferguson.org/blog/2017/10/06/how-to-install-opencv-and-python-using-homebrew-on-macos-sierra/), I set up a symbolic link between homebrew’s openCV install and the site-packages inside the ww2 venv folder.

Now it works!

That would have been nice to know!

I run VirtualBox on my machine to test Ansible playbooks and to run (my legal copy of) Windows 7. I’ve received noticed from VirtualBox that there was a new version of VirtualBox available. (I run 5.1.28. The notice says 5.1.30.) After several weeks of delay, I decide to check it out.

I go to the VirtualBox website and discover that VB is now up to version 5.2.2. This is interesting. I wonder what they fixed? I like that it’s on minor version x.x.2, so they’ve tracked down the errors. I like it. I’ll download it and test it.

It looks good. It starts up as usual. The Windows 7 VM starts as expected. OK, let’s check out ansible. (Record scratch!!!) It won’t run any more. Vagrant 1.9.7 insists that it needs a copy of VirtualBox to run. I have VB installed. This can’t be right. I’ll force it and try again. Still not go with vagrant.

Let me run a Google search. It turns out that vagrant 2 won’t run VirtualBox 5.2 until the next minor update. What’s the newest version of vagrant available? 2.0.1. That’s it. I download it and attach the machine to a good internet pipe to handle the playbook.yml file. It starts up finally. A wasted afternoon on a machine that was not fixed until 9 pm.

More deployment strangeness: Capistrano, part 0

As mentioned before, my web/database server infrastructure is becoming more complicated. I need to figure out a way to make things easier and repeatable for me. I’m going to test Ansible for server configuration. I also decided to try Capistrano for website deployment.

Originally, I was thinking of using Deployer. I liked the idea of having a PHP tool to deploy PHP websites, but then I realized I also want to deploy Node.js and Python WSDL sites. (By the way, deploying Python sites using Flask or Django as the web framework looks unusually complicated. More about that later.) For the PHP sites, I wrote a Phing script to collect everything I needed. My plan is for Capistrano to take that bundle and deploy that. We’ll see.

To use Capistrano, I need Ruby. I installed that somewhere when I took the Berkeley MOOC class.

>ruby –version
ruby 2.1.5p211 (2014-11-13 revision 48405)

Wow, that seems old. I’ll update it. After some google searches, I settle on this set of instructions to upgrade ruby. Their process in a nutshell:

  • install home-brew and git
  • install rvm. Over rbenv? Uh, OK.
  • install ruby
  • install any needed gems

Homebrew are already covered. Git works as expected. They also recommend the latest version of the macOS, along with Xcode and the Xcode command line tools. I’ve got that covered, too. Let’s keep moving.

Next: install gpg? What is that? It checks cryptographic security of the rvm download. OK, sure. I installed the security key as described. (When I went through this process on a different machine, I forgot about the security key and did not have any issues with the rvm download.)

Next: download rvm. It looks simple enough. After the installation, I get a notice about two versions of rvm running on my machine. Apparently, I need to either source my .bash_process file or reload a terminal window, which does the same thing. OK. that’s working now.

Next: use rvm to install ruby. Am I seeing things? Why is it installing ruby into my home directory? Well,iIt seems to work. Ruby has the correct version. Final stuff: update bundler and nokigiri using the gem installer.

Everything looks good. Next, Capistrano.

PHP weirdness

I use the stock PHP version installed for macOS Sierra. In my case, it’s PHP 5.6.30. It runs well on my local web apps and it’s useful for my development environment.

I’m not going to stick with it forever. The vagrant environment in Laravel (Homestead) uses PHP 7. The vagrant machine I’ve seen recommended for development comes with PHP 7. I’ll move to it eventually.

During the spring, I was trying to clean up my brew environment to allow for a cleaner install. I paid attention to the warnings, but I accidentally ran “brew upgrade” without any modifiers. I ended up installing a ton of stuff that I don’t use regularly, including PHP 7. “brew services” shows that it’s not running, so that’s OK, but still … I was not paying attention and now I have PHP 7.

I use composer to keep all my PHP packages and modules updated and tracked. I added a new module to a composer.json file, so I needed to update it. I ran composer and I got the following message:

“Your requirements could not be resolved to an installable set of packages.

Problem 1

– This package requires php ^5.5.38 but your PHP version (7.0.15) does not satisfy that requirement.”

It turns out that “which php” points to /usr/local/bin/php, which is the home brew install. When I check $PATH, it turns out that /usr/local/bin appears before /usr/bin, where the stock install of PHP 5 is located. That’s why PHP 7 is the PHP appearing in the command line.

Solutions?

1. Change the $PATH order

No! This is a bad idea. By changing this, a lot of the home brew installs will use the wrong version. It seems too much effort to fix one item.

2. Update Apache to use PHP 7.

I have to do it anyway, just not now.

3. Change the php requirement in composer.json

I could change the requirement to “>=5.6.30” instead of “^5.5.38”. The ^ requires that PHP stay within PHP 5. ‘>=‘ only requires that the PHP version be greater than or equal to that number. It’s a band-aid, but it gets the updates running for tonight.

 

 

I should have known that.

Recently, I upgraded my laptop to run macOS Sierra. Along with the OS updates, I also used brew to update python3. I was not paying attention and ended up upgrading PostgreSQL to the newest version, 9.6.2. I was unhappy that I was not paying attention, but I was glad that upgrades to PostgreSQL and MongoDB were also done with the upgrade to Python 3.

I use brew to start and stop PostgreSQL as needed, so I started it to check to see how it ran after the update. ‘psql’ gave me a version number, but I could not get it started to show me the schemas or tables from previous work. I also saw the following error message:

psql: could not connect to server: no such file or directory
Is the server running locally and accepting
connections on Unix domain socket “/tmp/.s.PGSQL.5432”?

Very strange error message, so I started doing Google searches to see what could be wrong. There some dead ends involving checking for a PID file and checking permissions in /usr/local/var/postgres. Eventually, I ran into what looks like the answer in two related locations: a blog post describing upgrades to PostgreSQL and another one describing the upgrade process using brew. Apparently, I had two versions of PostgreSQL on my machine.

> brew info postgresql

confirmed it. Both pages described what I have to do, so I’ll do it. I should have known I needed to do this. Now I know.

Adventures with Windows, part 2

As I mentioned earlier, I had trouble loading the free update to Windows 10. I now have a 5 GB file in my Windows 7 instance, but it never finished installing. It was stuck on 99% complete install for hours. I grew frustrated and decided to punt on the install.

Some websites mentioned that the Windows 10 update needed a fully patched and updates Windows 7 system in order to finish installing. It’s possible that may be true. It’s definitely true that I needed to patch Windows 7 as soon as possible. After installing an antivirus program (Avira) along with FireFox and Chrome, I decided to run the Windows update.

That did not go well either. I selected Windows Update from the Windows Control Panel and the “Other Programs” list on various occasions. I never got the impression that anything was happening. From other websites, I knew that the initial Windows update should take roughly 90-120 minutes. I would run Windows update 3 hours with  no indication that anything was happening.

I decided to try to install Windows updates directly from the browser. My browser was immediately updated to IE 11. Immediately is relative. It still took roughly 90 minutes to download and install 14 updates for IE 11 only. Once it was done, I went back to update.microsoft.com to see if that would list files to download. I was told to set the update back to Windows Update. I did not know any better, so now going to update.microsoft.com bounces me over to Windows Update again.

I may try downloading Windows 7 SP1 by itself. Maybe that will take care of some of the missing updates. It’s possible my home download speed is too slow to download any information quickly enough. It would be nice to go somewhere with a big pipe and not a lot of traffic. Where could that be?