installation of OpenCV, finally

I mentioned in another post about my plans for WorldWarIICasualtyProject.org. In short, the U.S. National Archives has scanned pages out of books that list American casualties that took place in World War II. I was curious to find out more, but discovered that those records did not exist in searchable form. I thought it would be interesting to figure out how to scan them (gif files, really!) and read the data as OCR.

I decided to go ahead on my initial plan: use a Python OCR module to read the scan. However … the Python module I tracked down (pytesseract) also required PIL (another module, part of the dependencies) and strongly suggested I install the python science packages. I figured I would need them at some point, so I installed numpy, scipy, matplotlib, scikit-image, scikit-learn, ipython, and pandas. ( https://www.learnopencv.com/install-opencv3-on-macos/)

At this point, I paused. I found several pages that suggested OpenCV be installed with Homebrew. That’s not a big deal because I use Homebrew for python 2/3. It gets confusing here. At one time, OpenCV was kept in a specialized area named “homebrew/science” but was moved to “homebrew/core”. I’m told “homebrew/science” is empty, so there should be no reason to link to it. We’ll see.

Note: use ‘> brew tap’ to list all taps connected for homebrew

Also note: opencv3 does not exist anymore. I think it has been renamed to opencv. Opencv2 has been renamed ‘opencv@2’. … So confusing …

Then there’s the question of linking OpenCV to “… Homebrew Python’s site-packages directory”. What? See https://www.learnopencv.com/install-opencv3-on-macos/

I’m sticking with these instructions: https://robferguson.org/blog/2017/10/06/how-to-install-opencv-and-python-using-homebrew-on-macos-sierra/ except for the part where I tap into homebrew/science (it doesn’t exist any more) and I install opencv3. (It’s been renamed to opencv).

2018-02-21 update

I installed OpenCV through homebrew. Lots of dependencies were installed. Interestingly enough, I can see opencv through the default homebrew python3 install, but not in virtual environment I created for custom work. In other words:

> python3
>>> import cv2
>>> cv2.__version__
‘3.4.0’

However, when I go to the virtual environment set up for ww2cp, I don’t see it.

> source ww2venv/bin/activate
(ww2venv) > python3
>>> import cv2
Traceback (most recent call last):
File “<stdin>”, line 1, in <module>
ModuleNotFoundError: No module named ‘cv2’

So, following the instructions here: (https://robferguson.org/blog/2017/10/06/how-to-install-opencv-and-python-using-homebrew-on-macos-sierra/), I set up a symbolic link between homebrew’s openCV install and the site-packages inside the ww2 venv folder.

Now it works!

Advertisements

Message digests OR I should have known that

I have lots of old SQL dumps stored in backups. I wanted to find a way to check to see if I was storing the same files over and over again. I did not want to check them line by line, because it would take too long. I remembered that message digests are a way to check to see if a file has been tampered. So, if I create a message digest of two files that I think are the same, a matching digest should (ideally) prove that they are the same.

By the way, what is a message digest? It’s “ … a cryptographic hash function containing a string of digits created by a one-way hashing formula”. ( https://www.techopedia.com/definition/4024/message-digest ). In other words, it is the result of sending a file or string through a one-way function and outputting the result. Ideally, it can be used to check to see if a file has been modified. If two files are related, but slightly different, they will generate two different message digests.

Back to digests. I like the idea of taking the sql dumps and generating a message digest. However, I noticed that the SQL dumps usually have a timestamp showing when the dump was created listed inside the SQL comments. This will automatically create a different digest. Can I remove the SQL comments and create a digest from that?

It turns out that I can. It works nicely.

> grep –regexp=“^–.*” <path-to-sql-dump>

shows all the SQL comments in the file

> grep –regexp=“^[^–.*]” <path-to-sql-dump>

shows everything but the SQL comments. Pipe that result into a digest function

> grep –regexp=“^[^–.*]” <path-to-sql-dump> | md5

shows the resulting digest using md5. Similarly, using “openssl sha1”, “shasum”, “shasum -a 512”, “shasum -a 512224” and “shasum -a 512256” will generate different digests, which can all be used to compare SQL commands in a SQL dump file.

I’m a little sad that “shasum” did not work completely. It adds the file name after the digest and hyphen, allowing storage of the digests. However, since the file is piped into the command, there is no file name to add to the end of the file. I’m sure there’s a way to add it to a file, though. Maybe something like?

> grep –regexp=“^[^–.*]” <path-to-sql-dump> | shasum; echo !!:2

then search for ‘- line-break’ and replace with ‘- ‘. … Maybe, maybe …

Adventure with Laravel Homestead, part 1

I thought it would be interesting to set up a Laravel instance to try it out. I did have a new project I wanted to develop, so I thought “How hard can it be to set up Laravel”? Well …

I have started using Vagrant to run some VMs (instead of VirtualBox by itself). I found the homestead box in the Vagrant Cloud, so I set that up to download and install.

> mkdir ~/Vagrant/laravel

> cd !!:$

> vagrant box add laravel/homestead

I wait a long while to download, but it finally arrives. I start it and look around, but I don’t see any of the programs I’m told will be associated with Homestead (mysql, nginx, etc). That’s odd.

The Laravel homestead documents also suggest cloning a repository. That did not make any sense, since I had a running Vagrant box. Since nothing was happening, I thought “Why not?”. The clone is downloaded into the same folder as the Laravel Vagrantfile. I follow the instructions to set up a Homestead.yaml file and look inside the Homestead folder to see if it’s there. It is, along with lots of others stuff, including … another Vagrantfile … That’s weird.

I make some simple changes to homestead.yaml and reload the original Vagrant box.

> vagrant reload –provision

I don’t see my changes. After some time, I wonder … What if I went inside the cloned folder and ran that Vagrantfile? It turns out that second Vagrantfile is the one that runs the homestead install. It seems weird to have a homestead box available in the Vagrant cloud and NOT have it be the one to use, but that’s what happened.

I make some changes in the homestead.yaml to set up a simple test site using the classic php test file.

<?php phpinfo(); ?>

Nothing displays. My first error message says “No input file specified”. I fixed that by pointing the map section inside homestead.yaml to the correct folder. Next error: “403 prohibited”. I thought it might be an nginx error, but I did not want to mess around too much with that just yet. However, I did check /var/log/nginx/homestead.test-error.log and I noticed something interesting:

“Unable to open primary script: /home/vagrant/code/test01/public/index.php (No such file or directory)”

OK. I guess I must wrap the simple php test page inside a Laravel template to make it display. Not a big deal, but one more thing I need to figure out. That will be for next time.

That was fun: Angular 5 updates

(This post should have been uploaded in November, 2017, when it happened. I should post more often.)

I noticed that the Angular.io docs were written with Angular 5 in mind. I especially noticed it when the Http methods did not load properly in my project. It turns out that HttpClient was moved to into @angular/common. However, my version of Angular had HttpClient still in @angular/http. I ended up with bad files when I followed the documentation. To stay current, I decided to  update my copy of Angular 4 to Angular 5.

I did a search on how to do the update and I ran across this site: angular-update-guide.firebaseapp.com. Their suggestions helped a lot. I did notice that when I ran the update, I kept seeing messages about ‘invalid’ modules. I’m still not sure why I saw that message, but I’ve seen odd messages from other updates before, so I filed it away to follow up in case the update did not work.

I run my development server with Angular CLI, so I entered ‘>ng serve’ to start the new server. It did not work. I saw an odd message saying my version of ‘angular/compiler-cli needs to be 2.3.1 or greater. Current version is 5.0.2’. Clearly, something was wrong with the angular-cli files. I considered updating the project.json file and updating everything with npm. It turns out I had to be more systematic to make sure everything related was updated properly.

I installed Angular-cli globally, so I had to remove it globally first. I also removed it from the dependency list in the project. I deleted the node modules to make sure every change shown in project.json was accepted correctly. I reloaded angular-cli globally and in the project dependencies again. Finally, I ran ‘>npm install’ to make sure the modules were fresh.

‘> ng serve’ worked as expected. That took longer than I thought.

nice to know: python imports

I am writing a Python command line app that cleans up data scraped from a web page and imports that data into associated tables inside a MySQL database. The top level of the directory holding the python files is getting crowded. I wanted to move the dependent modules into a directory and import those modules into the main file. I find out that I need to add that child directory into a sys.path list before I can import the modules. That’s good to know, but it’s too much for a command line script.

See here for more info:

error on part 0 of Ansible install

I’ve mentioned that I decided to use Ansible for my server configuration management. This installation has finally bubbled up as the first item on my to-do list. I looked around to see how to install it. Since I already have Homebrew installed on my mac, I saw two options:

  1. > brew install ansible
  2. > pip3 install ansible

I did not know any reason why these would be different, so I went ahead and ran ‘> brew install ansible’. That was a mistake. Ansible lists Python 2 as a dependency, which is not included in my Homebrew installation. Homebrew installs Python 2.7.13. I also have Python 3 running (3.6.2). I also found out there’s a third version of Python (2.7.10) which is part of the default installation in my laptop. 3 versions of Python on one machine. Wonderful!

I created a /etc/ansible/hosts file and did a test ping, which returns UNREACHABLE. That makes no sense, but I think I have to use some command line options to use the correct account.

Also, I don’t know which version of ansible is running. I may want to remove the Homebrew version and figure out how to use the python3 version. What a mess.

After a day of thought, I decided to check a few things:

> ansible —version

> ansible 2.3.2.0 (good)

… python version = 3.6.2 (what? Why not 2.7.13 or 2.7.10? Interesting.)

I checked the documents page at ansible.com. They say that ansible can run with python3 in one of two ways:

  1. > python3 <path/to/ansible> localhost -m ping
  2. > ansible localhost -m ping -e ‘ansible_python_interpreter=<path/to/python3>

To see what happens, I also tried

  1. > ansible localhost -m ping

It turns out all return good pings from localhost, so … I guess my installation is OK. Even so, next time, avoid using Homebrew for ansible installations.

More deployment strangeness: Capistrano, part 0

As mentioned before, my web/database server infrastructure is becoming more complicated. I need to figure out a way to make things easier and repeatable for me. I’m going to test Ansible for server configuration. I also decided to try Capistrano for website deployment.

Originally, I was thinking of using Deployer. I liked the idea of having a PHP tool to deploy PHP websites, but then I realized I also want to deploy Node.js and Python WSDL sites. (By the way, deploying Python sites using Flask or Django as the web framework looks unusually complicated. More about that later.) For the PHP sites, I wrote a Phing script to collect everything I needed. My plan is for Capistrano to take that bundle and deploy that. We’ll see.

To use Capistrano, I need Ruby. I installed that somewhere when I took the Berkeley MOOC class.

>ruby –version
ruby 2.1.5p211 (2014-11-13 revision 48405)

Wow, that seems old. I’ll update it. After some google searches, I settle on this set of instructions to upgrade ruby. Their process in a nutshell:

  • install home-brew and git
  • install rvm. Over rbenv? Uh, OK.
  • install ruby
  • install any needed gems

Homebrew are already covered. Git works as expected. They also recommend the latest version of the macOS, along with Xcode and the Xcode command line tools. I’ve got that covered, too. Let’s keep moving.

Next: install gpg? What is that? It checks cryptographic security of the rvm download. OK, sure. I installed the security key as described. (When I went through this process on a different machine, I forgot about the security key and did not have any issues with the rvm download.)

Next: download rvm. It looks simple enough. After the installation, I get a notice about two versions of rvm running on my machine. Apparently, I need to either source my .bash_process file or reload a terminal window, which does the same thing. OK. that’s working now.

Next: use rvm to install ruby. Am I seeing things? Why is it installing ruby into my home directory? Well,iIt seems to work. Ruby has the correct version. Final stuff: update bundler and nokigiri using the gem installer.

Everything looks good. Next, Capistrano.