Message digests OR I should have known that

I have lots of old SQL dumps stored in backups. I wanted to find a way to check to see if I was storing the same files over and over again. I did not want to check them line by line, because it would take too long. I remembered that message digests are a way to check to see if a file has been tampered. So, if I create a message digest of two files that I think are the same, a matching digest should (ideally) prove that they are the same.

By the way, what is a message digest? It’s “ … a cryptographic hash function containing a string of digits created by a one-way hashing formula”. ( https://www.techopedia.com/definition/4024/message-digest ). In other words, it is the result of sending a file or string through a one-way function and outputting the result. Ideally, it can be used to check to see if a file has been modified. If two files are related, but slightly different, they will generate two different message digests.

Back to digests. I like the idea of taking the sql dumps and generating a message digest. However, I noticed that the SQL dumps usually have a timestamp showing when the dump was created listed inside the SQL comments. This will automatically create a different digest. Can I remove the SQL comments and create a digest from that?

It turns out that I can. It works nicely.

> grep –regexp=“^–.*” <path-to-sql-dump>

shows all the SQL comments in the file

> grep –regexp=“^[^–.*]” <path-to-sql-dump>

shows everything but the SQL comments. Pipe that result into a digest function

> grep –regexp=“^[^–.*]” <path-to-sql-dump> | md5

shows the resulting digest using md5. Similarly, using “openssl sha1”, “shasum”, “shasum -a 512”, “shasum -a 512224” and “shasum -a 512256” will generate different digests, which can all be used to compare SQL commands in a SQL dump file.

I’m a little sad that “shasum” did not work completely. It adds the file name after the digest and hyphen, allowing storage of the digests. However, since the file is piped into the command, there is no file name to add to the end of the file. I’m sure there’s a way to add it to a file, though. Maybe something like?

> grep –regexp=“^[^–.*]” <path-to-sql-dump> | shasum; echo !!:2

then search for ‘- line-break’ and replace with ‘- ‘. … Maybe, maybe …

Advertisements

Adventure with Laravel Homestead, part 1

I thought it would be interesting to set up a Laravel instance to try it out. I did have a new project I wanted to develop, so I thought “How hard can it be to set up Laravel”? Well …

I have started using Vagrant to run some VMs (instead of VirtualBox by itself). I found the homestead box in the Vagrant Cloud, so I set that up to download and install.

> mkdir ~/Vagrant/laravel

> cd !!:$

> vagrant box add laravel/homestead

I wait a long while to download, but it finally arrives. I start it and look around, but I don’t see any of the programs I’m told will be associated with Homestead (mysql, nginx, etc). That’s odd.

The Laravel homestead documents also suggest cloning a repository. That did not make any sense, since I had a running Vagrant box. Since nothing was happening, I thought “Why not?”. The clone is downloaded into the same folder as the Laravel Vagrantfile. I follow the instructions to set up a Homestead.yaml file and look inside the Homestead folder to see if it’s there. It is, along with lots of others stuff, including … another Vagrantfile … That’s weird.

I make some simple changes to homestead.yaml and reload the original Vagrant box.

> vagrant reload –provision

I don’t see my changes. After some time, I wonder … What if I went inside the cloned folder and ran that Vagrantfile? It turns out that second Vagrantfile is the one that runs the homestead install. It seems weird to have a homestead box available in the Vagrant cloud and NOT have it be the one to use, but that’s what happened.

I make some changes in the homestead.yaml to set up a simple test site using the classic php test file.

<?php phpinfo(); ?>

Nothing displays. My first error message says “No input file specified”. I fixed that by pointing the map section inside homestead.yaml to the correct folder. Next error: “403 prohibited”. I thought it might be an nginx error, but I did not want to mess around too much with that just yet. However, I did check /var/log/nginx/homestead.test-error.log and I noticed something interesting:

“Unable to open primary script: /home/vagrant/code/test01/public/index.php (No such file or directory)”

OK. I guess I must wrap the simple php test page inside a Laravel template to make it display. Not a big deal, but one more thing I need to figure out. That will be for next time.

That was fun: Angular 5 updates

(This post should have been uploaded in November, 2017, when it happened. I should post more often.)

I noticed that the Angular.io docs were written with Angular 5 in mind. I especially noticed it when the Http methods did not load properly in my project. It turns out that HttpClient was moved to into @angular/common. However, my version of Angular had HttpClient still in @angular/http. I ended up with bad files when I followed the documentation. To stay current, I decided to  update my copy of Angular 4 to Angular 5.

I did a search on how to do the update and I ran across this site: angular-update-guide.firebaseapp.com. Their suggestions helped a lot. I did notice that when I ran the update, I kept seeing messages about ‘invalid’ modules. I’m still not sure why I saw that message, but I’ve seen odd messages from other updates before, so I filed it away to follow up in case the update did not work.

I run my development server with Angular CLI, so I entered ‘>ng serve’ to start the new server. It did not work. I saw an odd message saying my version of ‘angular/compiler-cli needs to be 2.3.1 or greater. Current version is 5.0.2’. Clearly, something was wrong with the angular-cli files. I considered updating the project.json file and updating everything with npm. It turns out I had to be more systematic to make sure everything related was updated properly.

I installed Angular-cli globally, so I had to remove it globally first. I also removed it from the dependency list in the project. I deleted the node modules to make sure every change shown in project.json was accepted correctly. I reloaded angular-cli globally and in the project dependencies again. Finally, I ran ‘>npm install’ to make sure the modules were fresh.

‘> ng serve’ worked as expected. That took longer than I thought.

nice to know: python imports

I am writing a Python command line app that cleans up data scraped from a web page and imports that data into associated tables inside a MySQL database. The top level of the directory holding the python files is getting crowded. I wanted to move the dependent modules into a directory and import those modules into the main file. I find out that I need to add that child directory into a sys.path list before I can import the modules. That’s good to know, but it’s too much for a command line script.

See here for more info:

error on part 0 of Ansible install

I’ve mentioned that I decided to use Ansible for my server configuration management. This installation has finally bubbled up as the first item on my to-do list. I looked around to see how to install it. Since I already have Homebrew installed on my mac, I saw two options:

  1. > brew install ansible
  2. > pip3 install ansible

I did not know any reason why these would be different, so I went ahead and ran ‘> brew install ansible’. That was a mistake. Ansible lists Python 2 as a dependency, which is not included in my Homebrew installation. Homebrew installs Python 2.7.13. I also have Python 3 running (3.6.2). I also found out there’s a third version of Python (2.7.10) which is part of the default installation in my laptop. 3 versions of Python on one machine. Wonderful!

I created a /etc/ansible/hosts file and did a test ping, which returns UNREACHABLE. That makes no sense, but I think I have to use some command line options to use the correct account.

Also, I don’t know which version of ansible is running. I may want to remove the Homebrew version and figure out how to use the python3 version. What a mess.

After a day of thought, I decided to check a few things:

> ansible —version

> ansible 2.3.2.0 (good)

… python version = 3.6.2 (what? Why not 2.7.13 or 2.7.10? Interesting.)

I checked the documents page at ansible.com. They say that ansible can run with python3 in one of two ways:

  1. > python3 <path/to/ansible> localhost -m ping
  2. > ansible localhost -m ping -e ‘ansible_python_interpreter=<path/to/python3>

To see what happens, I also tried

  1. > ansible localhost -m ping

It turns out all return good pings from localhost, so … I guess my installation is OK. Even so, next time, avoid using Homebrew for ansible installations.

More deployment strangeness: Capistrano, part 0

As mentioned before, my web/database server infrastructure is becoming more complicated. I need to figure out a way to make things easier and repeatable for me. I’m going to test Ansible for server configuration. I also decided to try Capistrano for website deployment.

Originally, I was thinking of using Deployer. I liked the idea of having a PHP tool to deploy PHP websites, but then I realized I also want to deploy Node.js and Python WSDL sites. (By the way, deploying Python sites using Flask or Django as the web framework looks unusually complicated. More about that later.) For the PHP sites, I wrote a Phing script to collect everything I needed. My plan is for Capistrano to take that bundle and deploy that. We’ll see.

To use Capistrano, I need Ruby. I installed that somewhere when I took the Berkeley MOOC class.

>ruby –version
ruby 2.1.5p211 (2014-11-13 revision 48405)

Wow, that seems old. I’ll update it. After some google searches, I settle on this set of instructions to upgrade ruby. Their process in a nutshell:

  • install home-brew and git
  • install rvm. Over rbenv? Uh, OK.
  • install ruby
  • install any needed gems

Homebrew are already covered. Git works as expected. They also recommend the latest version of the macOS, along with Xcode and the Xcode command line tools. I’ve got that covered, too. Let’s keep moving.

Next: install gpg? What is that? It checks cryptographic security of the rvm download. OK, sure. I installed the security key as described. (When I went through this process on a different machine, I forgot about the security key and did not have any issues with the rvm download.)

Next: download rvm. It looks simple enough. After the installation, I get a notice about two versions of rvm running on my machine. Apparently, I need to either source my .bash_process file or reload a terminal window, which does the same thing. OK. that’s working now.

Next: use rvm to install ruby. Am I seeing things? Why is it installing ruby into my home directory? Well,iIt seems to work. Ruby has the correct version. Final stuff: update bundler and nokigiri using the gem installer.

Everything looks good. Next, Capistrano.

PHP weirdness

I use the stock PHP version installed for macOS Sierra. In my case, it’s PHP 5.6.30. It runs well on my local web apps and it’s useful for my development environment.

I’m not going to stick with it forever. The vagrant environment in Laravel (Homestead) uses PHP 7. The vagrant machine I’ve seen recommended for development comes with PHP 7. I’ll move to it eventually.

During the spring, I was trying to clean up my brew environment to allow for a cleaner install. I paid attention to the warnings, but I accidentally ran “brew upgrade” without any modifiers. I ended up installing a ton of stuff that I don’t use regularly, including PHP 7. “brew services” shows that it’s not running, so that’s OK, but still … I was not paying attention and now I have PHP 7.

I use composer to keep all my PHP packages and modules updated and tracked. I added a new module to a composer.json file, so I needed to update it. I ran composer and I got the following message:

“Your requirements could not be resolved to an installable set of packages.

Problem 1

– This package requires php ^5.5.38 but your PHP version (7.0.15) does not satisfy that requirement.”

It turns out that “which php” points to /usr/local/bin/php, which is the home brew install. When I check $PATH, it turns out that /usr/local/bin appears before /usr/bin, where the stock install of PHP 5 is located. That’s why PHP 7 is the PHP appearing in the command line.

Solutions?

1. Change the $PATH order

No! This is a bad idea. By changing this, a lot of the home brew installs will use the wrong version. It seems too much effort to fix one item.

2. Update Apache to use PHP 7.

I have to do it anyway, just not now.

3. Change the php requirement in composer.json

I could change the requirement to “>=5.6.30” instead of “^5.5.38”. The ^ requires that PHP stay within PHP 5. ‘>=‘ only requires that the PHP version be greater than or equal to that number. It’s a band-aid, but it gets the updates running for tonight.