Message digests OR I should have known that

I have lots of old SQL dumps stored in backups. I wanted to find a way to check to see if I was storing the same files over and over again. I did not want to check them line by line, because it would take too long. I remembered that message digests are a way to check to see if a file has been tampered. So, if I create a message digest of two files that I think are the same, a matching digest should (ideally) prove that they are the same.

By the way, what is a message digest? It’s “ … a cryptographic hash function containing a string of digits created by a one-way hashing formula”. ( https://www.techopedia.com/definition/4024/message-digest ). In other words, it is the result of sending a file or string through a one-way function and outputting the result. Ideally, it can be used to check to see if a file has been modified. If two files are related, but slightly different, they will generate two different message digests.

Back to digests. I like the idea of taking the sql dumps and generating a message digest. However, I noticed that the SQL dumps usually have a timestamp showing when the dump was created listed inside the SQL comments. This will automatically create a different digest. Can I remove the SQL comments and create a digest from that?

It turns out that I can. It works nicely.

> grep –regexp=“^–.*” <path-to-sql-dump>

shows all the SQL comments in the file

> grep –regexp=“^[^–.*]” <path-to-sql-dump>

shows everything but the SQL comments. Pipe that result into a digest function

> grep –regexp=“^[^–.*]” <path-to-sql-dump> | md5

shows the resulting digest using md5. Similarly, using “openssl sha1”, “shasum”, “shasum -a 512”, “shasum -a 512224” and “shasum -a 512256” will generate different digests, which can all be used to compare SQL commands in a SQL dump file.

I’m a little sad that “shasum” did not work completely. It adds the file name after the digest and hyphen, allowing storage of the digests. However, since the file is piped into the command, there is no file name to add to the end of the file. I’m sure there’s a way to add it to a file, though. Maybe something like?

> grep –regexp=“^[^–.*]” <path-to-sql-dump> | shasum; echo !!:2

then search for ‘- line-break’ and replace with ‘- ‘. … Maybe, maybe …

Advertisements

My adventure with AWS, part 1

I decided to try out Amazon AWS to find out what it could do for me. As it turns out, I also has a project (WorldWarIICasualtyProject.org or ww2cp for short) that I wanted to host cheaply. Amazon AWS promises to do that.

First things first, I had to create an account. It’s not hard provided you have a credit card handy. Amazon tests the card to see if it has money (in case it’s a debit card). I chose to use a debit card first because there are no default brakes on spending. (Debit card runs out of money. I assume AWS stops services, but am I going to test it?) You have to protect your accounts to prevent others from spending your money. You also have to watch what you do decide to activate because there are no auto-shutdowns if you spend too much money. Live dangerously? Not really. Just figure out how to set alarms and stay on top of them.

AWS also recommends not using your root (or first) account for daily use, in case it gets compromised. I detoured over to IAM (Identity and Agent Management) and created a separate account that I would use every day. Oddly, it’s possible to assign almost all root powers to any child account, so, once again, be careful.

Once I was satisfied with the child accounts, I started testing S3 (Simple Storage Service). They operate on the concept of “buckets”, that hold pretty much everything. Amazon has built a pseudo-folder structure to allow some organization, but really everything goes in one big bucket.

One cool thing about S3 buckets is that they get mirrored to other nodes within a region. The idea is that this should make it easier to pull the data from the bucket regardless of where a browser is within a region. This becomes important when using Route53.

Route53 is Amazon’s version of DNS. I bought ww2cp from NameCheap.com and used them as the DNS to a placeholder while I figured out what I was going to do with the website. I discovered that I could have the S3 buckets I created earlier serve as a website, provided I let Route53 handle the DNS. Coolness!

Weird fact: Route53 assigned four name servers to resolve ww2cp. When I used nslookup to check for the correct IP address for the website, I would get a revolving set of four “web” servers instead of the one (parking) IP address I used before. I bet that has to do with the S3 mirrors I mentioned above.

Setting up Route53 to handle DNS is not hard. (There once was a time when AWS documentation was cryptic and undecipherable. If you read the same docs often enough, they make sense.) Anyway, I set up Route53 to handle the DNS services required to make the S3 bucket host the files for the website. I updated the name server information over at NameCheap … and nothing happened. For some odd reason, my changes to the name servers over at NameCheap kept reverting to their original settings. Eventually, I had to get NameCheap tech support involved to get the name server changes to stick, but it did.

Adventure with Laravel Homestead, part 1

I thought it would be interesting to set up a Laravel instance to try it out. I did have a new project I wanted to develop, so I thought “How hard can it be to set up Laravel”? Well …

I have started using Vagrant to run some VMs (instead of VirtualBox by itself). I found the homestead box in the Vagrant Cloud, so I set that up to download and install.

> mkdir ~/Vagrant/laravel

> cd !!:$

> vagrant box add laravel/homestead

I wait a long while to download, but it finally arrives. I start it and look around, but I don’t see any of the programs I’m told will be associated with Homestead (mysql, nginx, etc). That’s odd.

The Laravel homestead documents also suggest cloning a repository. That did not make any sense, since I had a running Vagrant box. Since nothing was happening, I thought “Why not?”. The clone is downloaded into the same folder as the Laravel Vagrantfile. I follow the instructions to set up a Homestead.yaml file and look inside the Homestead folder to see if it’s there. It is, along with lots of others stuff, including … another Vagrantfile … That’s weird.

I make some simple changes to homestead.yaml and reload the original Vagrant box.

> vagrant reload –provision

I don’t see my changes. After some time, I wonder … What if I went inside the cloned folder and ran that Vagrantfile? It turns out that second Vagrantfile is the one that runs the homestead install. It seems weird to have a homestead box available in the Vagrant cloud and NOT have it be the one to use, but that’s what happened.

I make some changes in the homestead.yaml to set up a simple test site using the classic php test file.

<?php phpinfo(); ?>

Nothing displays. My first error message says “No input file specified”. I fixed that by pointing the map section inside homestead.yaml to the correct folder. Next error: “403 prohibited”. I thought it might be an nginx error, but I did not want to mess around too much with that just yet. However, I did check /var/log/nginx/homestead.test-error.log and I noticed something interesting:

“Unable to open primary script: /home/vagrant/code/test01/public/index.php (No such file or directory)”

OK. I guess I must wrap the simple php test page inside a Laravel template to make it display. Not a big deal, but one more thing I need to figure out. That will be for next time.

That was fun: Angular 5 updates

(This post should have been uploaded in November, 2017, when it happened. I should post more often.)

I noticed that the Angular.io docs were written with Angular 5 in mind. I especially noticed it when the Http methods did not load properly in my project. It turns out that HttpClient was moved to into @angular/common. However, my version of Angular had HttpClient still in @angular/http. I ended up with bad files when I followed the documentation. To stay current, I decided to  update my copy of Angular 4 to Angular 5.

I did a search on how to do the update and I ran across this site: angular-update-guide.firebaseapp.com. Their suggestions helped a lot. I did notice that when I ran the update, I kept seeing messages about ‘invalid’ modules. I’m still not sure why I saw that message, but I’ve seen odd messages from other updates before, so I filed it away to follow up in case the update did not work.

I run my development server with Angular CLI, so I entered ‘>ng serve’ to start the new server. It did not work. I saw an odd message saying my version of ‘angular/compiler-cli needs to be 2.3.1 or greater. Current version is 5.0.2’. Clearly, something was wrong with the angular-cli files. I considered updating the project.json file and updating everything with npm. It turns out I had to be more systematic to make sure everything related was updated properly.

I installed Angular-cli globally, so I had to remove it globally first. I also removed it from the dependency list in the project. I deleted the node modules to make sure every change shown in project.json was accepted correctly. I reloaded angular-cli globally and in the project dependencies again. Finally, I ran ‘>npm install’ to make sure the modules were fresh.

‘> ng serve’ worked as expected. That took longer than I thought.

That would have been nice to know!

I run VirtualBox on my machine to test Ansible playbooks and to run (my legal copy of) Windows 7. I’ve received noticed from VirtualBox that there was a new version of VirtualBox available. (I run 5.1.28. The notice says 5.1.30.) After several weeks of delay, I decide to check it out.

I go to the VirtualBox website and discover that VB is now up to version 5.2.2. This is interesting. I wonder what they fixed? I like that it’s on minor version x.x.2, so they’ve tracked down the errors. I like it. I’ll download it and test it.

It looks good. It starts up as usual. The Windows 7 VM starts as expected. OK, let’s check out ansible. (Record scratch!!!) It won’t run any more. Vagrant 1.9.7 insists that it needs a copy of VirtualBox to run. I have VB installed. This can’t be right. I’ll force it and try again. Still not go with vagrant.

Let me run a Google search. It turns out that vagrant 2 won’t run VirtualBox 5.2 until the next minor update. What’s the newest version of vagrant available? 2.0.1. That’s it. I download it and attach the machine to a good internet pipe to handle the playbook.yml file. It starts up finally. A wasted afternoon on a machine that was not fixed until 9 pm.

Flask validation OR that would have been nice to know!

I’m building some sample sites to display my python skills. In one site, I’m building a app using Flask as the framework to show the web pages. In this page, I need to enter a username that will be used to collect some data from another site. Before I start writing the function to collect and massage the data, I want to be sure I could capture the username entered. I think “No problem”. The Flask module has several examples and tutorials on how to build a form and validate the data entered. I set it up … and it does not work.

I want to set up the same form in two places: as a form field in a navigation bar and as a separate form page, in case there was a problem using the original form field. In the navigation bar, I set up a simple form with one field, the username. The form would send the HTTP request to a submit page, where the form would be validated and sent back to the original form page with an additional  message OR stop and display the full form again in the same separate page (in case someone clicked the submit button in the nav bar without filling in the form field). In both forms, I was sent to the submit page, even after filling out the form field. The form data was never validated in the submit page, so that submit page was redrawn. Why?

I thought there was something wrong with the HTML generated by the flask templates, so I reviewed that HTML. It was fine. I figured out how to include bootstrap classes into flask forms, so that review was helpful. However, I was still stuck on the submit page. I thought there was something wrong with the HTML names and ids of the forms and form fields, but no. Everything was named properly. I reviewed my flask tutorial and the FlaskForm and flask-wtf module documentation pages. My forms.py and views.py pages were set up properly. Wait … What’s this bit about CSRF (cross site request forgery) tokens included in flask forms? That’s nice to use eventually, but I’m doing local testing. Why should I care about CSRF checking?

It turns out that the hidden CSRF field generated by a flask form is what is used to validate the form itself. If the CSRF field is not included in the original form, the form will never validate. That would have been nice to know.

I found one line in the (old) tutorial I used from Miguel Grinberg that says “The SECRET_KEY setting is only needed when CSRF is enabled, and is used to create a cryptographic token that is used to validate a form.” That line does not appear in sections describing form templates or form views, but at the top section describing configuration. I read nothing that confirms this in the flask docs or the flask-wtf docs. However, it appears to be true. The flask CSRF token is required to validate the form. No CSRF token:  no validation. A day and a half wasted.

nice to know: python imports

I am writing a Python command line app that cleans up data scraped from a web page and imports that data into associated tables inside a MySQL database. The top level of the directory holding the python files is getting crowded. I wanted to move the dependent modules into a directory and import those modules into the main file. I find out that I need to add that child directory into a sys.path list before I can import the modules. That’s good to know, but it’s too much for a command line script.

See here for more info: