Server automation, part 0

My infrastructure is slowly getting bigger, in spite of everything I do. I decided to research the current CI tools to decide what would be useful to use and what a potential employer would find useful. I’m trying to avoid learning another language. Puppet and Chef were ideas I was considering, but the need to become familiar with another language (Ruby) made me look at other options. On the other hand, I need to relearn Ruby anyways if I’m going to use Capistrano for deployment to remote servers. One thing at a time.

So far, I’m going round and round with Ansible and Salt, both in the Python universe. I’ve also run across something called StackStorm, which could be a possibility. (IFTTT for servers OR “event-driven automation”.) I’ll need to look at that another time to make valid decisions about that.

So far, it looks like Ansible would be easier to use, except for one issue. I’m still unclear if Ansible is useful with Python 3. I know that Ansible 2.2+ does run with Python 3, but it’s unclear if any related Ansible modules I might use are also compatible with Python 3.

I should probably try out at least two applications. For now, I’m going to try out Ansible.

Python errors: I should have known that

I needed to write some data to a MySQL database. I set up the MySQL Python connector without any real trouble. I tested it in a tiny Python page and it does connect to the correct database. Great!

I moved the connection code to a function on a new page and ran into trouble. I kept seeing a message saying something about “Reference error: weakly-referenced object no longer exists”. After a detour into weak references, I realized the issue was garbage collecting. Somewhere within my function, I had a object that was disappearing.

The connection function had no parameters. A connection object local to the function was created and the resulting cursor was returned. You should see the problem immediately. The connection object what the item that was disappearing. I rewrote the function to send back the connection object and then extract the cursor from the returned object. That error message went away.

I had another strange problem where updating a field was not allowed because of a type mismatch. I’m used to PHP, where weak or loose types are the norm. Once I realized that data going into the MySQL table also needs to match the correct type expected by the MySQL column, my problem is solved.

MySQL or PostGres? Why not both? Well …

I have some mini-projects that I would like to move off my laptop and onto the internet for further testing. Almost all my projects involve web-enabled databases, so I need to set up databases to handle the data used by the websites. For now, I want to stay away from NoSQL, which I something I don’t know yet. In that case, which to choose, MySQL or PostGres?

I’ve used MySQL forever. I think my first MySQL book (which I rarely use any more) discusses MySQL 3. I’ve appreciated the transition to MySQL 4, then 5. I know what it does. I have no trouble setting up MySQL PDO statements in PHP. I’m familiar with the MySQL Python modules and can also connect that way, too. However …

MySQL is owned by Oracle. Ever since it was bought by them, there has been an underlying question about how much support the community (free) edition would receive. Oracle’s latest financial snapshot came out recently (June 2016). They made most of their money with their cloud offering, as I understand it. They made no money in support or development. You have to wonder how much longer MySQL will keep going before it stagnates. I could move over to MariaDB, the “open source” version of MySQL, as strange as that sounds. I’m thinking that if I’m going to use a MySQL copy, why not use MySQL?

A few years ago, I took a Saas class through a Berkeley MOOC (CS 169, Agile Development Using Ruby on Rails). They set up accounts on Github and Heroku for their students. I liked how easy it was to migrate code and data onto the cloud using both platforms. My problem at the time was that I was not familiar with them, so it was one more thing I had to learn quickly while taking the class. Ruby favors Postgres as the database and Heroku made it easy to move the data through the command line interface installed through their app. If I’m going to host my projects on the cloud, then I should stick to what makes it easy.

However, I’m very familiar with MySQL. It’s easy to install. I know how to do basic hardening of the database. It’s easy to read/write to MySQL from PHP, using proper credentials. I’m starting over with Postgres. The stock version of PHP that comes with El Capitan does not handle PDO calls to Postgres by default. I have to compile them myself. Which Postgres do I install? Is Postgres.app really that much better? So many questions that I need to answer.

And now I’m looking at cloud services in addition to Heroku. AWS Elastic Beanstalk to use with Docker. Digital Ocean (super cheap). Linode (not as cheap, but I have considered them before.) Hmm …

Catching up with Python

I always forget that major software upgrades in OS X reset permissions in /usr/local. When I checked my brew install, I saw that message. I played around with write permissions only, but in the end, did what brew recommended and reset the permissions as described. That allowed me to update brew and move on to the next task: updating Python3.

I was not as far back as I expected. I was sitting on Python 3.4. The upgrades went smoothly and I’m back to Python 3.5.1. I thought I had modules installed, but I’m not getting any list of local modules installed. It’s very possible, since I never did much with on the desktop. That will be the next task.

Homebrew investigation

After the Mavericks upgrade, I noticed that my copy of Python3 had disappeared. I don’t see the point in staying with Python 2, when Python 3 exists, so I started working with that earlier this year. I discovered that others don’t share my view. For my own material (and as a way to learn something new), I installed Python 3 onto a Mac running Lion. Like I said, Mavericks got rid of it.

After some clean up, I discovered that I needed Python 3 to run the scripts that I had written before the upgrade. I had a clean system (more or less), and I did have some minor trouble when I installed Python 3 the last time. I looked at Homebrew again and thought it could be useful. However, I did not want to get caught by surprises again on my machine. Enter the department laptop.

There’s almost no development software installed on that machine. As it happens, when I ran the instructions on the homebrew web page, I was reminded that I had a non-dev machine. I had to install the Xcode command line tools. (In this case, I think that means GCC.) Any after a successful brew install, I used it to install Python 2 on the laptop.

Like I mentioned, Python 2 is already on the laptop. Brew installs everything into /usr/local and does not wipe out software that already exists, which is helpful. However, the python version was still the old one (2.7.5) and it was still pointing to the original location (/usr/bin/python). “> brew doctor” reminded me that my $PATH had expected locations in the wrong order. It helpfully gave me the one-liner to update my ~/.bash_profile, which I ran. Once it was done and I opened a new terminal window, I had the latest version of Python (2.7.8) and the correct path (/usr/local/bin/python). Upgrading setuptools and pip through pip worked as expected.

I feel confident enough to run brew on the main machine … another day.

How to set up Python 3 on OS X

I never remember, so I need to write this down

  1. Get Python 3 (from Python.org)
    • Hmm … Python 3.4.0 was released a few weeks ago. I may wait another week to see if 3.4.1 is released.
  2. install ‘distribute’, not ‘setuptools’
    • setup_tools is for Python 2, distribute is for Python 3
    • Use this command:
      “> curl -O http://python-distribute.org/distribute_setup.py
    • Go to the folder where it’s located and run
      “>python3 distribute_setup.py”
    • “easy_install” should be included
  3. Create a soft link to easy_install
    • For OS X, it should be in /Library/Frameworks/Python.Frameworks/Versions/3.3/bin, once it’s installed
    • I created an alias in my .profile, but it does not work with ‘sudo’
  4. Run ‘>sudo easy_install pip’ to install pip
    • It should be in /Library/Frameworks/Python.Frameworks/Version/3.3/bin
  5. set up another soft link to pip
    • “> ln -s /Library/Frameworks/Python.Frameworks/Versions/3.3/bin/pip pip”

Neat modules to consider

  • Beautiful Soup 4 (screen scraping)
  • virtualenv (isolated copies of Python)
  • httplib2 (replacement and enhancement for urllib.request.urlopen
  • mysql connector (connects Python to MySQL)

to me, more interesting stuff

The old laptop is running OS X Lion 10.7.5. It has Python 2.7.1 and Python 3.3.0. It also has Ruby 1.8.7.

I seem to remember I had Ruby 1.9.2 installed in an Ubuntu VM. That VM is not running well. (Something happened in a recent update and I can’t get the updates installed.)

I will definitely need to install JewelryBox for multiple Ruby installs. If I can, I’ll also update Python to 3.3.5, if possible. I can see I already have too many projects running all at the same time. This is not good.