Migrate from Boot2Docker to Docker Machine on OSX

Just learned that Docker deprecated official support to boot2docker and now official one is Docker Machine (at the time of writing 09/26/2015 is still in beta)

Step 1. is to Download binary from Docker Toolbox

Step 2. Stop boot2docker (boot2docker stop) and make sure to quit virtualBox. It might be a good opportunity to upgrade VirtualBox (make sure VirtualBox version is 4.3.28 or greater) 

Step 3. Run Docker Machine Installer. It will migrate your container etc for you.

Step 4. Once you get to Quick Start (see above) I open it in terminal. The biggest difference between Boot2Docker and Docker Machine is that Docker Machine allow you to create multiple machines so you have to specify which machine you are working on.

When migration is completed, installer will create default machine called "default" so to start the default docker machine, you specify docker machine (see below)

Now you should be able to run your familiar docker command but it didn't for me because I got timeout.... and looks like it is still looking at boot2docker vm which I already shutdown...

I realized that the following entries are in the .bash_profile. so I removed it and reload bash_profile

# docker:
    export DOCKER_CERT_PATH=/Users/naoko/.boot2docker/certs/boot2docker-vm
    export DOCKER_TLS_VERIFY=1
    export DOCKER_HOST=tcp://192.168.59.103:2376
alias docker="docker --tlsverify=false"

and add docker-machine version of it:

eval "$(docker-machine env default)"

reload the .bash_profile and ta-da!

Python API Load Test in 5 min with Locust - A modern load testing framework

I was looking for simple way to set up load test for my API application (Django with DRF).
I first tried multi-mechanize but I had a few issue installing dependencies such as Matlab and decided that I don't have time for this... and encountered Locust.

With this great example, I had my API Load Test up and running in matter of 5 minutes.

Writing Unit Test with Jasmine: How to test window.history

Here is how I wrote test to cover my back button directive in AngularJS.

my directive look like this:

app.directive('goBack', function ($window) {
    return {
        template: '<button id="button_goback" title="Go Back to Previous Page" class="btn btn-default" style="">Back</button>',
        restrict: 'E',
        replace: true,
        link: function (scope, element) {
            element.bind('click', function () {
                $window.history.back();
            });
        }
    };
});

and what I want to test is:

- I don't have to test JavaScript library to see if actually url was changed
- I want to make sure when this button is clicked, history.back is called

and here is my test look like:

it('should call history.back when back button is clicked', inject(function () {
        spyOn($window.history, 'back');
        element.find('button').click();
        expect($window.history.back).toHaveBeenCalled();
    }));

Being Productive: Human should never fix JSHint violation by hand (with WebStorm)

command + , to bring preference

search "jshint"

click "enable" and specify config file

above setting will allow you to see violation

then hold down (option + command + L) to bring the screen below

click "Run" to fix all JSHint issue

"Yay"

Continuous Integration and Deployment with Github

[CODE] ⇢ ⇢ ⇢ ⇢ ⇢ ⇢

Jenkins:

I didn't have good experience with Jenkins. In summary, something caused Jenkins to crash and I had to reinstall again... I repeated a few times and I decided that this should be done by someone else.
So I then went for hosted Jenkins. It was better but things weren't working everything perfectly just yet especially when creating test database.

I need super easy way to achieve Continuous Integration:

My co-worker mentioned that people at one conference was raving about codeship. Tested out free subscription. It is ridiculously simple and intuitive (just like when I found ansible while I was struggling with puppet). Best of all, there is small "?" button on bottom right corner and you can ask questions when you stuck and knowledgeable engineer would respond to you within 24 hours or so. Just fantastic.

Now to achieve Continuous Deployment:

What I end up doing is to setup deploy server which runs small flask (python micro framework) application and listens one type of request which is POST request from Github Webhooks.
Flask app will read payload from github and trigger ansible deployment script (via fabric) if commit / merge is made to master.
Once deployment is completed, it will post to private twitter account (this piece of code also on fabirc which is wrapper to ansible) that our team subscribes. The post message includes dev's commit message as well. I like this so far.
Because we are python shop and this approach will give us not only flexibility but ability to debug any issues easily.

Play with Angular: Get help from YEOMAN and Debugging with WebStorm

note: AngularJS is amazing.
warning: don't think in jQuery, think in AngularJS 

the developer is trying to "do jQuery" in the context of AngularJS. That's never going to work well. The view is the official record. Outside of a directive, you never, ever, never change the DOM. And directives are applied in the view, so intent is clear.

1. Let's start on the right foot: scaffolding workflow
2. Configure grunt server on WebStorm: so no need to start server from terminal app
3. Productive: Install AngularJS extension: more autocomplete to be productive
4. Debugging: Setup Debugger: so that we can be stress free (or minimum stress)
5. Process: Let generator do the job
6. Learning: Study & Get Help

1. Let's start on the right foot

Let Yeoman do the job. Yeoman will help us "scaffolding workflows for creating modern webapps, while at the same time mixing in many of the best practices that have evolved within the industry."

Install the required tools globally by running:

$ npm install -g yo

above command will installs grunt, karma etc
Karma: unit test framework specifically made for angular based application
Grunt: JavaScript task runner - automate various repetitive tasks such as js minification, reload server on change etc

$ npm install -g generator-webapp
$ npm install -g generator-angular 

I am going to call my application angular_play for this test project 
$ mkdir angular_play

open the directory you just created (angular_play in this case) from WebStorm

from WebStorm terminal interface, run the following to scaffold out a AngularJS project
$ yo angular  

install a dependency for your project from Bower
$ bower install angular-ui    

test your app     
$ grunt test

preview your app (formerly `grunt server`)              
$ grunt serve 

build the application for deployment             
$ grunt                            

Thank you YEOMAN!

2. Configure grunt server on WebStorm

"edit configuration"

"add new configuration" (command + N)
select Node.js and enter following parameters:

play "Grunt Server"

3. Install AngularJS extension 

WebStorm > Preferences > Install JetBeans plugins > Browse Repositories > "angluarjs" > double click search result and install

Ta-da! nice autocomplete and other support for all of the ng-goodness 

4. Setup Debugger

4.1. with WebStorm:

WebStorm > Edit Config > add "JavaScript Debug" > Specify file you want to debug > Click debug

Every time you reload page, it stops at breakpoint you specified!

4.2. with AngularJS Batarang Chrome Extension:

You can install AngularJS Batarang Chrome debugging extension for debugging and profiling. This performance tab comes in handy when you use function like $watch (listen for change)

4.3. Set "Pause on all exceptions" to halt when an exception occurs

Chrome > Dev Tools > Sources > toggle with icon looks like pig nose 

more about Chrome JavaScript Debugging

5. Let generator do the job

Why not use generator? 

from WebStorm terminal, just run command available 

6. Study & Get Help

egghead.io: web development training with AngularJS

read good AngularJS book
join AngularJS google group

Lastly....

feeling about AngularJS shown below (from Mr. Ben Nadel's blog) seems to be fairly common among devs. So let's not give up until you feel like "this is most awesome framework ever!"

Server Provisioning Tools

What's Ansible?

per ansible:

Ansible is a radically simple IT automation platform that makes your applications and systems easier to deploy. Avoid writing scripts or custom code to deploy and update your applications— automate in a language that approaches plain English, using SSH, with no agents to install on remote systems.

With vagrant, bringing server up from scratch and running with same state was just ridiculously easy.

Why did I choose Ansible over other tools?

My qualification and situation when I chose Ansible:

I am in an ultra time crunch right now but need to have provisioning tools up and ready in a few days
I am a developer and not full time sysadmin but I have administrated servers for last ten years mainly software level and nothing network level
I am proficient in shell and python

Process of selecting tools:

Quick Lookup: according to this article, SaltStack is rated #1 then puppet, chef and ansible... but ansible is simpler tool than puppet or chef. I like simple than complex...

Attempt 1: Last time I went to Django Conf, peopler were raving about SaltStack. So that was the very first one I tried. Installing software was easy. Test against localhost was successful. Attempt to talk to Server was not so successful and error wasn't verbose enough (I probably didn't know how to make it verbose) but I quickly moved off to seek tools that give me obvious error message.

Attempt 2: So next simple tool would be ansible. Yes. things are up and running in matter of 10 min. Amazingly easy to use. With -v option, error is verbose and clear and even give me tips how I may fix it. Plus amazing community. I posted question and got an answer within a few hours and solved the problem I was stuck on.

Our Ops wanted to use puppet because of robustness so now I am giving try puppet

Attempt 3: Installation of regular puppet required me to install 3 sets of DMG for osx whereas ansible, you just need pip install of a few packages.

Standard puppet seems to be fine. pp file extension was recognized by my favorite IDE Pycharm so I was pretty happy about it.

We wanted to bring up server (not just provisioning) with puppet which requires enterprise version. Ops setup PE (puppet enterprise) server but I have no idea where I go from here... in order to issue "puppet node_vmware create"

This enterprise version was confusing (for Unix - i use osx - 3.7GB worth of software is needed whereas for Linux you will need 300 to 400 mb of software)

So I just gave up until someone who have done can give me tip... (including watching youtube video)

Until then, my tools of choice had to be ansible with my limited knowledge (which I am very happy with)

I am planning to study puppet for next few weeks and post update. Also I will post some ansible tips.

fun with solr - day1: installation and simple example

I had urge of learning something new.
Per dice.com, there is a high demand of Solr skill. Solr is high performance NoSQL search platform/server built using Lucene Core, with XML/HTTP and JSON/Python/Ruby APIs, hit highlighting, faceted search, caching, replication, and a web admin interface.

Watched Yonik Seeley's presentation and Solr 4. The NoSQL Database on YouTube and was very fascinated by that.
List of sites who is already using is pretty impressive too and I just saw serious potential to our new app conversion.

So I decided to learn Solr as my weekend project!

before you have fun with Solr, you want to make sure java is installed.

to check, you just run: $ java -version
on my macbook, it was already installed
$ java -version
java version "1.6.0_51"
Java(TM) SE Runtime Environment (build 1.6.0_51-b11-457-11M4509)
Java HotSpot(TM) 64-Bit Server VM (build 20.51-b01-457, mixed mode)

but not on my EC2 instance so I installed with apt-get
$ sudo apt-get install openjdk-7-jre-headless

Install & Run Solr with Jetty

What is Jetty: Jetty is a pure Java-based HTTP server and Java Servlet container.
For playing purpose, single Solr instance with small Jetty version that already included in solr package is good enough. So let's get it started.

# download binary and unzeip.
$ curl -O http://apache.mirrors.hoobly.com/lucene/solr/4.4.0/solr-4.4.0.tgz
$ tar -xvf solr-4.4.0.tgz
# clean up
$ rm solr-4.4.0.tgz
$ cd solr-4.4.0/example
$ java -jar start.jar

and point your browser to: http://localhost:8983/solr/
Wow that was easy and Amazing interface!

if you are also setting up on EC2 you want to check following setup so that your server can serve tomcat via public ip
(1) go to Elastic IP to associate your instance
(2) go to Security Group > Inbound and make sure 8983 is open

 

Add and Retrieve document (example from Yonik's presentation)

naoko-macbook:~ nreeves$ curl http://localhost:8983/solr/update -H 'Content-type:application/json' -d'

 [

 {"id": "book1",

  "title": "American Gods",

  "author": "Nail Gaiman"

 }

 ]

 '

{"responseHeader":{"status":0,"QTime":64}}

naoko-macbook:~ nreeves$ curl http://localhost:8983/solr/get?id=book1

{
  "doc":
  {
    "id":"book1",
    "title":["American Gods"],
    "author":"Nail Gaiman",
    "_version_":1447566597403181056}}

How cool.

python: advanced sort with sorted and lambda

I was working on Django project. I had need of order by QuerySet in somewhat complex way.
Basically, I am listing family but list "Spouse" relationship at the top.

Say I have data structure like this:

class Family:
    def __init__(self, type, name):
        self.type = type
        self.name = name

    def __repr__(self):
            return repr((self.type, self.name))

fam = [
    Family("child", "Joi"),
    Family("child", "Leona"),
    Family("parent", "Toshiko"),
    Family("spouse", "Coy"),
    Family("parent", "Akira"),
]

In SQL statement (PostgreSQL) I can express like this.

SELECT * FROM family WHERE ......

ORDER BY CASE WHEN type='Spouse' THEN 'aaa' else type END

Easy enough.
Now I wasn't sure how to do this in Django.
With raw sql or extras filter code becomes pgSQL specific solution so I wanted to avoid if possible.
So I decided to go with pure python function "sorted" since set of data I will be dealing with is small (just family members... so usually like 4 or 5 and at most... 10?)

Here is how I was able to accomplish.

>>> print(fam)
[('child', 'Joi'), ('child', 'Leona'), ('parent', 'Toshiko'), ('spouse', 'Coy'), ('parent', 'Akira')]

>>> family = sorted(fam, key=lambda x: ("aaa" if x.type == "spouse" else x.type)) 
>>> print(family)
[('spouse', 'Coy'), ('child', 'Joi'), ('child', 'Leona'), ('parent', 'Akira'), ('parent', 'Toshiko')] 

I don't probably need to explain what lambda or sorted with key param is as documentation and thousands of other posts are doing great job but I wanted to post this because I could not find example that shows this level of "key" for sorted function.
"key" will be return ('aaa') for Spouse and (<type>) for other relationship.

for x in family:
    key=lambda x: ("aaa" if x.type == "spouse" else x.type)
    print("%(name)s has type of %(type)s" % {"name": x.name, "type":key(x)})


>>> Coy has type of aaa
>>> Joi has type of child
>>> Leona has type of child
>>> Akira has type of parent
>>> Toshiko has type of parent

Ah-ha!

If you want to further specify sort order, you can add sort by name followed by type like this too.

family = sorted(fam, key=lambda x: ("aaa" if x.type == "spouse" else x.type, x.name))

Hope this magic became clear to you as well.

python mock: Diffference between Mock and MagicMock

note: why use mock/fake/stub object for unit test:

Using mock objects allows developers to:

1. write and unit-test functionality in one area without actually calling complex underlying or collaborating classes
2. focus their tests on the behavior of the system under test (SUT) without worrying about its dependencies

When mock objects are replaced by real ones then the end-to-end functionality will need further testing. These will be integration tests rather than unit tests.

Mock and MagicMock objects create all attributes and methods as you access them and store details of how they have been used.

Mock allows you to assign functions (or other Mock instances) to magic methods and they will be called appropriately. The MagicMock class is just a Mock variant that has all of the magic methods pre-created for you (well, all the useful ones anyway) - (per http://www.voidspace.org.uk/python/mock/#quick-guide)

example:

>>> import mock
>>> mock1 = mock.Mock()
>>> mock2 = mock.MagicMock()

>>> mock2.__str__.return_value = 'foobarbaz'
>>> str(mock2)
'foobarbaz'
>>> mock2.__str__.assert_called_with()

>>> mock1.__str__.return_value = 'foobarbaz'
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: 'method-wrapper' object has only read-only attributes (assign to .return_value)

# ah, there it is, i have to configure __str___ behavior

>>> mock1.__str__ = mock.Mock(return_value='wheeeeee')
>>> str(mock1)
'wheeeeee'

SonarQube to analyze python code on mac

SonarQube is an open platform to manage code quality. As such, it covers the 7 axes of code quality:
- architecture & design
- duplications
- unit tests
- complexity
- potential bugs
- coding rules
- comments


This is the minimum configured way to install SonarQube and start analyzing your python code on your mac (I run 10.8.5 but I think this should work on older os x as well)
Estimated time of completion: 10 min

1. Install SonarQube (server to host result)
1. go to http://www.sonarqube.org/downloads/ and click "Download" link and unarchived (I downloaded sonar-3.7)
2. move unarchived directory to wherever you want to place (I created dir called sonar. so far, i have sonar/sonar-3.7/ and I will reference this dir as SONARQUBE_HOME) 
3. cd SONARQUBE_HOME/bin/macosx-universal-64
4. to start server: $ sh sonar.sh start
1. at this point, if you start server (), you should be able to see interface via browser @ http://localhost:9000/
2. Install SonarQube Runner (server to host result) 
1. Download from here (at the time of writing 2.2 is the latest version) and unarchived it
2. move unarchived directory to wherever you want to place (I moved under sonar I created at step 1. I now have sonar/sonar-runner-2.3/ and I will reference this dir as SONAR_RUNNER_HOME)
3. Edit SONAR_RUNNER_HOME/conf/sonar-project.properties
• sonar.projectKey=my:project
• sonar.projectName=project
• sonar.projectVersion=1.0
• sonar.projectDescription=project description
• sonar.source=path-to-project-dir
• sonar.language=py
3.  Install python plugin
1. Download from here (at the time of writing 1.1 is the latest)
2. copy downloaded jar file to SONARQUBE_HOME/extensions/plugins/
4. Restart the SonarQube server
5. Run SonarQube Runner to analyze
1. note: I was getting "Java heap space" error. In my case, runner needed 269m. You can solve this by increasing limit and run as follows (example shows to set it to 1g but you can do 512m etc as well): SONAR_RUNNER_OPTS=-Xmx1024m bin/sonar-runner
2. now you can review your result @ http://localhost:9000/