September 08, 2015

Triggering Cucumber tag from a Cucumber step



I am working on a legacy web application that has limited unit test and dependency on multiple applications.  To reduce the manual testing effort, we had to write end-to-end UI tests with help of Cucumber, Capybara.

We had to chain few tests (generally BAD PRACTICE) as there was no way to create the seed data in every end-to-end test. We had to reuse the data from other scenarios. Rather than going into further details, here is an example where you can trigger another cucumber tag from a step:


child_process.feature

Feature: As a Cucumber user
  I want to run other feature from a step

  @success_child_process
  Scenario: run job
    Given no exception is raised

  @failure_child_process
  Scenario: run job
    Given exception is raised

  Scenario: Child
    Given I trigger @success_child_process job
    And no exception is raised
    And I trigger @failure_child_process job


child_process_steps.rb


Given /^no exception is raised$/ do
  # do nothing here
end

Given /^exception is raised$/ do
  raise 'Exception is raised'
end

Given /^I trigger (@.*) job$/ do |tag|
  CucumberChildProcess.run_tag tag
end

cucumber_child_process.rb


require 'childprocess'

module CucumberChildProcess

  def self.run_tag(tag)
    logger.info "Triggering the child process for tag '#{tag}'"
    process = ChildProcess.build(find_executable('bundle'), 'exec', 'cucumber', '-t', tag)
    process.io.inherit!
    process.start
    logger.info "********* CHILD PROCESS LOGS for tag '#{tag}' - START **********"
    process.wait
    logger.info "********* CHILD PROCESS LOGS for tag '#{tag}' - END **********"
    raise "Child process failed while running the tag '#{tag}'. Check the log for more information. Exit code: #{process.exit_code}" unless process.exit_code == 0
    logger.info "Successfully executed the tag '#{tag}' as Child process"
  end

  def self.find_executable(cmd)
    ENV['PATH'].split(File::PATH_SEPARATOR).each do |path|
      next if path.empty?
      path = File.join(path, cmd)
      ['.exe', '.bat'].each do |ext|
        return path + ext if File.exists?(path + ext)
      end
    end
    raise "Unable to find command '#{cmd}' in the PATH"
  end
end

October 21, 2014

Upgrading to Yosemite - Rails development



It was very simple to upgrade to Yosemite from Mavericks.

Once the OSX upgrade is done, follow these steps to get back to action.


1) Open XCode to install the components
2) Install command line tools
        xcode-select --install
3) Reinstall Ruby
     CC=/usr/bin/gcc  rbenv install 2.1.2
4) Install bundler, libv8 and therubyracer gems
      gem install bundler
      gem install libv8 -v '3.16.14.7' -- --with-system-v8
      gem install therubyracer -v '0.12.1'

If you are using RubyMine, you have to install Java. Apple prompted me to download this when I opened RubyMine.

http://supportdownload.apple.com/download.info.apple.com/Apple_Support_Area/Apple_Software_Updates/Mac_OS_X/downloads/031-03190.20140529.Pp3r4/JavaForOSX2014-001.dmg

March 28, 2014

Bitcoin mining? - Had to close my AWS account

When I read about a similar incident in hacker news sometime back,  I felt sorry for that person and also happy that I had not exposed my secret key. I woke up this morning to see the same thing happen to my account.  I got an email this morning from AWS that my account was compromised and the secret key was available in github.

When I looked at that Github project, I was cursing myself. It was created a year back when I was taking part in a startup weekend event. I don't even remember checking in my secret key in the rush for delivering something over the weekend. Now it has come to haunt me. Someone had launched almost 20 "c3.8xlarge" instances in each region and too many spot instances. Estimated bill is already $300 in a day. It was exhausting for me to find every resource launched in every region and terminate it. I had to close my AWS account to stop further charges and sent an email to the support team and hoping that they would consider that my account was compromised. I am guessing it is bitcoin mining incident again.

I was using this account just for learning AWS and was launching EC2 micro/medium instances to try few things.  I assume most of the devs working in cloud projects would have their personal account to learn AWS and would never ever need to launch such Large instances. I hope that Amazon AWS would provide an option to create a developer account which gives access only to create micro/medium instances for learning AWS and not to worry about huge bill even if the key is exposed by mistake.

Hope that AWS support team helps me this time.

I wish I had the skills like Liam Neeson in "Taken" movie to go after them :)

EDIT:

Estimated bill is now $1534. I called up Amazon immediately and the customer care was very helpful. They informed me that they would arrange for one time credit and asked me to be careful with the Access keys. They also fixed my account and asked me to continue using it.

As suggested by Garp in the comments, the first thing I did is to delete my AWS access keys and created a IAM user with permission to create only t1.micro instances in us-east-1 region.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": "ec2:*",
      "Effect": "Allow",
      "Resource": "arn:aws:ec2:us-east-1:*",
      "Condition": {
         "StringEquals": {
            "ec2:InstanceType": "t1.micro"
         }
      }
    },
    {
      "Effect": "Allow",
      "Action": "elasticloadbalancing:*",
      "Resource": "*"
    },
    {
      "Effect": "Allow",
      "Action": "cloudwatch:*",
      "Resource": "*"
    },
    {
      "Effect": "Allow",
      "Action": "autoscaling:*",
      "Resource": "*"
    }
  ]
}

Amazon customer care is the best. 

August 12, 2013

Legacy project - improving code quality by identifying technical debt

I have always worked on new projects where we decide what technologies/framework/language to use. It is easier to follow good development practices from day one and Continuous delivery in a project is the new norm. More often large enterprises have a legacy projects that has to be maintained and enhanced with new features. 

I recently had an opportunity to work on a consulting assignment where we had to work with client team, up skill them with good development practices and move towards continuous delivery model for their business critical application. It was a large program with 7 scrum teams and each team having 6-8 members. We initially started with couple of scrum teams as a POC(Proof of Concept) and tried to identify the issues/blockers in the process. These are the steps we followed while we tried to improve the code quality of a legacy project.

1. Identifying what to fix
Every project that has good and bad parts of the code. First step in improving a legacy project is to identify what needs to be fixed in the existing system.  
  • List the high risk areas in the codebase based on experience - There will be some parts of the code that no developer wants to touch. Every time we make a change to that piece of code hell breaks loose. Identify those classes/modules and categorize it as High/Medium/Low.
  • Run static code analysis tool (like Sonar for Java project) - Static code analysis tools immediately shows you the current state of the code.
  • Generate Heatmap from Version Control - There are few parts of the code that gets modified very frequently. It is important to identify those classes so that it can be refactored to smaller classes, wrapped with unit tests to improve resilience. 
  • Heatmap - Identify frequent code checkins for fixing bugs. Based on the commit message identify the classes that get frequent bug fixs.     
2. Prioritizing the changes
  • Code which causes most of the defects
  • Code where the new enhancement work is going to happen
  • Bugs identified by the static code analysis tools
  • Refactor code to fix the cyclomatic complexity, large classes/methods, etc based on static code analysis (if it satisfies 1 & 2 conditions)
3. Items to exclude from scope
 This is one of the important lessons I learnt while dealing with legacy projects. If you come across a piece of code that is "working" and if there is no foreseeable changes happening in that code, don't fix/clean the code. Add it to the backlog of "code to be refactored" with a low priority. As a developer it is common to have an urge to clean everything, but we should keep in mind that lot of companies/projects have gone bust by spending their time and money on rewriting software. 
         
4. Scheduling the changes as part of on going work
Once the backlog of tech debt has been identified, we need to plan it as part of on-going project work. Avoid the idea of spending dedicate time (few sprints/months) just for refactoring. This would not fly well with business as they don't see any returns after few months of refactoring.     
I would recommend spending 10% - 40% of each sprint capacity to fixing the Tech debt based on the backlog you had come up with.
Also, In agile world, don't add story points to tech debt stories. You are not delivering new "business value" and it should reflect in the velocity. It would make it obvious to the business on how much time is invested in tech debt and over time velocity would increase as the tech debt decreases.

5. Prepare the team 
If the team is not familiar with TDD and other development practices, it would help in training the team before venturing into refactoring. 
          - TDD techniques
          - Design patters
          - Pair programming
          - Brown bag sessions
          - Code showcase
Few recommended books:
  • Refactoring
  • TDD
  • Working with Legacy code
  • Refactoring DB
  • Pragmatic Programmer 
6. Working on the improvements
  • Hire expert/coach to help team on the job (CRITICAL)
  • Break the silos (Architects, DBA, etc)
  • Retrospect frequently
  • Identify, raise and fix the blockers.
  • Improve efficiency
    • Invest in new infrastructure (Cloud - is not a differentiator, a necessity)
    • Find alternate tools (Tomcat instead of Websphere, Weblogic, etc, Local database for each developer)
    • Good IDE environment
      • Emphasis on using shortcuts
      • Never write code if IDE can generate
    • Faster build
    • Identify bottlenecks
7. Track the improvements
  • Review the static code analysis metrics
  • Review the code coverage metrics
  • Introduce code coverage ratchet (Good and bad.. )
  • Track individual commits
  • Build a dashboard
  • GREEN build is a must
  • Don't refactor if it is not in the prioritized list. This is a important item to track as it would increase the scope of work.
Testing Process & Deployment Process improvements:
Will be covered in the next blog post.    

April 24, 2012

Are you settling for mediocre solution?

"This is just an Intranet application"
"People would hardly use this feature"
"There will be only less than 100 users for this site"
How many times have you heard such arguments during discussions. I have heard similar comments many times and have also made such comments myself. I have just realized that I had been convincing myself with these arguments to settle with mediocre solution for some of the "interesting" problems.

There is one particular requirement in my recent project where I faced similar situation.

Initial approach:

We came up with some sort of a "conventional" solution to address a requirement. This approach was not very clean as it had couple of issues
- We were violating the general rule of the site to support browser back/forward actions
- We were using URL query param to hide/show few links
Even though we had these issues, I was convinced with the solution and wanted to go ahead with that. I had "strong" reasons (read as excuses :) ) for the same.

Reasons (Excuses):

  1. This application doesn't deserve "advanced/complex" features / This requirement is unwanted: This project has only 100 end users. We don't have to spend much time on building this "advanced/complex" features just for these users. I am not convinced yet that this requirement is needed.   

  2. Just enough design/This is an exceptional case: The conventional solution for this requirement works and we don't have to spend much time on this. We can violate the back/forward support for this requirement as it is an exception case.

  3. Story points: It is only a 2 (S/something) pointer and we can't spend much time on this story.

  4. This scenario would never happen: NO user would be performing these sequence of steps (Even though I don't have any metrics to support this claim)

But,
Luckily in my project, all mediocre/conventional solutions were challenged and the team was not convinced with my reasons. We had to come up with a better solution.

When we spent a little more time without thinking about excuses, we were able to come up with a better/cleaner solution. This out of the box solution met all our requirements and also made us feel good about. All it required was that little push from others and a little time to overcome the mediocre solution mode.

Now when I look back at my "strong reasons", everything seem to be flawed:
  1. I could not see the difference between unwanted feature and differentiators. Without these features (related to usability), our site would end up being mediocre. It would not be able to get more users on to the system.
  2. Looks like my notion of "Just enough design" also had "taking exceptions" for granted. In this particular case, I had already convinced myself that this requirement can take exceptions (like not supporting browser forward/backward) without exploring some more possibilities.
  3. Story points are just a metrics to track progress and it should not be used for killing innovation/thinking out of the box.
  4. I think, I had made up the "scenario would never happen" argument to support my "this is an exception" argument.

Lessons learnt:

Mediocre solutions come into existence only by convincing ourselves that it is the right thing to do in THIS  case/application. All it needs is a little more time to come up with a better alternative.
Team members should be constantly challenging each other on mediocre solutions in order to build a great product. It also helps the team to improve their technical and problem solving skills.

Are you coming up with excuses or a better product?