Philly ETE: Session 2

Airplanes to Application Development

-- John Kern

ObjectMentor guy talks big mechanical architecture and boils it down to incremental application development. At least that's my take up front. I'm also interested since I now consult at Boeing. Why not check out their most successful product in history (actually I have no idea if that's true, but it is pretty damn successful).

John is an aerospace engineer by training. This is "his angle" and affects the approach he takes to software development.

Rough timeline: early involvement and initial requirements were set in early 1990. The first commercial flight occurred in 1995! That's amazing to me. It's taken us half a year to successfully release a version of a product to the customers of my current client. We're not even talking about a green field project here. The Boeing 777 went from someone's head to a commercial flight in 6 years!

The development team consisted of 240 Design teams with up to 40 members each! Holy crap! They quickly learned lessons about communication to help them cope with such a massive overall team.

Risk mitigation was one of the most carefully adhered-to priciples during the building of the Boeing 777. They had a bunch of huge tests for awhile, and then were able to abandon them after they came across, learned and popularized the concept of CAD modeling. The result was that the Virtual 777 could be assembled, simulated, interference checked. Pretty amazing. Of course, this is actually easier to do with software. Sure, time has to be invested to create good, functional development and staging environments. But these are essential in my mind, not optional ways of adding value to the project.

(As an aside, I'm trying to eat my own dog food by creating a development environment for a project involving an MS Office client developed using C# and communicating with a Rails server on a remote host. It's not trivial. I'm not 100% there yet, but I believe it's important and I'll get there.)

What we end up with is a set of rhetorical questions gleaned from the success of the Boeing 777 project. It looks suspiciously like a checklist for agile software development ("Do you involve the client? test the process? deliver frequently?"). But I don't want to soft peddle this. John relates these questions to his own experience and has interesting takes on their real world application. Also, I learned how few of these things I'm actually doing at my client. Do I take a page out of Uncle Bob's book and just do it because it's the right thing to do? It gets complicated pretty quickly. Yes, I want to release more frequently. But I'm on just one team of many. Can I get just my team to release? What does that even mean in the context of my project? As it turns out, John's simple set of questions have started my mind grapes churning.

John mentioned something I had never heard of, called Semat. This looks suspiciously like YASM (Yet Another Software Manifesto -- I just made that up. You can use it if you want and call it your own. I don't mind). But I'll give it it's due and take a look before I throw it under a bus. (Note: there's also a Semat blog that might give a more informal but informative view of what Semat is all about). It's actually tough to tell if John supports Semat. I have noted that he is not listed as a signatory. OK, enough about Semat.

Thanks to John Kern for all of his insight.

Philly ETE: Session 1

Creating Multithreaded Code Using TDD

-- Venkat Subramaniam

I've chosen this session over the others, mainly because I think Venkat is a seriously smart programmer with a great deal of interesting knowledge to share. I'm curious to see how TDDing multithreaded code diverges from typical TDD practices.

Other questions I have before we start --

If I program in Ruby, should I ever do this, even if the TDD practices are awesome?

Will we get some TDD examples in an FP language like, for example, Clojure? That would be awesome. But I think I have a good enough grasp of Scala to follow along there too.

Interestingly enough, Venkat started out trying to disprove the validity of TDD in multithreaded programming. But in his attempt, he found that he actually WAS able to. Not only that, but the TDD methods he employed saved hours of debugging multithreaded code to find a couple of bugs.

Lots of talk about the importance of TDD. Probably a nice talk for inexperienced TDDers, but for me and the two Cyrus developers to my left, Venkat is preaching to the choir.

There won't be any slides during this talk. I have the highest respect for speakers who run their talks this way. It's risky and sometimes ugly, but it's the closest thing to real-world programming that we can get at a conference. And application to the real-world is what we're all trying to get out of these two days. But I digress.

We're up to the "Let's give it a shot" section. I'm excited. Running this example using Java on IntelliJ IDEA.

First we start with a "canary" test. It is a stupid test that does something like "assertTrure(true); I do something similar with my new Ruby test files, but I don't keep them. There might be a reason to keep them around, particularly when the application is multithreaded. We're starting our example with a couple of classes, MultiValueMap and MultiValueMapTest.

I'm doing my best to keep up with the code. However, I'm using vi (don't have IDEA installed here) so it's going to be rough around the edges. I might post what I have at some later point after I clean it up. No promises though.

Best argument for TDD "it prevents Whack-a-Mole software development."

Now to the multithreaded stuff. If we make a threading testcase, and to make the test pass we make our public void class a public synchronized void class, we run into a wall. We can't test this easily. We need to avoid sticking 'synchronized' in this case.

The next tool to reach for is mocking. But Venkat suggests a more stripped down version of mocking. The result: MyLockMock which extends, not a mocking library, but ReentrantLock.

After we get some threading tests passing, we look into exception handling. He forgoes putting 'try' and 'catch' statements until he can write a properly failing test. I lost the ability to keep up coding along with Venkat at this time, but I got to witness some interesting incremental TDD.

The message is repeated in a number of ways: make it as simple is possible, move slowly, test one discreet piece of functionality at a time. Simple advice that is much better given through coding that through a bunch of slides or beating us over the heads with it.

Venkat never created a single thread to test his multithreaded design. This was the 'revelation' that led him to believe that TDD was possible, even simple, with multithreaded code.

For C#, Venkat recommends anonymous delegates. Good to know in some of my own toy projects.

A personal aside:

Venkat is awesome! He's funny, engaging, takes the right amount of questions, rolls with the changes and questions of the audience and reflects it in his live coding. What an accomplished speaker. I'd watch this guy give a talk on milk pasteurization (or something else I care equally little about).

Philly ETE: A blow-by-blow Account

For the next two days I will try to keep a running log of my time here at the Philly Emerging Tech Conference. The blog posts will likely be rough around the edges because I won't spend much time editing. My goal here is to capture the most important parts of the talks I attend, information I pick up, and things I experience throughout the conference.

Enjoy and please feel free to comment, ask questions, or slam my opinions at your leisure.

Philly ETE : Keynote 1

Bad Code, Craftsmanship, Engineering, and Certification

-- Robert C. Martin

My first experience seeing Uncle Bob martin speak, and I'm very excited. I've read his recent blog post about certification so I'm prepared for something in this vein.

Bad Code: A Crap Odyssey - a video of a single file called two.java scrolling for minutes on end. 30,000 lines of code.

Bob's oft-stated axiom of programming: "The only way to go fast is to go well."

Boy Scout Rule - when you check a module in, it must be cleaner than the one you checked out. Check out the code, make your modifications and clenaup, and then perform one "random act of kindness" on them.

Length of functions - After a history of where the "20 lines max" suggestion came from, he suggested somewhere around 4 - 6 lines. He prefers lots and lots of little functions. Even in Ruby I probably don't adhere to a maximum of 6 lines per function. So a challenge is brought (at least to me) not 30 minutes into this conference.

Functions of extreme length, complexity, obscure variable names, etc... might work for one person. But as soon as the software team grows past the size of 1, any complexity is irresponsible to your fellow programmers.

Uncle Bob knows his function is small enough when it is no longer possible to extract a method. Using an IDE is the easiest way to "extract till you drop."

Number of Arguments - again, the answer to how many arguments a function should take is "as few as possible." But other strong statements were made -- "five is insane," "never pass booleans into your functions" (by the way this is because you are blatantly violating the rule of having each function do only one thing) -- and finally settles on no more than three arguments.

Variable names - nice, long names that describe what they do are preferable in private functions. If the variable has a very large scope (meaning it is accessed from many parts of the application), the name should be more terse. Use a sliding scale of scope to determine how long the variable name should be, but of course do not sacrifice understandability.

Classes - find ways to extract classes when you see large functions - especially is you see many indentations and conditionals. Uncle Bob's practice is to promote local variables to fields when extracting methods is not possible because the scope "returns multiple values." This obviates opportunities to create classes.

Craftsmanship - "it's not the next big thing." It's been around forever. It's simply committing yourself to doing a good job. It's understanding the rules and applying them, over and over, every day. A surprise to me is that there is actually a craftsmanship manifesto. Bob suggests signing it. I'll give it a close read and maybe sign it myself.

Discipline - Hardware has been changed and improved literally dozens of orders of magazine. Software development, however, has not changed nearly so much. We still rely on "assignment statements, if statements, and while loops." Functional programming is a movement to remove the assignment statement. But FP has a bigger view. The movement is towards multicore safety, multithreaded programming that will work through our processors. The functional language that Uncle Bob supports is.... drumroll... Clojure. Hey, me too!

TDD - "the controversy is long gone." Bob likens it to surgeons washing their hands before surgery. It used to be a controversial issue but "the jury's been in on this one for a long time."

Pairing, CI, etc... are also now axioms of best practices in software development. Bob's point here seems to be telling us how obvious it should be for developers to adopt these practices. Noted. I hope it has an effect on the 75% of the audience that does not even practice TDD.

"QA should find Nothing." -- this is the goal. Make them worry about their jobs.

Maybe the most inspiring part of the talk is his last point - "convincing management." Bob's advice -- "don't even ask them." It is the developer's job to do the best work she can. She can't do that without TDD, CI, Pairing. So do them. Don't ask permission from anybody. This is your career and your responsibility to design software the best way you know how.

Uncle Bob's Payroll Case Study: Use Case 2

---------------------------------------------------------
USE CASE 2: DELETING AN EMPLOYEE

Employees are deleted when a DelEmp transaction is received. The form of this transaction is as follows:

DelEmp [EmpID]

When this transaction is received, the appropriate employee record is deleted.

Alternative: Invalid or unknown EmpID
If the [EmpID] field is not structured correctly, or if it does not refer to a valid employee record, then the transaction is printed with an error message, and no other action is taken.
---------------------------------------------------------

A couple of new tests helped to drive my production code. Both were added to the RecordsManagerTest class, so I'll just post my additions:

class RecordsManagerTest < Test::Unit::TestCase
NON_EXISTENT_EMP_ID = 99

def test_delete_employee_deletes_employee_record
@employee = create_one_employee

assert_employee_was_deleted do
@records_manager.delete_employee(@employee.emp_id)
end
end

def test_delete_employee_emp_id_not_found
assert_equal RecordsManager::EMPLOYEE_NOT_FOUND_MESSAGE, @records_manager.delete_employee(NON_EXISTENT_EMP_ID)
end

private
def create_one_employee
@records_manager.add_employee(@new_employee_fields)
assert_employee_was_created
Employee.first
end

def assert_employee_was_created
assert_equal 1, Employee.count
end

def assert_employee_was_deleted
new_employee_count = Employee.count - 1
yield
assert_equal new_employee_count, Employee.count
end

end

As the tests indicate, a new method called 'delete_employee' is added to the RecordsManager class. Let's take a look at how that class looks now:

require 'app/models/salary'
require 'app/models/employee'

class RecordsManager
EMPLOYEE_NOT_FOUND_MESSAGE = "Employee not found.  Cancelling delete."
def add_employee(fields)
new_employee = Employee.new(fields)
return if new_employee.save
error_message_for new_employee
end

def delete_employee(emp_id)
return if Employee.delete_by_emp_id(emp_id)
EMPLOYEE_NOT_FOUND_MESSAGE
end

private

def error_message_for(employee)
employee.errors.full_messages.to_s
end
end

You'll notice the addition of an error message constant, EMPLOYEE_NOT_FOUND_MESSAGE. Originally, I created a module for RecordsManager errors and added my one static error message. Then I realized how much overhead I added just to manage one message and reverted. YAGNI, for now.

In the 'delete_employee' function, the RecordsManager delegates to Employee by way of 'delete_by_emp_id.' Here's the new Employee class:

require 'rubygems'
require 'dm-core'
require 'dm-aggregates'
require 'dm-validations'
require 'app/models/salary'

DataMapper.setup(:default, :adapter => 'mysql',
               :username => 'root',
               :password => 'even-fish',
               :database => 'payroll')
class Employee
include DataMapper::Resource

property :id,       Serial
property :emp_id,   Integer
property :name,     String
property :address,  Text

has 1, :salary

validates_present :emp_id, :name, :address
validates_is_unique :emp_id

def self.delete_by_emp_id(id)
 if emp = Employee.first(:emp_id => id)
   emp.destroy!
 else return false
 end
end
end

Thoughts:

• I had 'assert_difference' in place in my first test function before I realized that this is a method defined by the ActiveSupport library. So I kept the spirit of assert_difference, knowing exactly the 'difference' I wanted to test, and created 'assert_employee_was_deleted.' I like this method because it yields to a block just like assert_difference does, allowing me to be expressive in my test while hiding unnecessary implementation details in the private method.
• An interesting difference between ActiveRecord and DataMapper is in the comparison between the ActiveRecord 'find' functions and the DataMapper 'get' or 'first' functions. At first it felt funny to call Employee.first in order to find the employee record to delete. WAIT THIS STILL SEEMS FUNNY... Can't I use get with a key and value? Either way, I thought it important to add the uniqueness validation to :emp_id in order to feel safe in deleting an individual record.

Uncle Bob's Payroll Case Study: Use Case 1

Today I'm going to begin working through Uncle Bob's "Payroll Case Study" from AGILE SOFTWARE DEVELOPMENT. All of my implementations will be done in Ruby. Pick up your copy of AGILE SOFTWARE DEVELOPMENT and code along with me!

This section begins with the specification of a payroll system. The specs include descriptions of varying types of employees and how they are compensated. Uncle Bob then recommends that the reader "implement the first few use cases test-first." This sounds like a good idea. So I decided to implement the 7 use cases from the book on my own first. Here's the first one:

---------------------------------------------------------
USE CASE 1: ADD NEW EMPLOYEE

A new employee is added by the receipt of an AddEmp transaction. This transaction contains the employee's name, address, and assigned employee number. The transaction has the following three forms:

AddEmp [EmpID] "[name]" "[address]" H [hourly-rate]
AddEmp [EmpID] "[name]" "[address]" S [monthly-salary]
AddEmp [EmpID] "[name]" "[address]" C [monthly-salary] [commission-rate]

The employee record is created with its fields assigned appropriately.

Alternative: An error in the transaction structure
If the transaction structure is inappropriate, it is printed out in an error message, and no action is taken.
---------------------------------------------------------
I recognized (too?) quickly that I would need a database to manage the creating and deleting of employees, so I went with DataMapper. This decision was mostly made because I've never used DataMapper before and I thought this would provide me with an easy introduction.

Here are my files and tests to satisfy Use Case 1. I've provided observations on the crafting of this code below.
records_manager_test.rb

require File.join(File.dirname(__FILE__) ,'/test_helper')
require 'lib/records_manager'

class RecordsManagerTest < Test::Unit::TestCase

def setup
  DataMapper.auto_migrate!
  @any_id = 100
  @any_name = "name"
  @any_address = "address"
  @new_employee_fields = { :emp_id => @any_id,
                           :name => @any_name,
                           :address => @any_address,
                           :salary => new_salary('monthly')
                         }
  @employee_commission = 3.0
  @records_manager = RecordsManager.new
end

def teardown
  Employee.all.each { |e| e.destroy }
  Salary.all.each { |e| e.destroy }
end

def test_add_employee_creates_employee_record
  @records_manager.add_employee(@new_employee_fields)

  assert_employee_was_created
  assert_equal @any_id, new_employee.emp_id
  assert_equal @any_name, new_employee.name
  assert_equal @any_address, new_employee.address
end

def test_add_employee_creates_hourly_rate_employee
  hourly_salary = new_salary('hourly')
  assert_creation_of_employee_with hourly_salary
end

def test_add_employee_creates_monthly_rate_employee
  monthly_salary = new_salary('monthly')
  assert_creation_of_employee_with monthly_salary
end

def test_add_employee_creates_commission_employee
  commission_salary = new_salary('monthly', @employee_commission)
  assert_creation_of_employee_with commission_salary
end

def test_add_employee_fails_when_provided_no_emp_id
  assert_employee_not_created_with_nil :emp_id
end

def test_add_employee_fails_when_provided_no_name
  assert_employee_not_created_with_nil :name
end

def test_add_employee_fails_when_provided_no_ddress
  assert_employee_not_created_with_nil :address
end

private
def assert_employee_was_created
  assert_equal 1, Employee.count
end

def assert_creation_of_employee_with(salary)
  @new_employee_fields[:salary] = salary

  @records_manager.add_employee(@new_employee_fields)
  assert_equal salary, new_employee.salary
end

def assert_employee_not_created_with_nil(field)
  @new_employee_fields[field] = nil
  error_message = "must not be blank"
  assert_match error_message, @records_manager.add_employee(@new_employee_fields)
end

def new_salary(compensation, commission=nil)
  Salary.new :pay_rate => pay_rate(compensation),
             :compensation => compensation,
             :commission => commission
end

def pay_rate(type)
  return 5000 if type == 'monthly'
  return 12.50 if type == 'hourly'
end

def new_employee
  Employee.first
end

end

records_manager.rb

require 'app/models/salary'
require 'app/models/employee'

class RecordsManager

def add_employee(fields)
 new_employee = Employee.new(fields)
 return if new_employee.save
 error_message_for new_employee
end

private

def error_message_for(employee)
 employee.errors.full_messages.to_s
end
end

employee.rb

require 'rubygems'
require 'dm-core'
require 'dm-aggregates'
require 'dm-validations'
require 'app/models/salary'

DataMapper.setup(:default, :adapter => 'mysql',
               :username => 'root',
               :password => 'even-fish',
               :database => 'payroll')
class Employee
include DataMapper::Resource

property :id,       Serial
property :emp_id,   Integer
property :name,     String
property :address,  Text

has 1, :salary

validates_present :emp_id, :name, :address
end

salary.rb

require 'rubygems'
require 'dm-core'
require 'dm-aggregates'
require 'dm-types'
require 'app/models/employee'

DataMapper.setup(:default, :adapter => 'mysql',
                :username => 'root',
                :password => 'even-fish',
                :database => 'payroll')
class Salary
include DataMapper::Resource

property :id,             Serial
property :pay_rate,       Float
property :commission,     Float
property :compensation,  Enum['monthly','hourly']

belongs_to :employee
end

Thoughts:

• DataMapper is a heavyweight solution for the payroll system at this point. I cheated a little bit in looking at Use Case 2, which specifies that we should be able to delete employees. Still, it gave me an excuse to play with it and I'm pretty confident in it being a good solution as the software evolves.

• When I created a Salary model I thought immediately that I would need several types of Salary objects. My first thought was to use STI and I originally built it in for this use case. But then I reverted, eventually deciding that I could postpone this decision until the application gets a bit more complex. For now I'm just dealing with the pain of specifying a type of salary in my test cases.

• You'll notice that function 'add_employee' doesn't really take four or five arguments. It takes a hash because this seemed like a simpler solution. Now DataMapper handles the setting of fields and my RecordsManager object doesn't have to care what type of salary it's being passed. I'm not sure if that's in keeping with the specifications, but I'm going to leave it like that for now.

• I have many duplicated require statements in my models. Also, I'm duplicating my database setup. A future iteration should abstract this away and DRY up the models.

fooberific

Here is some code:

require 'rubygems'
require 'hpricot'
require 'syntax/convertors/html'

hpricot = Hpricot('generate_primes.rb')
convertor = Syntax::Convertors::HTML.for_syntax "ruby"
hpricot.search('//pre[@class="ruby"]') do |e|
e.inner_html = c.convert(e.inner_text, false)
end
puts hpricot.to_s