Category: Coding

The other day, I needed to replace all instances of the old-style Apache license header in the commons-net codebase with the new style license header. My first thought was to write a simple Perl script to extract and replace instances of the license, however I decided to give Ruby a shot. The result is below: LicenseScanner.rb will scan the given directory and all subdirectories for Java source files and attempt to locate and replace any instances of older-style source headers. Being used to Perl for this kind of thing, you really appreciate some of the Perl-isms built into Ruby, for instance:

• “Here” variables – see the definition of @@new_asl_license;
• First class regular expression support, via // and =~.

Here is the source:

require "find"
# Written by Rory Winston <rwinston@apache.org>
# 
class LicenseScanner
  # Older, incorrect license headers - Replace
  @@asl_patt_10 = /(\/\*(.*) \* The Apache Software License, Version 1.1(.*?)\*?(?!\/)\*\/)/mis
  @@asl_patt_20 = /(\/\*(.*) \* Licensed under the Apache License, Version 2.0(.*?)\*?(?!\/)\*\/)/mis
  @@asl_patt_other = /(\/\*(.*) \* Copyright 200[0-9](-200[0-9])? The Apache Software Foundation(.*?)\*?(?!\/)\*\/)/mis

  # New corrected header - Leave as-is
  @@new_asl_pattern = /(\/\*(.*) \* Licensed to the Apache Software Foundation (ASF)(.*?)\*?(?!\/)\*\/)/mis

  # The new license header 
  @@new_asl_license = <<EOL
/*
 * Licensed to the Apache Software Foundation (ASF) under one or more
 * contributor license agreements.  See the NOTICE file distributed with
 * this work for additional information regarding copyright ownership.
 * The ASF licenses this file to You under the Apache License, Version 2.0
 * (the "License"); you may not use this file except in compliance with
 * the License.  You may obtain a copy of the License at
 *
 *      http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */
EOL

  @prompt = true

  def initialize(dir)
    @dir = dir
    @str = ""
  end

  def initialize(dir, prompt)
    @dir = dir
    @prompt = prompt
  end

  def scan
    Find.find(@dir) do |path|
      Find.prune if [".",".."].include? path
      readFile(path) if File.basename(path) =~ /(.*).java$/
    end
  end

  def readFile(filename)
    @str = ""
    @filename = filename
    @str = IO.read(@filename)
    @str.gsub!("\015", "")
    scanFile
  end

  def scanFile
    if @str =~ @@asl_patt_10
      puts "Found Apache 1.x License in #{@filename}"
      replaceLicense(@@asl_patt_10)
    elsif @str =~ @@asl_patt_20
      puts "Found other Apache 2.0 License in #{@filename}"
      replaceLicense(@@asl_patt_20)
    elsif @str =~ @asl_patt_other
      puts "Found Apache 1/2 license in file in #{@filename}"
      replaceLicense(@asl_patt_other)
    elsif @str =~ @new_asl_pattern
      puts "Correct license found in #{@filename}"
    else
      puts "No license found in #{@filename}"
      insertLicense
    end
  end

  # Replace an existing Apache 2.0 license with the new format
  def replaceLicense(pattern)
    resp = confirm "Replace license in #{@filename}"

    case resp
      when "y","Y"
        # Open existing file and truncate
        puts "Replacing license in #{@filename}..."
        srcFile = File.new(@filename, "w+")
        srcFile.puts  @str.sub(pattern, @@new_asl_license)

        srcFile.close
      when "n", "N"
        puts "Skipping..."
    end
  end

  # Insert a new license at the beginning of the file
  def insertLicense
    resp = confirm "Insert license in #{@filename}"

    case resp
      when "y","Y"
        puts "Inserting license in #{@filename}..."
        srcFile = File.new(@filename, File::TRUNC | File::RDWR)
        srcFile.puts  @@new_asl_license
        srcFile.puts @str
        srcFile.close
      when "n", "N"
        puts "Skipping..."
     end

  end

  def confirm(msg)
    if @prompt != true
      puts "#{msg} [y/n]?"
      resp = gets
      resp.chomp!

      while (resp !~ /[YyNn]/)
        puts "#{msg} [y/n]?"
        resp = gets
        resp.chomp!
      end
    else
      resp = "y"
    end

    resp
  end

end

# Usage: LicenseScanner.new("path/to/src/dir", auto-replace [true/false])
scanner = LicenseScanner.new("c:/sandbox/net/src/main/java", true)
scanner.scan

A post on the Jakarta User mailing list piqued my interest this week. The poster had noticed that they were getting significantly different results for some statistical measures from the output of Commons::Math vs. what Excel was producing. This, if true, would be a pretty serious situation. Excel’s calculation engine is proven and very mature (I know one of the guys who works on it, and he’s a genius), so any discrepancy would seem to point to Commons::Math.

Needless to say, the best way to verify this is with a simple “spike”, as the agile guys would say. So I fired up Excel, and using its random number generator, produced 20,000 normally distributed numbers with a standard deviation of 1,000 and a mean of 5,000. I used the Tools > Data Analysis add-in to do this, but you could also use the =NORMINV(rand(),mean,standard_dev) function.

When this was complete, I exported the data to a text file, and read in the values and calculated some simple stats using Commons::Math. Here is the sample program, if you’re interested:


package uk.co.researchkitchen.math;

import java.io.BufferedReader;
import java.io.FileReader;
import java.io.IOException;

import org.apache.commons.math.stat.descriptive.moment.Mean;
import org.apache.commons.math.stat.descriptive.moment.StandardDeviation;
import org.apache.commons.math.stat.descriptive.moment.Variance;
import org.apache.commons.math.stat.descriptive.rank.Median;

public class TestPrecision {
  
  static final int DATA_SIZE = 20000;
  
  double[] data = new double[DATA_SIZE];
  
  public static void main(String[] args) throws IOException {
    TestPrecision testPrecision = new TestPrecision();
    // Book1.txt is an exported Excel spreadsheet containing
    // 20,000 normally distributed numbers with a mean of ~5,000
    // and a stdev (sigma) of ~1,000
    testPrecision.calculateStats("c:\\temp\\Book1.txt");
  }
  
  public void calculateStats(String filename) throws IOException {
    BufferedReader br = new BufferedReader(new FileReader(filename));
    
    int count = 0;
    String line = null;
    while ((line = br.readLine()) != null) {
      double datum = Double.valueOf(line);
      System.out.println(datum);
      data[count++] = datum;
    }
    
    System.out.println("Read " + count + " items of data.");
    
    System.out.println("Standard deviation = " + new StandardDeviation().evaluate(data));
    System.out.println("Median = " + new Median().evaluate(data));
    System.out.println("Mean = " + new Mean().evaluate(data));
    System.out.println("Variance = " + new Variance().evaluate(data));
  }

}

I then went back to Excel and calculated the same measures in there (calculating both the 1/(N-1) sample and 1/N population standard deviation and variance). Here are the tabulated results:

As suspected, they are almost identical, bar some rounding differences, at least an order of magnitude closer than the figures in the post (I told Excel to limit the precision to 10 digits, hence some figures seem a smaller precision). I don’t know what precision Excel uses internally for these calculations. It may be interesting to write a BigDecimal-based equivalent of the statistics package in Commons::Math.

There may be other reasons why the numbers given in the example don’t match, but unless I’m missing something obscure, or specific to the use case shown, it looks like the issue is not with the internal implementation of [math] (which incidentally looks like a very very neat little toolkit).