Narkisr

Tripping

Sat, 08 Oct 2011 00:01:27 -0400

Securing your Linux server takes the same Unix philosophy as other parts of the system, meaning you have an assortment of tools each does its own special thing and does it well.
One of these tools is Tripwire an intrusion detection tool that safe guard the servers filesystem integrity, Tripwire creates a hash DB of a configurable files list, on each check tripwire report back on suspicious changes.
In this post ill showcase how to set it up on your Ubuntu server, starting with installation, while some posts suggest installing from source id rather go with the pretty latest version in the repos:

 $ sudo aptitude install tripwire

During the installation you will be asked for a site and local passphrase, generate some good passwords for those and get ready for configuration.

Tripwire configuration consists of two main files, twcfg.txt and twpol.txt (both under /etc/tripwire), its suggested to encrypt and back them up after configuration is done (so that the bad guys won't know what folders your watching>?), the twpol (policy) file include which folders/files to watch for changes and the severeness level that such changes imply.
Unfortunately the Ubuntu package generates a file that point to /proc, this is problematic since it contains files that represent process (those are bound to change), go ahead and apply the following in the policy file:

#
# Critical devices
#
(
  rulename = "Devices & Kernel information",
  severity = $(SIG_HI),
)
{
        /dev            -> $(Device) ;
        /dev/mem                                        -> $(Device)   ;
        /dev/null                                       -> $(Device)   ;
        /dev/zero                                       -> $(Device)   ;
        /proc/devices                                   -> $(Device)   ;
        /proc/net                                       -> $(Device)   ;
        /proc/tty                                       -> $(Device)   ;
        /proc/sys                                       -> $(Device)   ;
        /proc/cpuinfo                                   -> $(Device)   ;
        /proc/modules                                   -> $(Device)   ;
        /proc/mounts                                    -> $(Device)   ;
        /proc/dma                                       -> $(Device)   ;
        /proc/filesystems                               -> $(Device)   ;
        /proc/interrupts                                -> $(Device)   ;
        /proc/ioports                                   -> $(Device)   ;
        /proc/kcore                                     -> $(Device)   ;
        /proc/self                                      -> $(Device)   ;
        /proc/kmsg                                      -> $(Device)   ;
        /proc/stat                                      -> $(Device)   ;
        /proc/loadavg                                   -> $(Device)   ;
        /proc/uptime                                    -> $(Device)   ;
        /proc/locks                                     -> $(Device)   ;
        /proc/version                                   -> $(Device)   ;
        /proc/meminfo                                   -> $(Device)   ;
        /proc/cmdline                                   -> $(Device)   ;
        /proc/misc                                      -> $(Device)   ;
}

Another spot I had to adjust is the /root part:

# These files change the behavior of the root account
(
  rulename = "Root config files",
  severity = 100
)
{
        /root                           -> $(SEC_CRIT) ; # Catch all additions to /root
        /root/.bash_history             -> $(SEC_CONFIG) ;
        /root/.bashrc                     ->  $(SEC_CONFIG) ;
        /root/.profile                     ->  $(SEC_CONFIG) ;
}

Now we are ready to apply our configuration changes:

$ sudo twadmin --create-polfile --cfgfile /etc/tripwire/tw.cfg --site-keyfile /etc/tripwire/site.key /etc/tripwi
re/twpol.txt

Lets create our initial DB:

  $ sudo tripwire --init --cfgfile /etc/tripwire/tw.cfg --polfile /etc/tripwire/tw.pol --site-keyfile /etc/tripwire/site.key --local-keyfile /etc/tripwire/narkisr.com-local.key

And run our first check:

   $ sudo tripwire --check
    # ...
    
    Rule Name                       Severity Level    Added    Removed  Modified 
    ---------                       --------------    -----    -------  -------- 
    Invariant Directories           66                0        0        0        
    Tripwire Data Files             100               0        0        0        
    Other binaries                  66                0        0        0        
    Tripwire Binaries               100               0        0        0        
    Other libraries                 66                0        0        0        
    Root file-system executables    100               0        0        0        
    System boot changes             100               0        0        0        
    Root file-system libraries      100               0        0        0        
    (/lib)
    Critical system boot files      100               0        0        0        
    Other configuration files       66                0        0        0        
    (/etc)
    Boot Scripts                    100               0        0        0
    Security Control                66                0        0        0
    Root config files               100               0        0        0
    Devices & Kernel information    100               0        0        0

  Total objects scanned:  12050
  Total violations found:  0

The report list all the changes that were made to you system since the last check, lets play a bit:


   $ sudo touch /bin/bla
   $ sudo tripwire --check
   # ... 
    -------------------------------------------------------------------------------
    Rule Name: Root file-system executables (/bin)
    Severity Level: 100
    -------------------------------------------------------------------------------

    Added:
    "/bin/bla"

    Modified:
    "/bin"

As you can see tripwire warns us about the new file, lets fix it up:


   $ sudo rm /bin/bla
   $ sudo tripwire --check

   #... /bin has still been changed
    -------------------------------------------------------------------------------
    Rule Name: Root file-system executables (/bin)
    Severity Level: 100
    -------------------------------------------------------------------------------

    Added:
    "/bin/bla"

    Modified:
    "/bin"

   $ sudo tripwire --update -Z low --twrfile /var/lib/tripwire/report/narkisr.com-20111007-185529.twr
   # ... 
    Remove the "x" from the adjacent box to prevent updating the database
    with the new values for this object.

    Modified:
    [x] "/bin"

Tripwire marks changes with "[x]" giving us the options to accept changes that were made as legal, further checks will not warn us about them.
The last thing left is to automate the reporting to that we can follow if things break on a regular basis:

      $ sudo cat /etc/cron.d/tripwire
      0 2 * * * /usr/sbin/tripwire --check | /usr/bin/mail "narkisr@somewhere.com" -s "Tripwire Check" 2>&1

Ubuntu PXE-ing, not a walk in the park

Fri, 30 Sep 2011 05:28:27 -0400

Ubuntu is one of the most user friendly systems both on the desktop side and on the server side, still there is one part in which Ubuntu requires a bit of tinkering which is pre-seeding automatic desktop installations.
Most examples out there are targeted for server installations, in this post will explain how to setup a complete automatic installation for Ubuntu desktop (PXE boot with preseeding).
Note that ill be using cobbler as the PXE server and apt-mirror but I won't be covering those in detail (there is enough material out there about these).

First a couple of perquisites:

Get Ubuntu alternate cd, the plain desktop cd cannot be used in automatic installations.
Install Cobbler on the server from which your going to serve your installations, following these instructions.
Setup apt-mirror, you can follow my guide.

Now that we are all set will import the alternate cd into cobbler:

  $ sudo mount -o loop ubuntu-11.04-alternate-amd64.iso /media/loop
  $ sudo cobbler import --name=ubuntu-alternate  --path=/media/loop --breed=ubuntu --os-version=natty

Next we will add entries to our /etc/apt/mirror.list file, this is required in order to enable our local mirror during the installation:

  deb http://host/mirrors/ubuntu natty main main/debian-installer restricted restricted/debian-installer

Now comes the fun part, preseeding, iv found it to be a black art that involves a lot of trial and error, feed the following preseed file within cobbler, will go through the important options starting with using our local apt-mirror during installation:

# Mirror settings
# setting country string manual is crucial otherwise the mirror setting will not work!
d-i     mirror/country string manual
d-i     mirror/http/hostname string mirror-url
d-i     mirror/http/directory string /ubuntu
d-i     mirror/http/proxy string
d-i     mirror/udeb/components multiselect main, restricted

Next is partitioning without Lvm which is a bit tricky:

# partitioning
d-i     partman-auto/method string regular
d-i     partman-lvm/device_remove_lvm boolean true
d-i     partman-lvm/confirm boolean true
d-i     partman/confirm_write_new_label boolean true
d-i     partman/choose_partition        select Finish partitioning and write changes to disk
d-i     partman/confirm boolean true
d-i     partman/confirm_nooverwrite boolean true
d-i     partman/default_filesystem string ext4

Now that we partitioned we can proceed to package selection, if we don't tell the installation to setup the Ubuntu desktop we will be left without any GUI, the following sets this up:

tasksel tasksel/first multiselect ubuntu-desktop

Specifying more packages is possible using:

# Making it puppet ready
d-i     pkgsel/include string byobu vim openssh-server puppet

You can now create a profile within cobbler that will be inherited by systems to be installed, make sure to set:

Mac address for systems (so that cobbler will connect the dots).
The preseed file within the profile.
The following kernel opts priority=critical locale=en_US (without this the preseed will be ignored).

All things should have been ready now for installation but unfortunately cobbler has the following bug, cobbler creates the following /var/lib/tftpboot/pxelinux.cfg/MAC-ADDRESS file for each system:

default linux
prompt 0
timeout 1
label linux
        kernel /images/ubuntu-alternate-x86_64/linux
        ipappend 2
        append initrd=/images/ubuntu-alternate-x86_64/initrd.gz  locale=  priority=critical locale=en_US text  auto url=http://192.168.x.x/cblr/svc/op/ks/system/playing hostname= domain=local.lan suite=natty

The last file within this file states the kernel options that will be used for the system, notice the empty hostname value, this value takes precedence on what ever value we set in the preseed file, making us a bit jammed. The only workaround is to somehow set this value right before the installation takes place, ill update this post if once ill reach the final solution but right now ill be using a simple web service to patch this file.

Pin puppet free

Sat, 04 Jun 2011 01:23:27 -0400

One of the best things I like about Linux (and Ubntnu in particular) is the package management, yet the main drawback of non rolling distros repositories is that you get might stuck with old package versions until the next release comes along.
Usually you can find a PPA or a vendor package that gives you the new goodies, in the case of puppet your left with 2.6.4, while you could build puppet from source (its ruby after all) the integration with OS is lost.

After some frantic googling all iv found was this which states the ability of using debian packages as the source for latest puppet version.
Reading about apt pinning and the strong recommendation to use source packages instead lead me to the following solution:

echo "deb-src http://ftp.us.debian.org/debian/ wheezy main" | sudo tee -a /etc/apt/sources.list
sudo aptitude update
cd /tmp/
sudo apt-get build-dep puppet
sudo apt-get -b source puppet

We add debian wheezy repository to our apt source.list and update, then we install the build dependencies of puppet and build puppet packages from source, we make sure to do all this in /tmp since the packages will be created under the cwd.

$ ls /tmp
puppet-2.6.8  
puppet_2.6.8.orig.tar.gz 
puppetmaster-passenger_2.6.8-1_all.deb
puppet_2.6.8-1_all.deb 
puppet-common_2.6.8-1_all.deb 
puppet-testsuite_2.6.8-1_all.deb
puppet_2.6.8-1_amd64.changes
puppet-el_2.6.8-1_all.deb
vim-puppet_2.6.8-1_all.deb
puppet_2.6.8-1.debian.tar.gz
puppetmaster_2.6.8-1_all.deb
puppet_2.6.8-1.dsc
puppetmaster-common_2.6.8-1_all.deb

Now in order to install puppetmaster:

$ sudo dpkg -i puppetmaster-common_2.6.8-1_all.deb
$ sudo dpkg -i puppetmaster_2.6.8-1_all.deb

Installing puppet client:

$ sudo dpkg -i puppet-common_2.6.8-1_all.deb 
$ sudo dpkg -i puppet_2.6.8-1_all.deb

Its important to mention that this procedure might not work for other packages since Ubuntu and Debian are not binary compatible.

P3wn Windows from Ubuntu

Fri, 21 Jan 2011 00:34:00 -0500

I don't run windows, still there are cases that I need to remotely run a windows tool in an automated fashion, up till now I didn't find a working solution, psexec is the tool of choice on the windows platform, googeling will lead you to this forum post that mentions winexe.
Finding how to use the tool and installing the latest version (0.9.1) on Ubuntu 10.10 took me some time that I hope this post will spare.
Most sites offer to compile winexe right from zenoss repo going this route will lead you to 0.8 version running which didn't work with a remote win7, another (less mentioned) option is on sourceforge, the project page offers 0.91 source tar, lets head on to business:

 $ wget http://tinyurl.com/4t8vjwl && tar -xvzf 4t8vjwl
 $ cd winexe-0.91/source4
 $ ./autogen.sh
 $ ./configure
 $ make 
 $ bin/winexe
    winexe version 0.91
    This program may be freely redistributed under the terms of the GNU GPLv3
    ...

Using the tool is as follows:

 $ bin/winexe -U windows-domain/user%pass //machine-name "cmd"  
 C:\ we are in windows land

Note that the machine name is that under the windows domain and not dns name, the user must have admin remote priviliges in the domain (not just on the machine) in order to bypass that you can regedit your way out, I wish you all a windows-less future :)

apt-mirror, local speedup

Sat, 08 Jan 2011 04:31:00 -0500

One of the biggest advantages of Linux are package manager like apt yum and pacman, its really easy to get your software installed via a single command.
Still installing package via your WAN can be time/bandwidth consuming, there are two main solutions:

Caching tools like Apt-Cacher that cache installed packages in a local proxy server, the main drawback is that the first one who installs a package waits the most.
Mirroring tools like apt-mirror that mirror an entire repository onto a local server, the main drawback of mirroring is the initial sync operation bandwidth (can take 50Gb) and storage capacity.

Iv choose to use mirroring mainly because the time a single machine will have to wait the first time, lets start by installing apt-mirror:

$ sudo aptitude install apt-mirror

The main configuration file is /etc/apt/mirror.list:

############# config ##################
set base_path    /media/ubuntu-mirror
set limit_rate 15k
set mirror_path  $base_path/mirror
set skel_path    $base_path/skel
set var_path     $base_path/var
set nthreads     10
set _tilde 0
############# end config ##############

deb-i386 http://archive.ubuntu.com/ubuntu maverick main restricted universe multiverse
deb-i386 http://archive.ubuntu.com/ubuntu maverick-security main restricted universe multiverse
deb-i386 http://archive.ubuntu.com/ubuntu maverick-updates main restricted universe multiverse
#deb-i386 http://archive.ubuntu.com/ubuntu maverick-proposed main restricted universe multiverse
#deb-i386 http://archive.ubuntu.com/ubuntu maverick-backports main restricted universe multiverse

deb-i386-src http://archive.ubuntu.com/ubuntu maverick main restricted universe multiverse
deb-i386-src http://archive.ubuntu.com/ubuntu maverick-security main restricted universe multiverse
deb-i386-src http://archive.ubuntu.com/ubuntu maverick-updates main restricted universe multiverse
#deb-i386-src http://archive.ubuntu.com/ubuntu maverick-proposed main restricted universe multiverse
#deb-i386-src http://archive.ubuntu.com/ubuntu maverick-backports main restricted universe multiverse


deb-amd64 http://archive.ubuntu.com/ubuntu maverick main restricted universe multiverse
deb-amd64 http://archive.ubuntu.com/ubuntu maverick-security main restricted universe multiverse
deb-amd64 http://archive.ubuntu.com/ubuntu maverick-updates main restricted universe multiverse
#deb-amd64 http://archive.ubuntu.com/ubuntu maverick-proposed main restricted universe multiverse
#deb-amd64 http://archive.ubuntu.com/ubuntu maverick-backports main restricted universe multiverse

deb-amd64-src http://archive.ubuntu.com/ubuntu maverick main restricted universe multiverse
deb-amd64-src http://archive.ubuntu.com/ubuntu maverick-security main restricted universe multiverse
deb-amd64-src http://archive.ubuntu.com/ubuntu maverick-updates main restricted universe multiverse
#deb-i386-src http://archive.ubuntu.com/ubuntu maverick-proposed main restricted universe multiverse
#deb-i386-src http://archive.ubuntu.com/ubuntu maverick-backports main restricted universe multiverse

clean http://archive.ubuntu.com/ubuntu

Notice limit_rate and nthreads, both will affect how much bandwidth will be used during sync, iv also disabled source downloads, we can now run the initial sync:

$ apt-mirror
  Downloading 90 index files using 10 threads...

The initial sync will take some time, feel free to stop it at any stage (it will resume once restarted), once its done will head on to set regular update job, edit /etc/cron.d/apt-mirror to the following:

 #
 # Regular cron jobs for the apt-mirror package
 #
 0 4 * * * apt-mirror  /usr/bin/apt-mirror >> /var/spool/apt-mirror/var/cron.log 2>&1

Unlike the default setting Im redirecting stderr and also appending to the log file, in order to serve apt requests from this server will use Nginx:

$ cd /var/www
$ sudo ln -s /media/ubuntu-mirror/mirror/archive.ubuntu.com/ubuntu ubuntu

Last thing left is to edit /etc/apt/sources.list to use our new local mirror:

#deb cdrom:[Ubuntu 10.10 _Maverick Meerkat_ - Release amd64 (20101007)]/ maverick main restricted
# See http://help.ubuntu.com/community/UpgradeNotes for how to upgrade to
# newer versions of the distribution. 

deb http://mirror-host/ubuntu/ maverick main restricted

## Major bug fix updates produced after the final release of the
## distribution.
deb http://mirror-host/ubuntu/ maverick-updates main restricted

...

apt-getting was never faster.

Automation, automation, automation! [0..2]{automation}!

Fri, 20 Aug 2010 02:21:07 -0400

There are always tasks that look too small to worry about, copying a file pasting that line, the comforting sound of repetition.
Uh? as devs we are better than that, we can program complex logic in commercial settings there is no reason not to apply our knowledge to every day tasks, let the machines do what they do best, repeat!
For example the enjoyable task of copying the contents of cd's (remember those?) onto your hard drive, the drill is quite simple you pop in a cd, head on to the file manager gui select all files, enter copy and move into the destination folder, oh and don't forget to create a unique folder for each copied cd, now repeat that 50x times, can we do better?

PREFIX = '/home/ronen/temp/'
last_cd_in = lambda{ %x(dmesg | grep ISOFS).split("\n").last }

def mounted_at
	res = nil
        res_ok = lambda{res && res.split(' ')[1]}
	10.times {
		puts 'another attempt'
	       	res = `cat /proc/mounts | grep sr0`; break if res_ok.call
		sleep 1
       	}
        notify 'failed to find mount point' and exit 1 if !res_ok.call	
	res.split(' ')[1].gsub('040',' ') 
end

def notify (msg) 
	`notify-send '#{msg}' -t 10`
end

def sh(cmd)
	res = `#{cmd}`
	notify("#{cmd} failed") and exit(1) if $?.to_i != 0 
	res
end

last = nil

eject_action = lambda {
	sh 'eject'
	last = last_cd_in.call
	puts 'place the cd in, press enter when your done'
	gets
}

eject_action.call

while(true)
	sh 'eject -t'
	begin
		curr = last_cd_in.call
	end while last==curr
	dest ="#{PREFIX}#{%x(uuid)}"
	sh "mkdir #{dest}"
	if !system("cp -rv #{mounted_at()}/* #{dest}")
		notify 'Failed to copy cd, please check media!'
	end
	eject_action.call
end

This script enables the following functionality:

Cdrom tray is popped out and the user is prompted to enter a new one in.
A unique destination folder is created (using uuid).
The contents is copied to the destination.
The cd is popped out ready for another round.

As for the implementation its Ruby mixed with good old unix tools like cp, mkdir eject and some /proc trickery to keep track of new mounted cd path, I find this combination a win win, we use Ruby instead of bash and use standard unix tools set instead of writing them ourself, Ruby has excellent unix process management API's (as any one who used Java for that can approve).
Notify-send is a nice decor (Ubuntu notification system) that growls at us when things go wrong (keeping us free to roam into more interesting domains).

Another annoying task is waiting for tests to end in Intellij, this is not using our time, wouldn't it be great to be notified when tests fail? (not supported OOTB):

require 'rubygems'
require 'open4'

def numeric?(object)
	true if Float(object) rescue false
end


while (true)
        pid=`jps -v | grep AppMain | awk '{print $1}'`
	if numeric?(pid)
		puts "JUnit found at PID #{pid}"
		opid, stdin, stdout, stderr = Open4::popen4("strace -p #{pid}")
		Process::waitpid2 opid
		lines = stderr.readlines
		if lines.last.include?  '255'
			`notify-send 'tests failed!' -u critical`
		else
			`notify-send 'all tests passed!' -u critical`
		end
	end
	sleep 2.5
end

Same idea again, using Linux tool box with Ruby to automate our task, this script polls all Java processes for AppMain (Intellij main class runner), once found strace is attached to it in order to grab the exit code and notify-send the matching return status, the theme is repeating levrage the operating system and Ruby in order to turn a mundane task into an enjoyable one!

Fasten your seatbelt

Sat, 03 Jul 2010 03:07:42 -0400

Using your IDE effeciently is an integral part of being a proficient developer, Intellij offers a multitude of options to turn your coding experience into a fast mouseless one.
While most developers memorize the key bindings they tend to less explore other options of cutting key strokes away, in this post ill cover couple of those, while the examples ill be showing are in Java land the ideas should be useful in other languages as well.

How many times did you find yourself writing the following during a typical programming day? (at least I hope you did):

  @Test
  public void someTest(){
    // ..
  }

You type the public void someTest proceeded by the curely braces, then you head up and add the @Test annotation, quite repetative isn't it? it would be nicer to be able to do something like:

  test <Tab>

   expands to -> 

  @Test
  public void <cursor>(){

  }

  // now you enter the test name

  @Test
  public void someTest(){

  }

In order to enable this head on to the settings (using Alt+Ctrl+S) and search "Live Templates", add a new template called test with the following content:

@Test
public void $NAME$ {
}

Make sure to set the context as Java Code and your set to go, other templates that I like are iter (foreach loop), psvm (public void main), and thru (throw runtime).

Another optimization helps us to find out early when we forget to implement an interface method:

   public interface Bar {
    void someMethod();
   }

   public class Foo implements Bar {

    // a long list of other methods


    public void someMethod(){// oops
      // intellij default comment
    }

   // yet some more code
   }

Discovering it couple of minutes later during (hopfully) a test is quite annoying, wouldnt it be nice to get warned if you forgot to implement it?

   public class Foo implements Bar {

    // a long list of other methods


    public void someMethod(){
      throw new RuntimeException("not implemented yet");
    }

   // yet some more code
   }

Head on to settings and look for "File Templates", select "implemented method body" and replace it with "throw new RuntimeException("not implemented yet");", this will make un-implemented methods fail loud and clear.

Our last optimization involves finals, mutating local variables and method parameters is limiting readability and make it harder to write thread safe code:

    public class NotReadable{

      public void wacky(int i) {

      //.. post some logic with j

       i = j;


      // multiple lines after, do we really know what i is now?

       return i + 4;
      }
    }

Making them final solves this however going past each one manualy is really tidiouse so most devs just avoid it, well there is an easy way to automate it, head on settings and search for inspections, look for "decleration can have final modifier" and enable it:

    public class NotReadable{

      public void wacky(int i) {// intellij highlights i, alt + enter and select "fix all"
        // ...
      }
    }

In post iv introduced three quick methods of saving key strokes, our goal is not count how many keys we press but rather to remove distractions in order to concetrate better on the problem we are trying to solve.

APT Bandwidth Diet

Sat, 17 Apr 2010 01:57:27 -0400

In this post ill cover how to sync your Ubuntu installation without sacrificing a large amount of bandwidth, its commons to have a number of machines with the same packages installed the question is how we sync them all up (especially clean installs) without using up bandwidth.
One could setup a proxy if the machines share the same local network however for distributed scenario's (multiple networks) this won't do.

A nice solution is apt-move (from the man page):

apt-move - move cache of Debian packages into a mirror hierarchy.

apt-move enables us to create a mirror from our local apt cache, we can then distribute the mirror on a removable drive using an iso format, the target machines will add the iso image by using apt-cdrom to the sources list, using the local filesystem instead the web.

The first thing is to make sure that apt-move won't delete any thing from the cache:

   $ sudo aptitude install apt-move
   # within /etc/apt-move.conf make sure to set COPYONLY to yes => COPYONLY=yes

Now lets create the mirror:

 # clearing old packages
 $ sudo aptitude autoclean
 # clearing the apt-move target
 $ sudo rm -rf /mirrors/debian
 # this dumps packages from /var/cache/apt/archives into /mirrors/debian
 $ sudo apt-move -d karmic update
 $ cd /mirrors/debian
 # this isn't created by apt-move, a possible bug?
 $ sudo mkdir -p dists/karmic/non-free/binary-i386
 # this solves permission issues
 $ sudo chmod 777 . -R
 $ sudo apt-ftparchive packages pool/main/ | gzip -9c > dists/karmic/main/binary-i386/Packages.gz
 $ sudo apt-ftparchive packages pool/non-free/ | gzip -9c > dists/karmic/non-free/binary-i386/Packages.gz

Take a look into /mirrors/debian/pool folder, youll be able to see all the packages that the mirror contains, note that packages which are not within the cache folder will be missing even if you installed them before (we will see later how to bypass this), lets create the ~/Release.dummy file (the mirror Release file):

APT::FTPArchive::Release {
 Origin "APT-Move";
 Label "APT-Move";
 Suite "karmic";
 Codename "karmic";
 Architectures "i386";
 Components "main restricted";
 Description "Ubuntu Updates CD";
};

Place the Release file within the mirror folder:

 $ cd /mirrors/debian
 $ rm dists/karmic/Release
 $ apt-ftparchive -c ~/Release.dummy release dists/karmic/ > Release
 $ mv Release dists/karmic/

Now will need a gpg key in order to sign our Release file, in this stage we will also create the iso:

 $ gpg --gen-key
 $ gpg -bao dists/karmic/Release.gpg dists/karmic/Release
 # no need for this folder
 $ rm -rf .apt-move
 # creating the name of the iso
 $ mkdir .disk
 $ echo Ubuntu-Updates `date +%Y-%m-%d` > .disk/info
 # will need the public key within the iso
 $ gpg --export -a "Your Name" > public.key
 # creates the iso file
 $ mkisofs -r -A "Ubuntu Updates `date +%Y%m%d`" -o ~/ubuntu-updates.iso   /mirrors/debian

Now copy this iso to your favorit portable media in order to use it on the target machine:

   $ sudo mount -o loop /path/to/iso/ubuntu-updates.iso /cdrom
   $ sudo apt-key add /cdrom/public.key
   # during the issue of this command the /cdrom gets unmounted, mount -o loop again before pressing enter
   $ sudo apt-cdrom add -d=/cdrom
   # mount the iso again due to the last command

We can now install packages from the iso, lets intall java (assuming you had it in you cache):

 $ sudo apt-get install sun-java6-jdk
 # the install should point to local filesystem & not the web (note from section).
  Selecting previously deselected package sun-java6-jre.
  (Reading database ... 146576 files and directories currently installed.)
  Unpacking sun-java6-jre (from .../sun-java6-jre_6-15-1_all.deb) ...
  sun-dlj-v1-1 license has already been accepted
  Selecting previously deselected package sun-java6-bin.
  Unpacking sun-java6-bin (from .../sun-java6-bin_6-15-1_i386.deb) ...
  sun-dlj-v1-1 license has already been accepted
  ...

In the case that you had a package that was cleared from the cache you can restore is by using:

 # re-install with only download flag
 $ sudo apt-get install -d package-name --reinstall

I hope that this will save you time post the near 10.04 release ;)

Update Couchdb via FUSE

Sat, 10 Apr 2010 01:48:49 -0400

FUSE is one of my favorite projects it really makes FS development accessible, it can abstract almost any resource as a file system making it first class citizen in the UNIX realm,
an option that I was thinking about is to represent a Couchdb database as a folder. A possible use case might be a Dropbox/Ubuntu-one like clone (taking an advantage on Couchdb excellent replication) another one might be "greping" documents, the entire UNIX set will be able to process them.

Apparently im not the first to think about this, enter Couchdb FUSE, this project enables the manipulation of document attachments (files), the install process under Ubuntu is a bit undocumented, I had to figure it out myself:

    $ sudo aptitude install libfuse2 fuse-utils# installing fuse
    $ sudo aptitude install python-setuptools # this installs easy_install
    $ sudo aptitude install python-dev libfuse-dev # fuse-python requires it
    $ sudo easy_install couchdb
    $ sudo easy_install couchdb-fuse
    $ sudo easy_install fuse-python

Now we should be able to mount a single document attachments like so:

    $ mkdir couchmount
    $ curl -X PUT http://localhost:5984/fuse
    $ curl -X POST http://localhost:5984/fuse -H 'Content-Type: text/javascript; charset=utf-8' -d '{}' # an empty document, next step will use its id
    $ couchmount http://localhost:5984/fuse/6a53b4e951ea836b7fef55e7afafcd7a  couchmount
    $ touch couchmout/bla
    $ curl -X GET http://localhost:5984/fuse/6a53b4e951ea836b7fef55e7afafcd7a 
    {"_id":"6a53b4e951ea836b7fef55e7afafcd7a","_rev":"2-d9a4307289432688cd5a2f2d5c605f88","_attachments":{"bla":{"stub":true,"content_type":"","length":0,"revpos":2}}}

This is all nice but lacks couple of features that id like to have:

Mounting an entire DB & exposing all the documents.
Make the entire document editable (not just the attachments).
Enable full CRUD access to the DB.

I guess that this calls for another solution, do you have any thing in mind?

Artifactory meets VPS

Mon, 05 Apr 2010 05:28:27 -0400

If your using Maven/Gradle/Lein/Buildr to build your projects than you must have stumbled upon one common issue which is where to host your artifacts,
syncing your local repository between machines is impossible without a proxy, a nice solution is to host it on a VPS.
I didn't find a single clear source of information on how to do so in a safe manner so ill try to cover the main points in this post.
Before going further please take note to the following disclaimer: Im not a security expert, any damage cause by following these steps is your responsibility.

Lets start with the main components that we are going to use, first is Artifactory my favorite maven proxy that runs on an embedded jetty server out of the box, its common to place a reverse proxy in front of Java server in order to gain performance (mainly for static content) and enhanced security.
Iv chosen to deploy Nginx as our proxy since its very lite on resources, offers good performance and has excellent documentation.

In order to forward traffic from Nginx to Artifactory will need to add the following section to nginx.conf:

    server {
                server_name     example.com;
                listen          80;


                location ~ ^/artifactory/(.*)$ {
                        access_log      /var/log/nginx_67_log;
                        proxy_pass      http://127.0.0.1:8081;
                        proxy_redirect  off;
                        proxy_set_header        Host            $host;
                        proxy_set_header        X-Real-IP       $remote_addr;
                        proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
                }
    }

Now that we proxy the traffic from Nginx to Artifactory we need to start thinking on how to protect Artifactory from the outside world, one of the simplest forms of protecting online resources is using basic authentication, in this configuration the user has to provide user name & password in order to access a url:

     server {
                server_name     example.com;
                listen          80;


                location ~ ^/artifactory/(.*)$ {

                        auth_basic "Restricted";
                        auth_basic_user_file /some/path/htpass; 

                        access_log      /var/log/nginx_67_log;
                        proxy_pass      http://127.0.0.1:8081;
                        proxy_redirect  off;
                        proxy_set_header        Host            $host;
                        proxy_set_header        X-Real-IP       $remote_addr;
                        proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
                }
    }

We added two new lines to the configuration, the htpass file is the file that contains the user details which are required for authentication, in order to generate this file:

$ sudo aptitude install apache2-utils
$ pwgen -s 50 # make sure to use strong passwords!
$ htpasswd -c htpass username

Now we should be set to access Artifactory using the credentials we created above, in order to enable maven to use these credentials we can set the following in the settings.xml file (on our client machine):

    
        
            
                Artifactory
                username
                password
                
                    
                        username
                        password
                    
                
            
        
        
            
                *
                repo
                http://example.com/artifactory/repo
                Artifactory

Wait!, while all this should work it is totally insecure!
Basic authentication is nothing more than a formalized method of sending user name and password out on the clear via http, its very easy to intercept.
We must make sure that the authentication info is send on an encrypted channel using ssl, in order to enable that we have to:

Create a self signed certificate for our server (or buy one from a CA).
Configure Nginx to use the certificate and change the protocol to ssl on the Artifactory urls.
Import the certificate to our client (Java or a browser in our case).

Will start with the certificate creation, ill go ahead with the self signed one, this is quite the standard procedure with the only exception of using AES256 instead of DES3:

    $ cd /usr/local/nginx/conf
    $ openssl genrsa -aes256 -out server.key 1024 #  generating a private rsa key
    $ openssl req -new -key server.key -out server.csr # create a certificate based upon the key, when asked "Common Name (eg, YOUR name) []:" make sure to use your proper domain name!
    $ cp server.key server.key.org # see next
    $ openssl rsa -in server.key.org -out server.key # removing the pass phrase of the key
    $ openssl x509 -req -days 365 -in server.csr -signkey server.key -out server.crt # sign certificate with private key (clients use it).

Now lets go ahead & make Nginx use ssl:

server {
            server_name     example.com;
            listen          443;
            ssl on;

            ssl_certificate /usr/local/nginx/conf/server.crt;
            ssl_certificate_key /usr/local/nginx/conf/server.key;
            access_log /var/log/nginx/ssl.access.log;
            error_log /var/log/nginx/ssl.error.log;


            location ~ ^/artifactory/(.*)$ {

                    auth_basic "Restricted";
                    auth_basic_user_file /some/path/htpass;

                    access_log      /var/log/nginx_67_log;
                    proxy_pass      http://127.0.0.1:8081;
                    proxy_redirect  off;
                    proxy_set_header        Host            $host;
                    proxy_set_header        X-Real-IP       $remote_addr;
                    proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
                    proxy_read_timeout 500;
                    proxy_send_timeout 600;
            }
    }

server { # making sure that the redirects that artifactory makes will remain under https
            listen          80;
            server_name     example.com;


            location ~ ^/artifactory/(.*)$ {
              rewrite ^(.*) https://$server_name$1 permanent;
            }
}

Now all that is left is to import the certificate to our clients, in the case of a browser just add an exception & import it, in the case of Java based tools like Maven we need to import the certificate using Java keytool:

        sudo keytool -import -alias example.com -file /some/path/server.crt -keystore /usr/lib/jvm/java-6-sun/jre/lib/security/cacerts

Note that since all Artifactory related traffic is now https so we need to adjust our settings file as well:

    
        
        
            
                *
                repo
                https://example.com/artifactory/repo
                Artifactory

Now that we feel secure & fuzzy all round there is still one issue to consider which is brute force attacks, nothing stops any one from pounding our server to death using tools like John the Ripper.

A tool to fight such attacks is fail2ban, this tool polls log files after patterns & triggers iptable drop instructions upon matches, lets go ahead & install it:

       $ sudo  aptitude install fail2ban
       $ sudo cp /etc/fail2ban/jail.conf /etc/fail2ban/jail.local

Add the following defenition to the jail.local file:

[artifactory]

enabled = true
port = https
filter = nginx-auth
logpath = /var/log/nginx/ssl.error.log
maxretry = 3
bantime  = -1

Now lets add the filter to a new file under /etc/fail2ban/filter.d/nginx-auth.conf:

[Definition]

failregex = no user/password.*client\: <HOST>
user.*authentication failure.*client\: <HOST>
user.*not found.*client\: <HOST>
user.*password mismatch.*client\: <HOST>

Thats a lot of configuration mayham, let me know if you found it useful!

The world as a sequence

Sat, 03 Apr 2010 04:34:40 -0400

Hey there! welcome to my new blog, after long time of blogsphere absence iv returned with a new self hosted blog that iv developed, this little project gave me a couple of insights that id like to share.

One of the main abstractions that Clojure uses is that of a Sequence, a seq is an interface (a java one) that represents a logical immutable persistent list, any data structure that expose this interface enables a vast number of core functions to our disposal.
Nothing up to this point should sound out of the ordinary except that the real fun begins when we define our own sequences!

Lets take for example the entries of a Jar file, by using repeatedly on the archive input stream we get a seqence of entries to work with, this turns the stream into a first class Clojure citizen that we can doseq on, in this case to extract the entries in the unjar function:

(defn- jar-stream-seq [stream]
  (take-while #(not= nil %) (repeatedly #(.getNextEntry stream))))

; extract-entry emitted for brevity
(defn unjar [src dest]
  (with-open [jar-stream (JarArchiveInputStream. (file-in-stream src))]
    (let [dir (file dest)]
      (doseq [entry (jar-stream-seq jar-stream)] (extract-entry entry dir jar-stream)))))

Another example is CouchDb to Clojure state syncronization, most applications that interacts with CouchDb (or any other persistent storage) hold some state that need to be updated in order to reflect lastest changes.
Instead of polling the DB we can (in the case of CouchDb) listen to a broadcasted via a url (http://localhost:5984/db/_changes?feed=continuous&heartbeat=1000):

    {"seq":1,"id":"43badb35255a94b4ab5c3168553d1acc","changes":[{"rev":"2-8630b20788a2fd2618d85d2239a31f7f"}],"deleted":true}
    {"seq":2,"id":"1ed79fc5a1ddfaaf08a73e68b1f5432e","changes":[{"rev":"2-a72a134b273510eaa247c142fc62fd51"}],"deleted":true}
    {"seq":3,"id":"df76d2644cef31df9f9e4e6db29e614d","changes":[{"rev":"5-1bbc0c3f12cf3b3f23ef7f2fb99a6e8e"}]}

This channel publishes all the changes in a json format that we can consume (can you guess already how?):

(defn- bytes-seq [input]
  (take-while #(not= -1 %) (repeatedly #(. input read))))

(defn- changes-seq []
  (let [url (URL. (couch str "/_changes?feed=continuous&heartbeat=1000&since=" (last-seq)))
        uc (. url openConnection)]
    (. uc connect)
    (filter (comp not blank?)
      (map #(reduce (fn [r c] (str r c)) "" %)
        (partition-by #(if (= \newline %) true)
          (map char (bytes-seq (. uc getInputStream))))))))
    
(defn follow-couch-changes []
  (doseq [change (changes-seq) :let [id (read-id change)]]
    (when (not-design-id id)
      (dosync
        (alter posts-map assoc id (read-with-date id))
        (alter posts init-posts)))))

The bytes-seq gives a bytes sequence that is mapped into chars and partitioned into lines under changes-seq, note that since we use heartbeat of 1000 millis (keeping the connection open) we get empty lines that are filtered out, the processing of these events is made using the standard Clojure functions.
Another important aspect of this is that we can process infinite sequences (as in this case).

We can think of many parts of the outside world as a collection of sequences waiting to be processed by the same functional constructs that form the base of Clojure, Taking this approach we get an idiomatic Clojure code that really show how powerfull the language is.

Cool Groovy techniques

Sat, 09 Jan 2010 06:13:00 -0500

Groovy is a versatile & very powerful language to code with, as a Java super set it enables a Java programmer to write its code with all the known Java idioms & progress into Groovy land in small steps, in this post ill present two techniques that I find to be very powerful.

The first is closure initialization, in Java land its common that when we have an object A which uses another object B with non trivial construction code (which depends on A's input) than we write all the initialization code of B inside A constructor:

      class ComplexInit { // our B
          // ... some state that need to be initialized by users of this class
      }

      class UsesComplex {// our A
        private ComplexInit v
        def UsesComplex(){
          v = new ComplexInit()
          //..  some complex initialization logic which depends on UsesComplex   }
      }

Another common practice is to create a factory that will encapsulate the initialization of ComplexInit class taking A as input:

     class ComplexInitFactory {
        def createComplexWith(UsesComplex u){
          // initialization code which depends on
        }
     }

Groovy offers another way which I named "closure initialization":

      class UsesComplex{
         private ComplexInit val = {
           def result = new ComplexInit()
             // more logic here
           }()

        def UsesComplex(){
          // ComplexInit is initialized above
        }
     }

Note that we are embedding a closure within the UsesComplex class & call it immediately after it has been defined, this resembles static initializer code blocks in Java except that it not static at all, the anonymous closure has access to its surrounding scope & can access other data members during its run:

      class UsesComplex{
        def today = new Date()
        private ComplexInit val = {
          def result = new ComplexInit()
          result.a = today
          result
        }()
      }

By separating our constructors into initialization blocks we modularize the construction of our object, making our code more readable & DRY (no separation between deceleration & initialization).
Another interesting technique is dynamic method dispatch, in Groovy we can dispatch methods like so:

      class Target{
        def methodA(){ println 'hey'}
      }

      new Target().methodA()
      new Target()."methodA"()// this works too!

This means that we can dispatch methods according to run time logic:

      class Target{
        def calcA(){'a'}
        def calcB(){'b'}
        def calcC(){'c'}
        def notCalc(){'d'}
      }

      def t = new Target()
      def calcs = t.metaClass.methods.findAll{it.name.startsWith('calc')}.collect{it.name}
      calcs.each{println t."$it"()}// will print a b c

Its true that Java reflection has similar capabilities however the Java API is very verbose & not as elegant, use this technique with care since it may result with hard to read code!

Thats about it for this post, let me know if you find these techniques useful in your code.

Synching your enviroment around

Sat, 31 Oct 2009 08:34:00 -0400

Ubuntu 9.10 has just came out & iv rushed to upgrade, upgrading my 9.04 to 9.10 via the upgrade manager is a working option but usually id rather do a clean install & get a fresh system to poke around.
The main issue that clean install brings is restoring your old environment which includes installed programs, settings files (.bashrc, .vimrc etc..), non packaged applications (usually the latest & greatest versions of Clojure, Groovy etc..).
Another issue is the need to restore these settings on multiple machines while keeping it all consistent between them, there must a better way than working it all manually!

Well the solution that iv come with involve two tools, Dropbox & AutomateIt, Dropbox is no more than a fancy folder to the cloud synchronizer which enables me to access all my settings file from any machine, lsym-ing my .bashrc, .vimrc & any other file that id like to share across machines into the Dropbox folder is all that is required.

The second part involves AutomateIt, which is a configuration management framework written in Ruby, an easy way of re-creating symlinks from my Dropbox folder onto a new system is using the following AutomateIt script:

        HOME = ENV['HOME']
        DROP = "#{HOME}/Private/Dropbox"
        def link(src,dst)
          ln_s("#{DROP}/#{src}","#{HOME}/#{dst}")
        end
        # Vim
        link "vim/.vimrc",".vimrc"
        link "vim/.vim",".vim"
        link "vim/.vimclojure-2.1.2",".vimclojure"
        # Bash
        link "BashEnv/.bashrc",".bashrc
        link "BashEnv/.inputrc",
        # Languages
        link "prog-langs/.clojure",".clojure"
        link "prog-langs/.groovy",".groovy"
        link "prog-langs/.jruby",".jruby"

The nice thing about it is that it hides all the nitty gritty details that id have to figure out when using plain Ruby (or any other environment for that matter), AutomateIt wraps common tasks with a bash like DSL, unlike bash this DSL is portable across multiple unix systems & it hides many of the complexities that follow.
AutomateIt performs actions only when they are required, this solves annoying cases that rise when scripts run more than once on a system (e.g. appending text to files).

Another cool feature is package management, AutomateIt is capable of interacting with multiple packaging systems (yum, apt, gem etc..) in a transparent way, its easy to replicate installed packages onto multiple systems:

        # bash
        %w(terminator rlwrap).each{ |p| package_manager.install p}

        # programming
        %w(vim git-core).each{ |p| package_manager.install p}

        # communication
        %w(deluge openssh-server).each{ |p| package_manager.install p}

        # misc
        %w(gpodder gnome-do).each{ |p| package_manager.install p}

        # installing dropbox
        download_manager.download 'http://www.getdropbox.com/download?dl=packages/nautilus-dropbox_0.6.1_i386_ubuntu_9.10.deb' , :to => '/tmp/nautilus-dropbox_0.6.1_i386_ubuntu_9.10.deb'
package_manager.install ({'nautilus-dropbox' => '/tmp/nautilus-dropbox_0.6.1_i386_ubuntu_9.10.deb'} , :with => :dpkg)

Using these tools has made configuration nirvana a bit closer to me & hopefully to you :)

Dont overflow yourself!

Fri, 24 Jul 2009 01:57:00 -0400

Clojure has many nice less highlighted features, one of these is overflow protection, this can be a true life saver:

        user=> (unchecked-add 1 2); not safe but might be faster
        3
        user=> (unchecked-add (. Integer MAX_VALUE) 2); oops
        -2147483647
        user=> (+ (. Integer MAX_VALUE) 2) ; using safe arithmetic again
        2147483649
        user=> (+ (. Integer MAX_VALUE) 23)
        2147483670
        user=> (class (+ (. Integer MAX_VALUE) 23)); Clojure does the right thing for us
        java.lang.Long

It does pack more than just concurrency up its sleeve.

Genericsfaction, getting some generics satisfaction

Tue, 30 Jun 2009 00:26:00 -0400

Java generics have a lot of short shortcoming, most of which are the result of the religious backward compatibility that Java is known for.
One of the most annoying "features" of generics is type erasure in which the compiler removes (almost) any trace of generic types meta data.

Iv been pondering about this issue & had a simple idea, what if we could add back in the meta data that the erasing process removed?
So, iv decided to create a proof of concept in order to show how this might work and named it genericsfaction.

Using genericsfaction requires an additional code manipulation stage in which the source code is scanned & gets its AST manipulated (meta data is introduced), each manipulated source file is dumped into a separate folder (named gen), all that is left is to compile the files under gen (instead of the original src).

At the moment there are only two enrichments implemented, the first is method input parameters annotation enrichment:

        public void twoTypeParamsMet(@GenMeta(stringRep = "((Object ()) (Object ()))", classes = { Object.class, Object.class }) Map <Object, Object> map) {
          // method code
        }

The second is variable initialization enrichment:

        public static void main(String [] args) {
          List<Integer> list = (List <Integer>)MetaProxy.newInstance(new ArrayList<Integer>(),"(Integer ())",new Class []{Integer.class});
        }

The reason that annotations aren't used in this case is due to this limitation.

There is no release to play with yet, however the source is up on github (mostly Clojure), I am planing on making a release as soon as ill finish to implement a validation module that uses this meta data in runtime.

Newify your Object

Sat, 23 May 2009 11:21:00 -0400

Update a generous Anonymous has left a comment that the following special macro form does the same as the newify function that i present later on:

        (def nested-object (-> GrandFather. Father. GrandSon.))

Feel free to skip this entry (unless your interested in an alternative implementation).

Instantiating large nested object graphs can be tedious in any language Clojure included:

        (def nested-object
          (new Grandson (new Father (new Grandfather))))

All those new methods call get old pretty quickly and are not DRY, to remedy this will take advantage on a cool property of Clojure Homoiconicity.
Homoiconicity basically means that the code is represented by a basic data structure of the language itself (lists in the case of Clojure), this means that manipulating the AST is easy as pie!
It would be nice if we could write something like the following:

        (def nested-object (newify '(Grandson (Father (Grandfather)))))

The newify function takes a nested list of objects & adds new before each object, post that the function evaluates the resulting list:

        (defn newify [e] (eval (newify-imp e)))
        ; required since used before defined
        (defn- newify-rest [e] )

        (defn- newify-imp [e]
          (let [add-new (partial cons 'new) add-rest (partial cons (first e))]
            (if (not (empty? e))
              (-> (newify-rest e) add-rest add-new) e)))

        (defn- newify-rest [e]
          (map #(if (list? %) (newify-impl %) %) (rest e)))

The newify-imp & newify-rest functions are mutually recursive, newify-imp adds the new symbol & the first list element to the result of the newify-rest function result, the newify-rest maps each list in the rest of the list to its newified value.

This implementation could be transformed into a recursive macro that will spare us from the list escaping:

        (def nested-object (newify '(Grandson (Father (Grandfather)))))
        ; no escaping here
        (def nested-object (newify (Grandson (Father (Grandfather)))))

Ill have to dig that option later on.

NILFS on Jaunty

Fri, 01 May 2009 01:56:00 -0400

The latest release of Ubuntu includes the long awaited Ext4 FS (works flawlessly on my system).
Ext4 is faster & more secure but still lacks the ability to manage FS snapshots (ZFS excels in that, but runs only in FUSE on linux).
An interesting alternative is NILFS:

NILFS is a log-structured file system supporting versioning of the entire file system and continuous snapshotting which allows users to even restore files mistakenly overwritten or destroyed just a few seconds ago.

NILFS maintains a repo for Hardy, the Jaunty repos contain only the user land tools which won't do us much good since we need the kernel module as well, this leaves us only with the option of installing it from source (still quite easy).

    # a perquisite
    $ sudo aptitude install uuid-dev
    # installing kernel module, result module resides in /lib/modules/2.6.28-11-generic/kernel/fs/nilfs2/nilfs2.ko
    $ wget http://www.nilfs.org/download/nilfs-2.0.12.tar.bz2
    $ tar jxf nilfs-2.0.12.tar.bz2
    $ cd nilfs-2.0.12
    $ make
    $ sudo make install
    # installing user land tools
    $ wget http://www.nilfs.org/download/nilfs-utils-2.0.11.tar.bz2
    $ tar jxf nilfs-utils-2.0.11.tar.bz2
    $ cd nilfs-utils-2.0.11
    $ ./configure
    $ make
    $ sudo make install

Creating a file system on a file (ideal for playing around):

    $ dd if=/dev/zero of=mynilfs bs=512M count=1
    $ mkfs.nilfs2 mynilfs

The FS is only a mount away:

    # mounting the file as a loop device
    $ sudo losetup /dev/loop0 mynilfs
    $ sudo mkdir /media/nilfs
    $ sudo mount -t nilfs2 /dev/loop0 /media/nilfs/

Now lets create a couple of files:

$ cd /media/nilfs
    $ touch 1 2 3
    # listing all checkpoints & snapshots, on your system list should vary
    $ lscp
    CNO        DATE     TIME  MODE  FLG   NBLKINC       ICNT
    7  2009-05-01 01:08:09   ss    -         12          6
    13  2009-05-01 19:05:34   cp    i          8          3
    14  2009-05-01 19:05:59   cp    i          8          3
    15  2009-05-01 19:07:09   cp    -         12          6
    # creating a snapshot
    $ sudo mkcp -s
    # 15 is the new snapshot (mode is ss)
    $ lscp
    CNO        DATE     TIME  MODE  FLG   NBLKINC       ICNT
    7  2009-05-01 01:08:09   ss    -         12          6
    13  2009-05-01 19:05:34   cp    i          8          3
    14  2009-05-01 19:05:59   cp    i          8          3
    15  2009-05-01 19:07:09   ss    -         12          6
    16  2009-05-01 19:08:59   cp    i          8          6

    # our post snapshot file
    $ touch 4

Now lets go back in time into our snapshot, NILFS enables us to mount old snapshots as read only FS (while the original FS is still mounted):

    $ sudo mkdir /media/nilfs-snapshot
    $ sudo mount.nilfs2  -r  /dev/loop0 /media/nilfs-snapshot/ -o cp=15 # only snapshots works!
    $ cd /media/nilfs-snapshot
    # as we might expect
    $ ls
    1 2 3

NILFS has some interesting features, its not production ready yet however it sure worth looking after its development.

MetaClasses can bite!

Fri, 06 Mar 2009 05:30:00 -0500

Groovy has MOP classes built in like the MetaClass class, this class enables (among other things) objects introspection, we could for example use its getProperties() method in order to get the list of all the meta properties of an object and copy their values into another object:

      class ParentModel {
        def String parentProp

        /**
        * Copying the current object properties into the other object
        */
        def copyInto(other){
          this.metaClass.getProperties().findAll{it.getSetter()!=null}.each{metaProp->
           if(metaProp.getProperty(this)!=null){
             metaProp.setProperty(other,metaProp.getProperty(this))
           }
          }
          other
        }
       }

The usage is quite simple:

        def parent = new ParentModel(parentProp:"parent")
        def copiedParent = parent.copyInto(new ParentModel())
        assert copiedParent.parentProp.equals(parent.parentProp)

While this code runs perfectly fine with object that have identical classes it has some unexpected side effects when one of the classes derives from another:

      class DerivingModel extends ParentModel{
           def derivedProp
      }

      def parent = new ParentModel(parentProp:"parent")
      def derived = new DerivingModel(derivedProp:"derived")
      def copiedDerived = parent.copyInto(derived)
      assert copiedDerived.parentProp.equals(parent.parentProp)
      assert copiedDerived.derivedProp.equals("derived")// throws groovy.lang.MissingPropertyException:
      // No such property: derived

How can this be? the DerivingModel class has the derivedProp but after the copyInto invocation its missing!
Lets see what the getProperties method actually returns:

        def parent = new ParentModel(parentProp:"parent")
        parent.metaClass.getProperties().each{println it.name}// prints: class, parentProp, metaClass

It seems that we don't only get the parentProp that we defined but also the metaClass and class properties, this means that in the derived case the copyInto method has set the metaClass and class values of ParentModel into the DerivedModel instance (causing it to lose access to its derivedProp).
The fix is quite simple:

      class ParentModel {
        def String parentProp
        def excludes = ['metaClass']
        def copyInto(other){
          this.metaClass.getProperties().findAll{it.getSetter()!=null}.each{metaProp->
           // we must not set the metaClass of the current object into the other
           if(metaProp.getProperty(this)!=null && !excludes.contains(metaProp.name)){
              metaProp.setProperty(other,metaProp.getProperty(this))
           }
          }
          other
        }
      }

There is no need to exclude the class property since it keeps the original even if set.

Clojure on youtube

Mon, 02 Mar 2009 00:41:00 -0500

Iv really enjoyed watching the Intro to Clojure series:

If your interested what the fuss is about check it out!

GroovyShell bindings validation

Sat, 28 Feb 2009 09:24:00 -0500

Groovy 1.6 got released not long a go with a load of new features, one of these is AST transformations which basically lets developers manipulate/access Groovy's AST before it gets loaded into the JVM.
This feature enables some really cool MOP options to mind however iv decided to try it in solving a non MOP-ish issue.

The GroovyShell is a very versatile and useful tool that enables us to evaluate any Groovy script file at runtime:


def bindings = [lastDay:2]
        def script = """
          import java.util.Calendar
          Calendar cal = Calendar.instance
          cal.set(2009,Calendar.FEBRUARY,2)
          def ghday = cal.time
          cal.set(2009,Calendar.MARCH,lastDay)
          def spring = cal.time
          def days = (ghday..spring).size()
          println days
        """
        def shell = new GroovyShell(new Binding(bindings))
        shell.evaluate(script)

The script get its input parameters from the bindings hash, forgeting to pass an input variable (lastDay in this case) will cause a groovy.lang.MissingPropertyException to be thrown.

This is quite reasonable during development time but not in other scenarios in which the script is written in one machine & evaluated on a remote machine.
The exception gets thrown only post its evaluation which might be too late (deep in the execution flow), wouldn't it be nice if it was possible to validate this at will on the client side?

We could scan the script for suspect variables that might not have been initialised, such scanning might be possible by inspecting the script AST, looking after variables that weren't declared in the current script, Groovy code visitors to the rescue!

The following implementation matches the bill perfectly:

        package com.jdftm.script.valid
        import org.codehaus.groovy.ast.ClassCodeVisitorSupport
        import org.codehaus.groovy.ast.expr.MethodCallExpression
        import org.codehaus.groovy.control.SourceUnit
        import org.codehaus.groovy.ast.expr.VariableExpression
        import org.codehaus.groovy.ast.expr.DeclarationExpression

        class ClassCodeVisitorSupportImp extends ClassCodeVisitorSupport {

            def declared = []

            def existsByDefault=['context','args']

            def bindings

            def notInBindings = {name -> bindings?.keySet().find{it.equals(name)} == null}

            def notDeclared = {name -> declared?.find{it.text.equals(name)} == null && !existsByDefault.contains(name)}

            def notValid = {notDeclared(it.text) && !it.hasInitialExpression() && notInBindings(it.text)}

            def suspectVariables = []

            public void visitMethodCallExpression(MethodCallExpression methodCall) {
             def variables=methodCall.arguments.expressions.findAll{(it instanceof VariableExpression)}
             suspectVariables
            }

            public void visitDeclarationExpression(final DeclarationExpression decleration){
             declared
            }

            protected SourceUnit getSourceUnit() {
              return null;
            }
        }

This visitor keeps a suspects list of all the variables that are either not in the declared variables or in the bindings hash.
The following class encapsulates the visitor usage:

        package com.jdftm.script.valid
        import org.codehaus.groovy.ast.ClassCodeVisitorSupport
        import org.codehaus.groovy.ast.ModuleNode
        import org.codehaus.groovy.control.SourceUnit
        import org.codehaus.groovy.control.CompilationUnit
        import org.codehaus.groovy.ast.ClassNode
        import org.codehaus.groovy.control.Phases;

        class GroovyScriptValidator {

            def validate(script,bindings){
              def ast=getAST(script)
              def impl=new ClassCodeVisitorSupportImp()
              impl.bindings=bindings
              impl.visitClass((ClassNode)ast.getClasses().get(0));
              impl.suspectVariables.flatten()
            }

            def ModuleNode getAST(String source) {
             SourceUnit unit = SourceUnit.create("Test",source);
             CompilationUnit compUnit = new CompilationUnit();
             compUnit.addSource(unit);
             compUnit.compile(Phases.SEMANTIC_ANALYSIS);
             return unit.getAST();
            }
        }

Validating the script is now simple:

        package com.jdftm.script.valid

        /**
        *
        * @author ronen
        */
        class InParamValidationTest {

          public static void main(String[] args) {
            def bindings = [lastDay:2]
            def script = """
                import java.util.Calendar

                Calendar cal = Calendar.instance
                cal.set(2009,Calendar.FEBRUARY,2)
                def ghday = cal.time
                cal.set(2009,Calendar.MARCH,lastDay)
                def spring = cal.time
                def days = (ghday..spring).size()
                println days
            """

            def validator = new GroovyScriptValidator()
            def suspects = validator.validate(script,bindings)
            suspects.each{println it.text}
            if(suspects.size()==0){
              def shell = new GroovyShell(new Binding(bindings))
              shell.evaluate(script)
            }
          }
        }

AST introspection is one cool feature to have!

Demystifying Spring wiring process

Sat, 17 Jan 2009 07:52:00 -0500

Many Java applications these days make use of Spring IoC, taking the initialization management from the hand of the developer has many benefits but also has a couple of down sides like the lose of some control on the wiring process itself.

Most of the time its a none issue however there are times in which Spring fails to resolve a dependency and all that the developer is left with is a cryptic stack trace to follow.

Iv been thinking on how to make it possible to track down Spring weaving process and see at each stage what has been wired up, basically there are three approaches that iv considered:

The first is to hook into the injection process by some public API, iv decided to drop it since i didn't find any API which notifies me upon beans creation and wiring.

The second was to wire some Aspect into Spring injection process, iv started to read Spring source code and saw that pin point all the spots in code in which the wiring takes place is very tedious and error prone to start with.

The third option was to think on Spring as an isolated container that has to interact with the JVM in order to create and inject objects into each other, tracing this interaction sholud give us the log of the injection process itself!

A little more digging (see org.springframework.beans.BeanUtils) revealed that Spring uses Java reflection API (java.lang.reflect.Constructor.newInstance, java.lang.reflect.Field.set) in order to build/wire beans, armed with this knowledge we are only left to devise a method for tracing the calls to these methods.

A possible option is Btrace, Btrace is a Java agent modeled after DTrace which is capable of instrumenting running Java classes, using the following Btrace script would activate the tracing of the wiring process on all the classes under the com.jdftm package:

      package jdftm;

      import com.sun.btrace.annotations.*;
      import com.sun.btrace.AnyType;
      import static com.sun.btrace.BTraceUtils.*;
      import java.lang.reflect.Constructor;
      import java.lang.reflect.Method;
      import java.lang.reflect.Field;

      @BTrace public class ReflecTracer {
        @OnMethod(
         clazz="/java.lang.reflect.Constructor/",
         method="/newInstance/")
        public static void anyNewInstance(Constructor con,Object[] args) {
         if(con!=null && matches("public\\scom\\.jdftm(\\.|\\w*)*\\(\\)",str(con))){
           println(strcat(strcat("Object ",str(con))," got constructed"));
         }
        }

        @OnMethod(
         clazz="/java.lang.reflect.Field/",
         method="/set/")
        public static void anyFieldSet(Field field,Object obj,Object value) {
          println(strcat("Set invoked on ",str(field)));
        }
      }

Using the script with Btrace involve two steps, compilation:

btracec ReflecTracer.java

Agent attaching (attach before context creation, debugging is an option):

btrace <process id, use jps> jdftm/ReflecTracer.class

The following beans wiring:

        public class BeanA {
         @Autowired
         private BeanB b;
        }

        public class BeanB {
         @Autowired
         private BeanA a;
        }

Produces the following traces:

    Object public com.jdftm.spring.context.BeanA() got constructed
    Object public com.jdftm.spring.context.BeanB() got constructed
    Set invoked on private com.jdftm.spring.context.BeanA com.jdftm.spring.context.BeanB.a
    Set invoked on private com.jdftm.spring.context.BeanB com.jdftm.spring.context.BeanA.b

An interesting idea that i had is to create a graph out of these traces and give the developer even better visualization on the process.

Learning some Haskell

Sat, 10 Jan 2009 11:19:00 -0500

Well iv been studying a little Haskell lately, mainly by reading Learn You a Haskell for Great Good!, I can only recommend any one to follow this book, my next read will be Real world Haskell.
Functional programing rulz!

Pure Java entertainment

Fri, 26 Dec 2008 00:08:00 -0500

Iv really enjoyed this presentation about Java puzzlers.
The main moral of this presentation in my mind is that we should relay more on static tools in order to avoid such corner cases (there is no way to remember them all).
Take a look im sure that youl enjoy it as I did.

VMware recovery methods

Thu, 18 Dec 2008 10:46:00 -0500

Iv been using VMware server and player for a time now and usually the hosted OS is XP, it happens to be that iv had the "luck" to witness the crash/corruption of such images and have devised two nifty tricks/methods of data recovery that i think worth sharing.
The first is for data recovery for an unbootable image, it matches cases in which you don't seem to be able to boot XP up but the image itself is readable & loaded by VMware (you see the black screen loading), the basic idea is to change the first boot device of the image to a Linux live cd (Ubuntu works perfectly), after booting up all you need to do is to mount the NTFS folder and copy it out (by scp, thumb drive, etc..).
The second one is more complex and comes to handle a case in which the image dose not load at all (vmware complains about a corrupted vmx file), in order to recover from this state there are a couple of prequisits:

VMware server installed.

You have another working XP image with a vmx file.

The corrupted image has a snapshot of a working state

The solution in this case is to copy the working vmx file into the folder of the corrupted image, load this image in VMware server and replace all the hard drive entries with the corrupted machine images, now all that is done is to revert to the last saved snapshot and see how the system is restored!
I hope that these two little methods help you as much as they helped me.

Java statefull callbacks

Tue, 16 Dec 2008 11:32:00 -0500

Any programmer comes to know callbacks at some point or another, in Java it looks something like this:

public interface Callback {
 void doCall();
}

public class CallbackCosumer {
 public void execute(final Callback callback){
  callback.doCall();
 }
}

// usage
public class Main {
 public static void main(String[] args) {
  final CallbackConsumer consumer = new CallbackConsumer();
  consumer.execute(new Callback() {
   public void doCall() {
    System.out.println("doing my callback thingy");
   }
  });
 }
}

The biggest downside is that its hard to pass calling context state into the rigid callback interface, we could try something like:

public class CustomCallback <T> implements Callback {

 private <T> state;

 public void doCall() {
  System.out.println(state);
 }

 public void setState(<T> state) {
  this.state = state;
 }
}

Note that this implementation is limiting, the doCall method has to work for any type T, custom behavior will require sub classing (not reasonable expectation from the callback provider).
A somewhat better (the most elegant that iv found to date) solution is to take advantage of another interface:

public interface StatefullCallback<T> extends Callback{
 void addState(T state);
}

// Now we have a custom state object
public static class CustomState {
 // fields ..
}

// And usage with a type matching logic
public static void main(String[] args) {
 final CallbackConsumer consumer = new CallbackConsumer();
 final StatefullCallback<CustomState> callback = new StatefullCallback<CustomState>() {
  private CustomState state;

  public void addState(final CustomState state) {
   this.state = state;
  }

  public void doCall() {
   // some custom logic that uses the custom state object
  }
 };
 callback.addState(new CustomState());
 consumer.execute(callback);
}

Its quite easy to see how its possible to pass multiple types of state objects into the callback execution context and do it with relative ease, closure-blessed languages require no such hacking.