Narkisr.com

Rules, not what you recall

Mon, 27 Apr 2020 00:00:00 +0200

Intro

Like most developers I used to think on rule engines as a interesting technology that never took off, a common misconception around them was that they would enable domain experts to encode rules into them without the need of having developers in this process (reducing cost/implementation time).

As expected this never worked in practice and as projects failed interest dwindled and they were left as an historical relic together with visual programming.

In this post I will try to revive the interest in using a rule engine and cover how I use Clara rules in an interesting domain - operation flow automation.

So what are rule engines?

A Rule engine include 3 main types of components:

Facts, information that our system get from the world.
Rules, how those facts effect our world
Queries that we may run in order to find what kind of rules were activate and why.

The underlying implementation uses a directed acyclic graph and an algorithm named RETE for efficient computation on top of a large sets of rules and facts.

* If the above rings a bell of Logic programming and expert system you are not mistaken with the later usually being implemented in Prolog.

A basic rule in Clara is composed from a left hand side (a match on facts) and the right hand side (the effect of a match) separated by '=>', in this case we match on facts that have a :state value of ::start :

(ns foo)

(defrule example
  "An example rule"
  ; What to match on
  [?e <- ::start]
  =>
  ; The effect of the match
  (println "foo")
  (insert! {:state ::done :failure false}))

Lets note a couple of interesting points here:

The RHS doesn't have to be side effect free! (see that lovely println)
The RHS can insert new facts into the system.
We can capture the incoming matched fact in variables (?e), this allows us to pass in arbitrary information to the rules.
Our facts are plain old Clojure maps (we can create them from any arbitrary input).

Our rule engine state (RETE remember?) is an immutable data structure (turtles all the way down) that we store in an atom:

(defn initialize []
  (atom (mk-session 'foo)))

(def session (initialize))

If we want to insert new facts into it (while keeping our old view of the world) we can just reset it:

(defn update- [facts]
  (let [new-facts (reduce (fn [s fact] (insert s fact)) @session facts)]
    (reset! session (fire-rules new-facts))))

(commment
  ; update our view of the world
  (update- {:state :foo/start :yet :another :key 1}))

We can also query our session and ask what happened:

(defquery get-failures
  "Find all failures"
  []
  [?f <- :re-flow.core/state (= true (this :failure))])

Existing approaches

Lets take a step back and cover what a conditional is composed out of:

A predicate (the when of an IF statement).
The result of a match on a predict (the logic that happens after a match).
The computation flow (the cause and effect graph).

We use conditional logic when we try to describe the flow of steps in a distributed process like for example a remote backup restore test:

Create a VM with a volume to restore our data to
Provision the VM with the required tooling and prepare our volume
Check that we are ready to start the restore process (our VM is ready and our volume has the correct size)
Restore the data and verify that it was successful
Notify on success or failure

There are a number of ways to model this kind of flow, the simplest one is to use conditionals:

(defn restore-backup [backup]
  (let [{:keys [created?] :as creation-result} (create-vm)]
     (if-not vm-create?
         (throw (ex-info "failed to create the VM" creation-result))
         (let [{:keys [provisioned?] :as provisioning-result} (provision)]
            ; rest of the flow
            ))))

There are a number of issue with this approch:

As Cyclomatic complexity explodes its really hard to reason about the code.
Composition is challenging since trying to compose one chain of conditional logic with another is fragile.
The only way to understand failure points in the code is by painstaking debugging and printing.

Another approach is to use finite state machines (FSM for short), the following FSM has a :start state and a number of transitions defined:

(defsm restore
  [[:start
    ::create -> (create-vm)]
   [:created
    true ->  (provision)
    false -> (report-error)]
   [:provisioned
    true ->  (restore)
    false -> (report-error)]
  [:restored
    true ->  (printrln "success")
    false -> (report-error)]
   ])

On a first glance this looks promising as it frees us from the conditional nesting but there are still a number of issues:

Location information is still complected making change in a large set of FMS's is still tricky.
They are not easy to compose (think about combining the restore FSM with a new notification FSM)
They lack introspection (understand why we reached a certain state within them).

There are other approaches that solve the same problem but I would like to claim (while not providing a rigorous mathematical proof) that they are reducible to the two approaches mentioned above.

The case fore Clara rules

Coming back to the same problem space (restoration flow) our code now is composed out of a two sets of rules, the first one is the creation and provisioning logic:

(ns re-flow.setup
  ;...
  )

(defrule creating
  "Create VM"
  [?e <- ::creating []]
  =>
  (insert! (assoc ?e :state ::created :failure (create-vm)))


(defrule provisioning
  "Provisioning"
  [?e <- ::created (= ?failure false)]
  =>
   ; ....
   (info "provisioning")
   (insert! (assoc ?e :state ::provisioned :failure (provision))))

Note that we didn't specify the order of execution we only provided the predicate and the effect of a match.

Our restoration logic is completely decoupled from the creation logic (it resides in a different namespace), it does not care on how the instance is coming to life it only concern itself with what it should do once its up and running:

(ns re-flow.restore
  ;...
  )

(defrule check
  "Check that the ?e fact is matching the ::restore spec"
  [?e <- ::start]
  =>
  (insert! (assoc ?e :state ::spec :failure (not (s/valid? ::restore ?e)))))

(defrule create
  "Triggering the creation of the instance"
  [?e <- ::spec [{:keys [failure]}] (= failure false)]
  =>
  (info "Starting to run setup instance")
  (insert!  (assoc ?e :state :re-flow.setup/creating :spec instance)))

(defrule restore
  "Restoring information"
  [?e <- ::provisioned [{:keys [failure]}] (= failure false)]
  =>
  (insert! (assoc ?e :state ::restored :failure (restore)))

The check rule matches on the ::start state fact where ?e is captured and verified against a spec, the second rule trigger the creation flow if the ::spec fact failure value is false.

In order to launch this flow we insert a fact into our session:

; We provide the backup information as a map within the fact
(update- {:state :re-flow.restore/start :backup {:source "s3://" ...})

We may use the query we already defined to find what rule fail:

(defn run-query [q]
  (query @session q))

; The query result is a native Clojure datastucture
(run-query get-failures)

This approach has a number of clear advantages:

Logic is decoupled its easy to make changes to one rule without breaking other rules.
Composition is easy, one set of rules can trigger other rules from other namespaces.
Tracking failure and other information is easy (just write queries)
We get the execution engine for free (computation graph).

A number of other key points that we should cover:

The engine does not deal with concurrency however its immutable which allows us to set the new engine state using native Clojure Refs.
The Rules are not persisted to disk by default if we choose to do so we can serialize its state and persist it.
The engine state is not a distributed data structure, if we have multiple JVM's we can shard our facts based on keys (think Kafka consumers).

Summary

Iv found Clara rules to be a really powerful tool for automation if you are curious check Re-core code, ill conclude this post with my own variation on Greenspuns tenth rule:

"Every complex yml/json that describes a complex pipeline is interpreted by an ad hoc informally-specified, bug-ridden, slow implementation of a Forward-chaining rules engine"

Pipelines are computation graphs

Footnotes:

Martin fowler mentioning the misconception.
The connection between Expert systems and rule engines.

Speculative execution

Tue, 20 Feb 2018 00:00:00 +0100

Intro

One of the characteristics of good tools is that they allow creative exploration of ideas, in this post ill cover some speculative flows that Re-ops may support in the future.

The flows are not implemented but Re-ops makes them all possible due to its basic architecture and design.

Voice up

There are a lot of cases where it makes sense to create/start a VM/Physical machine, lets take for example your media server, or a remote development machine, we need those only on specific times yet we usually run them 24/7, but what if we could trigger them based on our needs?

(defn plex-on []
  (run (start) | (alexa :reply "ok")))

(defn update-all []
  (run (update) | (alexa :reply "ok"))

; start VM instances by saying 'Alexa start plex on'
(wemo (alexa :event "plex power on") (plex-on))

; updating all the machine by saying 'Alexa update all'
(wemo (alexa :event "update all") (update-all))

Now this flow isn't implemented now, I suspect it should pretty easy to implement by exposing a WeMO like interface and pretending to be one or more devices.

Another good options (tough it will require a bit more setup) is to use MQTT, the main issue here is exposing the MQTT broker to Alexa.

Sensors

Another cool idea is to integrate Re-ops with sensors:

; our closet instances (including the physical hypervisor big-iron)
(def closet (Hosts. {} ["foo" "bar" "big-iron"]))

; Stopping our closet instances when it gets too hot (beyond 70c)
(def check-temp [t]
  (when (> t 70)
    (run (stop hs) | (email to-from "stopped all closed instances, room too hot"))

(watch :temp once-an-hour (check-temp (sensor :temperture))

Here we would stop all the VM's and the hosts if our server room (aka our closet) gets too hot thus preventing hardware meltdown while we are not near, we would need to exposing an MQTT end point and the rest should be pretty easy to follow.

A more far out scenario is automating our machines based on sensor detecting power output from solar panels (is it sunny enough or not) if there isn't enough power draw we would rather not to use costly energy from the wall, our mining farm will start/stop based on the power costs:

; our mining instances (including the hypervisor big-iron)
(def mining-pool (Hosts. {} ["coin-1" "coin-1" "big-iron"]))

; We produce more then 10KW of power lets start mining
(defn power-ok []
   (run (start mining-pool) | (pretty "pool resumed"))

; We produce less then 10KW of power lets stop mining
(defn low-power []
  (run (stop mining-pool) | (pretty "pool stopped"))

(defn adjust-pool []
  (if (> (check-power) 10))
     (power-ok)
     (low-power))

(watch :power-check once-an-hour adjust-pool)

Scanning

Running a scanning tool on a regular basis is a good way for checking compliance and detect potential security issues:


(def suspicious-ports #{25 ...})
(def scanner (Hosts. {} ["scanner"]))

(defn scan []
  (run (open-ports scanner) | (pick (fn [ports] (suspicious-ports ports))) | (email "suspicious ports found"))

(watch :nmap (every-day 22) (scan hosts))

Using the scanner from a dedicated host enables us to:

Selectively allow password-less sudo access instance for the specific VM.
Have scanning agents running within segmented networks, thus controlling which ports/networks are exposed.

Spot instances

In AWS we can bid for EC2 instances and offer a max price that we are willing to pay for them, for some workloads (like batch processing) this can be highly cost effective.

The main down side is that we do this bidding only within EC2, what if we could have a cost decision tree that makes a dynamic decision whether to use one provide or another:

; some long batch job
(def batch (Hosts. {} ["worker-1" "worker-2" "master"]))

; AWS cost per hour bellow $0.0058 per Hour
(defn use-ec2 []
  (run (destroy batch) | (create batch aws-spot-instance)))


; AWS too expansive we will run locally
(defn use-kvm [c]
  (run (destroy batch) | (create batch kvm-medium)))

(defn pick []
   (if (< (cost :ec2-cost) 0.0058)
     (use-ec2)
     (use-kvm)))

; The cost changes will be collected once an hour so we won't flap too often.
(watch :aws-spots once-an-hour pick)

Summary

In this post we explored some of the more speculative (yet possible) flows that Re-ops can enable, having a flexible and fully programmable tool can really lead to interesting ideas and implementation.

Footnotes:

WeMo home automation.
MQTT protocol,
AWS Spot instances

When types are eating away your sanity

Sun, 04 Feb 2018 00:00:00 +0100

Intro

Iv been programming in Java for more than 15 years and in every project I kept seeing the same pattern:

public class PersonTo {
  public String name;
  public Integer age;
}

public class PersonModel {
  public String name;
  public Integer age;
  public boolean isRegistered;
}

public class ServiceA {

  public void personsService(String json){
   PersonTo person = parse(json);
   PersonModel model = process(person);
   persist(model);
  }
}

If you lucky you have a set of conversion logic in place (iv seen a lot of duplicated code that ignores the issue and just copy and past the same logic):

public class PersonToConvertor<PersonModel> {
   public PersonModel convert(PersonTo person) {
    //.. the pleasure of conversion logic
    return personModel;
   }
}
public class PersonModelConvertor<PersonTo> {
   public PersonTo convert(PersonModel person) {
    //.. the pleasure of conversion logic
    return personTo;
   }
}

We have a simple json -> PersonTo -> PersonModel conversion chain, why we have those conversions? because each part of the system expect to get the data in a certain shape and form and because its better OOP modeling.

Still this is a simple example and you can imagine the piles and piles of code that deals with this issue, it does not matter if your programming in Java/Haskell/Go/Elm at the end of the day slurping data into your system will cost you.

Not only that but if the data types/structure changes you may find yourself in a huge refactoring and moving/shaking a lot of types/methods/functions around (a ripple effect).

But wait, is there an alternative?

Bringing back sanity

As a matter of a fact there is, almost any functional dynamic language does not require such wacky conversions, we can just work on the raw data:

(defn service-a [json]
    (-> json parse process persist))

; once parsed the person is a simple hashmap datastructure that we manipulate
(defn process [person]
   (assoc person :is-registered false))

Oh wait, that is too simple your cheating!, what about type checking your code? making sure that its correct and all.

In the words of Rich Hickey:

And I like to ask this question: What's true of every bug found in the field?

Audience reply: Someone wrote it? Audience reply: It got written.

It got written. Yes. What's a more interesting fact about it? It passed the type checker.

With all that in mind there are cases in which verifying structure the data between components makes sense, enter clojure.spec:

(s/def ::name string?)
(s/def ::age integer?)

(s/def ::person (s/keys :req [::name ::age]

This allows us to check the validity in run time and check that different parts of the system conform to the specifications of others.

We have magic X that converts our types!

There are "shorter" ways of converting types, such as:

Code generation, we generate our types and convertors we don't write them manually, the business isn't generated and its bound to those multiple types increasing the maintenance cost, in addition you maintain a whole set of tools and schema so its not free.
Automated mapping libraries, similar to code generation you still have to maintain a huge set mapping between the types also the types themself don't go away you still have multiple types describing the same entity.
We work on the raw parsed data matching on the JSON:

class CC[T] {
  def unapply(a:Option[Any]):Option[T] = if (a.isEmpty) {
    None
  } else {
    Some(a.get.asInstanceOf[T])
  }
}

object M extends CC[Map[String, Any]]
object L extends CC[List[Any]]
object S extends CC[String]
object D extends CC[Double]
object B extends CC[Boolean]

for {
  M(map) <- List(JSON.parseFull(jsonString))
  L(languages) = map.get("languages")
  language <- languages
  M(lang) = Some(language)
  S(name) = lang.get("name")
  B(active) = lang.get("is_active")
  D(completeness) = lang.get("completeness")
} yield {
  (name, active, completeness)
}

Imagine this logic for n JSON documents in your system, it gets ugly pretty quickly.

Summary

When using types in the context of unstructured data be aware to the trade-offs, if your system deals with a huge variety of JSON/CSV or random HTML you will be spending a lot time in juggling type conversion instead of focusing on the business logic itself.

Its hard to quantify but ~30% is not an exaggerated number from my experience in large Java projects, other languages with better type systems might get away with less but it still a considerable overhead.

Footnotes:

Simple Made easy.
Dozer is an example for an "automated" bean to bean mapping framework.
How to parse JSON in Scala using standard Scala classes?

The balancing game

Wed, 17 Jan 2018 00:00:00 +0100

Intro

Dependencies exists in multiple shapes and forms in our software, some are easy to detect while other are hidden and insidious.

Common to all dependencies is that they have a great effect on the systems we create, in this short post ill try to make you aware on why its crucial to manage them properly and the damage that poor management can cause.

Where can we find them?

In external libraries our code depends upon.
Services that our systems use (Databases, external API's etc..).
Operating system, packages its running, its kernel version etc..
The hardware we deploy on, the network setup.
People we depends upon like developers, clients, investors and of course users.

There are many other places that dependencies creep from just stop and think about it for a second.

Good or bad?

Like most things in life it depends (pun intended), any dependency can turn into a liability in the future.

It can make your code harder to change, OS upgrades harder or cause security issues, that said using the right dependency can reduce the amount of work you need to do while increasing reuse and security.

Managing dependencies is a balancing game and a delicate one at that, it takes experience and trial and error to learn which dependencies are good and which are better to avoid.

Do's and Don'ts

Map the dependencies you have and track them under an SCM.
Use dependency managers to manage them (create them if none exists).
Trim dependencies once in a while, they tend to accumulate and grow like a wild plant.
Keep dependencies up to date, postponing updates for too long can cause your system to stagnate (there are tools that track out of date dependencies).
Minimize dependencies, sometimes its better to copy and paste a single function (assuming its self isolated) than to include a entire library.
Don't over minimize and re-invent stuff if it exists already NIH.

Summary

Dependencies can really change the faith of a projects (and companies) once a dependency has set in it may be really hard to detach, be cautious and aware to both the challenge and reward that your dependencies present.

Footnotes:

Lein Ancient and Gradle Versions are tools to track and report old dependencies.
Gradle, Lein, Bundler, Pipefile are good examples of code dependency management tools.
Apt, DNF and Chocolatey manage packages on a variety of OSes.
Configuration management tools such as Puppet and Chef manage dependencies of deployed components.

PI crypt

Sat, 16 Apr 2016 00:00:00 +0200

Intro

Raspberry PI devices are on the rage now, they are cheap portable and run Linux, using such a device on a remote site (be it for remote backup or pentesting) requires you to keep sensitive data on remote sdcards/USB drives etc.

In this post ill cover how to setup root encrypted RPi.

Setup

Iv used a lot of resources (see footnotes) and posts in order to get a working solution using Ubuntu 15.10 as the host and Raspbian Jessie lite as the Pi OS image.

First we will setup the sdcard:

# prepare the image
$ wget https://downloads.raspberrypi.org/raspbian_lite_latest
$ unzip raspbian_lite_latest
# we assume the sdcard 
$ dd if=2016-03-18-raspbian-jessie-lite.img of=/dev/sdb bs=4M

Entering the chroot:

$ aptitude install qemu-user-static
# chrooting mostly same as in kali
$ mkdir -p /mnt/chroot/boot
$ mount /dev/sdb2 /mnt/chroot/
$ mount /dev/sdb1 /mnt/chroot/boot/
$ mount -t proc none /mnt/chroot/proc
$ mount -t sysfs none /mnt/chroot/sys
$ mount -o bind /dev /mnt/chroot/dev
$ mount -o bind /dev/pts /mnt/chroot/dev/pts
$ cp /usr/bin/qemu-arm-static /mnt/chroot/usr/bin/
# And we are home
$ LANG=C chroot /mnt/chroot/
$ uname -a
 Linux foo 4.2.0-25-generic #30-Ubuntu SMP Mon Jan 18 12:31:50 UTC 2016 armv7l GNU/Linux

Setting up initramf (yes your are running arm binaries on your x86 machine):

# Comment out the single line in this file (or else apt-get wont work)
$ vi /etc/ld.so.preload
$ apt-get update
$ apt-get install busybox cryptsetup dropbear -y

Our first mkinitramfs (the small dropbear that will load prior to the OS):

# find the kernel version for initramfs
$ ls -l /lib/modules/ |awk -F" " '{print $9}'
4.1.19+
4.1.19-v7+

# You can ignore the 'Unsupported ioctl: cmd=0x5331' at the end
$ mkinitramfs -o /boot/initramfs.gz 4.1.19-v7+
$ update-rc.d ssh enable
# set the root password for the init image
$ passwd
# Now edit and change/add root=/dev/mapper/crypt_sdcard cryptdevice=/dev/mmcblk0p2:crypt_sdcard 
$ vi /boot/cmdline.txt
# add 'initramfs initramfs.gz 0x00f00000'
$ vi /boot/config.txt

Copy the private ssh key that will enabled you to ssh to the dropbear instance (in which you enter the decryption password):

$ cat /etc/initramfs-tools/root/.ssh/id_rsa

At this point we will load modules (bypassing 'debian /sbin/cryptsetup: not found raspbian') and change the fstab to point to crypt_sdcard and:

$ cat /etc/fstab
proc /proc proc defaults 0 0
/dev/mmcblk0p1 /boot vfat defaults 0 2
/dev/mapper/crypt_sdcard / ext4 defaults,noatime 0 1

# make sure to load the initramfs modules 
$ cat /usr/share/initramfs-tools/modules
aes
chainiv
cryptomgr 
krng
sha256
sha224
sha512
sha1
xts
cbc
ecb
ctr
dm-crypt
dm-mod
# And yet again
$ cat /etc/initramfs-tools/modules
aes
chainiv
cryptomgr 
krng
sha256
sha224
sha512
sha1
xts
cbc
ecb
ctr
dm-crypt
dm-mod
# append the following to /usr/share/initramfs-tools/hooks/cryptroot (after the askpass copy_exec):
$ cat /usr/share/initramfs-tools/hooks/cryptroot | grep /sbin/cryptsetup -A 5
copy_exec /sbin/cryptsetup
copy_exec /sbin/dmsetup
copy_exec /lib/cryptsetup/askpass
# new section
for mod in aes chainiv cryptomgr krng sha256 sha224 sha512 sha1 xts cbc ecb ctr dm-crypt dm-mod; do
 add_crypto_modules $mod
done

# ignore the cryptsetup errors since we don't have encryption set yet
$ mkinitramfs -o /boot/initramfs.gz 4.1.19-v7+

We will un-chroot and copy our work aside so we can copy it back once the drive is encrypted:

$ exit
$ umount /mnt/chroot/boot
$ umount /mnt/chroot/sys
$ umount /mnt/chroot/proc
# Backup our work thus far
$ mkdir -p /mnt/backup
$ rsync -avh /mnt/chroot/* /mnt/backup/
# finalize the un-chrooting
$ umount /mnt/chroot/dev/pts
$ umount /mnt/chroot/dev
$ umount /mnt/chroot

We will move to setting up the encrypted drive

$ echo -e "d\n2\nw" | fdisk /dev/sdb
# note the use of 131072 which is where sdb1 ends for this specific image
$ echo -e "n\np\n2\n131072\n\nw" | fdisk /dev/sdb

Extrac the SDcard (so the new partitioning scheme will be detected) and move to encrypting the new partition:

$ cryptsetup -v -y --cipher aes-cbc-essiv:sha256 --key-size 256 luksFormat /dev/sdb2
$ cryptsetup -v luksOpen /dev/sdb2 crypt_sdcard
$ mkfs.ext4 /dev/mapper/crypt_sdcard

We can now restore our backup into the new encrypted partition:

$ mkdir -p /mnt/encrypted
$ mount /dev/mapper/crypt_sdcard /mnt/encrypted/
$ rsync -avh /mnt/backup/* /mnt/encrypted/
$ umount /mnt/encrypted/
$ rm -rf /mnt/backup
$ sync
$ cryptsetup luksClose /dev/mapper/crypt_sdcard

You can now insert the SDcard into the PI and carry on to logging in.

The login process is now composed of two steps, first logging into the dropbear session and decrypt the drive, the second is the usual ssh login:

# find the IP by viewing the pi load screen
$ ssh -i private root@<ip>
# Enter your key
$ /scripts/local-top/cryptroot
# trigger the boot process
$ kill -9 `ps | grep -m 1 'cryptroot' | cut -d ' ' -f 3`
# once the pi loads you should be able to login as usual
$ ssh pi@<ip>

Summary

The processes of setting the root encrypted image is quite contrived and manual at the moment, the guides out there aren't complete or up to date, this guide is a starting point for a more automated approach.

Footnotes:

The original tutorial for Kali Linux which gives a good outline (doesn't work as is on Raspbian tough).
A post which mainly helped me to solve 'qemu: uncaught target signal 4' error.
A post that helped solved the 'debian /sbin/cryptsetup: not found raspbian' errors

Security onion vagrant

Sat, 27 Dec 2014 00:00:00 +0100

Intro

Setting up IDS systems such as snort has always been a tedious task, Security onion (or SO for short) is a solution that aims to change that.

It includes not only snort but also ossec (a HIDS system), ELSA for central logging management, Snorby a dashboard for IDS events.

SO publishes a PPA which is Ubuntu 12.04 compatible and distributes a livecd ISO for setting it up on physical machines.

In this post ill go through on how to setup SO vagrant box, enabling us to do fast iterations and provisioning development.

BOX setup

The first step is creating the SO box itself using packer:

$ git clone git://github.com/narkisr/packer-security-onion.git 
# won't work on a headless machine
$ make virtualbox/security-onion-12.04.4

The box itself is quite heavily customized due to the requirements of SO, once the box was created we import it like any other:

$ vagrant box add security-onion-12.04.4_puppet-3.7.3 box/virtualbox/security-onion-12.04.4_puppet_3.7.3.box

Now we are ready to start rolling vagrant.

Vagrant

Perquisites:

Opskeleton defines the project layout of the following Vagrant/Puppet code base so make sure to have.
SO itself requires a running GUI (so no headless servers).
Have a mirror port ready on your switch connected to a physical interface (if you want to capture meaningful traffic), note that you can customize the mirror port vagrant uses (using VAGRANT_MIRROR).

$ git clone git://github.com/opskeleton/security-onion-sandbox.git
# Wont work on a headless machine
$ gem install bundle
$ bundle install
$ librarian-puppet install
# in order to customize mirror port use VAGRANT_MIRROR=ifc
$ vagrant up

Once the VM UI is running enter a terminal and start sosetup:

$ sosetup

On its first invocation the wizard will guide you through the network setup, choose eth0 as the management interface and eth1 as the monitor interface, reboot once done.

Once the machine is up run sosetup again to setup Sguil and Snorby u/p

$ sosetup

Once done you can either head on to localhost (from within the VM) or from the host use the forwarded ports snorby.

Footnotes:

Sadly automating the setup wizard using Puppet didn't workout, you can pass an answer file but the text based wizard does not work proprerly while being ran from the Puppet exec call.
openssh server didn't work well so the vagrant box is using dropbear instead.

Nodifying Clojure

Sat, 18 May 2013 00:00:00 +0200

Intro

In this post ill cover how to integrate ClojureScript (cljs in short) with nodejs, the use case I was trying to solve was to write hubot integration script with a product Im working on.

Hubot scripts are written using Coffeescript but iv really missed the power of Clojure under my finger tips so iv decided to write a nodejs library with all the main logic in cljs. I wanted to have a full development stack with build, dependency management and testing support.

A key point to understand is that a cljs/node project is living on dual platforms at once, it has dependencies rooted in its nodejs package.json file and some in its cljs project.clj not to mention two build procedures.

Setting up:

The tool chain required in setting up a project include:

lein the Clojure build tool we will be using lein-cljsbuild plugin in order to compile our cljs code.
npm which is the nodejs package management tool, I highly recommend using nvm for setting it up.

The project structure includes:

├── lib # our compiled cljs js file
├── Makefile # nodejs build tasks
├── node_modules # npm local libraries
├── package.json # npm dependencies
├── project.clj # lein cljs build and dependencies
├── src # cljs source
├── test # compiled test js file
└── test-cljs # cljs test code

The project.clj contains our cljs build settings and dependencies:

(defproject celestial-node "0.1.0-SNAPSHOT"
  :description "Celestial nodejs integration"
  :url "https://github.com/narkisr/celestial-node"
  :license {:name "Apache V2" :url "https://www.apache.org/licenses/LICENSE-2.0.html"}
  :dependencies [[org.clojure/core.incubator "0.1.2"]
                 [org.bodil/redlobster "0.2.1"]
                 [litmus "0.2.0-SNAPSHOT"]]
  
  :plugins  [[lein-cljsbuild "0.3.0"]]
  
  :cljsbuild {
              :builds
              {:prod 
               {:source-paths ["src"]
                :compiler {
                           :target :nodejs
                           :output-to "lib/celestial.js"
                           :optimizations :simple
                           :pretty-print true}}
               :test
               {:source-paths ["test-cljs"]
                :compiler {
                           :target :nodejs
                           :output-to "test/celestial.js"
                           :optimizations :simple
                           :pretty-print true}}}}
  )

Notice that there are two builds with two different outputs, one for production use (lib/celestial.js) and one for testing (test/celestial.js), this enable us to keep test logic out of our production code (in fact the test run executes as a part of the main) , in order to build the cljs output we could run:

 # one time compilation
 $ lein cljsbuild once 
 # auto compilation on save
 $ lein cljsbuild auto

Testing:

Another key aspect is a testing framework, most cljs testing frameworks deal with testing browser code (and require phantom js), the only one that iv found to run well on nodejs and has a cljs DSL is litmus which wrapps around mocha.js , currently its not up on Clojars so build it locally using lein install.

The following show cases how a litmus test looks like:

 (describe "system manipulations"
   (given "we store a new system"  
      (it-async "creates a new id"    
        (create-system (slurp-edn "fixtures/redis-system.edn") 
            (fn [body] (reset! sys-id (aget body "id")) (done))
            (fn [error] (println "failed" error) (done))))
      (it-async "we can fetch it back"
        (get-system @sys-id 
            (fn [body] (equals? (aget body "type") => "redis") (done))
            (fn [error] (println error))))    
          ))

Due to the async nature of nodejs we use done fn to mark when a test is indeed done, the DSL is BDD oriented, in order to run the tests we run the 'make test' defined in our Makefile (which is a part of the nodejs build system), the Makefile consists of:

REPORTER = dot
  
test:
        @NODE_ENV=test ./node_modules/.bin/mocha \
        --reporter $(REPORTER) \
        --ui tdd -t 3000
    
test-w:
        @NODE_ENV=test ./node_modules/.bin/mocha \
        --reporter $(REPORTER) \
        --growl \
        --ui tdd \
        --watch

package: clean
        lein cljsbuild once
        mkdir pkg
        cp lib/celestial.js pkg
        cp -r fixtures pkg
        
clean:  
      rm -rf pkg

.PHONY: test test-w

As you can see the test target invokes the mocha binary located under npm's node_modules library which is defined in the package.json file:

{
  "name": "celestial-node",
  "preferGlobal": false,
  "version": "0.1.0",
  "author": "Ronen Narkis <narkisr@gmail.com> (http://narkisr.com)",
  "description": "Celestial API for nodejs",
  "repository": {
    "type": "git",
    "url": ""
  },
  "license": "Apache-V2",
  "dependencies": {
    "npm": ">= 1.1.2",
    "request": "=2.21.0"
  },
  "scripts": {
    "test": "make test"
  },
  "devDependencies": {
    "chai": "*",
    "mocha": "*"
   },
  "optionalDependencies": {},
  "engines": {
    "node": "0.6 || 0.7 || 0.8"
  },
  "homepage": "",
  "main": "lib/celestial.js"
}

Nodejs integration:

The package.json includes all our nodejs dependencies (remeber the duality part?), we can add and use any nodejs module from our cljs code, in order to use a nodejs module:

; nodejs require
(def request (js/require "request")) 

; js interop usage
(defn GET [url s f]
  (.get request (<< "~(@conf :url)~{url}") (clj->js (opts)) (response- s f)))

In order to expose cljs api to nodejs consumers we need to export a ns:

(ns celestial.core ...)
 
(defn main [])

; required even if we don't use a main
(set! *main-cli-fn* main)

; export this ns as a part of nodejs module
(aset js/exports "core" celestial.core)

With that all in place we can require our cljs module from any other npm enabled project:

{
  "name": "hubot-celestial",
  "version": "0.0.1",
  "description": "Hubot script for interacting with Celestial",
  "main": "main.js",
  "scripts": {
  },
  "repository": "git://github.com/narkisr/hubot-celestial.git",
  "author": "narkisr.com",
  "license": "Apache-V2",
  "dependencies": {
    "coffee-script": "~1.4.0",
    "hubot": "~2.4.8",
    "hubocator": "0.1.2",
    "cli-table" : "0.2.0",
    "underscore" :"1.4.4",
    "celestial-node":"git://github.com/narkisr/celestial-node.git"
  }
}

Requiring the api:

celestial = require ('celestial-node')
celestial.core.get_system(id,((b)-> msg.send systemTable(b)), ((e)-> msg.send e))

Summary

Iv found cljs and nodejs to be a very powerfull (yet a bit young) combo to work with, it offers yet another very rich platform to run Clojure code on top of. I hope you will find it useful too.

Footnotes:

celestial-node (most of the code preseted)
hubot-celestial

Asus TF101 Cyanogenmod

Sat, 13 Apr 2013 00:00:00 +0200

Intro

In this post ill try to cover in detail how to root your Asus TF101 device. Some of the difficuly in rooting android devices is lack of clear undestanding of the process combbined with lack of proper documentation, I hope to clear a bit of the myst in this post.

DISCLAIMER, I take no responsility if you brick you device or any other damage that this might inflict on your device, proceed at your own peril!

Scheme

A root session is composed from 4 steps:

Rooting your device (basicly enabling root user access to underlying Linux system).
Install ClockworkMod Recovery(http://forum.xda-developers.com/wiki/ClockworkMod_Recovery) which is a recovery tool installed on the device, this enables unpacking of custom Android distributions on your device (like cyanogenmod ")."
Booting into CWM and installing a custom ROM from a predownloaded zip file.
Instaling Google aps from a pre download zip file (the reason this is required is because Google forbided the bundling of its closed source product with the custom ROM).

Lets begin with rooting the device, we will follow this method, first we will install adb and fastboot (both are android tools for interacting with the device) as described here :

$ sudo add-apt-repository ppa:nilarimogard/webupd8
$ sudo apt-get update
$ sudo apt-get install android-tools-adb android-tools-fastboot

Now enable usb debbuging in your device and connect it to your pc, it should be detected by adb:

$ adb devices                                                                  
List of devices attached 
xxxxxxxxxxxxxxx device

Now we will download TF101_Root.zip which includes a script and blob file that will uploaded via adb

$ wget http://ubuntuone.com/5xwKVscUxwgdnudw589ERd -O TF101_Root.zip
$ unzip TF101_Root.zip
# Now I follow root_tf101.sh
$ adb push recoveryblob /sdcard/
$ adb shell mv /data/local/tmp /data/local/tmp.bak
$ adb shell ln -s /dev/block/mmcblk0p4 /data/local/tmp
$ adb reboot 
# wait to device to boot before proceeding
$ adb shell dd if=/sdcard/recoveryblob of=/dev/block/mmcblk0p4
$ adb shell exit
$ adb reboot 
# wait to device to boot before proceeding
$ adb push Superuser-3.0.7-efghi-signed.zip /sdcard/
$ adb reboot

Now at the last boot follow the script maunal steps:

Shut down the Transformer manually and boot into recovery. To boot into recovery hold Power + Volume Down buttons and next confirm with Volume Up buked. Rogue XM recovery will boot.
Select the option to wipe cache, wipe Dalvik cache, then choose Install zip file from internal storage and choose the Superuser-3.0.7-efghi-signed.zip.

Congrates you have root access now.

Now lets head on to installing CWM, from your device download cwm-6.0.2.1-notouch-hybrid.zip as referenced from this forum entry , reboot into recovery mode (same as before) and apply the zip, reboot again and use the volume-up and power button to enter into CWM recovery.

Once you see the CWM recovery loading and working we can proceed with installing the unofficial cyanogenmod as reviewed here , from within your device browser download v0.8.0.zip and Google apps 4.2_gapps-jb-20121119.zip , now boot into CWM and apply the v0.8.0.zip from internal sdcard, also clear the device and dalvik caches reboot into the new Android OS.

Note that it will be missing Google apps but we still need to boot into the new OS before applying the second Google apps zip, now its safe to boot into CWM and apply the 4.2_gapps-jb-20121119.zip, one last reboot and you should be greated with Google acount setup dialog.

Waiting for your future

Tue, 24 Jul 2012 00:00:00 +0200

So, you've got an action that you want to run in parallel using multiple cores, in pure Java land you have the old fashioned Thread.start which takes a Runnable implementation, another option is the ExecutorService which uses a pool to store a collection of reusable threads (in fact its the same pool that agents use). Clojure on the other hand has the future macro which takes a body of code and runs it on a seprate thread:

(def f (future (println "hello")))
  
@f ; will block here until it executes

Futures use an unbound thread pool which is good for IO bound actions, lets assume we have a sequence (a large one at that) that serves as an input to that heavy IO operation, we want to parallelize the operation on multiple cores, the naive approach will be:

 (defn action [i]
    (Thread/sleep 4000)
    (println i))

(defn -main []
  (map (future (action %)) (range 1000)))

Running this code will result with no future run, the main thread will exit before any future gets the chance to execute (since its async), in order to wait for them to run we need to deref all futures,

 (defn -main []
  (map deref (map (future (action %)) (range 1000))))

This looks good and indeed when we run it we notice the following output

0
1 
.. ; omitted for brevity 
.. 
31
nil nil nil  ; after a delay 
32
..
..
63 ; order is random

So whats going on here? we a get a lot less concurrency then we'd like, what we see here is Clojure chunking in action, since the inner map is lazy we produce 32 futures in each chunk deref them and carry on to the next, we could force all futures by using doall:

(defn -main []
  (map deref (doall (map (future (action %)) (range 1000)))))

This poses another problem, we issue 1000 future and hammer on, this might cause the IO target to grind into a halt (be it filesystem or remote web service), a more controled method is required. One option is to use Java's own ExecutorService and submit actions into that (using a bounded pool) and in fact thats exactly what pallet thread enables us to do.

Now we want to implement two main requirements, the first generate a blocked amount of running actions, the second make sure not to block the execution of actions (by chunking for example) if threads are available to handle them:

(use '[pallet.thread.executor :only (executor execute)])  
 
(def pool
  (executor {:prefix "foo" :thread-group-name "foo-grp" :pool-size 4}))

(defn bound-future [f]
 {:pre [(ifn? f)]}; saves lots of errors
  (execute pool f))

(defn wait-on [futures]
  "Waiting on a sequence of futures, limited by a constant pool of threads"
  (while (some identity (map (comp not future-done?) futures))
    (Thread/sleep 1000) 
    )) 

(defn -main []
  (wait-on (map #(bound-future (fn []  (action %))) (range 1000))))

The print will now look like

0
1
2
3
; a short delay for the next 4 actions to kick into action
1
..

I found this method to offer a balance between the spread of work and predictable load on the target.

Raspberry foo

Fri, 20 Jul 2012 00:00:00 +0200

Iv been lucky to get my hands on a raspberry pie, from the get go it was clear that this device isn't ready for the faint hearted, while there is abundant content around for how to start going the pi is a moving target and things change quickly, this post is a quick start for getting pie going.

In this post will cover

Downloading sdcard image and \"dd'ing\"
Enabling SSH and updating Pi's frimware.
Enabling wifi with the 8192cu chipset:p

Downloading and dding

The pi site offers both Arch and Debian images , iv went ahead with Debian mainly since its closer to Ubuntu, note that there are [beta] (http://rpi-developers.com/frs/download.php/file/20) " versions out there but iv found the stock one working best.".

 $ sudo dd bs=1M if=debian6-19-04-2012.img of=/dev/sdb

When its done push the sdcard into its slot, make sure to connect the screen before powering up (iv had issue with connecting DVI cable once it was running),
as the boot is done connect it to the internet and run all the updates (note sudo password is raspberry by default, change it afterwards).

$ sudo aptitude update
$ sudo aptitude upgrade

SSHing

Its time to get our monitor back, we will enable SSH so we can work remotely:

# boot enable
$ sudo update-rc.d ssh

# ssh won't start without this
$ ssh-keygem -b 1024 -t rsa -f /etc/ssh/ssh_host_key 
$ ssh-keygen -b 1024 -t rsa -f /etc/ssh/ssh_host_rsa_key 
$ ssh-keygen -b 1024 -t rsa -f /etc/ssh/ssh_host_dsa_key 

# enable firewall
$ sudo aptitude install ufw
$ sudo ufw allow 22

Now will update Pi's frimware:

# updating frimware
$ sudo wget --no-check-certificate http://goo.gl/1BOfJ -O /usr/bin/rpi-update 
$ sudo chmod +x /usr/bin/rpi-update
$ sudo apt-get install git-core
# avoiding http://tinyurl.com/cbauu4w  
$ sudo ldconfig
$ sudo rpi-update

Wifi

The hardest part in setting up the pi is enabling wireless, this boils down to two main reasons, power consumption and driver compatibility (kernel version wise). The pi is powered by a 5v power supply (usb phone charger), this means that it has little power to spare for devices connected to its usb ports, high power long range wifi dongle require too much juice.

Among the low bandwidth cost adapters that iv found the EW-7811UN draws very little power, in addition it has a very low profile making it barely visible on the pi board, now comes the fun part of making it work.

The info of how setting it up is scattered and inconsistent, the best source out there for setting it up is this forum post by MrEngman , the post provides a shell script and instructions on how to use it, under compatibility section you can see which driver version matches a kernel version

$ uname -a 
Linux raspberrypi 3.1.9+ #171 PREEMPT Tue Jul 17 01:08:22 BST 2012 armv6l GNU/Linux

# The matching driver version 
rtl8188cus driver versions and compatible Linux versions
-------------------------------------------------------- 

Driver tar file: 8192cu-20120701.tar.gz (http://dl.dropbox.com/u/80256631/8192cu-20120701.tar.gz)
For kernel versions:

Linux raspberrypi 3.1.9+ #171 PREEMPT Tue Jul 17 01:08:22 BST 2012 armv6l GNU/Linux

Run the installation script:

$ sudo ./install-rtl8188cus.sh

Lets inspect the results:


# the 8192cu driver is in place
$ sudo ls /lib/modules/3.1.9+/kernel/net/wireless
8192cu.ko  cfg80211.ko	lib80211.ko  lib80211_crypt_ccmp.ko  lib80211_crypt_tkip.ko  lib80211_crypt_wep.ko

$ sudo cat /etc/network/interfaces | grep wlan0 -A 5 
allow-hotplug wlan0

auto wlan0

iface wlan0 inet dhcp
wpa-ssid "ssid"
wpa-psk "pass"

$ ifconfig | grep wlan
wlan0     Link encap:Ethernet  HWaddr 00:1f:1f:f2:ff:f8

Enjoy your pie!

Narkisr.com

Rules, not what you recall

Intro

So what are rule engines?

Existing approaches

The case fore Clara rules

Summary

Footnotes:

Speculative execution

Intro

Voice up

Sensors

Scanning

Spot instances

Summary

Footnotes:

When types are eating away your sanity

Intro

Bringing back sanity

We have magic X that converts our types!

Summary

Footnotes:

The balancing game

Intro

Where can we find them?

Good or bad?

Do's and Don'ts

Summary

Footnotes:

PI crypt

Intro

Setup

Login

Summary

Footnotes:

Security onion vagrant

Intro

BOX setup

Vagrant

Footnotes:

Nodifying Clojure

Intro

Setting up:

Testing:

Summary

Footnotes:

Asus TF101 Cyanogenmod

Intro

DISCLAIMER, I take no responsility if you brick you device or any other damage that this might inflict on your device, proceed at your own peril!

Scheme

Waiting for your future

Raspberry foo

In this post will cover

Downloading and dding

SSHing

Wifi