The Reasoned Schemer

I’m working through The Reasoned Schemer and had some trouble seeing clearly how results are returned when using run n. After Googling a bit, I found a clear walkthrough of the logic on an obscure message board. I’m reposting here to make sure it doesn’t disappear.

The key insight is that the n in run n refers not to the number of goals that should be tried, but rather the number of answers that should be returned, if available. Full text below:

Posted: Mon Jan 14, 2008 2:19 pm

Hi all,

I’m currently re-reading the Reasoned Schemer, but am having a bit of
difficulty understanding how values are computed for recursive
functions. For example, in chapter 2, the recursive function lolo and
section 24 asks what is the value of the following:

(run 5 (x)
(lolo ((a b) (c d) . x)))

The first value (), is obvious to me. The variable x is fresh, so it
is associated with () via the nullo check on line 4 of the method.

Here’s my explanation for how the second value is derived. Since we
are asking for another value, and the expressions are evaluated within
the conde, we refresh x and try the next line. At this point, the caro
associates the fresh variable x with the cons of the fresh variable a
and the fresh variable d-prime (introduced by the caro). Then, listo
is called on a. Since a is fresh, this call associates a with ().
Since this question succeeds, we try the answer. The cdro associates x
with the cons of fresh variable a-prime (introduced by the cdro) and
the fresh variable d. Then, lolo is called on d, and so d (being
fresh), is associated with ().

Since a-prime and a co-share, and a is (), and d and d-prime co-share,
and d is (), x can be successfully associated with the cons of () and
(), so the result is (()).

Here’s where I get confused. This invocation of run 2 has actually
produced #s 3 times, once to produce () and twice to produce (())
(once in each of the calls to listo and the recursive call to lolo).
However, we have *asked* for only two goals to be met, in order to
produce two values. If I invoke “run 3 …,” however, which conde
lines in which recursive frames are being evaluated, and having their
associations preserved? Or rather, how is the result expanding beyond
(())?

At the high level, I understand why the answer is (() (()) (() ()) (()
() ()) (() () () ())). But I’m still missing something at the lower
level that makes it harder for me to understand how some of the more
advanced examples, like the adder, work.

Thanks,
Joe
=====================

Posted: Tue Jan 15, 2008 3:30 pm

(run 2 (q) g1 g2 g3) does *not* specify that at most two goal
invocations should succeed, or that at most two #s goals should be
tried. Rather, run 2 specifies that we want two answers (if there are
two answers to be had), regardless of how many goals must be tried,
must succeed, or must fail in order to get those answers.

Hope this helps.

–Will

Share:

Here’s an example. We want to find the largest index i such that haystack[i] <= needle. Needle = 15, and haystack is a sorted list of the first 8 prime numbers, indexed by binary numbers:

000     2
001     3
010     5
011     7
100     11
101     13
110     17
111     19

Note first that the haystack index requires no more than 3 bits. Therefore we start with the highest order bit set: b=100. (This font denotes binary numbers here.) Is haystack[100] <= 15? Yes, 11<=15. Observe this means that the first bit of the index we are looking for is 1. We look at the next bit. Is haystack[110] <= 15? No, 17 <= 15 is false. This means the 2nd bit is 0. Finally, is haystack[101] <= 15? Yes, 13 <= 15. Therefore the index we are looking for is 101.

In essence, the main loop to find the index i is simply:

    for( ; b ; b >>= 1 )
        if( haystack[i|b] <= needle ) i |= b;

where b is the value with 1 bit set, that began with b=100 and i=0.

It is the natural structure of the bits within the index variable that automatically tracks both the upper and lower bounds of the search window for each iteration. Compare this with the less efficient and more verbose arithmetic performed in the main loops of standard binary search algorithms.

The full C++ code for the improved binary search algorithm fbsearch() is:

// Binary search revisited.
 
// Define only one of these:
// #define SETBIT_FAST
// #define SETBIT_FASTER
#define SETBIT_FASTEST
 
#ifdef SETBIT_FAST
#include <math.h>
#endif
 
// Return 1 << log_2(list_size-1), or 0 if list_size == 1.
// This sets the initial value of b in fbsearch().
inline unsigned init_bit( unsigned list_size )
{
#ifdef SETBIT_FAST
    return list_size == 1 ? 0 :
        1 << int( log(list_size-1) / M_LN2 );
#endif
#ifdef SETBIT_FASTER
    return list_size == 1 ? 0 :
        1 << ( sizeof(unsigned) << 3 ) - 1
             - __builtin_clz(list_size-1);
#endif
#ifdef SETBIT_FASTEST
    unsigned b;
    __asm__ ( "decl %%eax;"
              "je DONE;"
              "bsrl %%eax, %%ecx;" // BSR - Bit Scan Reverse (386+)
              "movl $1, %%eax;"
              "shll %%cl, %%eax;"
              "DONE:" : "=a" (b) : "a" (list_size)
    );
    return b;
#endif
}
 
// Return the greatest unsigned i where haystack[i] <= needle.
// If i does not exist (haystack is empty, or needle < haystack[0])
// then return unsigned(-1). T can be any type for which the binary
// operator <= is defined.
template <typename T>
unsigned fbsearch( const T haystack[], unsigned haystack_size,
                   const T& needle )
{
    if( haystack_size == 0 ) return unsigned(-1);
    unsigned i = 0;
    for( unsigned b = init_bit(haystack_size) ; b ; b >>= 1 )
    {
        unsigned j = i | b;
        if( haystack_size <= j ) continue;
        if( haystack[j] <= needle ) i = j;
        else
        {
            for( b >>= 1 ; b ; b >>= 1 )
                if( haystack[i|b] <= needle ) i |= b;
            break;
        }
    }
    return i || *haystack <= needle ? i : unsigned(-1);
}
 
// Example Usage
#include <iostream>
using namespace std;
 
int main()
{
    const int sorted_list[] = { 2, 3, 5, 7, 11, 13, 17, 19, 23 };
    const unsigned list_size = sizeof(sorted_list)/sizeof(int);
    int needle = 15;
    cout << "fbsearch(sorted_list,"<<list_size<<','<<needle<<") = "
         << fbsearch(sorted_list,list_size,needle) << endl;
    // fbsearch(sorted_list,9,15) = 5
    return 0;
}

Ancillary notes:

• The function init_bit() sets the initial value of b to having a single bit set in the highest position for indexing the array list. The three SETBIT_FAST, SETBIT_FASTER, and SETBIT_FASTEST code blocks within it are simply 3 different ways of calculating the initial value of b, whose speed ranks are the reverse of their cross-platform compatibility. The first method should compile everywhere, and simply calculates it from a base-2 logarithm. The second method uses the faster GNU __builtin_clz method (thanks Chao Xu for the suggestion) that counts the highest order bit from the left. The third and fastest method uses the Intel assembly instruction BSR that counts it from the right.
• The null return value of unsigned(-1) (=4294967295 for 32-bit unsigned) is the unique value that will never otherwise be returned. Therefore it can safely be checked for by the calling function without risk of misinterpretation.
• The one else block can safely be removed without changing the overall functionality. It exists to simply save having to check that the array index is in bounds on every iteration, since it is at a point in logic where that condition is guaranteed.

Share:

Say you have a large corpus of web documents and you want to group them together by some notion of “similarity”. For instance, we may want to detect plagiarism or find content that appears on multiple pages of a site.

In this scenario, it’s impractical to do a pairwise comparison of all documents. Fortunately, we can use simhashing.

Broadly speaking, simhashing is a algorithm that calculates a “cluster id” (the minimum hash, or minhash) from the content. Because the minhash for an item is calculated independently of the other items in the set, minhashing is an ideal candidate for MapReduce.

Ryan Moulton has written a wonderful article on Simhashing. I’m not going to repeat his content here, so if you’re unfamiliar with simhashing I encourage you to go and read his article first.

In his article, Ryan sketches the proof that the probability that any two sets (in this case, documents) share the same minhash is equal to their Jaccard similarity coefficient. This is a really neat result because we are able to get the Jaccard index without having to actually compare the intersection of the two sets directly.

cascading-simhash

I’ve created a library for calculating simhashes in Hadoop. It’s written in Clojure and Java and uses Casacalog and Cascading.

To use it, you 1) input tuples consisting of a (document_id, body) and 2) define how to tokenize your body. The job emits tuples of the form (minhash, document_id, body). You can then use minhash as the key for your next phase. (All records that share a minhash are potential duplicates.)

The library can be called from either Clojure or Java. Additionally, Simhash returns a Flow so you can use it in your Cascade if you want to make it part of a bigger pipeline.

A Java example

Here’s a quick example on how to use the library from Java:

/**
 * Simple Simhash - an example of how to use Simhash
 *
 * To run this example:
 *   lein uberjar
 *   lein classpath > classpath
 *   java -cp `cat classpath`:build/cascading-simhash-1.0.0-SNAPSHOT-standalone.jar simhash.examples.SimpleSimhash "test-resources/test-documents.txt"
 **/
public class SimpleSimhash {
  private static final Logger LOG = Logger.getLogger( SimpleSimhash.class );
 
  /**
   * Create a tokenizer that is a subclass of clojure.lang.AFn and
   * implements invoke(Object body)
   **/
  public static class Tokenizer extends AFn {
 
    /**
     * Your tokenization logic goes here
     *
     * @param String body
     * @return something seq-able
     */
    public Object invoke(Object body) throws Exception {
      String b = (String)body;
      return b.split(" ");
    }
  }
 
  public static void main( String[] args ) {
    Tap inputTap = new Hfs( new TextDelimited( 
                                new Fields("docid", "body"), "\t" ),
                            args[0] );
    Tap outputTap = new StdoutTap();
 
    // create the flow
    Flow simhashFlow = Simhash.simhash(inputTap, outputTap, 
                                       2, // combine n-th lowest minhashes (e.g. 2) 
                                       SimpleSimhash.Tokenizer.class);
    simhashFlow.complete(); // or add to your Cascade, etc
  }
}

Notice a few things here:

• We’re inputting a tap of two fields: (docid, body)
• The 2 parameter is the number of minhashes to combine. In this case, we will combine the 2 lowest hashes to create one minhash. This parameter controls the overlap required for a match. In this case, the two sets much share the same 2 minhashes in order to match.
• The Tokenizer is a subclass of clojure.lang.AFn. Override the invoke(Object) method and you will be passed the body the current record. In this case, we’re tokenizing by doing a simple String split.

If you’ve checked out the source you can run it like this:

lein uberjar
lein classpath > classpath
java -cp `cat classpath`:build/cascading-simhash-1.0.0-SNAPSHOT-standalone.jar simhash.examples.SimpleSimhash "test-resources/test-documents.txt"

Given:

# test-resources/test-documents.txt
# docid \t body
DocA	my dog has fleas
DocB	my dog has fleas
DocC	my dog has hair
DocD	see spot run
DocE	We hold these truths

We get:

RESULTS
-----------------------
23fd68296bc65391799c8c441faf4403c729256f	DocE	We hold these truths
402183e1cbc52e7c87eb230c281f35e4b27c2a39	DocD	see spot run
49c31c1459a7603bd5680d11285a5716c4ba3903	DocA	my dog has fleas
49c31c1459a7603bd5680d11285a5716c4ba3903	DocB	my dog has fleas
58e5a2035461323a37102e22273c9b25cbb9df61	DocC	my dog has hair
-----------------------

A Clojure example

Similarly, here’s how to run the library from Clojure. This time we use bi-grams as the tokens.

(ns simhash.examples.bigrams
  (:use 
   [simhash core util]
   [cascalog api testing])
  (:require 
   [simhash [taps :as t] [ops :as ops]]
   [clojure.contrib.str-utils :as stu])
  (:gen-class))
 
(defn my-source [path]
  (<- [?docid ?body]
      ((hfs-textline path) ?line)
      (ops/re-split-op [#"\t" 2] ?line :> ?docid ?body)
      (:distinct false)))
 
(defn tokenize 
  "tokenize into bi-grams (sliding window)"
  [body]
  (map
   (fn [tokens] (stu/str-join " " tokens))
   (partition 2 1 (stu/re-split #"\s+" body))))
 
(defn -main [& args]
  (?- (stdout) 
      (simhash-q (my-source (first args))
                 2 ;; number of minhashes
                 tokenize)))

A few things to point out about the Clojure example:

• simhash-q is just a Cascalog query. Unlike the Java example (which required a Tap as the input) simhash-q can accept any other Cascalog query as the input.
• You must use gen-class on the namespace that holds your tokenize function. This is because Cascading will serialize your Flow and it has a hard time with functions generated at run-time. Generally speaking, if the tokenize function isn’t aot compiled into a class you’re going to run into problems.

The project also includes a tokenizer for extracting text from HTML documents. For examples see tokenizers.html_text.clj for an example on how to write a tokenizer in Clojure. See HtmlSimhash.java for a Java example on how to use it.

Summary

Simhashing in MapReduce is a quick way to find clusters in a huge amount of data. By using Cascading and Cascalog we’re able to work with MapReduce jobs at the level of functions rather than individual map-reduce phases.

Have any data you need clustered? Try cascading-simhash and let me know how it goes!

Learn more about big data by following me on twitter.

You can get the jars via clojars:
leiningen:

  [cascading-simhash "1.0.0-SNAPSHOT"]

maven

<dependency>
  <groupId>cascading-simhash</groupId>
  <artifactId>cascading-simhash</artifactId>
  <version>1.0.0-SNAPSHOT</version>
</dependency>

View the source on github.

References

1. http://knol.google.com/k/simple-simhashing
2. http://en.wikipedia.org/wiki/Jaccard_index
3. http://en.wikipedia.org/wiki/MinHash

Share:

Why is XOR the default way to combine hashes?

I posted this question on Stackoverflow and got this excellent and concise answer from Greg Hewgill:

Assuming random (1-bit) inputs, the AND function output probability distribution is 25% 0 and 75% 1. Conversely, OR is 75% 0 and 25% 1.

The XOR function is 50% 0 and 50% 1, therefore it is good for combining uniform probability distributions.

The distributions become clear when we chart the output of each of the operations:

If you are combining two random bits with AND you have a 75% chance of 1. Similarly, if you combine two random bits with OR you have a 75% chance of 0.

My friend Rob Zinkov [1] explains it by framing it in terms of entropy:

[When combining hashes] essentially you want an operation that maximizes entropy since lower entropy implies the hash is storing less data than it appears.

With AND and OR lots of the variation in bits is being removed, so the hash you end up with tend to be biased to towards 1s (as in the OR case) or 0s (as in the AND case)

That bias means each bit is storing slightly less information (since that bias is present).

Summary: If you need to combine two hashes, XOR them together and you’ll have a better chance at maintaining the entropy of the original hashes.

[1] Rob runs the L.A. Machine Learning meetup group. If you’re in L.A. AND interested in machine learning, stop by sometime.

Share:

If you’re not familiar with PermGen space, it is a portion of memory reserved for the JVM itself. It is used for storing information about Classes and interned Strings.

When you intern a String the JVM stores a single copy of that String in the PermGen space. This can save RAM because only one copy of the String will exist in the system. It can also speed up == comparisons for two interned Strings because you only have to compare the reference not the characters.

The problem is, the PermGen space is typically very small (64m is a common default). So if you have many classes or a lot of interned Strings, you can easily blow out the PermGen space.

This problem was showing up in our crawler and we traced it to how we were parsing robots.txt.

robots.txt is a convention website owners can use that will instruct
crawlers how to act while they are on their site. All polite crawlers
use them. For example:

Disallow: /no-crawl/
Allow: /
Sitemap: http://www.foo.com/sitemap.xml

In our crawler, we’ve written a custom robots.txt parsing library: clj-robots (github).

In clj-robots, there was one section of the code where we were taking the left hand side of the robots.txt and converting it to a keyword. This made for cleaner code than comparing Strings. Since there are only a fixed number of robots.txt directives, this should be safe, right?

It turns out it isn’t safe. First, you don’t know what people are actually going to put in their robots.txt. Second, what we forgot was that many sites don’t have a robots.txt, but they don’t return an empty 404, they often return their custom 404 HTML page. What happened was we were parsing an HTML page as a robots.txt and then interning everything that looked like a robots.txt directive.

The result was a spectacular “java.lang.OutOfMemoryError: PermGen space” after just a few minutes. The general principle here is that you should never allow user-generated input become an interned String.

Lessons learned:

• PermGen stores Classes and interned Strings
• Clojure’s keyword interns a String
• Don’t call keyword on user-generated input
• A profiling tool (e.g. JProfiler) can be your best friend in these situations

References:

• Java GC FAQ
• Keyword.java from Clojure Core
• interned Strings : Java Glossary

Share:

Every CS student knows how to write a binary search algorithm. The question is, are we really making full use of the fact that we are using a binary computer to do a binary search? While it is true that computers are good at doing arithmetic, they are even better at doing pure bit logic. Often times our grammar school arithmetic way of thinking can get in the way of solving a problem that has a more elegant solution in terms of lower-level bit logic. Binary search in an ordered list is one such problem.

Your homework problem: re-write the typical textbook binary search in an ordered list algorithm, making it faster, without doing any arithmetic within the main loop of the search algorithm (underlying pointer arithmetic by direct access of array elements is ok).

Solution in C++:

// Fastest binary search
#include <iostream>
#include <math.h>

int main()
{
    const int sorted_list[] =
        { 2, 3, 5, 7, 11, 13, 17, 19, 23, 29 };
    const size_t list_size = sizeof(sorted_list)/sizeof(int);
    int i = 7;
    cout << "fbsearch(sorted_list,"<<list_size<<‘,’<<i<<") = "
         << fbsearch(sorted_list,list_size,i) << endl;
    return 0;
}

Technical Notes:

• Feel free to add a statement within the loop that returns the index if found. Note that for large n, you will spend more time on average checking for equality than the time you will save by returning early.
• Instead of dividing by ln(2), feel free to save the constant 1/ln(2) and multiply by it instead.

So what’s going on here? Take a list of the first 10 prime numbers, for example, and enumerate them using base-2:

0000   2
0001   3
0010   5
0011   7
0100  11
0101  13
0110  17
0111  19
1000  23
1001  29

If we are given one of the items on the list, say 13, then we are looking for its 4-bit index. Each bit is a yes/no question. The first bit asks, “Are you at index 1000 or higher?” If no, then the second bit asks, “Are you at index 0100 or higher?” If yes, then the third bit asks, “Are you at 0110 or higher?” If no then the last bit asks, “Are you at 0101 or higher?” By now we have asked all 4 questions, and thus have all 4 bits of the index.

Moral of the story: Sometimes it’s better to think like a silicon being than a carbon one.

p.s. Happy birthday, Nate!

Share:

Here’s a quick example on how to use it:

;; lein dependency: [net.htmlparser.jericho/jericho-html "3.1"]
(ns foo.preprocess
  (:import 
   [java.io File BufferedInputStream FileInputStream]
   [net.htmlparser.jericho Source TextExtractor]))
 
(defn extract-text 
  "given File returns a String of the extracted text"
  [f]
  (let [source (Source. (BufferedInputStream. (FileInputStream. f)))]
    (.toString (TextExtractor. source))))
 
(def filename "data/some-index.html")
(extract-text (java.io.File. filename))

TextExtractor has sensible defaults and ignores the css and javascript by default. See the TextExtractor class for more details.

Share:

You can find the code on github.

Usage:

    BoyerMoore.search(haystack, needle)   # returns index of needle or nil

Examples:

Basic search in string:

    BoyerMoore.search("ANPANMAN", "ANP")   # => 0
    BoyerMoore.search("ANPANMAN", "ANPXX") # => nil 
    BoyerMoore.search("foobar", "bar")     # => 3

You can also search an array of tokens:

    BoyerMoore.search(["<b>", "hi", "</b>"], ["hi"])         # => 1 
    BoyerMoore.search(["bam", "foo", "bar"], ["foo", "bar"]) # => 1 
    BoyerMoore.search(["bam", "bar", "baz"], ["foo"])        # => nil

A token can be a regular expression:

    BoyerMoore.search(["Sing", "99", "Luftballon"], [/\d+/]) == 1
    BoyerMoore.search(["Nate Murray", "5 Pine Street", "Los Angeles", "CA", "90210"], [/^\w{2}$/, /^\d{5}$/]) == 3

Notes:

The regular-expression token matching is a bit of a hack and will be fairly slow because every hash miss is compared against every regular expression key. You probably shouldn’t use the regular expression token search for anything more than a toy.

Download the Boyer-Moore string search algorithm in Ruby.

Share:

You can find the code on github

My solution’s main ideas are adapted from these two pages:

• http://getsetgames.com/2009/08/21/cocos2d-and-uiscrollview/
• http://blog.proculo.de/archives/180-Paging-enabled-UIScrollView-With-Previews.html

My contribution is combining the UIScrollView with previews with Cocos2d and cleaning it up.

If you haven’t tried to implement this before it might not be obvious why this is tricky to implement. Apple’s UIScrollView allows you to have a view which scrolls and optionally snaps to pages. The effect you see in the video above (and in Angry Birds level selection and many other apps) shows a preview of each panel on either side. This let’s you easily see if a next or previous page exists and you see a preview of that page.

The problem is that Apple’s UIScrollView doesn’t let you set the width of the frame, so you can’t page less than a whole screen (well, a whole width of theUIScrollView, more on that later).

To get around this I originally tried writing my own paging controller. If you’ve tried this you’ll know that it is extremely tricky to get the same interaction dynamics as Apple’s. (For instance, pull out your phone and play with the Photo application. Notice if you just drag slowly you lack enough inertia to go to the next page so it will snap back to the frame your are on. If you flick fast over a small area the page will skip to the next frame. Etc.) While at first glance the rules seem easy to reimplement, you have to cover a lot of edge cases to recreating the familiar paging interaction.

So ideally we need to figure out a way to use Apple’s UIScrollView and we should save ourselves a lot of work.

Like we said above, a UIScrollView will only page the width of the entire UIScrollView. So to get this preview effect you can create a UIScrollView that is less than the width of the entire screen. The problem here is that any touches that lie outside of that UIScrollView (say on the edge of the screen won’t be sent to the UIScrollView.

Our solution, (again, borrowed largely from the above links) looks like this:

The idea is this:

• We create a CCMenu and add it to a CCLayer
• The UIScrollView is resized to the width of our panel images (smaller than the whole screen)
• The UIScrollView transforms its scrolling action into moving the position of the CCLayer containing our CCMenu
• We create a full-screen TouchDelegatingView that simply forwards its touches on to the UIScrollView

More Details

In Jacob’s Shapes (JS), we have a GameController which knows all of the levels. For the sake of this example, we’re just going to store all the level names in an NSArray.

# HCUPPanelScene.m (in onEnter)
NSArray* panelNames = [NSArray arrayWithObjects: 
    @"amazon", @"arctic",
    @"brkfst", @"camp", 
    @"city", nil];
int numberOfPages = [panelNames count];
 
// create an empty layer for us to work with
CCLayer* panels = [CCLayer node];

Custom CCMenu and CCMenuItem

We use a custom subclass of CCMenu and CCMenuItem, NMPanelMenu and NMPanelMenuItem, respectively. NMPanelMenu tweaks how the current item is determined. Overriding NMPanelMenuItem allows us to add metadata about the panel, play sounds, and optimize how we use the images for selected panels.

# HCUPPanelScene.m
NMPanelMenu* menu = [NMPanelMenu menuWithItems: nil];
float onePanelWide = -1;
 
// Now add the panels
for(int i=0; i < numberOfPages; i++) {
    NSString* currentName = [panelNames objectAtIndex:i];
    CCSprite* pane2 = [CCSprite spriteWithFile:[NSString stringWithFormat: @"%@-panel.png", currentName]];
    NMPanelMenuItem* menuItem2 = [[NMPanelMenuItem alloc] initFromNormalSprite:pane2 
                                                                selectedSprite:pane2
                                                                  activeSprite:pane2
                                                                disabledSprite:pane2
                                                                          name:currentName
                                                                        target:self selector:@selector(levelPicked:)];
    menuItem2.world = i;
    menuItem2.name = currentName;
    [menu addChild: menuItem2];
    [menuItem2 release];
    // set onePanelWide to be the width of the first panel
    if(i==0) onePanelWide = [pane2 textureRect].size.width;
}

Here we used CCSprite#spriteWithFile, but in JS we use Zwoptex-created sprite sheets for the panels and then create sprites from those frames. This makes a huge difference in the load time of this scene when you have 20 panels. In JS, instead of loading 20 textures (one for each panel) we only load 2 textures, each containing 10 panels each.

JS is graphics heavy and we definitely had to pay attention to file sizes to keep it under 22MB. Originally I had created two versions of each panel, one for “off” and one with a glow for “on” (active/selected). Each of the panels as a transparent png was somewhere around 100k. So 100k x 2 for each state x 20 panels was somewhere around 4MB just for this single scene.

We decided to sacrifice a bit of the quality of the glow for the “on” state and just create one transparent image for the glow and reuse that for every panel.

To use the glow a portion of our NMPanelMenuItem looks like this:

# NMPanelMenuItem.m
-(void) activate
{
    isActive_ = YES;
    // play sound here
    [super activate];
}
 
-(void) draw
{
    if(isActive_) {
        [self.activeImage draw];
        if(self.showGlow) [self.glow draw];
    } else {
        [super draw];
    }
}

Where self.glow is a CCSprite attached to the NMPanelMenuItem.

Adding the Cocos2d Panels

Next we need to setup some basic options for how much padding we want and what the total width of the panels layer is going to be. Then we add the panels to our scene and set the position.

# HCUPPanelScene.m
float padding = 15;
float totalPanelWidth = onePanelWide + padding*2;
float totalWidth = numberOfPages * totalPanelWidth; // (wait, do we need padding in here?)
 
int currentWorldOffset = 0;    // current world number. 
// int currentWorldOffset = 1; // Try changing to 1 and see what happens
 
[menu alignItemsHorizontallyWithPadding: padding*2];
 
// add our panels layer
[panels addChild:menu];
[self addChild:panels];
 
// set the position of the menu to the center of the very first panel
menu.position = ccpAdd(menu.position, ccp(totalWidth/2 - totalPanelWidth/2, 0));

Note that the panels are the visual representation but we haven’t added in any scrolling dynamics. To do that we need to add a UIScrollView.

Adding the UIScrollView

Here we do two things:

1. Add our CocosOverlayScrollView which is only one panel wide (less than the whole screen). If we had this layer only then we wouldn’t be notified of touches on the edge of the screen.

2. We add the TouchDelegatingView which is full screen. The TouchDelegatingView will delegate any touches it receives to our paging scroll view

# HCUPPanelScene.m
// Note that we're only concerned with a horizontal iPhone. If your game is
// vertical, change accordingly
touchDelegatingView = [[TouchDelegatingView alloc] initWithFrame:CGRectMake(0, 0, 320, 480)];
scrollView = [[CocosOverlayScrollView alloc] initWithFrame:CGRectMake(0, 0, 320, totalPanelWidth)
                                                  numPages: numberOfPages
                                                     width: totalPanelWidth
                                                     layer: panels];
touchDelegatingView.scrollView = scrollView;
 
// this is just to pre-set the scroll view to a particular panel
[scrollView setContentOffset: CGPointMake(0, currentWorldOffset * totalPanelWidth) animated: NO];
 
// Add views to cocos2d
// We called it a TouchDelegatingView, but it actually isn't containing anything at all.
// In reality it is just taking up any space under our ScrollView and delegating the touches. 
[[[CCDirector sharedDirector] openGLView] addSubview:touchDelegatingView];
[[[CCDirector sharedDirector] openGLView] addSubview:scrollView];
 
[scrollView release];
[touchDelegatingView release];

You can configure your UIScrollView options by simply changing the code in CocosOverlayScrollView#initWithFrame:numPages:width:layer. (Note that this class was originally written by Alexander Repty)

The TouchDelegatingView simply delegates any touches it receives to the CocosOverlayScrollView.

And there you have it! Feel free to fork and make any changes to the code and send me a pull request.

What do you think? Have any ideas for cleaning it up? Leave your comments below!

Share:

I was trying some remote git repo tests last night, and when I went to push my changes, I received this warning message:

warning: updating the currently checked out branch; this may cause confusion, as the index and working tree do not reflect changes that are now in HEAD.

On remote server I created a directory and did “git init“. Then cloned it from my local machine, did changes, committed, and then push. All seemed straightforward there. Any thoughts?

The issue git is warning you about is that your remote has both the repository and a working copy. That is, on the remote server you have a directory project/ with files in it (the working copy) and the folder project/.git (the repository).

If you push from your local machine to the remote, you will only be updating files in the repository and not the working copy. That is, the non-git files will not be changed. This can be confusing because you might log into the remote after you push an expect the working copy to be different.

To deal with this possible confusion git init provides a --bare option. What this does is create the repository only (no working copy). You can then push and pull from the remote like you might a central svn server.

Let me show an example. Say I have an existing git repository on my local machine and I want to create a new remote to back it up. My workflow would look like this:

ssh me@myserver.com
mkdir ~/git/newproject.git
cd ~/git/newproject.git
git init --bare
exit
git remote add myserver me@myserver.com:/home/nmurray/git/newproject.git
git push myserver master

If you want, you could even chain these together as a single command:

ssh me@myserver.com "mkdir ~/git/newproject.git && cd ~/git/newproject.git && git init --bare" && echo git remote add myserver me@myservercom:/home/nmurray/git/newproject.git

Hope this helps!

Share:

I like to use Cascading (or cascalog) for my Hadoop jobs, but out of the box Cascading doesn’t support using SequenceFiles that were created outside of Cascading. That is to say, Cascading requires that your SequenceFiles values be an instance of Tuple.

The solution is to create your own Scheme that parses a SequenceFile according to your own format. In my case I just want to parse each line as the string list.

The code is simple but may not be obvious for a first-time Cascading user. I hope this will save someone a few minutes.

    package com.xcombinator;
 
    import java.io.IOException;
 
    import cascading.tap.Tap;
    import cascading.tuple.Fields;
    import cascading.tuple.Tuple;
    import cascading.tuple.TupleEntry;
    import cascading.tuple.Tuples;
    import cascading.scheme.SequenceFile;
    import org.apache.hadoop.mapred.JobConf;
    import org.apache.hadoop.mapred.OutputCollector;
    import org.apache.hadoop.mapred.SequenceFileInputFormat;
    import org.apache.hadoop.mapred.SequenceFileOutputFormat;
 
    /**
     * A SequenceFileAsText is a type of {@link SequenceFile}, however the
     * SequenceFile has been created outside of Cascading and is assumed to have a
     * value of a string.
     */
    public class SequenceFileAsText extends SequenceFile
      {
      /** Field serialVersionUID */
      private static final long serialVersionUID = 1L;
 
      /** Protected for use by TempDfs and other subclasses. Not for general consumption. */
      protected SequenceFileAsText()
        {
        super( null );
        }
 
      /**
       * Creates a new SequenceFileAsText instance that stores the given field names.
       *
       * @param fields
       */
      public SequenceFileAsText( Fields fields )
        {
        super( fields );
        }
 
      @Override
      public Tuple source( Object key, Object value )
      {
        if (value instanceof Tuple)
        {
          return (Tuple) value;
        }
        else if (value instanceof Comparable)
        {
          return new Tuple((Comparable) value);
        }
        else if (value != null)
        {
          return new Tuple(String.valueOf(value));
        }
        else
        {
          return new Tuple((Comparable)null);
        }
      }
 
    }

Share:

Git Cheat Sheet Preview

I realize there are a number of cheatsheets for git already. However, I wanted a simple, one-page sheet specifically for my attendees.

You can download it here:

• git cheatsheet pdf
• git cheatsheet LaTeX source

Like it? Hate it? Find a typo? Leave your feedback in the comments!

Here are my raw notes from the talk:

;; -*- mode: Markdown; -*-

# How to read:
commands are indented
actions to perform while presenting are marked with @
left to right

# Welcome
see progit.org
what is version control

why use it:

  * backup/restore
  * synchronization sharing
  * track changes
  * ownership
  * branching and merging

who has used subversion 

git
  * you've heard its distributed
  * b/c branching and merging

pace - slow, no slides

leave with practical understanding

# Install & Config

    sudo port install git-core +svn
    git config --global user.name "Nate Murray"
    git config --global user.email "nate@natemurray.com"

# Basic Commands

    cd ~
    mkdir -p projects/demo       # explain only a little
    cd projects/demo
    git init
    git status                   # nothing here
    ls -a                        # talk .git repository vs. working copy
    echo "version 1" > README.txt
    git status                   # untracked file
    git add README.txt
    git status                   # changes to be committed
    git commit -m "added version one of the file"
    git status                   # clean

stop, draw the picture of the local operation phases - e.g. svn vs. git

> Principle 1: (almost) everything is local

so now that you know about the staging area, lets do it again

    echo "new file" > sheep.rb
    git status                   # draw untracked
    git add sheep.rb
    git status                   # draw staged
    git commit -m "added"

    cat README.txt                 # draw unmodified
    echo "version 2" > README.txt
    git status                   # draw modified
    git commit -a -m "updated version" # shorthand for git add
    git status

Tips:

    git config --global alias.st status
    git st

# Git Internals

* Before we can talk about branching you *have* to understand how git (tried to avoid this)
* files and folders

three objects -  @ Draw first commit

  * blob        - raw data
  * tree        - folder (stores blobs and trees)
  * commit      - snapshot of the repo + meta 

You won't need to use `git cat-file` on a daily basis. however, understanding
the concepts we're going to talk about is really important for branching.

    git log # view the log
    git show ----  # first commit, whatever it is

    git cat-file -p  ---- # first commit
    git cat-file -p  ---- # tree
    git cat-file -p  ---- # blob

draw the rest using git `cat-file`

    git log           # show the log again
    git cat-file -p ---- # second commit

draw the picture. point out the parent connection.
note committer / author

    git cat-file -p ---- # tree

note here there are two blobs!

finish drawing out the second commit
* git stores reference to first file.
* snapshot of the *whole project*
* git stores each file once
* filename is in the `tree` 

draw the last commit

     git log
     git cat-file -p ---- # third commit

> Principle #2 : Git commits are snapshots

* A commit in git is a snapshot of the entire project, not just a list of diffs.
* snapshot is based on the SHA hash function. guarantees file integrity

# refs/branches

questions?

@ stop. redraw commits as *linear* . looking only at commits

ready to define a branch
a branch is a pointer to a commi
text file with a sha. thats it. 

start with one branch called `master`

    git branch

bash prompt

    # skip this
    tree .git/refs/
    cat .git/refs/heads/master
    git log
    # compare the SHAs

update diagram by adding a `ref` to our commit. (`master`). 

@ draw circle pointing to commit

create testing branch

# branching

So lets create another branch:

    git branch testing
    git branch

only created, didn't switch. just created a ref pointing to this
commit

@ update diagram

How does git know what branch we are "on"?

special ref called `HEAD` that points to the local branch
since we are still on master HEAD points to master

@ add HEAD

To switch working copy, use the `git checkout`

    git checkout testing
    git branch

HEAD moves from `master` to `testing`

@ update diagram

master and testing point to the same commit, working directory isn't changed

checkout means something different in git than it does in svn.
checkout in git to switch our working directory to a particular commit. 

now make changes:

    cat README.txt
    echo "we are on the testing branch!" > README.txt
    cat README.txt
    git commit -a -m "updated the readme"
    git log

@ update diagram, adding new commit. move the testing ref and the HEAD ref with it

add a "test"

    echo "this is a test" > test.rb
    git add test.rb                    # stage it for our commit
    git commit -m "added a test"       # now commit
    git log

@ update diagram - should have two commits

hotfix - scenario: you need to switch back to master

    git checkout master
    ls

@ move HEAD

so notice two things.
1) switching to this branch was fast - everything is local
2) our file test.rb is absent

and if we

    cat README.txt

it says 'version 2' just like we would expect

    echo "applying fix" >> sheep.rb
    cat sheep.rb
    git commit -a -m "applied important fix"
    git log
    git cat-file -p ---- # last commit

@ draw the new commit, and draw its reference back to the parent. move HEAD and master

now fixed, can push into production
and get back to work in `testing`

    git checkout testing
    cat README.txt
    cat test.rb

This is a general pattern:

> Principle #3: Branching is cheap, use it often

If you are working on a particular feature, create a branch. 

If you're coming from svn, making frequent branches might seem unnatural.
in svn, a branch is global -> namespace issues.
vs. git: private branches
name your branch 'test' and it won't collide with anyone elses

But branching itself isn't that useful unless its easy to merge.

* how many of you have merged a branch in svn?
* how many of you enjoyed it?

merging is one of git's strength and git makes it relatively easy

# merging

    cat sheep.rb

two branches: `master` and `testing` - need to merge

    git checkout master
    git merge testing
    git show HEAD

instead of a 'parent' we have a line that says 'merge'
a merge commit has more than one parent

@ draw the commit object
@ draw lines to the commits

    gitx

sometimes merging doesn't go as planned - conflicts

    git checkout -b breaker

this is shorthand for create and then checkout a new branch based on the
current HEAD

    vi sheep.rb # changing fix
    git commit -a -m "changed the fix"
    git checkout master
    vi sheep.rb # improving fix
    git commit -a -m "improved the fix"

@(update diagram, adding breaker and master refs)

    git merge breaker
    git status

there are many diff viewing tools.
* perforce
* opendiff - from apple

    git mergetool -t opendiff

I don't really like using the visual tools.
Sometimes you need character level editing

    vi sheep.rb
    git add sheep.rb
    git commit -a

talk about merge with conflicts

@ update diagram draw new merge commit

    gitx

Questions?

# Remotes

Everything so far on one machine. 

I work offline (I take the train)
If I break something I can rollback see where I was an hour ago 

want to share our changes.
might seem scary or messy because changes to totally independent lines of the code.
but in practice its not a problem.

svn version numbers are incremental - so two repos would get out of
step
no easy way of merging two separate repostories. 

git blob identifiers are a SHA of the content.
if the same content is created anywhere in the universe you'll still
have the same SHA

git doesn't care about where your commits come from or how you get them

Protocols:
  * ssh
  * git
  * http
  * local file system

sample project on our github

    cd ..
    open http://XXX/nmurray/simple-echo
    git clone git@XXX:nmurray/simple-echo.git
    cd simple-echo
    git log

svn checkout just HEAD
vs. git - whole repo

To be able to collaborate with others you have to manage 'remote repositories'.
When you clone a project, you have a default remote called 'origin'. 

    git remote -v

Remotes are pointers to other repositories that are _usually_ over the network.
'pull' and 'push' changes.

    vi README.mkd
    # make a change
    git commit -a -m "make a change"
    git push

If someone else makes a change:

    git pull origin master

This means pull from `origin` the branch `master` into local branch `master`. You can often to just

    git pull

which means pull from origin whatever branch Im on (i.e. HEAD) into this branch.

Now let's say someone pushes a change and I make a change
I can't push unless I pull first. This is good.

# remote forks

So that is while we are on the same line. What if were on different lines?

@(open up webbrowser again)

Bh also has forked my project. But when we say forked, all the means is he has
created his own development line from some of my commits

    git remote add bh git@XXX:bhenderson/simple-echo.git
    git remote -v

Now you shouldn't be surprised to learn that adding the remote doesn't change
anything. First we have to `fetch` hist changes

    git fetch bh

`fetch` brings his commits into my repo but again, doesnt change my working copy.

fetch brought branches + commits into repo
work with those branches just like any other branch.

    git branch -a

So you see here we have 

* `master`, which is our local master
* we have at the bottom `origin/master` which is the origin where we pulled from branch master
* and then we have `bh/master`, which is bhendersons master branch

These are all regular branches: they are just pointers to commits. We
can even checkout as branch 

    git checkout bh/master

scary message

    git checkout master

So how would we merge bhendersons changes with our own? I'm sure you could guess by now. Simply:

    git merge bh/master # don't press enter!!!

But lets take it up a notch.
say you didn't want to merge bh changes in your master branch.
real world, you might not know if his changes would merge cleanly
don't want to mess up your master branch.  

What we are going to do is
* create a new branch,
* merge bhs branch in THAT branch
* then we're going to merge to master.

It will make more sense when we do it. Lets try:

Okay we first want to create a new branch based on our master

    git checkout -b bh-merge
    git branch -a 

Now lets merge his changes

    cat simple-echo.rb
    git merge bh/master
    cat simple-echo.rb
    git log                  # see bh as the author of the commit

okay everything was clean! *phew* now lets go back to master

    git checkout master
    git merge bh-merge
    git log

and there we go! merged nicely.
now I don't need bhendersons merge branch anymore, so lets delete it

    git branch -d bh-merge
    git branch -a

git is distributed

Instead of one central server, that everyone has to sync to,
* independent lines of work can go on.
* If someone creates something good in their branch, they just tell people about it.
* permission-less 

you can see why it is so good for open-source development

questions about branching?

# Advanced

* tagging
* rebase
* cherry pick
* git bisect
* hooks
* tracking branches
* submodules
* interactive staging
* squashing commits
* git-svn
* setting up your own server
* patches via email
* gitjour

Share:

So far I’ve tried aleph, compjure/ring/jetty and saturnine.

The compojure stack is by far the cleanest, but it’s geared more
towards synchronous request/response cycles. I’d like to use
something like to EventMachine and saturnine seems the closest to
that goal. However, saturnine’s current HTTP implementation is
lacking.

Now before I could contribute to saturnine I first needed to understand netty. If you’re trying to learn netty, I recommend you first read the users guide and then jump straight to the API docs on ChannelPipeline.

Next, I needed to write a basic HTTP server using netty. This post on StackOverflow and this sample code on netty’s website helped me get a basic HTTP server up an running.

Below is a a nieve translation of the netty sample code into clojure. Note that this is not listed here as an example of sexy clojure code, but rather a starting point for someone looking to get dirty with the netty libraries in clojure.

    (ns xcombinator.netty.server
      (:gen-class)
      (:use clojure.contrib.import-static)
      (:import
         [java.net InetSocketAddress]
         [java.util.concurrent Executors]
         [org.jboss.netty.bootstrap ServerBootstrap]
         [org.jboss.netty.channel Channels ChannelPipelineFactory
           SimpleChannelHandler SimpleChannelUpstreamHandler]
         [org.jboss.netty.channel.socket.nio NioServerSocketChannelFactory]
         [org.jboss.netty.buffer ChannelBuffers]
         [org.jboss.netty.handler.codec.http HttpRequestDecoder
           HttpResponseEncoder DefaultHttpResponse]
    ))

    (import-static org.jboss.netty.handler.codec.http.HttpVersion HTTP_1_1)
    (import-static org.jboss.netty.handler.codec.http.HttpResponseStatus OK)
    (import-static org.jboss.netty.handler.codec.http.HttpHeaders$Names CONTENT_TYPE)

    (declare make-handler)

    (defrecord Server [#^ServerBootstrap bootstrap channel])

    (defn start
      "Start a Netty server. Returns the pipeline."
      [port handler]
      (let [channel-factory (NioServerSocketChannelFactory.
                              (Executors/newCachedThreadPool)
                              (Executors/newCachedThreadPool))
            bootstrap (ServerBootstrap. channel-factory)
            pipeline (.getPipeline bootstrap)]
        (.addLast pipeline "decoder" (new HttpRequestDecoder))
        (.addLast pipeline "encoder" (new HttpResponseEncoder))
        (.addLast pipeline "handler" (make-handler))
        (.setOption bootstrap "child.tcpNoDelay", true)
        (.setOption bootstrap "child.keepAlive", true)
        (new Server bootstrap (.bind bootstrap (InetSocketAddress. port))))) 

    (defn stop-server
      {:doc "Stops a Server instance"
       :arglists '([server])}
      [{bootstrap :bootstrap channel :channel}]
      (do (.unbind channel)
          (.releaseExternalResources bootstrap)))

    (defn http-response
      [status]
      (doto (DefaultHttpResponse. HTTP_1_1 status)
        (.setHeader CONTENT_TYPE "text/plain; charset=UTF-8")
        (.setContent (ChannelBuffers/copiedBuffer
                       (str "Success: " status) "UTF-8"))))

    (defn make-handler
      "Returns a Netty handler."
      []
      (proxy [SimpleChannelUpstreamHandler] []
        (messageReceived [ctx e]
          (let [c (.getChannel e)
                cb (.getMessage e)
                ]
            (println "HTTP request from" c)
            (.write c (http-response OK))
            (-> e .getChannel .close)))

        (exceptionCaught
          [ctx e]
          (let [throwable (.getCause e)]
            (println "@exceptionCaught" throwable))
          (-> e .getChannel .close))))

    (comment
      (def *server* (start 3335 make-handler))
      (stop-server *server*)
    )

Here’s a project.clj that will load up the right dependencies:

    (defproject server "0.0.1"
      :description "Simple Netty HTTP server"
      :repositories [["JBoss" "http://repository.jboss.org/maven2"]]
      :dependencies
        [[org.clojure/clojure "1.2.0-beta1"]
        [org.clojure/clojure-contrib "1.2.0-beta1"]
        [org.jboss.netty/netty "3.2.0.BETA1"]
        [log4j/log4j           "1.2.14"]]
      :dev-dependencies [[autodoc              "0.7.0"]
                         [lein-clojars         "0.5.0-SNAPSHOT"]
                         [lein-run "1.0.0-SNAPSHOT"]
                         [swank-clojure "1.2.1"]]

      :namespaces [xcombinator.netty.server])

Then lein swank and evaluate the (def *server*... line in your
REPL. You’ll have a simple netty HTTP server running on port 3335.

Share:

I was looking for an easy way to replicate ruby’s fileutils library in clojure and I came across a version by Mark McGranaghan in his clj-garden project which wraps around the Apache Commons IO library.

I extended Mark’s library by using multimethods to allow the use of strings (rather than requiring File objects) for parameters.

Usage

user=> (use 'clj-file-utils.core)
nil
user=> (exist "foo.txt")
false
user=> (touch "foo.txt")
nil
user=> (exist "foo.txt")
true
user=> (rm "foo.txt")
nil
user=> (file "foo.txt")
#<File foo.txt>
user=> (.getParent (file "/path/to/foo.txt"))
"/path/to"

As A Dependency

leiningen

[clj-file-utils "0.1.1"]

maven

<dependency>
  <groupId>clj-file-utils</groupId>
  <artifactId>clj-file-utils</artifactId>
  <version>0.1.1</version>
</dependency>

Code

• jashmenn/clj-file-utils on github

Share:

The tf-idf weight (term frequency-inverse document frequency) is a weight often used in information retrieval and text mining. This weight is a statistical measure used to evaluate how important a word is to a document in a collection or corpus. [1]

To calculate tf-idf we need the following four values:

• The number of times a term appears in a given document (n_i,j)
• The total number of terms in a given document (sum k, n_k,j)
• The number of documents that contain a given term (|{d : t_i E d}|)
• The total number of documents in the corpus (D)

Mathematical Details

We want to score the importance of term t_i in document d_j.

Term frequency is defined by:

Where n_i,j is the number of occurrences of term t_i in document d_j.

Inverse document frequency is defined by:

Where:

• D is the total number of documents in the corpus and
• |{d : t_i E d}| is the number of documents in which the term t_i appears.

Then:

Refer to [2] for more information on tf-idf.

Operation Input

Last time we discussed the technique of taking a group of records, calculating a value from that group and emitting each record with the calculated value attached. We called this operation a BufferedSum. We’re going to build on our previous work and create a reusable component (called a SubAssembly) for calculating tf-idf using Cascading on Hadoop.

To make our tf-idf operation we need to decide what the input arguments will be. Last time, we used an input corpus of the format (document_id, body) and emitted (document_id, term, term_count_in_document) for all terms in each document. This last tuple will be the input format to our tf-idf operation.

Creating a SubAssembly

The general format of a SubAssembly in Cascading is as follows:

public class MySubAssembly extends SubAssembly {
  public MySubAssembly(Pipe pipe) {
    // do something with `pipe`
    setTails(pipe); // must register all assembly tails
  }
}

In our operation we are assuming that the total number of documents in the corpus is known or could be found with a simple MapReduce job. Given that we have the total number of documents we take that number as the input to our SubAssembly:

public class TfIdfIndexSubAssmbly extends SubAssembly {
  private Integer totalNumberOfDocuments;
  public TfIdfIndexSubAssmbly(Pipe pipe, Integer totalNumberOfDocuments) {
    this.totalNumberOfDocuments = totalNumberOfDocuments;
    // do something with pipe
    setTails(pipe); // must register all assembly tails
  }
}

Gathering Variables

To compute our final tf-idf score, we first need to compute the intermediate variables.

total_terms_in_document

Given our input is (document_id, term, term_count_in_document) then we already have the first variable n_i,j. We can now calculate the total number of terms in each document:

// input: (document_id, term, term_count_in_document)
// emits: (document_id, term, term_count_in_document, total_terms_in_document)
pipe = new GroupBy(pipe, new Fields("document_id"));
pipe = new Every(pipe,
    new BufferedSum(new Fields("total_terms_in_document"),
                   new Fields("term_count_in_document"),
                   new Fields("document_id", "term", "term_count_in_document")),
    Fields.SWAP);

Remember that BufferedSum takes three arguments:

• The name of the Field to output
• The name of the Field to sum
• The other Fields to “pull through” the operation

So here we are grouping by document_id, and summing term_count_in_document for each group and placing the value into the field total_terms_in_document.

number_of_documents_containing_term

Next we need to calculate the number of documents that contain each term. We’ve already grouped by document_id and term, therefore we know we only have one record for a given document_id/term pair.

Rather than counting the number of document_id/term pairs directly we are simply going to assign a count of 1 to each record and then sum that value. This allows us to reuse the code we’ve written for BufferedSum.

// calculate the number of documents containing each term
// input: (document_id, term, term_count_in_document, total_terms_in_document)
// emit:  (document_id, term, term_count_in_document, total_terms_in_document, number_of_documents_containing_term)
pipe = new Each(pipe, new Insert(new Fields("term_in_doc"), 1), Fields.ALL); // we're going to sum these, easier than creating BufferedCount
pipe = new GroupBy(pipe, new Fields("term"));
pipe = new Every(pipe,
    new BufferedSum(new Fields("number_of_documents_containing_term"),
                   new Fields("term_in_doc"),
                   new Fields("document_id", "term", "term_count_in_document", "total_terms_in_document", "term_in_doc")),
    Fields.SWAP);

Here we group on term and for every term group we calculate the number of documents that contain that term. Note that if you have a very large corpus some groups may become memory constrained as very common words such as “the” have groups containing nearly the entire corpus (it would be a good idea to remove stop-words during pre-processing).

After we’ve calculated the value for number_of_documents_containing_term we don’t need the term_in_doc field any longer. Using Cascading’s Identity operation we can reorder and discard fields.

 pipe = new Each(pipe, //  reorder and rm fields
    new Fields("document_id", "term", "term_count_in_document", "total_terms_in_document", "number_of_documents_containing_term"),
    new Identity());

total_documents

Next we do a hard insert of the number of documents. Again, you can calculate this value with a relatively simple MapReduce job (e.g. use a counter), but here probably not the best place to do it. Because you have to count every document in the corpus, it would be better to calculate the number of documents when you are generating the input documents file.

int D = this.totalNumberOfDocuments;
pipe = new Each(pipe, new Insert(new Fields("total_documents"), D), Fields.ALL);

The Insert operation simply inserts the number of documents into the tuple stream.

Calculating Tf-idf with a Custom Operation

Now that we have all four values we can calculate tf-idf. We are going to create a custom operation to do this. We will perform this operation on each Tuple in the Pipe with the following:

// calculate tf * idf
// input:  (document_id, term, term_count_in_document, total_terms_in_document, number_of_documents_containing_term, total_documents)
// emit:   (document_id, term, term_count_in_document, total_terms_in_document, number_of_documents_containing_term, total_documents, tf, idf, tfidf)
pipe = new Each(pipe, new TfIdfOperation(), Fields.ALL);

To create an operation in Cascading you simply subclass BaseOperation. For example:

private static class MyOperation extends BaseOperation implements Function {
  public MyOperation() {
    super(new Fields("out_field_1", "out_field_2"));
  }
  public void operate(FlowProcess flowProcess, FunctionCall functionCall)
  {
    TupleEntry inputTuple = functionCall.getArguments();
    // take values form inputTuple
    // transform them to make outputTuple
    Tuple outputTuple = new Tuple("a value 1", "a value 2");
    functionCall.getOutputCollector().add(outputTuple);
  }
}

Here are the key things you need to do to create a Cascading operation:

• subclass BaseOperation and implement Function (there are other types of operations)
• call super and declare the names of the Fields this operation will be emitting. See BaseOperation for details
• functionCall.getArguments() returns a TupleEntry containing the input Tuple and input Fields.
• functionCall.getOutputCollector() is the OutputCollector you can use to emit Tuples from this operation.
• call outputCollector.add() to emit a Tuple. You can emit 0..n Tuples.

For our tf-idf operation we want to emit three Fields: tf, idf, and tfidf. We do this with the following code.

// note that we're using a private nested class. This is not required.
private static class TfIdfOperation extends BaseOperation implements Function {
  public TfIdfOperation() { super(new Fields("tf", "idf", "tfidf")); }
  public void operate(FlowProcess flowProcess, FunctionCall functionCall)
  {
    TupleEntry arguments = functionCall.getArguments();
    // tf
    Double termCount  = arguments.getDouble("term_count_in_document");
    Double totalTerms = arguments.getDouble("total_terms_in_document");
    BigDecimal tf = new BigDecimal(termCount / totalTerms);
    // idf
    Double totalDocuments  = arguments.getDouble("total_documents");
    Double td = arguments.getDouble("number_of_documents_containing_term");
    BigDecimal idf = new BigDecimal(Math.log(totalDocuments / (1 + td)));
    // tfidf
    BigDecimal tfidf = tf.multiply(idf);
    functionCall.getOutputCollector().add(
        new Tuple( tf.toPlainString(), idf.toPlainString(), tfidf.toPlainString() ));
  }
}

When we convert, say, a Double to a string then we often get an exponent (e.g. 0.02948E7). The exponent can be cumbersome to work with so we use BigDecimal to convert the number into a string without an exponent using toPlainString().

Building the Index

Now that we have the term, document_id, and tfidf score we can build our index. First we strip out the unnecessary fields and move the term to the front.

// input:  (document_id, term, term_count_in_document, total_terms_in_document, number_of_documents_containing_term, total_documents, tf, idf, tfidf)
// emit:   (term, document_id, tfidf)
pipe = new Each(pipe, // reorder and rm some fields
    new Fields("term", "document_id", "tfidf"),
    new Identity());

In our last step we want to build a single row that tells us “given a term, what documents are most relevant to it”. We format this list of (document_id, score) pairs as a JSON hash (JSONObject). (Formatting our records in this way is called using “stripes”.)

pipe = new GroupBy(pipe, new Fields("term"));
// "stripe" our group e.g.:
// input: (term,
//          (document_id_1, tfidf_1),
//          (document_id_2, tfidf_2),
//          ...)
// emit: (term, {document_id_1:tfidf_1, document_id_2:tfidf_2})
pipe = new Every(pipe,
    new Fields("document_id", "tfidf"),
    new JSONTupleAggregator(new Fields("scores"), "JSONObject"),
    new Fields("term", "scores"));

JSONTupleAggregator is an operation that can be found in the cascading.json project. It takes a group of tuples and emits them as either a JSONArray (nested list) or JSONObject (hash).

Using the Index

To use the index, simply perform a CoGroup on the term in your right-hand-side to the term in our index. See Cascading’s documentation on CoGroup for more information.

Full Code Listing

package com.xcombinator.cascading.pipes;
import com.xcombinator.cascading.operations.*;
import com.xcombinator.cascading.operations.buffers.*;
import cascading.tuple.Fields;
import cascading.pipe.*;
import cascading.operation.regex.RegexSplitter;
import cascading.operation.Identity;
import cascading.operation.text.DateParser;
import cascading.operation.Insert;
import cascading.flow.FlowProcess;
import cascading.tuple.TupleEntry;
import cascading.operation.BaseOperation;
import cascading.operation.Function;
import cascading.operation.FunctionCall;
import cascading.pipe.Each;
import cascading.pipe.Every;
import cascading.pipe.GroupBy;
import cascading.tuple.Tuple;
import cascading.operation.Debug;
import cascading.json.operation.aggregator.*;
import java.math.BigDecimal;
/**
 *
 * The goal of this SubAssembly is to create an index that can be used to find
 * the most relevant document given a term.
 * Required: you need to input the number of total documents.
 * Input: list of (document_id, term, count). e.g. :
 *
 *   (document_id,  term,  2)
 *   (document_id,  term2, 1)
 *   (document_id2, term2, 3)
 *   # etc
 *
 * where `count` is the number of times that term appears in that document
 *
 *
 * Emits: list of (term: {document_id_1:score_1,document_id_2,score_2...})
 *
 * Note that the tuple emitted is all String representation of the decimal
 * numbers. This is to allow easy (and correct) parsing if you write to a file
 * after this.
 *
 * It is assumed you've already done any normalization of the terms such as stemming etc.
 */
public class TfIdfIndexSubAssmbly extends SubAssembly {
  private static class TfIdfOperation extends BaseOperation implements Function {
    public TfIdfOperation() { super(new Fields("tf", "idf", "tfidf")); }
    public void operate(FlowProcess flowProcess, FunctionCall functionCall)
    {
      TupleEntry arguments = functionCall.getArguments();
      // tf
      Double termCount  = arguments.getDouble("term_count_in_document");
      Double totalTerms = arguments.getDouble("total_terms_in_document");
      BigDecimal tf = new BigDecimal(termCount / totalTerms);
      // idf
      Double totalDocuments  = arguments.getDouble("total_documents");
      Double td = arguments.getDouble("number_of_documents_containing_term");
      BigDecimal idf = new BigDecimal(Math.log(totalDocuments / (1 + td)));
      // tfidf
      BigDecimal tfidf = tf.multiply(idf);
      functionCall.getOutputCollector().add(
          new Tuple( tf.toPlainString(), idf.toPlainString(), tfidf.toPlainString() ));
    }
  }
  private Integer totalNumberOfDocuments;
  public TfIdfIndexSubAssmbly(Pipe pipe, Integer totalNumberOfDocuments) {
    this.totalNumberOfDocuments = totalNumberOfDocuments;
    // calculate the total terms in each document. note that the input set is
    // smaller because we've already counted the occurrence of each term in
    // each document
    // input: (document_id, term, term_count_in_document)
    // emits: (document_id, term, term_count_in_document, total_terms_in_document)
    // pipe = new Each(pipe, new Debug(true));
    pipe = new GroupBy(pipe, new Fields("document_id"));
    pipe = new Every(pipe,
        new BufferedSum(new Fields("total_terms_in_document"),
                       new Fields("term_count_in_document"),
                       new Fields("document_id", "term", "term_count_in_document")),
        Fields.SWAP);
    // calculate the number of documents containing each term
    // input: (document_id, term, term_count_in_document, total_terms_in_document)
    // emit:  (document_id, term, term_count_in_document, total_terms_in_document, number_of_documents_containing_term)
    pipe = new Each(pipe, new Insert(new Fields("term_in_doc"), 1), Fields.ALL); // we're going to sum these, easier than creating BufferedCount
    pipe = new GroupBy(pipe, new Fields("term"));
    pipe = new Every(pipe,
        new BufferedSum(new Fields("number_of_documents_containing_term"),
                       new Fields("term_in_doc"),
                       new Fields("document_id", "term", "term_count_in_document", "total_terms_in_document", "term_in_doc")),
        Fields.SWAP);
    pipe = new Each(pipe, //  reorder and rm some fields
        new Fields("document_id", "term", "term_count_in_document", "total_terms_in_document", "number_of_documents_containing_term"),
        new Identity());
    // here we do a hard-insert of the number of documents. again, you
    // could/should calculate this with MR, but this is not the place. It
    // would be better to calculate the number of documents when you are
    // generating the input documents file.
    int D = this.totalNumberOfDocuments;
    pipe = new Each(pipe, new Insert(new Fields("total_documents"), D), Fields.ALL);
    // now calculate tf * idf
    // input:  (document_id, term, term_count_in_document, total_terms_in_document, number_of_documents_containing_term, total_documents)
    // emit:   (document_id, term, term_count_in_document, total_terms_in_document, number_of_documents_containing_term, total_documents, tf, idf, tfidf)
    pipe = new Each(pipe, new TfIdfOperation(), Fields.ALL);
    pipe = new Each(pipe, // reorder and rm some fields
        new Fields("term", "document_id", "tfidf"),
        new Identity());
    pipe = new GroupBy(pipe, new Fields("term"));
    // "stripe" our group e.g.:
    // input: (term,
    //          (document_id_1, tfidf_1),
    //          (document_id_2, tfidf_2),
    //          ...)
    // emit: (term, {document_id_1:tfidf_1, document_id_2:tfidf_2})
    pipe = new Every(pipe,
        new Fields("document_id", "tfidf"),
        new JSONTupleAggregator(new Fields("scores"), "JSONObject"),
        new Fields("term", "scores"));
    // must register all assembly tails
    setTails(pipe);
  }
}

Share:

Its pretty easy once you know how to do it:

• Create a JobConf that defines the parameters for the streaming job
• Send up the hadoop-*-streaming.jar with your cascading job by putting it in your jar
• Send up the scripts (python, in this case) by using the -file option
• Send up any other dependencies, corpora, etc. by using the -file, -cacheFile, or -cacheArchive options (See the Hadoop Streaming page for more details)

Resources

NLTK

To generate the nltkandyaml.mod zip file do the following:

# download nltk and unzip
cd nltk
zip -r nltkandyaml.zip nltk yaml
mv nltkandyaml.zip nltkandyaml.mod

Note that this technique is taken from Cloudera

WordNet

The WordNet zip file needs to be flat. e.g. don’t zip up the files with a subdirectory. You could create this file like so:

# download and unzip the wordnet corpus
cd wordnet
zip -r ../wordnet-flat.zip *

Streaming Script

In python, we’ll be using zipimport.zipimporter to import the nltk libraries from a zip file. In Hadoop 0.20.0, Hadoop didn’t decompress our wordnet-flat.zip file automatically (but we’ve heard reports that it will, but I’m not sure which versions). For us the .zip file was placed in lib relative to the pwd of the script. This allowed us to keep the WordNet corpus as a zip and read it in that format.

wn = WordNetCorpusReader(nltk.data.find('lib/wordnet-flat.zip'))

(In this code we’re not using the python reducer.)

#!/usr/bin/env python 
import os
import re
import sys
import zipimport
 
importer = zipimport.zipimporter('nltkandyaml.mod')
yaml = importer.load_module('yaml')
nltk = importer.load_module('nltk')
punct = re.compile('[^\w\s]+')
 
from nltk.corpus.reader import wordnet
from nltk.corpus.reader import WordNetCorpusReader
 
nltk.data.path += ["."]
wn = WordNetCorpusReader(nltk.data.find('lib/wordnet-flat.zip'))
 
def mapper(args):
  line = sys.stdin.readline();
  try:
    while line:
      line = line.strip()
      word = line
      all_synonyms = []
 
      string_synsets = wn.synsets(word)
 
      for synset in string_synsets:
        synonyms = [lemma.name for lemma in wn.synset(synset.name).lemmas]
        synonyms.pop(0)
        for synonym in synonyms:
          synonym = re.sub("_", " ", synonym)
          all_synonyms.append(synonym) 
 
      print "\t".join([word, ','.join(all_synonyms)])
      line = sys.stdin.readline()
  except "end of file":
    return None
 
# we're not using this, but we could
def reducer(args):
  for line in sys.stdin:
    line = line.strip()
    print line
 
if __name__ == "__main__":
  if sys.argv[1] == "mapper":
    mapper(sys.argv[2:])
  elif sys.argv[1] == "reducer":
    reducer(sys.argv[2:])

Cascading Code

Here’s the bulk of the code that will achieve the effect we want. Like last time, we’re using two intermediate taps as the input and output of the streaming job. Also, we’re just using TextLine files for simplicity. If you don’t want the intermediate files hanging around, look at the comments towards the bottom for some example code on how to remove the files when the job is finished running.

package com.xcombinator.hadoopjobs.cascadingstreamingtest;
 
import cascading.cascade.*;
import cascading.flow.Flow;
import cascading.flow.FlowConnector;
import cascading.flow.MapReduceFlow;
import cascading.operation.aggregator.Count;
import cascading.operation.regex.*;
import cascading.pipe.*;
import cascading.scheme.*;
import cascading.tap.*;
import cascading.tuple.Fields;
import cascading.operation.Identity;
import java.util.Properties;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.conf.Configured;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapred.FileInputFormat;
import org.apache.hadoop.mapred.FileOutputFormat;
import org.apache.hadoop.mapred.JobConf;
import org.apache.hadoop.mapred.TextInputFormat;
import org.apache.hadoop.mapred.TextOutputFormat;
import org.apache.hadoop.mapred.lib.IdentityMapper;
import org.apache.hadoop.mapred.lib.IdentityReducer;
import org.apache.hadoop.util.Tool;
import org.apache.hadoop.util.ToolRunner;
import org.apache.log4j.Logger;
import cascading.operation.Debug;
import org.apache.hadoop.streaming.StreamJob;
import java.io.IOException;
 
/**
 * An example file to use a Hadoop Streaming job in cascading
 */
public class Main extends Configured implements Tool
  {
  private static final Logger LOG = Logger.getLogger( Main.class );
 
  public int run(String[] args)
  {
    JobConf conf = new JobConf(getConf(), this.getClass());
    Properties properties = new Properties();
    FlowConnector.setApplicationJarClass(properties, this.getClass());
 
    CascadeConnector cascadeConnector = new CascadeConnector();
    FlowConnector flowConnector = new FlowConnector(properties);
 
    String inputPath  = args[0];
    String outputPath = args[1];
    String intermediatePath1 = args[1] + "-mr-input";
    String intermediatePath2 = args[1] + "-mr-output";
 
    Scheme textLineScheme = new TextLine();
 
    Tap sourceTap = new Hfs(textLineScheme, inputPath);
    Tap intermediateTap1 = new Hfs(textLineScheme, intermediatePath1);
    Tap intermediateTap2 = new Hfs(textLineScheme, intermediatePath2);
    Tap sinkTap   = new Hfs(textLineScheme, outputPath);
 
    // create our first flow, sink to the intermediateTap
    Pipe wsPipe = new Each("wordsplit", 
        new Fields("line"), 
        new RegexSplitGenerator(new Fields("word"), "\\s+"), 
        new Fields("word"));
 
    Flow parsedLogFlow = flowConnector.connect(sourceTap, intermediateTap1, wsPipe);
 
    // Create a pipe and set our mr job for it 
    Pipe importPipe = new Pipe("mr pipe");
    Flow mrFlow;
 
    try {
      JobConf streamConf = StreamJob.createJob( new String[]{
          "-input", intermediateTap1.getPath().toString(), 
          "-output", intermediateTap2.getPath().toString(),
 
          // straight unix
          // "-mapper", "/bin/cat",
          // "-reducer", "/usr/bin/wc"
 
          // ruby
          // "-mapper", "src/main/ruby/word_count_mapper.rb",
          // "-reducer", "src/main/ruby/word_count_reducer.rb",
          // "-file", "src/main/ruby/word_count_mapper.rb",
          // "-file", "src/main/ruby/word_count_reducer.rb"
 
          // python
          "-mapper", "python synsets.py mapper",
          "-reducer", "org.apache.hadoop.mapred.lib.IdentityReducer",
          "-file", "src/main/python/synsets.py",
          "-file", "resources/nltkandyaml.mod",
          "-file", "resources/lib/wordnet-flat.zip",
          });
      mrFlow = new MapReduceFlow("streaming flow", streamConf, intermediateTap1,
        intermediateTap2, false, true);
    } catch(IOException ioe) {
       ioe.printStackTrace();
       System.exit(1);
       return 1;
    }
 
    // create our third "regular" cascading pipe. this is a bit contrived, but
    // the idea is substitute all 'e's with 'x's. it's just here to show how to
    // take the input of a streaming job back into cascading
    Pipe subPipe = new Pipe("subber");
    subPipe = new Each(subPipe,
        new Fields("line"),
        new RegexReplace(new Fields("linx"), "e", "x", true),
        new Fields("linx"));
    Flow subFlow = flowConnector.connect(intermediateTap2, sinkTap, subPipe);
 
    Cascade cascade = cascadeConnector.connect(parsedLogFlow, mrFlow, subFlow);
    cascade.complete();
 
    // to get rid of the intermediate files you could do this:
    // Path tmp = tap.getPath();
    // FileSystem fs = tmp.getFileSystem(conf);
    // fs.delete(tmp, true);
 
    return 0;
  }
 
  public static void main(String[] args) throws Exception 
  {
    int res = ToolRunner.run(new Configuration(), new Main(), args);
    System.exit(res);
  }
 
  }

Share:

Share:

Testing multi-process erlang gen_servers can be tricky. Typically one relies simply on pattern matching to verify that the response matches what you would expect.

{expected, Response} = gen_server:call(Pid, hi).

As long as the gen_server call hi returns expected as the first element of the tuple, then the tests pass.

The technique is also the same when building client-server code where both client and server are gen_servers. The common case is to simply test one side at a time; test the response of all client calls and then (independently) test the responses of the server calls.

What about asynchronous cast?

gen_server:call is convenient because it is synchronous and returns a value.
gen_server:cast, on the other hand, is asynchronous and always returns the atom ok. This can make casts difficult to test.

gen_server_mock is a library to mock gen_server processes that expect specific, ordered sets of messages. It allows you to unit test gen_servers by verifying they are receiving the expected set of messages.

Example 1

     {ok, Mock} = gen_server_mock:new(),
     gen_server_mock:expect(Mock, call, fun({foo, hi}, _From, _State) -> ok end),
     gen_server_mock:expect_call(Mock, fun({bar, bye}, _From, _State) -> ok end),
 
     ok = gen_server:call(Mock, {foo, hi}),  
     ok = gen_server:call(Mock, {bar, bye}),  
 
     ok = gen_server_mock:assert_expectations(Mock)

This Mock expects two calls: {foo, hi} and {bar, bye}. Since Mock receives both of these messages, assert_expectations does not raise any errors.

What is verified

gen_server_mock:assert_expectations(Mock) verifies that:

1. all expected messages were received
2. no messages were received that were not expected

You can catch the exit by using the following:

     Result = try gen_server_mock:assert_expectations(Mock)
     catch
         exit:Exception -> Exception
     end,
     % etc...

Example 2

     {ok, Mock} = gen_server_mock:new(),
 
     gen_server_mock:expect_call(Mock, fun(one,  _From, _State)            -> ok end),
     gen_server_mock:expect_call(Mock, fun(two,  _From,  State)            -> {ok, State} end),
     gen_server_mock:expect_call(Mock, fun(three, _From,  State)           -> {ok, good, State} end),
     gen_server_mock:expect_call(Mock, fun({echo, Response}, _From, State) -> {ok, Response, State} end),
     gen_server_mock:expect_cast(Mock, fun(fish, State) -> {ok, State} end),
     gen_server_mock:expect_info(Mock, fun(cat,  State) -> {ok, State} end),
 
     ok = gen_server:call(Mock, one),
     ok = gen_server:call(Mock, two),
     good = gen_server:call(Mock, three),
     tree = gen_server:call(Mock, {echo, tree}),
     ok = gen_server:cast(Mock, fish),
     Mock ! cat,
 
     gen_server_mock:assert_expectations(Mock)

Currently three types of messages are supported: call, cast, and info.

The signature of the fun of each expectation is the same as the corresponding
gen_server:handle_*. So the fun for expect_call has the same signature as handle_call: fun(Request, From, State). See man gen_server for more information.

However, the return value of the fun must be one of:

    ok |                  
    {ok, NewState} |
    {ok, ResponseValue, NewState} |

Anything else will be an error. Note that you can change the state of your Mock by returning NewState.

Arbitrary, non-gen_server messages are handled with expect_info, e.g. Mock ! cat fulfills the expect_info in the example above.

References

• Github Repo (Patches readily accepted)
• Work inspired by this post
• Mocks Aren’t Stubs

Share:

My goal was to temporarily revert one or many commits that I had pushed to the remote server.

http://cheat.errtheblog.com/s/git — specifically the “Fix mistakes / Undo” section was helpful.

What I found:

git revert -n <sha>

#run this for each commit you would like to “undo”

(the -n makes it so that you are not actually creating a commit, but staging the reverse of the changes made by your <sha> commit in your index. git status will show you this)

git ci -a # apply your revision
git push #origin to master (these steps effectively created one commit that was the product of reversing all the commits you picked in the git revert -n step)

now your index looks like:

now, lets say, the time has come to reapply your commits. Because you didn’t just do

git reset --hard <commit3>

or something like that, all you have to do is git reset –hard <sha1> which will “undo your undo”

then

git push #origin master

again and you are back to where you were.

Share:

I want desperately to know how to detach a process from one terminal, and re-attach it to another, without using screen from the get-go. The more I read about it, the more I figure that the answer (you can’t) probably has more to do with my lack of understanding of how processes and terminals work. I read a great post here that introduced me to disown -h (careful) and nohup, some really great bash builtins. I thought, ok, lets try it. This is where I got stuck.

I tried

while true; do sleep 10; done &
disown -h %1
exit

The disown builtin handles a problem with background processes: From the bash man page (and also the above blog)

The shell exits by default upon receipt of a SIGHUP

therefore killing your process you ‘thought’ you put in the background, and then logged out to go home for the night. Disown tells the jobspec not to accept a SIGHUP, and the -h switch tells it to remain in the jobs table. I thought, cool, maybe if it stays in the the jobs table, I could also transfer it to another jobs table of another tty. (no, you can’t…)
but now I had a process on my hands that wasn’t attached to a terminal, and would just run forever unless I rebooted.

The while loop itself didn’t have a process ID, which is interesting, and because of the nature of while, the sleep commands PID kept changing, so a normal ps aux | grep slee[p] | awk '{print $2}' | xargs kill -9 wasn’t working. (This post is loosing topic fast, but the slee[p] in the above command was a cool trick I learned so that I didn’t need a grep -v grep in there).

I *did* find that I could use ps to figure out the ppid (parent process ID) and just kill -9 that, but I was also interested in knowing for sure that it wasn’t in charge of doing something else important. A little digging around, and I came across the UNIX utility pstree which of course didn’t come on my mac, but I quickly figured out that it could be installed with

sudo port install pstree

Yesterday, I had done a similar thing with the UNIX command watch, which also nicely installed using port
And, for those who don’t know, the UNIX command watch is a great poller utility, that will display the first screen’s worth of output of any command, and update it on a regular basis.

I used ps | grep to find the ppid of the sleep process, then ran this command:

watch "pstree $PPID"

This was way cool, as every ten seconds, I watched as the PID of sleep (the child process of this bash process I had just found) changed.

Take away:

sudo port install watch
sudo port install pstree

Share: