Tapestry trick for caching autogenerated images

At work I am currently using Tapestry 5.0 to create a portal for the game we are working on "Call of the Kings". This has given rise to a lot of stress, but working with it more and more convinces me that it was a good choice. I should warn though that it is not a flat learning curve - consider it more like a wall that can be climbed if you try very hard. Once climbed it will grant you enormous potential.

The trick I am going to talk tell about it how to make caching for generated images/pages/etc. where you know that the only thing affecting them is the request-parameters (called the context in Tapestry5.0 language, available during "Page Activation").

First of all you must understand that the context for a tapestry page is encoded as "subfolders". Lets say that the context expects 2 integers and a string, then it would look like this:

http://localhost/yourproject.isnice.com/pagenameinlowercase/42/32/thestring

Now this is VERY nice for bookmarking, it is nice for remembering, it is fun when testing etc. It is however very different from what we are used to. Also it leads to the following question:

What if I had a directory structure like this "pagenameinlowercase/42/32/" and inside the top folder had a file called "thestring" what would Tomcat/Jetty/etc. do with the request?

Jup, it would render the static file in preference of calling your onActivate(42, 32, "thestring") on your page called "PageNameInLowerCase". We will come back to this later.

Now these parameters are all of type string to start with so Tapestry uses something called a "Coercer". This you have to inject using IoC into your page - don't ask that is the wall you will climb. Once you have this little thing going it's easy to convert between types (and using IoC you can even add more type coersions for your own types).

Ok, that was a lot of stuff with no real point - so here goes the actual case:

We have a heraldry generator in our game, it takes some base images and combines them into flags/shields etc. Only one player can have any given combination, but since you choose 4 images and we have a lot of them I guess there are a few billion combinations. Now these images are used a lot on the portal to identify players, but only few of the possible combinations are actually used. Therefor we would like to generate them on the fly - but as the promise stated, this is a case where ONLY the request-parameters (Tapestry: context) have influence on the result.

Now, the shields we generate take 4 integers and a size (since we also want to scale them). So a request to our images generator-page looks like this:

http://portal.cotk.net/showheraldryshield/shield/301050/301000/200782/110881/64/76

Now the first parameter "shield" is just to tell it that we are going to need a shield (not a flag or something like that). The next 4 are the actual heraldry components and the last one is the size. Now the trick is that in our component we can easily access these parameters and convert them to integers. It looks something like this:

@OnEvent(value="activate")
public StreamResponse onActivate(Object[] context)
{
  String type = (String)context[0];
  if("shield".equals(type))
  {
    return _heraldryImageProvider.getHeraldryShield(
        _typeCoercer.coerce(context[1], Integer.class), 
        _typeCoercer.coerce(context[2], Integer.class),
        _typeCoercer.coerce(context[3], Integer.class),
        _typeCoercer.coerce(context[4], Integer.class),
        _typeCoercer.coerce(context[5], Integer.class),
        _typeCoercer.coerce(((String)context[6]).replace(".png", ""), Integer.class)
    );
  }
  else
  {
    return null;
  }
}

Basically we convert parameter with indexes 1-6 to integers... But what is that replace ".png" thing at the end?

Yes, that is the trick, since if the request had looked like this:

http://portal.cotk.net/showheraldryshield/shield/301050/301000/200782/110881/64/76.png

The last parameter would not have been "76" but "76.png". It is very smart that I strip off this part, because now when generating a shield that I have not generated before I can just store it as a plain png-image on the server. Next time someone requests this specific heraldry (and they will since I intend to keep it) Tomcat will serve them the stored image.

Conclusion
Just by taking advantage of a parameter (Tapestry: context) encoding we can implement caching just by placing the result at a location that would correspond to the first request. Then Tomcat/Jetty/etc. will serve the cached version. This could be used for HTML/JS/Jpeg/SWF/etc. but in case the result is something a little more "live" than an image you should watch out that you don't give your visitors an easy injection-point on your web server. But in general this falls under the normal "Watch out what you do with arguments given to you by a user"-paranoia that one must have when dealing with a server-application.

Ubuntu: Network Manager will not forget!

For a long time I have had a wired and wireless network shared with the rest of the apartments where I live. This has been all nice - except one of the Wireless APs is just within reach of where I often sit, but far from good enough to be useful. This should not pose a problem, except I told the Network Manager about the AP's ESSID and told it the key to it. Now it insist on switching to this AP every now and then, rendering my connection almost useless.

I have tried to search the internet for a solution but the words "remove", "essid" and "network manager" tend to bring up pages about someone who cannot connect to the internet...

But today, I suddenly found it!!

This is where the magic takes place:

~/.gconf/system/networking/wireless/networks

The path seems obvious - retro spectrum.

Multiple Eclipse Setups

This morning I saw that I now have 4 launch icons for Eclipse on my desktop (that is all I have). And I though I might share with you why.

In a number of projects I have found Eclipse to be the best IDE. It has support for many languages, it is easy to work with and actually also quite easy to extend. Now, when I first tried it many years ago, I did not have the biggest computer and since I develop in a lot of different languages it did not take long before my Eclipse installation was so bloated with plug-ins that it could hardly start.

The lesson learned was that Eclipse should be kept slim - with a minimum of plug-ins. This in turns has made it impossible to develop anything than JAVA in it. I therefor ventured out and decided that my current machine could handle at least the PDT extension also. This was correct it could handle it. But the PDT had a different idea about what F11 (launch last launched in JDT) was used for and this ended up making me very angry.

Now, I knew that it was possible to have multiple workspaces (I have that already to keep my Java projects and websites apart). But I did not know that it was also possible to have separate setups with different sets of Plug-ins.

It took me a while to get it working, and I am properbly not doing it the right way, but it works very well. I now have a full functional Java-IDE, PHP-IDE, CPP-IDE and alternative Java-IDE. I rarely have to mix the languages in one project, and even if I did I could just create a new Eclipse setup and install the mixture of plug-ins needed.

So, the "code" (actually it is just a small shell-script but I guess you get the idea):

/opt/eclipse/eclipse \
        -Xmx1024M \
        -data /home/emanuel/sources/java/workspace/ \
        -configuration "file:///home/emanuel/sources/java/workspace/Eclipse_Configuration"

As you can see, adding the -data makes it easy to start in a preselected workspace. Using -configuration instructs eclipse to use the given directory for settings files. Then all you have to do when installing plug-ins is to place them somewhere "special" for the given setup. The fact that the configuration directory is inside my workspace is just to ease things for me - it could be placed anywhere and be used for several workspaces.

Iterable-mutable-hashtable

For a long time I have been trying to find a data structure for a set of problems that we kept running into at work. What we wanted was a container that was both fast for lookups (hashtable) good for iteration (linked list) - and with the special feature that you could add and remove from the list while iterating it.

The two first are true for most implementations of the java interface Map. But the last requirement is not met in these standard implementations - as they all throw an ConcurrentModificationException. The reason we wanted the ability to concurrently modify the container was that a lot of iterating over children of objects resulted in the children self-destructing, hence removing them-selves from the container we were currently iterating. Now, one could argue that we could just use the iterators "remove()" method to do this, but since it was not for the parent to decide child would have to somehow know about the current iterator... I guess you see the code cluttering up down that train of thoughts.

So, after many ideas that seemed to be working on the paper, but have some flaw once implemented I came up with the LinkedList-Hashtable-mix that this entry is all about.

Basic idea
The idea is really simple: create a class where insertions where made both to a linked list (for iterations) and a hashtable (for lookups). This did not work in practice since the LinkedList did not like concurrent modifications either - but the idea worked, so I just had to create my own implementation of a linked list.


Implementation
Actually implementing the thing took a few hours once the idea was in place. Mainly because the container had to handle serialization, and serialization of a linked list cannot be done in the standard serialization way (due to stack overflow).

Code

import java.io.IOException;
import java.io.Serializable;
import java.util.Collection;
import java.util.Enumeration;
import java.util.Hashtable;
import java.util.Iterator;
import java.util.LinkedHashMap;
import java.util.Set;
import java.util.Vector;
import java.util.Map.Entry;


/**
 * The following is much like the {@link LinkedHashMap} except it allows concurrent
 * modifications.
 * 
 * The order is not guarantied after serialization.
 * 
 * @author Emanuel Greisen
 */
public class ThreadSafeHashtable<K,T> implements Iterable<T>,Serializable
{
 private static final long serialVersionUID = 2131757156391825454L;
 protected Hashtable<K,ThreadSafeHashitem<T>> lookup_table = 
  new Hashtable<K,ThreadSafeHashitem<T>>();
 transient ThreadSafeHashitem<T> first_item;
 transient ThreadSafeHashitem<T> last_item;

 
 private void writeObject(java.io.ObjectOutputStream out) 
  throws IOException
 {
  out.defaultWriteObject();
 }
     
 private void readObject(java.io.ObjectInputStream in) 
  throws IOException, ClassNotFoundException
 {
  in.defaultReadObject();
  
  // OPTIMIZE: postpone this untill we need it. Loads of unserializations 
  //           on the server does not use all tables, hence 
  //           the relinking everying for iteration is not nessesary.
  if(lookup_table != null)
  {
   for(Entry<K,ThreadSafeHashitem<T>> item_entry : lookup_table.entrySet())
   {
    if(first_item == null)
    {
     first_item = last_item = item_entry.getValue();
    }
    else
    {
     last_item.next = item_entry.getValue();
     item_entry.getValue().prev = last_item;
     last_item = item_entry.getValue();
    }
   }
  }
 }
 
 public Iterator<T> iterator()
 {
  return new ThreadSafeHashtableIterator<K,T>(first_item);
 }
 
 /**
  * Put an object in the hashmap, note that if any iterators are open the addition 
  * will not be visible untill these iterators have closed
  * or the object has been serialized/deserialized.
  * @param key
  * @param value
  * @return
  */
 public T putObject(final K key, final T value)
 {
  if(key == null)
   throw new IllegalArgumentException("'key' may not be null.");
  
  ThreadSafeHashitem<T> old = lookup_table.get(key);
  if(old != null)
  {
   T old_val = old.value;
   old.value = value;
   return old_val;
  }
  
  ThreadSafeHashitem<T> new_item = new ThreadSafeHashitem<T>(value);
  if(last_item != null)
  {
   last_item.next = new_item;
   new_item.prev = last_item;
   last_item = new_item;
  }
  else
  {
   first_item = new_item;
   last_item = new_item;
  }
  lookup_table.put(key, new_item);
  return null;
 }
 
 /**
  * Remove all objects in the hashtable.
  */
 public void clear()
 {
  // Nuke all "next"-refs
  ThreadSafeHashitem<T> item = first_item;
  while(item != null)
  {
   ThreadSafeHashitem<T> next_item = item.next;
   item.next = null;
   item.prev = null;
   item = next_item;
  }
  lookup_table.clear();
 }
 
 /**
  * Return true if the hashtable contains a certain key.
  * @param key
  * @return
  */
 public boolean containsKey(K key)
 {
  return lookup_table.containsKey(key);
 }
 
 /**
  * Get the object that is assosiated with the given key.
  * @param key
  * @return
  */
 public T getObject(K key)
 {
  ThreadSafeHashitem<T> item = lookup_table.get(key);
  if(item != null)
  {
   return item.value;
  }
  return null;
 }

 /**
  * The number of objects in the hashtable.
  * @return
  */
 public int size()
 {
  return lookup_table.size();
 }

 public Collection<T> values()
 {
  Vector<T> vals = new Vector<T>();
  for(Entry<K, ThreadSafeHashitem<T>> entry : lookup_table.entrySet())
  {
   vals.add(entry.getValue().value);
  }
  return vals;
 }

 public Enumeration<K> keys()
 {
  return lookup_table.keys();
 }

 public T removeObject(K key)
 {
  if(key == null)
   throw new IllegalArgumentException("'key' may not be null.");
  
  ThreadSafeHashitem<T> item = lookup_table.remove(key);
  if(item != null)
  {
   // Link the previous to the next
   if(item.prev != null)
   {
    item.prev.next = item.next;
   }
   else
   {
    // we were the first (update first_item)
    first_item = item.next;
   }
   
   // Link the next to the prev
   if(item.next != null)
   {
    item.next.prev = item.prev;
   }
   else
   {
    // We were the last, (update last_item)
    last_item = item.prev;
   }
   
   item.removed = true;
   return item.value;
  }
  return null;
 }
 
 /**
  * Note that this method is NOT safe to modify in.
  * TODO: make a copy.
  * @return
  */
 public Set<K> keySet()
 {
  return lookup_table.keySet();
 }

 public void addAll(ThreadSafeHashtable<K, T> table)
 {
  for(K k : table.keySet())
  {
   putObject(k, table.getObject(k));
  }
 }
}

class ThreadSafeHashitem<T> implements Serializable
{
 private static final long serialVersionUID = 1906364262566366088L;
 T value;
 transient ThreadSafeHashitem<T> prev;
 transient ThreadSafeHashitem<T> next;
 transient boolean removed;
 public ThreadSafeHashitem(T value)
 {
  this.value = value;
 }
}

class ThreadSafeHashtableIterator<K,T> implements Iterator<T>
{
 ThreadSafeHashitem<T> item;

 public ThreadSafeHashtableIterator(ThreadSafeHashitem<T> item)
 {
  this.item = item;
 }
 
 public boolean hasNext()
 {
  // Skip all the removed items.
  while(item != null && item.removed)
  {
   item = item.next;
  }
  return item != null;
 }

 public T next()
 {
  T val = item.value;
  item = item.next;
  return val;
 }

 public void remove()
 {
  //NOT SUPPORTED: iterator.remove();
 }
}

Created a blog

Well, I just guess it was time to create my self a blog. Just like everyone else I have something to say every now ans then - and the CMS system on my website is just too much trouble for a comment now and then.

I guess this blog will be - like many others - without any purpose other than to get things out that would otherwise have stayed in...

Hope that some of it will be of interest to you or someone you know.