Gremlin’s Fireside Chat

Bit of an exciting week for me. Last Friday a long contract finally came to an end and since I had six weeks notice of it I had signed up for the “Open Data 18 hour Challenge” on 4th and 5th July which from the blurb on their website was about local government data sets which have recently started to be released in Ireland. I was expecting something like the tech conferences I have attended in the past – some presentations, discussions maybe the odd coding and brain storming sessions; But boy was I wrong. If I had known what it was to be about before hand I would probably never have signed up for the challenge, but I am SO GLAD that I did. In short the challenge was to generate business idea’s that used open data and maybe in the future take them forward to a product and a new High-Tech business.

The challenge was run over two days when on the first day was almost a pure brainstorming session where you as part of a table of techies, academics, entrepreneurs you came up with as many idea’s as you could on using the open data sets (Fingal County Council datasets and Dublin dataset ) Each table had to create two or three posters from the best ideas they could come up with – and there were loads. Once the posters were created and everyone had a look at them there was a short presentation on each one of them after which everyone voted on which ones were the best idea’s. Once all the votes were cast the idea’s with the least votes were eliminated and then we were asked to pin our names to the projects we wanted to do the next day – I chose one called “BizFit”. We were then told at the end of the next each of the ten teams had to do a five minute “Dragon’s den” style presentation to a panel of judges who would make the the final choices.

The BizFit idea is conceptually a simple one in that the idea is to use demographic data to either allow a small-to-medium company to find the best location for their business or for someone like an estate agent find the best kind of company for a particular empty business unit. With a great team of six people we had to develop the idea, discover the competitors and work out revenue streams etc for the final slide based presentation.

Anyway to cut a long story short our BizFit team beat off stiff competition to win the £4000 euro prize. Needless to say our team were stunned at our success.

Not too sure what we are going to do next. There is talk about entering the launchpad process to see if we can take the idea further. I hope we do since its an interesting business idea.

If you want more information on the challenge and the results from the Fingal Open Data Site, New Tech Post, Silicon Republic and Digital Times

I learn’t a lot on the process of business idea generation and what is needed to initiate the process of turning an idea into something more concrete. The “buzz” in the air on both days was palpable and I got to meet a lot of great people whom I hope to maintain contact with.

I am working on a project, and am spending a fair bit of time trying to get the Java Virtual Machine to perform better. This is very much a black art and in reality there is not much out there in the ether to look at. I did however find this excellent page on the JVM Parameters you can use which provides much more information that “java -help” which only lists the basics.
The only downside is I would like to more information on what a load of the parameters do…. Anyway hope you find this useful.

If like me you are a java developer on unix this little blog might be of some use if you have lots of jars and are prone to “which jar contains the class X” where X could be riomhaire.roster.model.User and you find your self in a long search. What I do is ….

Have a file in my home directory called “classlist.txt” which contains the output of “jar -t” after some text replacements to remove non classes and to replace “/” with “.” so we can get the class names correct. Each jar dump includes the jar name and location.

The script is fairly simple being :

#####################################################################
echo “##### ${1} ####”
jar -tf ${1} | sed ’s/\//./g’ | grep “\.class$”
echo “”
#####################################################################
and which I call ‘listJarClasses’

I invoke the script via the shell command line:

find / -name “*\.jar” -exec listJarClasses {} >>classLocations.txt \;

at the end I end up with a file which can look like:

##### ./ant/etc/ant-bootstrap.jar ####
org.apache.tools.ant.Main.class

##### ./ant/lib/ant-antlr.jar ####
org.apache.tools.ant.taskdefs.optional.ANTLR.class

##### ./ant/lib/ant-apache-bcel.jar ####
org.apache.tools.ant.filters.util.JavaClassHelper.class
org.apache.tools.ant.util.depend.bcel.AncestorAnalyzer.class
org.apache.tools.ant.util.depend.bcel.DependencyVisitor.class
org.apache.tools.ant.util.depend.bcel.FullAnalyzer.class

……

Then if I want to find a class I open the file and search for the class name I am looking for and look for the preceeding jar name.
Hope this is of some use.

In my last piece about instrumentation I (hopefully) showed the importance of instrumenting your code – in many ways it is one of the major “unwritten� requirements in any project – along with a good logging strategy. The tough question is what do you instrument and what are the options?

As with most things in development there are many options and variants – you just need to choose what feels right for your project. A passionate and professional developer after a few years in the minefield of software development gets to know what will and more importantly what will not work in a situations.

Ninety percent of software development involves something (user/software) invoking a method or command on some target object. Some of the things you can track from that is:

1. How long the invocation took.

2. The result of the invocation (success, error etc).

3. The average time invocation of this method/command.

4. The maximum time of any invocation.

5. The minimum time of any invocation.

6. The invocation/min/max/average times for 1-5 for the last n commands.

7. Values 1-5 but broken down by success/error etc – can help if “success� times take a lot longer than the errors.

8. How much memory was used.

9. How many times method/command has run.

10. Tallies on errors.

11. Deviation counters – How many times <1 second, <10 second, <100 sec, <1000 sec and so on. For All/success/error etc.

12. Tallies by hour, day, week, month etc.

13. Throughput calculation by the second, minute,hour,day depending on the domain.

14. Error rates by second, minute,hour,day depending on the domain.

15. Drop rates – in some systems requests can time out. You might need to track these.

16. Uptime.

17. Load – number of concurrent invocations etc.

18. Memory usage (very very rough in Java).

19. Data transfer – if you can calculate request and response sizes.

This list is by no means complete, and if you have any suggestions then drop me a line at gary(at)garyleeson.com.

You can also from the above work out what invocations were executing concurrently which can help in figuring out the hog processes when certain “odd� things occur that depend on when and what things are running; For example when two processes use a lock to access some resource. When run individually everything runs fine. When they run concurrently one gets the lock and the other has to wait; if the process that has to wait has a hard time constraint then you might have issues.

Another thing to do is make sure that all this is kept as an in-memory database for performance reasons – using a traditional DB backend such as postgres could have quite an impact; this does not prevent you from building in a mechanism to flush to backing store during quiet periods or every 5 minutes or so.