Monday, November 25, 2013

JIRA Agile Extended Kanban Board

JIRA Agile does not keep the column width as you add more columns resulting on unusable boards when the number of columns reach certain point and your display device is not big enough.

Here are a couple of hacks I have tried so far until JIRA Agile provides a fix for this issue. All credits here for Jose Garcia who helped me tweak the JIRA standard Kanban Board with javascript plus CSS hacks.

TamperMonkey script

This is our preferred option so far. I have built a Chrome Tampermonkey JIRA Extended Rapidboard plugin. A similar plugin could be tested and released for Firefox GreaseMonkey plugin.

Javascript Injector add-on

I started using this at the beginning but it was buggy in chrome for windows (In a MAC it worked kind of OK)
  1. Install Javascript Injector chrome plugin
  2. Set url as http://your.jira.url/secure/RapidBoard.jspa
  3. Paste the snippet in “script”:
    var clone = $("#ghx-column-header-group").clone();
    clone.attr("id", "newHeader").css("background", "#FFF").css("position", "absolute").css("width", "1465px").css("margin-top", "-90px");
    
    $("#ghx-column-header-group").remove();
    $("#ghx-rabid").append(clone);
    $("#ghx-pool").css("width", "1500px");
    $("body").removeClass("ghx-scroll-columns").css("overflow-y", "hidden !important");
    $("#ghx-work").attr("id", "ghx-work1").css("overflow-x", "scroll").css("overflow-y", "hidden").width("2000px");
    
  4. Hit “Inject Now”. You have some options there like unchecking "autorun" or using regex for "url" so most likely you will be able to introduce some more customization for your own needs like when needing something different depending on the specific board.

Other approaches

You can certainly build your own extension. For example it would make sense to have the width for the whole board available for setup as well as turning the script ON/OFF. These features are available via TamperMonkey or GreaseMonkey but I have to agree the interfaces might be a little bit scary for non Javascript programmers.

Friday, November 22, 2013

When the SSL certificate expires one liner

echo | openssl s_client -connect ${host}:${port} 2>/dev/null | openssl x509 -text | grep "Not After"

Wednesday, November 20, 2013

Got OWASP? Tomcat.tomdept vulnerability or bad hardening?

We have known this rule for ages: Do not run services you do not need. Hardening servers is mainstream already, and yet malware gets through those services that should not be running.

Why would someone run the Tomcat "manager" application? It is just one of the first things you should remove when you install your brand new tomcat.

Not doing so will only increase your chances to get compromise with malware like Tomcat.tomdept.

Monday, November 18, 2013

Disk full, beyond resource leaking it could lead to increased business risk

We do our best to identify big files and directories, delete them and so on. But is that enough? We live in a world of abundance and think that pouring more hardware resources is the way to go when we get that "Disk full" error or alike. As a consequence you get developers using better hardware than what a server might have.

This combined with the lack of performance and stress testing ends up hiding important code problems which lead to resource leaking (memory, file system, CPU) and pop up in the servers at a latest phase.

If you constraint resources in developer machines on purpose then you might be able to find some of those problems quicker.

In a developer machine you will see the disk full:

$ ssh dev@desktop.sample.com df -h
Filesystem                           Size  Used Avail Use% Mounted on
/dev/sda1                             34G   32G  4.0K 100% /

Time to use lsof to find out the open files:
$ ssh dev@desktop.sample.com lsof >~/lsof.txt
After a reboot I got back 25% of the file resources:
$ ssh dev@desktop.sample.com df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1        34G   23G  8.9G  73% /

Now it is time to analyze lsof:
$ sort -n -k 7 ~/lsof.txt | tail -1
java       1645        dev  202w      REG                8,1 9086951424     530660 /home/dev/.local/share/Trash/expunged/555119177 (deleted)

The 7th column gives us the size so we sort by its numeric value and get the last record which contains the biggest consumer. It tells us there is a 9GB file which was deleted but it is still use by tomcat (process 1645). Most likely there is a resource leak. 

How can we find it? Stop any automated processes in charge of deleting files and run lsof when you run out of HDD space again. It should tell you exactly which file is that and you should be able to look into your source code for the resource not being closed. In Java 7 try-with-resources should be used, previously we used to use libraries or simply those of us coming from C would be way more careful when operating with resources. In anyway look into your IDE or compiler options that could help identifying not closed streams. Java developers should turn on their warnings in their IDEs and cleanup the classes they touch. If this leak is not picked by compiler or IDE warnings then reach the community to find out why. Probably findbugs could help and if not reach them out, they will be more than happy to help as far as I can tell.

I have found in my years as developer that we get "overwhelmed" by alerts, compiler warnings and many other "inconveniences" and as a result we ignore them all. All this happens until the team faces the challenge from IT arguing that the software they have built is not efficient.

Quality of code is important and as in any other business defines it's mere future. Code quality is about Risk management and such as a developer you should not ignore warnings.

Happy (and responsible) coding!



Friday, November 15, 2013

Apache and Tomcat mod_proxy [warn] Proxy client certificate callback: (sample.com:443) downstream server wanted client certificate but none are configured

This warning was coming up un apache logs:
[Fri Nov 15 16:03:13 2013] [warn] Proxy client certificate callback: (sample.com:443) downstream server wanted client certificate but none are configured

Expired or not currently valid Certificate

The certificate might be expired or it could have been issued for a date in the future. You can check the validity using:
openssl s_client -connect sample.com:443 | openssl x509 -noout -dates
depth=0 /C=Argentina/ST=FL/L=Buenos Aires/O=My Company, LLC/OU=Operations/CN=sample.com/emailAddress=it@sample.com
verify error:num=18:self signed certificate
verify return:1
depth=0 /C=Argentina/ST=FL/L=Buenos Aires/O=My Company, LLC/OU=Operations/CN=sample.com/emailAddress=it@sample.com
verify return:1
notBefore=Jan 24 13:29:12 2012 GMT
notAfter=Jan 21 13:29:12 2024 GMT
Recreating the certificate resolved the issue.

Tomcat miss configuration

The SSL Connector was having the below configuration set to "optional" but when using apache as a reverse proxy for load balancing this configuration is not needed. We should use the default which is "none":
SSLVerifyClient="none"

Wednesday, November 13, 2013

Updating Ubuntu Packages through PPA Ubuntu 12.04 with SVN 1.7 svn status -u svn: The path '.' appears to be part of a Subversion 1.7 or greater working copy. Please upgrade your Subversion client to use this working copy.

Personal Package Archives are not trusted however if you know the pusblisher you could at least manage the risk. Subversion 1.7 is not available for Ubuntu 12.04 so you need to trust the svn PPA if you want to install it in your Ubuntu desktop and avoid:
$ svn status -u
svn: The path '.' appears to be part of a Subversion 1.7 or greater
working copy.  Please upgrade your Subversion client to use this
working copy.
I see a lot of posts encouraging to modify /etc/apt/sources.list. Make sure if you do so you revert the changes after the installation. you should never have to edit manually sources.list. To install a particular package out of a PPA:
 
sudo add-apt-repository -y ppa:svn/ppa
sudo apt-get update
sudo apt-get -y install subversion
Note that this will add some files:
$ ls -al /etc/apt/sources.list.d/
-rw-r--r-- 1 root root  238 Dec  6 14:29 svn-ppa-saucy.list
To remove the ppa repos:
 
sudo add-apt-repository --remove -y ppa:svn/ppa
You will notice the list files has now size=0
 
$ ls -al /etc/apt/sources.list.d/
-rw-r--r-- 1 root root  0 Dec  6 14:33 svn-ppa-saucy.list

A Java Runtime Environment must be available in order to run ... No Java Virtual Machine was found in the path

In some Ubuntu Desktops (while not in all) I have found the below error when trying to run Eclipse or Talend from gnome shortcuts:
A Java Runtime Environment must be available in order to run ... No Java Virtual Machine was found in the path ...
I solved it adding the -vm option to the command, for example:
${path-to-talend-binary} -vm ${path-to-jdk}/bin
Why gnome does not find the path when it is included in the profile of the user? Running "${path-to-talend-binary}" directly from the path does work.

Saturday, November 09, 2013

Find Talend Components in use from command line

Here is how to find out a list of components for a particular Talend job:

JVM OutOfMemory troubleshooting - Eclipse MAT to rescue Talend jobs

I will illustrate with an example how to use Eclipse MAT to debug OutOfMemory issues in standalone Talend jobs.

Talend Open Studio for Data Integration generates a shell file that packages a java command. When that shell script returns OutOfmemory errors we need to proced exactly the same way we would troubleshoot OutOfmemory errors in any JVM running process. We need to generate a Heap Memory Dump file (*.hprof) and analyze it with a tool to find out if we are holding more objects in memory than actually needed.

The first thing we need to do is to narrow down the OutOfMemory to a command line:
job/job/job_run.sh --context_param ...
Then we need to ddd the necessary flags to the shell script to get the heap dump file:
java -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath="/tmp/dumps"
Now we run the script again and we notice the message:
java.lang.OutOfMemoryError: Java heap space
Dumping heap to /tmp/dumps/java_pid5394.hprof ...
Note that HeapDumpPath is actually not a directory but the generated file when running JDK 8 so you will see a different message indicating that /tmp/dumps is actually your hprof file.
java.lang.OutOfMemoryError: Java heap space
Dumping heap to /tmp/dumps ...
You will need to append the extension in order to load it in Eclipse MAT or simply use the file name directly from the JVM flag:
java -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath="/tmp/dump.hprof"
Our file has been generated so it is time to load the Heap Memory Dump file (*.hprof) into Eclipse Memory Analyzer (MAT)

Just go to "Menu|File|Open File". Once loaded select "Leak Suspects Report". The pie chart should identify major problem suspect(s) and scrolling down that page you can drill down:



Click in "Details" for each problem suspect, for example:



Look how in the case of Talend the whole consumption is practically happening in the main() method. Talend just produces a huge Java class with a main method. Drill into "Accumulated Objects by Class" available towards the bottom of the page:



As you can see dom4j is used for parsing most likely big XML content (instead of SAX for example). Clicking on the link for objects you will be able to navigate through their herarchy. With Object Query Language (OQL) you can literally inspect anything in the hierarchy. Locate the OQL icon and click on it and get ready to type queries. The result will be a similar hierarchy than the one you get when inspecting individual objects if you just "select * from" the object but you can drill down getting hints about the actual loaded data with OQL. All you need is to look into fields of the object for which you can use the hirarchy inspection or even Javadocs directly:



The solution is most likely to use a different configuration for the faulty component or in the case of a deficiency of it look for an alternative. BTW if your heap is too high you can always limit memory consumption to minimize the size of the hprof file. Most likely with smaller memory footprint the memory leaking will be still revealed by an excesive usage of certain classes of objects.

Profile the application before getting surprising OutOfMemory

You can generate heap dumps from running programs at any time. You just need the pid of the running process, for example:
$ jps
2017 Bootstrap
13667 Jps
13650 talend_sample
$ jmap -dump:format=b,file=/tmp/talend_sample.hprof 13650
Dumping heap to /tmp/talend_sample.hprof  ...
Heap dump file created

Kanban: Show stale issues to reach Kaizen Moments. Show issues which status have not changed for some time with JIRA Agile

Agile Software Development needs to be done as simple as possible. Lean thinking is about that, it is about making sure the SDLC gets to a stage where it cannot be simpler. This is what the philosophy about continuous improvement is about.

High quality production starts with great culture supported by lean processes in place.

Bottlenecks have to be identified and if the culture is right then team will speak out issues in the daily meeting. However using a tool to remind those who forget about the issues they are having would be great and that is exactly what JIRA allows. You can mark issues that have been stale for 2 days for example using the below JQL for "Card Color"
NOT (Status changed AFTER -2d)
Of course identifying the bottleneck is a great part of the equation but resolving it is the most important part of it. Project Managers, Coachers, Coordinators etc should not be spending time chasing what people are doing or where they are stuck. They should be spending time removing roadblocks and eliminating bottlenecks. A tool can be helpful at identifying but intelligence (only found in human brains at the time of this writing) is needed to resolve the challenges we face.

Be happy when you find out issues, be worry when you find none. If you hear from the team they have no issues but you see there are stale items in the board then clearly you have reached a Kaizen moment. Without bottlenecks there will be no process improvement.

Wednesday, November 06, 2013

Share your Google Drawing via SVG with plain HTML

SVG has been there for a while and if you don't care about unsupported old and most likely vulnerable browsers then you should be fine just downloading as SVG your Google Drawings hosted in Google Drive and then pasting the content of them (plain XML) inside any web page like for example your wiki. You can use the SVG as external image or inline (as I show below).

It will not be difficult to get your Google Drawings and build presentations out of them. And yes presentations which lose no quality in bigger screens and which resize nicely depending on the existing canvas.

Mediawiki

In the case of Mediawiki you need the below in LocalSettings.php to be able to use html directly in pages:
$wgRawHtml = true;
Now you can include any html code like the <svg> tag inside a literal <html> tag like in:
<html><?xml version="1.0" standalone="yes"?>

<svg version="1.1" viewBox="0.0 0.0 960.0 720.0" fill="none" stroke="none" stroke-linecap="square" stroke-miterlimit="10" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink"><clipPath id="p.0"><path d="m0 0l960.0 0l0 720.0l-960.0 0l0 -720.0z" clip-rule="nonzero"></path></clipPath><g clip-path="url(#p.0)"><path fill="#000000" fill-opacity="0.0" d="m0 0l960.0 0l0 720.0l-960.0 0z" fill-rule="nonzero"></path><path fill="#cfe2f3" d="m165.11215 69.882095l0 0c-1.0062714 -7.7441444 2.2975006 -15.410381 8.509399 -19.74564c6.211899 -4.3352585 14.242477 -4.5792274 20.684082 -0.62838364l0 0c2.2818146 -4.5027695 6.458191 -7.611622 11.265884 -8.3861885c4.807678 -0.77456665 9.681976 0.87612915 13.148499 4.4527817l0 0c1.9438171 -4.0825005 5.7605133 -6.8254166 10.095764 -7.2554245c4.3352356 -0.43000793 8.575455 1.5137558 11.216019 5.1415367l0 0c3.5117798 -4.3274384 9.099106 -6.1494713 14.344315 -4.6776924c5.245224 1.4717789 9.206223 5.9730225 10.169052 11.556019l0 0c4.3025208 1.2290115 7.886444 4.353321 9.825775 8.565704c1.939331 4.212387 2.0438538 9.099625 0.28652954 13.399002l0 0c4.236725 5.774544 5.2278137 13.469048 2.6033936 20.212059c-2.6244507 6.743019 -8.470093 11.521484 -15.355469 12.552162c-0.04852295 6.3285675 -3.3627625 12.135574 -8.665298 15.182762c-5.3025208 3.0471878 -11.76532 2.8587189 -16.897324 -0.4927597c-2.1859589 7.579544 -8.338715 13.156502 -15.800034 14.321419c-7.461334 1.164917 -14.893646 -2.291031 -19.085907 -8.8747635c-5.1388397 3.24514 -11.305008 4.17997 -17.10756 2.5935898c-5.8025513 -1.5863724 -10.752701 -5.560318 -13.733795 -11.025391l0 0c-5.251251 0.6435318 -10.328476 -2.2056427 -12.711884 -7.133484c-2.3833923 -4.927841 -1.5655975 -10.885323 2.0475159 -14.915779l0 0c-4.68425 -2.8872223 -7.0744324 -8.616432 -5.924164 -14.200073c1.1502686 -5.583641 5.5803223 -9.756432 10.980072 -10.34243z" fill-rule="nonzero"></path><path stroke="#000000" stroke-width="2.0" stroke-linejoin="round" stroke-linecap="butt" d="m165.11215 69.882095l0 0c-1.0062714 -7.7441444 2.2975006 -15.410381 8.509399 -19.74564c6.211899 -4.3352585 14.242477 -4.5792274 20.684082 -0.62838364l0 0c2.2818146 -4.5027695 6.458191 -7.611622 11.265884 -8.3861885c4.807678 -0.77456665 9.681976 0.87612915 13.148499 4.4527817l0 0c1.9438171 -4.0825005 5.7605133 -6.8254166 10.095764 -7.2554245c4.3352356 -0.43000793 8.575455 1.5137558 11.216019 5.1415367l0 0c3.5117798 -4.3274384 9.099106 -6.1494713 14.344315 -4.6776924c5.245224 1.4717789 9.206223 5.9730225 10.169052 11.556019l0 0c4.3025208 1.2290115 7.886444 4.353321 9.825775 8.565704c1.939331 4.212387 2.0438538 9.099625 0.28652954 13.399002l0 0c4.236725 5.774544 5.2278137 13.469048 2.6033936 20.212059c-2.6244507 6.743019 -8.470093 11.521484 -15.355469 12.552162c-0.04852295 6.3285675 -3.3627625 12.135574 -8.665298 15.182762c-5.3025208 3.0471878 -11.76532 2.8587189 -16.897324 -0.4927597c-2.1859589 7.579544 -8.338715 13.156502 -15.800034 14.321419c-7.461334 1.164917 -14.893646 -2.291031 -19.085907 -8.8747635c-5.1388397 3.24514 -11.305008 4.17997 -17.10756 2.5935898c-5.8025513 -1.5863724 -10.752701 -5.560318 -13.733795 -11.025391l0 0c-5.251251 0.6435318 -10.328476 -2.2056427 -12.711884 -7.133484c-2.3833923 -4.927841 -1.5655975 -10.885323 2.0475159 -14.915779l0 0c-4.68425 -2.8872223 -7.0744324 -8.616432 -5.924164 -14.200073c1.1502686 -5.583641 5.5803223 -9.756432 10.980072 -10.34243z" fill-rule="nonzero"></path><path fill="#000000" fill-opacity="0.0" d="m159.95244 94.723076l0 0c2.2105103 1.3624878 4.7641754 1.980545 7.3181 1.7711868m3.345108 20.278572c1.0982971 -0.13459778 2.17482 -0.41960907 3.201828 -0.847702m27.638 9.279144c-0.77246094 -1.2131119 -1.4192047 -2.50943 -1.9292145 -3.866867m36.816498 -1.5800171l0 0c0.39852905 -1.3818665 0.6567383 -2.8041 0.77033997 -4.242981m24.79132 -10.446487c0.05166626 -6.737732 -3.6026306 -12.90686 -9.39328 -15.857582m22.145233 -16.90593c-0.93777466 2.2943268 -2.3694153 4.32959 -4.182617 5.946213m-5.9288025 -27.911522l0 0c0.15975952 0.9265022 0.23373413 1.8669281 0.2208252 2.8082428m-24.733795 -9.686291l0 0c-0.87602234 1.079483 -1.5977478 2.2857895 -2.1427002 3.5813599m-19.169281 -1.4679413l0 0c-0.46684265 0.980484 -0.81544495 2.0180054 -1.037796 3.0886612m-23.376953 0.84482574l0 0c1.3630524 0.8359947 2.6240387 1.842205 3.7552948 2.9965286m-32.948395 17.377605l0 0c0.13868713 1.0673447 0.3578186 2.1215286 0.655365 3.1526947" fill-rule="nonzero"></path><path stroke="#000000" stroke-width="2.0" stroke-linejoin="round" stroke-linecap="butt" d="m159.95244 94.723076l0 0c2.2105103 1.3624878 4.7641754 1.980545 7.3181 1.7711868m3.345108 20.278572c1.0982971 -0.13459778 2.17482 -0.41960907 3.201828 -0.847702m27.638 9.279144c-0.77246094 -1.2131119 -1.4192047 -2.50943 -1.9292145 -3.866867m36.816498 -1.5800171l0 0c0.39852905 -1.3818665 0.6567383 -2.8041 0.77033997 -4.242981m24.79132 -10.446487c0.05166626 -6.737732 -3.6026306 -12.90686 -9.39328 -15.857582m22.145233 -16.90593c-0.93777466 2.2943268 -2.3694153 4.32959 -4.182617 5.946213m-5.9288025 -27.911522l0 0c0.15975952 0.9265022 0.23373413 1.8669281 0.2208252 2.8082428m-24.733795 -9.686291l0 0c-0.87602234 1.079483 -1.5977478 2.2857895 -2.1427002 3.5813599m-19.169281 -1.4679413l0 0c-0.46684265 0.980484 -0.81544495 2.0180054 -1.037796 3.0886612m-23.376953 0.84482574l0 0c1.3630524 0.8359947 2.6240387 1.842205 3.7552948 2.9965286m-32.948395 17.377605l0 0c0.13868713 1.0673447 0.3578186 2.1215286 0.655365 3.1526947" fill-rule="nonzero"></path></g></svg></html>

Blogs

Here is a Cloud SVG from Google Drawing which has been included using the inline svg tag above:

Friday, November 01, 2013

OSX Mavericks still without RSS Reader integrated in Mail.app

This is one of those simple things Mail.app still does not support (after Lion). I am unsure why it has not been included but if you use Mail with Exchange then you will be able to read RSS with the below *only* if you have outlook client side constantly pulling the RSS which makes this method not practical really unless Microsoft would support pulling the RSS from server side:
  1. Add the feed via Outlook from a Windows machine. Check the Account Settings in Outlook looking for the RSS Feeds tab and "change" your feed to make sure it has the correct "update limit". In my case for Mediawiki I found that the default stopped the feed from being refreshed so I unchecked the "Use the publisher update recommendation". I had to "Send/Receive tab | Send & Receive group | Send/Receive | Define Send/Receive Groups | Group Name | All Accounts | Schedule an automatic send/receive every n minutes check box | When Outlook is Offline | Schedule an automatic send/receive every n minutes"
  2. See the feed inside the "RSS Feeds" folder in Mail.app

Test connectivity through netcat

You want to test that networking is properly configured before you actually run eny services. For example you are migrating from mod_jk to mod_proxy so your DMZ should be able to connect to tomcat on port 8443 (HTTPS) instead of 8009 (AJP). You cannot make any changes so far but it is Friday and you want to make sure the routes through DMZ firewalls are working correctly. Just put a listener in the tomcat server and use a client to connect to it from Apache. So in tomcat:
$ nc -l 8443
In Apache use a client:
$ nc tomcatServerDomainOrIp 8443
Hello
And you should get what you type (Hello) in the server side if the connectivity is correct. This is of course testing TCP traffic on port 8443.

Followers