Thursday, December 15, 2011

Two hours with Rasmus Lerdorf, the creator of PHP

I had the pleasure to meet Rasmus Lerdorf, the creator of PHP in a two hour session organized by South Florida PHP Users Group in Nova South Eastern University.

As one of the earliest PHP adopters I share a lot of feelings for the language and it was great to see so many young people in the room together with not so young like me ;-) and some older than me. It is great to see a language going on for so long. I would say it was a celebration of more than 15 years of PHP coding.

Even though I have been working more with python, ruby and Java for the last 5 years PHP always comes back: I need to patch a built in PHP software, configure it or even get a quick snippet and modify it to get my work done.

So with a lot of quotations here is an extract of the introductory ideas this MIT awarded "top 100 innovators in the world" shared with us.

The idea behind PHP was always "the need for speed and performance". Rasmus "did not worry about correctness but about resolving the problem". He is a guy that "does not love to program" and that is precisely why you want to use PHP because instead he actually "loves to solve problems".

"You have to add non scalable features in order to make php not scalable". The "documentation was always done before the function was implemented". He constantly "insisted in having lot of examples as part of the documentation"

PHP was born while "thinking in the ecosystem". Basically shared hosting demanded control on memory and CPU (QoS) to adequate existing servers to the needs of multiple applications.

PHP "runs crappy code extremely fast" so it is better for startups than other languages like Java. It simply scales well.

I asked the question about why he considered Java not scalable and he clarified:
-It does scale but it comes out of the box with features that allow the requests to be sticked in memory, a thing php by default does not allow. Also when it comes to go beyond one server it is then more difficult in java just because of this.

Of course one can argue sticky sessions are enough for most projects. Yes the user session will expire and then you will need to re login but things like remember-me will help (not that I like it really). Not big deal considering how many times this actually happens, I would say nobody complained to me on the clusters I have setup for which in really weird circumstances a shared clustered session was used.

He described some features that make PHP even faster today:
  1. libevent allows event driven programming ala nodejs.
  2. zeromq for better and simplest socket programming. I have to add that messaging is in fact something I would say will gain more and more momentum within the software community. The world is asynchronous as the creator of Erlang language said said.
  3. FastCGI Process Manager (FPM) allows dividing the workload while sending jobs to workers. This is the PHP way to say no to threads.

Rasmus talked about performance and brought some examples providing some guidance to improve it:
  1. Use wisely your datastores. Do not try to replace sql with nosql, it simply does not work that way.
  2. Turn off logging in production (error_reporting(-1) is expensive).
  3. Use strace: Look for ENOENT, excessive stats (lstat), check your realpath_cache_size.
  4. Use a profiler: callgrind, xdebug, xhprof, xhgui.
  5. Set default timezone (look at phpinfo).
  6. Watch for inclusions. Too many inclusions degrade performance. An example of a bad application is Magento which includes thousand pf files.
  7. HipHop-PHP can be used to do static analysis. They need to parse php better than the php parser itself because their goal is to translate PHP to native code. Be prepared to wait for the compiler but this is a good exercise to find problems in your code.

Here are some architecture reminders/suggestions from Rasmus:
  1. Use different domain/subdomain for static assets.
  2. Keep cookies short (MTU size maters and will translate in more roundtrips when it is overflown).
  3. Multiply by 5 times the amount of cores and that is the amount of real concurrent users. I have to say I have been calculating allowed concurrent users for years and the metric about just the cores is not that simple. It even depends on what the application is using of course so a word of advice from my end would be take a look at your system metrics for example with vmstat, top and other system commands. The use of automated stress tests driven by for example jmeter are a must do in my opinion.
  4. Use out-of-band processing with Gearman, PHP-FPM or custom via ZeroMQ.
  5. Tweak ORM and caching.

Finally he talked about the PHP 5.3 features, how PHP 6 development was stopped, the problems inherent to support Unicode and the efforts that are still being made in the current version 5 to support little by little more Unicode functionality. Here are some of those new features:
  1. Better performance through lot of code optimization.
  2. Support for closures.
  3. Namespaces. There is a big debate around the decision to use a backslash instead of a dot for the namespaces. To be honest I do feel that as awkward but as Rasmus said "we will need to get used to it".
  4. Late Static Binding support.
  5. Garbage Colector which you should not need in web apps but if you have long running scripts then ii will definitely help.
  6. NEWDOC which is like HEREDOC but does not perform any parsing inside the string block.
  7. Remember "Go To Statement Considered Harmful", the famous letter from Edsger Dijkstra? The controversy is still on as PHP allows "goto" to eliminate verbosity or the use of break. In my opinion the discussion about the name is secondary. You can use break or continue to a a label in Java for example so if a goto has just the meaning of breaking the loop to a variable I think is awkward but but I am fine with it. However allowing goto to go to any part of the code block is not precisely a multi level break or continue, it is something more than that. If that is harmful, confusing, miss leading or not I leave it to the discussion. Perhaps just adjusting the multi level sysntax from PHP to accept a letter instead of a "magic" number would be a better approach.
  8. DateInterval/DatePeriod classes.
  9. date_create_from_format
  10. FastCGI Process manager (FPM)
  11. For people moving to ngnix from apache .php_ini allows for custom php directives just like .htaccess for Apache.
  12. Traits: Compiler assisted copy and paste to resolve the lack of multiple inheritance.

Thank you Rasmus for PHP. Let's celebrate how lucky we are that not a single language could ever win all battles. Different languages exist to resolve similar problems unders different scenarios. This is not any different than we, human beings. We need different skills in a team as much as we need different languages in modern computing.

and different human beings exist to reso

Workspace defines a VM that does not contain a valid jre/lib/rt.jar in Snow Leopard OSX

I was getting this error today after trying to use "mvn eclipse:eclipse" to generate Eclipse project files out of a maven project:
[WARNING] Workspace defines a VM that does not contain a valid jre/lib/rt.jar: /System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home

There is a proposed patch for this bug but at the time of this writing I could not figure out a better way than running the below commands:
cd /System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home
sudo mkdir -p jre/lib
cd jre/lib
sudo ln -s ../../../Classes/classes.jar rt.jar

Remove context from url from Tomcat applications

If you are like me you love tomcat simplicity and you are used to deploy your WAR files or exploded directories in webapps folder directly. However in order to serve URLs without the context included you will need to stop that practice.

You might come with a hack like I discover a while back, but as you can read there that will result in the same application being deployed several times.

Here is how to configure tomcat (Tested in tomcat 7) to serve content from two different URLs
  1. Make your websites resolve to real IPs. For production you rely on external DNS, for other environments you rely on internal DNS and many times in you /etc/hosts:
    $ vi /etc/hosts
    ...
    127.0.0.1   bhubdev.nestorurquiza.com
    127.0.0.1   bhubdev2.nestorurquiza.com
    ...
    
  2. You need to add them both to Engine section in conf/server.xml:
    ...
     <host name="bhubdev.nestorurquiza.com"  appBase="bhub-app"
        unpackWARs="true" autoDeploy="true"
        deployOnStartup="true" />
      <host name="bhubdev2.nestorurquiza.com"  appBase="bhub2-app"
        unpackWARs="true" autoDeploy="true"
        deployOnStartup="true" />
    
    
  3. Create these folders (two per domain as you can see). Be sure to change to proper paths in your system:
    $ mkdir /Users/nestor/Downloads/apache-tomcat-7.0.22/bhub-app
    $ mkdir /Users/nestor/Downloads/apache-tomcat-7.0.22/conf/Catalina/bhubdev.nestorurquiza.com
    $ mkdir /Users/nestor/Downloads/apache-tomcat-7.0.22/bhub2-app
    $ mkdir /Users/nestor/Downloads/apache-tomcat-7.0.22/conf/Catalina/bhub2dev.nestorurquiza.com
    
  4. Restart tomcat.
  5. Deploy the WAR files as:
    /Users/nestor/Downloads/apache-tomcat-7.0.22/bhub-app/ROOT.war
    /Users/nestor/Downloads/apache-tomcat-7.0.22/bhub2-app/ROOT.war
    
  6. Alternatively deploy the exploded WAR file as:
    unzip bhub-app.war -d /Users/nestor/Downloads/apache-tomcat-7.0.22/bhub-app/ROOT/
    unzip bhub-app2.war -d /Users/nestor/Downloads/apache-tomcat-7.0.22/bhub2-app/ROOT/
    

Now if you use maven and you want to deploy from there you probably have something like the below if you are still deploying to webapp folder (using ant tasks to deploy either the WAR file or the exploded directory)
...

    ${MVN_TOMCAT_HOME_DEPLOY}
...

    
    
        
    

...

    Deploying WAR locally
    

...

You will need to update that to point to the new directories and not to webapps anymore.
...

  ${CATALINA_HOME}
...

    
    
        
    

...

    
    

...

Do not forget to add to your profile the needed for Maven environment variable:
$ vi ~/.profile:
#old needed variable
#export MVN_TOMCAT_HOME_DEPLOY=/opt/tomcat/webapps
#new needed variable
export CATALINA_HOME=/opt/tomcat
$ source ~/.profile

Wednesday, December 14, 2011

Monitor Event Log in Windows 2008

I already described how we can monitor event logs from windows but that procedure will not work for Windows 2008 and Windows 7 because eventtriggers.exe has been deprecated.

Here is a new script you will need to use in Windows 7/2008 together with "Windows Task Scheduler". The comments on the top of the script should be straightforward to understand how to get an email alert every time an application ERROR event is registered in the Event logs.

'''''''''''''''''''''''''''''''''''''''''''''''
'
' c:\scripts\events\sendEventErrorByEmail.vbs
'
' @Author: Nestor Urquiza
' @Created: 12/14/2011
'
'
' @Description: Alerts a Windows Admin there are errors in Event Viewer. 
' It could be scheduled to run every maxMinutes but
' Using it as an action for a custom Scheduled Task with a trigger on event filters:
'
' Task Scheduler Library: Create Task | Triggers | New Trigger | Begin the Task On an Event | Settings Custom | New Event Filter | Event Level Error | By Log | Event Logs | Windows Logs | Application
'
'
'
'

'
'
' @Compatibility: Tested so far in WindowsXP, Vista, 7, 2000, 2003, 2008
'
'
' @Parameters
' 1. A prefix body message in case specific errors are to be sent 
'    (a combination of batch and eventtriggers will do the trick)
'
'
' @Filters: I am filtering only "Application" events. Change the SQL query if you want to apply a different filter or not filter at all
'
'
'
''''''''''''''''''''''''''''''''''''''''''''''''

'Constants
strSmartHost = "mail.sample.com"
strSmartPort = 25
maxMinutes = 1
strComputer = "."
emailFrom = "donotreply@nestorurquiza.com"
emailTo = "nurquiza@nestorurquiza.com"

'System config
Set wshShell = WScript.CreateObject( "WScript.Shell" )
strComputerName = wshShell.ExpandEnvironmentStrings( "%COMPUTERNAME%" )
Set objSWbemServices = GetObject("winmgmts:" _
 & "{impersonationLevel=impersonate}!\\" & strComputer & "\root\cimv2")
Set colTimeZone = objSWbemServices.ExecQuery _
 ("SELECT * FROM Win32_TimeZone")
For Each objTimeZone in colTimeZone
 offset = objTimeZone.Bias
Next

'Parameters
Dim strBody
If (Wscript.Arguments.Count > 0) Then
  strBody = Wscript.Arguments(0)
End If

'Start date to look for events
dtmDate = DateAdd("n",-maxMinutes,Now())
dateToWMIDateString = Year(dtmDate) & padZeros(Month(dtmDate)) & padZeros(Day(dtmDate)) & padZeros(Hour(dtmDate)) & padZeros(Minute(dtmDate)) & padZeros(Second(dtmDate)) & ".000000" & offset

'Get events matching the query
Set objWMIService = GetObject("winmgmts:" _
    & "{impersonationLevel=impersonate}//" & _
        strComputer & "\root\cimv2")
Set colLogFiles = objWMIService.ExecQuery _
    ("Select * from Win32_NTLogEvent " _
        & "Where Logfile='Application' and Type='Error' and TimeGenerated > '" &  dateToWMIDateString & "'" )

'Accumulate all events dates and details
For Each objLogFile in colLogFiles
    dtmInstallDate = objLogFile.TimeGenerated
    WMIDateStringToDate = CDate(Mid(dtmInstallDate, 5, 2) & "/" & _
     Mid(dtmInstallDate, 7, 2) & "/" & Left(dtmInstallDate, 4) _
         & " " & Mid (dtmInstallDate, 9, 2) & ":" & _
             Mid(dtmInstallDate, 11, 2) & ":" & Mid(dtmInstallDate, _
                 13, 2))
    WMIDateStringToDate = DateAdd("n", offset, WMIDateStringToDate)
    details = details & vbCrLf & WMIDateStringToDate  & " - [" & _
                   objLogFile.Type & "] " & _
                   objLogFile.Message
    'Wscript.Echo details
Next

'Send email with details about matching events
If (Not IsNull(details) And details <> "") Then
    'Prepare email
    Set objEmail = CreateObject("CDO.Message")
    objEmail.From = emailFrom
    objEmail.To = emailTo
    objEmail.Subject = "[" & strComputerName & "] " & "Event Viewer Alert"
    If (Not IsNull(strBody) And strBody <> "") Then
        objEmail.Textbody = strBody & ". "
    End If
    objEmail.Textbody = objEmail.Textbody & details

    'Custom server
    objEmail.Configuration.Fields.Item("http://schemas.microsoft.com/cdo/configuration/sendusing") = 2
    objEmail.Configuration.Fields.Item("http://schemas.microsoft.com/cdo/configuration/smtpserver") = strSmartHost
    objEmail.Configuration.Fields.Item("http://schemas.microsoft.com/cdo/configuration/smtpserverport") = strSmartPort
    objEmail.Configuration.Fields.Update

    'Send it
    objEmail.Send
End If

Function padZeros(dtmDate)
If Len(dtmDate) = 1 Then
    padZeros = "0" & dtmDate
Else
    padZeros = dtmDate
End If
End Function

BTW you will notice the Event filter contains the below which could give you some hints to research even more powerful ways to control different event alerts:

    
      
    
  
Here is how to get SSL authentication or TLS Authentication support.

Wednesday, December 07, 2011

Arial Font from JasperReports for free and legally

If you are planning to ship documents to your clients and you need to use one of the commercial most used fonts like Arial you are in your way to pay a good chunk of money. There is not even a clear price on what they might cost because basically it depends on your application (read the license goes with how much money you are making with your product).

Thanks to the information I found in this post it took me little time to legally use a close to the proprietary Arial font in our application. If you plan to distribute or license your application you will need to buy Arial font or find a different project with a more commercial friendly license (like BSD) as the license for Liberation fonts is GPL. must then

  1. Download and extract the Red Hat Open Source GPL licensed Liberation Fonts
  2. From iReport Preferences on a Mac or Options for the rest of the mortals navigate through "iReport | Fonts | Install Font", then import "LiberationSans-Regular.ttf" from the exploded directory created as part of the above step. Click next and pick the bold, Italic and BoldItalic versions for LiberationSans. LiberationSans is the closest font to Arial. The rest of the Liberation Fonts are closer to Times New Roman and Courier New fonts
  3. Pick a good family name that will later use in your style fontName property, for example "LiberationSans". Check the "Embed this font in the PDF document"
  4. Include your font in a hello world jrxml like I show below and run it from iReport. Play with the properties and you should see bold and italic rendered as well (I am in this example striking through and underlining as well).
    <?xml version="1.0" encoding="UTF-8"?>
    <jasperReport xmlns="http://jasperreports.sourceforge.net/jasperreports" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://jasperreports.sourceforge.net/jasperreports http://jasperreports.sourceforge.net/xsd/jasperreport.xsd" name="hello" language="groovy" pageWidth="595" pageHeight="842" whenNoDataType="AllSectionsNoDetail" columnWidth="555" leftMargin="20" rightMargin="20" topMargin="20" bottomMargin="20">
     <property name="ireport.zoom" value="1.0"/>
     <property name="ireport.x" value="0"/>
     <property name="ireport.y" value="0"/>
     <style name="Default" isDefault="true" fontName="LiberationSans" fontSize="20" isBold="true" isItalic="true" isUnderline="true" isStrikeThrough="true"/>
     <background>
      <band splitType="Stretch"/>
     </background>
     <title>
      <band height="79" splitType="Stretch">
       <staticText>
        <reportElement x="12" y="10" width="531" height="57"/>
        <textElement/>
        <text><![CDATA[Hello JasperReports!!!]]></text>
       </staticText>
      </band>
     </title>
     <pageHeader>
      <band height="35" splitType="Stretch"/>
     </pageHeader>
     <columnHeader>
      <band height="61" splitType="Stretch"/>
     </columnHeader>
     <detail>
      <band height="125" splitType="Stretch"/>
     </detail>
     <columnFooter>
      <band height="45" splitType="Stretch"/>
     </columnFooter>
     <pageFooter>
      <band height="54" splitType="Stretch"/>
     </pageFooter>
     <summary>
      <band height="42" splitType="Stretch"/>
     </summary>
    </jasperReport>
    
  5. Now you should see from iReport your font correctly embedding in the resulting PDF
  6. From iReport navigate to the Fonts settings as I explained before and then pick "LiberationSans" and export it as liberation-sans.jar
  7. Include the jar in your application classpath
  8. Now your app is serving close to Arial font styled content

Wednesday, November 30, 2011

svn dump stderr output causing false positives

We were getting these annoying alerts every time the svn backup was running. Reason was 'svn dump' by default sends feedback like the below to the stderr:
* Dumped revision 0.
* Dumped revision 1.
* Dumped revision 2.
...

From the help command the reason and the workaround came clear:
# svnadmin dump --help
dump: usage: svnadmin dump REPOS_PATH [-r LOWER[:UPPER] [--incremental]]

Dump the contents of filesystem to stdout in a 'dumpfile'
portable format, sending feedback to stderr.  Dump revisions
LOWER rev through UPPER rev.  If no revisions are given, dump all
revision trees.  If only LOWER is given, dump that one revision tree.
If --incremental is passed, the first revision dumped will describe
only the paths changed in that revision; otherwise it will describe
every path present in the repository as of that revision.  (In either
case, the second and subsequent revisions, if any, describe only paths
changed in those revisions.)

Valid options:
  -r [--revision] ARG      : specify revision number ARG (or X:Y range)
  --incremental            : dump incrementally
  --deltas                 : use deltas in dump output
  -q [--quiet]             : no progress (only errors) to stderr

Just using the --quiet option does the trick:
svnadmin dump -q $MASTER_REPO >  $dump_file_path/dumpfile

Monday, November 28, 2011

Jasper Reports Excel Cross Sheet Formulas

A member of the Data team brought my attention to an iReports bug affecting Excel Output containing formulas which reference a following sheet. If the formula is let us say referring Sheet1 from Sheet2 there is no problem but if it refers Sheet2 from Sheet1 Jasper will fail to generate a correct XLS file. This is not the case when we try to generate an XLSX file. In that case it does work as expected.

When running with "Excel 2007 (XLSX) Preview" the cell does get what we expect:
=mySheet2!A1

However when we pick "XLS Preview" we get:
=#REF!A1

Here is the screen shot showing a successful rendering when using XLSX:
Here is the failure when using XLS:

This is clearly a problem related to different exporters. I still need to test this from Java with different exporter options but it would be ideal if a simple cross sheets formula like this could work from either XLS or XLSX files.

Below is the source code for this report. I have picked one jrxm posted in forums and I have added a couple of lines just to make sure I generate two different spreadsheets.

The output contains two sheets which reference each other through a simple formula that brings the content of the A1 cell from one sheet to the other in both directions. As stated at the beginning the formula in Sheet1 will fail when exporting to XLS but will succeed when exporting to XLSX. The formula in Sheet 2 will always succeed.

<?xml version="1.0" encoding="UTF-8"?>
<jasperReport xmlns="http://jasperreports.sourceforge.net/jasperreports" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://jasperreports.sourceforge.net/jasperreports http://jasperreports.sourceforge.net/xsd/jasperreport.xsd" name="formulaSample" pageWidth="595" pageHeight="842" whenNoDataType="AllSectionsNoDetail" columnWidth="535" leftMargin="0" rightMargin="0" topMargin="0" bottomMargin="0">
 <property name="net.sf.jasperreports.export.xls.detect.cell.type" value="true"/>
 <property name="net.sf.jasperreports.export.xls.one.page.per.sheet" value="true"/>
 <property name="net.sf.jasperreports.export.xls.sheet.names.all values" value="mySheet1/mySheet2"/>
 <property name="ireport.zoom" value="1.0"/>
 <property name="ireport.x" value="0"/>
 <property name="ireport.y" value="0"/>
 <import value="net.sf.jasperreports.engine.*"/>
 <import value="java.util.*"/>
 <import value="net.sf.jasperreports.engine.data.*"/>
 <title>
  <band height="262" splitType="Stretch">
   <textField>
    <reportElement key="textField-1" x="0" y="0" width="68" height="23"/>
    <textElement/>
    <textFieldExpression><![CDATA[new Integer(2)]]></textFieldExpression>
   </textField>
   <textField>
    <reportElement key="textField-2" x="0" y="23" width="68" height="23"/>
    <textElement/>
    <textFieldExpression><![CDATA[new Integer(1)]]></textFieldExpression>
   </textField>
   <textField>
    <reportElement key="textField-3" x="0" y="46" width="68" height="23">
     <propertyExpression name="net.sf.jasperreports.export.xls.formula"><![CDATA["mySheet2!A1"]]></propertyExpression>
    </reportElement>
    <textElement/>
    <textFieldExpression><![CDATA[]]></textFieldExpression>
   </textField>
   <break>
    <reportElement x="0" y="116" width="1" height="1"/>
   </break>
   <textField>
    <reportElement key="textField-3" x="0" y="163" width="68" height="23">
     <propertyExpression name="net.sf.jasperreports.export.xls.formula"><![CDATA["mySheet1!A1"]]></propertyExpression>
    </reportElement>
    <textElement/>
    <textFieldExpression><![CDATA[]]></textFieldExpression>
   </textField>
   <textField>
    <reportElement key="textField-2" x="0" y="140" width="68" height="23"/>
    <textElement/>
    <textFieldExpression><![CDATA[new Integer(9)]]></textFieldExpression>
   </textField>
   <textField>
    <reportElement key="textField-1" x="0" y="117" width="68" height="23"/>
    <textElement/>
    <textFieldExpression><![CDATA[new Integer(7)]]></textFieldExpression>
   </textField>
  </band>
 </title>
</jasperReport>

Note that this example contains a useful hint that allows to quickly come up with a test case that can be shared with the community. This is just using java.lang.* package to include hard coded values in text fields (that later become Excel cells) without the need to having a datasource defined (Use "Empty Datasource" to run this jrxml)

Last but not least we set the property "whenNoDataType="AllSectionsNoDetail"" because we are not using any datasource. If you are rendering the jrxml with no datasource from Java you will need it otherwise you will just get blank pages or corrupted XML files depending on the options you use.

Wednesday, November 23, 2011

The plugin 'org.apache.maven.plugins:maven-jetty-plugin' does not exist or no valid version could be found

This issue happens when the repository does reach the Mortbay release repo http://jetty.mortbay.org/maven2/release so to be safe I always add the below to the build/plugins node in the pom.xml.

<plugin>
        <groupId>org.mortbay.jetty</groupId>
        <artifactId>maven-jetty-plugin</artifactId>
        <version>6.0.1</version>
      </plugin>

Tuesday, November 22, 2011

SWFUpload plugin in Self Signed Certificate websites

When you are developing you will not buy an SSL certificate but instead you will use a self signed one. Adobe Flash is used to provide cool multiple file upload capabilities through the use of the SWFPlugin but it will fail if the Certificate is self signed.

This was affecting our QA team today and thanks to this post I figured a solution for the problem:

  1. In IE double click the "Certificate Error" icon
  2. Pick Install Certificate. Pick the defaults and finish

For some people this was not working still so I suggested to restart browsers and even Windows, you know that kind of stuff you need to do when suspicious right? At least one of my team members reported cleaning cookies resolved the problem for him (of course after following the above steps)

So SSL support based on self-signed certificates is buggy in flash and you can expect the same for SWFUpload. Look for SSL in the Known issues page for a similar statement.

Cannot determine realm for numeric host address

I could not connect today to one of our servers using password less public key ssh login. Using -vvv ssh option would show:
debug1: An invalid name was supplied 
Cannot determine realm for numeric host address 

So from the Red Hat target I inspected the secure log:
$ tail -f /var/log/secure
...
Nov 22 11:37:03 t2215179 sshd[23123]: Authentication tried for admin with correct key but not from a permitted host (host=tools, ip=xxx.xxx.xxx.xxx).
...

This was resolved by adding the server IP to /etc/hosts in the client
$ vi /etc/hosts
...
xxx.xxx.xxx.xxx tools
...

Friday, November 18, 2011

Android on the HP Touchpad

Update 20140610: The links for the original steps were not longer working so I followed http://liliputing.com/2014/06/use-touchpad-toolbox-install-android-erase-webos-hp-touchpad.html and I was able to get kitkat in the touchpad! The only thing that took me sometime was to find the right combination of files (gapps-kk-20140606-signed.zip, update-TWRP-jcs-dm-tenderloin-20140512.zip, cm-11-20140409-SNAPSHOT-jcsullins-tenderloin.zip). MAX OSX does not support MTP so you will need a program like http://www.android.com/filetransfer/ to transfer files to the android kitkat.

Update 20131002: I have been happy now for some time running Android on the HP Touchpad. Still running the old CyanigenMod7 which just crashed and reinstalled following the steps below.

Just follow instructions from cyanogenmod touchpad port team, Download section. Be sure you install everything including ClockworkMod.

Then install GoogleApps so you get Market application:
  1. Boot into Android mode
  2. Connect TouchPad to Computer
  3. Touch the notification icon so you are able to turn on USB
  4. From the computer drop the GoogleApps zip file (for example gapps-gb-20110828-signed.zip) into the TouchPad
  5. Reboot the TouchPad (Power + Home buttons for 15-20sec will do it in case just Power does not give you the reboot option)
  6. In the boot menu quickly move using the volume keys to option "ClockworkMod". Press Home button to accept
  7. choose "install zip from sdcard", then "choose zip from sdcard" and pick the GoogleApps zip file.
  8. Reboot

Thursday, November 17, 2011

Hide link or anchor after click with jquery

This is a common need when you provide what I call a naked user interface meaning, well I am waiting for my front end engineer and I provide just the bare minimums so the whole functionality can be used. But this is also a need for complicated user interfaces where sometimes a link is just supposed to trigger something, you want to stay in the page and you do not want the link to be available again (avoiding double submission for example).

So here is the JQuery code that will keep the link text but will change it by just a spanned text:
    //
    // Hide link after click
    //
    $('.hideAfterClickLink').click(function(e) {
     $(this).replaceWith('' + $(this).text() + '')
    });

Friday, November 11, 2011

Tomcat Servlet Context initialized twice or multiple times

Tomcat Servlet Context initialized twice or multiple times
I have seen this issue both in JBoss and Tomcat and in both cases this is about a miss configuration. It is worth to be said this issue affects the agility of the developer because extra time needs to be spent just waiting for tomcat to finish its initialization. This is also an issue that hits production systems many of which have very slow startup just because of miss configurations.

The first thing that you need to do is to include a log INFO level message when the servlet context is initialized like below:
public class ApplicationServletContextListener implements
        ServletContextListener {
...
    Logger log = LoggerFactory.getLogger(ApplicationServletContextListener.class.getName());
    @Override
    public void contextInitialized(ServletContextEvent event) {
       ...
       log.info("Servlet Context initialized");
...

Now whenever you restart your server you should get only one line:
2011-11-11 16:29:38,027 INFO [com.nestorurquiza.utils.web.ApplicationServletContextListener] -   Servlet Context initialized

If you get more than one look for errors in server.xml. Just to show an example of something that actually hit me Today this was the configuration in one of my servers:
...
<Host name="bhubdev.nestorurquiza.com"">
  <Context path="" docBase="/opt/tomcat/webapps/nestorurquiza-app"/>
...

Just changing for the below fixed the issue:
...
<Host name="bhubdev.nestorurquiza.com" appBase="webapps/nestorurquiza-app"/>
...

However it also screwed the root context you get with the previous configuration. If you still need the application to respond to the root context you will need to deployed as ROOT.war as explained in the official documentation.

Thursday, November 10, 2011

Excel Jasper Report with default Zoom using a Macro

I could not find a way to specify a default zoom for an Excel Report generated from iReport/Jasper Reports so I had to apply the Macro hack I recently documented.

Here is the Visual Basic for Applications code that make this possible. It zoom out the whole workbook to 75%:
Private Sub Workbook_Open()
    If Application.Sheets.Count > 1 Then
        Worksheets("Macro").Visible = False
    End If
    Application.ScreenUpdating = False

    For Each ws In Worksheets
        ws.Activate
        ActiveWindow.Zoom = 75
    Next

    Application.ScreenUpdating = True
End Sub

Wednesday, November 09, 2011

Excel Jasper Reports With Macros

I am a big fan of separation of concerns. Did I say this before? :-)

When it comes to View technologies there is a miss conception to think about View part as "passive" when in reality we have seen "active" Views for years with Javascript in the center of our HTML pages. Well, the same applies for PDF and Excel.

So I don't understand why providing Macro support in a Jasper Report should be considered wrong and since especially in the Financial Industry Excel is a core technology where Macros are simply needed I decided to present here a solution to provide Excel Jasper Reports containing Macros.

Some times the macro is needed by business while other times the macro can save time with formatting features that are unavailable in the iReport/Jasper last version. For the second case I do recommend to fill a feature request because it is a hack to force a formatting capability using a Macro and your users might not be OK with code running in their Excel which imposes a big security concern. But for sure this hack allows to move while keeping separation of concerns, in other words liberating Java developers from the need to touch code, recompile and redeploy (The Controller) just to fulfill a need in formatting that is not available where it should be available (The View).


Create an Excel Workbook containing the Macro to be applied to the Excel Jasper Report

  1. Create a brand new Excel Workbook
  2. Rename Sheet1 as 'Macro' and delete the rest of them
  3. Select from the Menu: Tools | Macro | Visual Basic Editor
  4. Right click on "This Workbook" and select "View Code". Paste the below in the code pane:
    Private Sub Workbook_Open()
        If Application.Sheets.Count > 1 Then
            Worksheets("Macro").Visible = False
        End If
        MsgBox "Hello Excel Jasper Report!"
    End Sub
    
  5. Save the Workbook as "myTemplate.xls", close and open. A pop up should come up automatically. This code as you already noticed has a hack which basically allows to hide the unique sheet we are forced to leave in the workbook because of an Excel imposed limitation.
  6. Now we are ready to use myTemplate.xls as the template out of which our Jasper Report will be built.

Use the Excel Template from jasper API

First, you will need to extend the exporter. I am showing here how to do it with JRXlsExporter which uses POI API but the same idea should be applicable to other exporters.
package com.nestorurquiza.utils;

import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;

import net.sf.jasperreports.engine.export.JRXlsExporter;

import org.apache.poi.hssf.usermodel.HSSFWorkbook;
import org.apache.poi.hssf.util.HSSFColor;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

public class CustomJRXlsExporter extends JRXlsExporter {
    private static final Logger log = LoggerFactory.getLogger(CustomJRXlsExporter.class);
    
    private InputStream is;
    
    public InputStream getIs() {
        return is;
    }

    public void setIs(InputStream is) {
        this.is = is;
    }

    @Override
    protected void openWorkbook(OutputStream os) {
        if(is != null) {
            try {
                workbook = new HSSFWorkbook(is);
                emptyCellStyle = workbook.createCellStyle();
                emptyCellStyle.setFillForegroundColor((new HSSFColor.WHITE()).getIndex());
                emptyCellStyle.setFillPattern(backgroundMode);
                dataFormat = workbook.createDataFormat();
                createHelper = workbook.getCreationHelper();
                return;
            } catch (IOException e) {
                log.error("Creating a new Workbook when I was supposed to use an existing one.", e);
            }
        }
        super.openWorkbook(os);
    }
}

Second just use the Custom Exporter passing the input stream to your Excel Template file.
...
CustomJRXlsExporter exporter = new CustomJRXlsExporter();
String xlsTemplate = ctx.getParameter("xlsTemplate");
if( xlsTemplate != null ) {
    String xlsTemplateFilePath = getReportsPath() + "/xlsTemplate/" + ctx.getParameter("xlsTemplate");
    exporter.setIs(new FileInputStream(xlsTemplateFilePath));
}
...

You end up with your report and a message box popping up from the macro code "Hello Excel Jasper Report!".

A more realistic example? I will probably start documenting some of them in the near future so search this blog for jasper excel macro.

This technique is simple but powerful. I hope it is integrated somehow in the different exporters as it was proposed 5 years ago.

As I have said there I think iReport should be able to use a hint so the front end report designer can get the formatting they want using an Excel Template without the need to wait for an external API implementation.

Monday, November 07, 2011

error running shared postrotate script for /var/log/mysql

I am sure the error below can be related to many different circumstances:
/etc/cron.daily/logrotate:
error: error running shared postrotate script for '/var/log/mysql.log /var/log/mysql/mysql.log /var/log/mysql/mysql-slow.log '
run-parts: /etc/cron.daily/logrotate exited with return code 1

In my case after cloning an ESX Ubuntu VM where MySQL was installed into some other servers where we did not need mysql and after uninstalling mysql I could still see mysqld running.

Just running the below command was enough to stop the alert:
$ sudo apt-get autoremove
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following packages will be REMOVED:
  mysql-server-5.1 mysql-server-core-5.1
...

Sunday, November 06, 2011

User impersonation with Spring and LDAP

Impersonating a user (also known as "switch a user" or "do as user") in a web application is an important feature for doing Quality Assurance (QA) or any Manual User Acceptance Test (UAT).

With Spring you can achieve it using the SwitchUserFilter. If you are using LDAP here are the relevant pieces.

...
<beans:bean id="ldapUserDetailsService" class="org.springframework.security.ldap.userdetails.LdapUserDetailsService">
  <beans:constructor-arg><beans:ref bean="ldapUserSearch"/></beans:constructor-arg>
  <beans:constructor-arg><beans:ref bean="ldapAuthoritiesPopulator"/></beans:constructor-arg>
  <beans:property name="userDetailsMapper" ref="customUserDetailsContextMapper" />
</beans:bean>
...
<beans:bean id="ldapUserSearch" class="org.springframework.security.ldap.search.FilterBasedLdapUserSearch">
    <beans:constructor-arg type="String"><beans:value>ou=people,o=nestorurquiza</beans:value></beans:constructor-arg>
    <beans:constructor-arg type="String"><beans:value>mail={0}</beans:value></beans:constructor-arg>
    <beans:constructor-arg><beans:ref bean="ldapContextSource"/></beans:constructor-arg>
</beans:bean>
 ...                            
<beans:bean id="switchUserProcessingFilter" class="org.springframework.security.web.authentication.switchuser.SwitchUserFilter">
        <beans:property name="userDetailsService" ref="ldapUserDetailsService" />
        <beans:property name="switchUserUrl" value="/admin/switchUser" />
        <beans:property name="exitUserUrl" value="/admin/switchUserExit" />
        <beans:property name="targetUrl" value="/" />
</beans:bean>
...
<http auto-config="true" use-expressions="true" access-decision-manager-ref="accessDecisionManager" disable-url-rewriting="true">
    ...
 <custom-filter  after="FILTER_SECURITY_INTERCEPTOR" ref="switchUserProcessingFilter" />
    ...
    <intercept-url pattern="/admin/**" access="hasAnyRole('ROLE_ADMIN','ROLE_PREVIOUS_ADMINISTRATOR')" />
... 
<beans:bean id="ldapContextSource" class="org.springframework.ldap.core.support.LdapContextSource">

    <beans:property name="url" value="${ldap.url}" />
    <beans:property name="userDn" value="${ldap.userDn}" />
    <beans:property name="password" value="${ldap.password}" />
</beans:bean>
...
<beans:bean id="ldapAuthProvider"
        class="org.springframework.security.ldap.authentication.LdapAuthenticationProvider">
  <beans:constructor-arg>
    <beans:bean class="org.springframework.security.ldap.authentication.BindAuthenticator">
      <beans:constructor-arg ref="ldapContextSource"/>
      <beans:property name="userDnPatterns">
        <beans:list><beans:value>mail={0},ou=people,o=nestorurquiza</beans:value></beans:list>
      </beans:property>
    </beans:bean>
  </beans:constructor-arg>
  <beans:constructor-arg ref="ldapAuthoritiesPopulator"/>
</beans:bean>
...
<beans:bean id="ldapAuthoritiesPopulator" class="org.springframework.security.ldap.userdetails.DefaultLdapAuthoritiesPopulator">
      <beans:constructor-arg ref="ldapContextSource"/>
      <beans:constructor-arg value="ou=groups,o=nestorurquiza"/>
      <beans:property name="groupSearchFilter" value="uniquemember={0}" />
</beans:bean>
...    
<beans:bean id="ldapTemplate" class="org.springframework.ldap.core.LdapTemplate">
    <beans:constructor-arg ref="ldapContextSource" />
</beans:bean>
...
I18n messages:
switchUser=Impersonate
switchUserExit=Be yourself
A link to impersonate in the users listing page should be available only for admins:
<security:authorize access="hasRole('ROLE_ADMIN')">
                        | <a href="<spring:url value="/admin/switchUser?j_username=${employee.email}" htmlEscape="true" />">
                            <spring:message code="switchUser"/></a>
                </security:authorize>
A link to switch back to admin user should be available only if the user is being impersonated:
<security:authorize access="hasRole('ROLE_PREVIOUS_ADMINISTRATOR')">
                        | <a href="<spring:url value="/admin/switchUserExit" htmlEscape="true" />">
                            <spring:message code="switchUserExit"/></a>
                    </security:authorize>
The way you use it is hitting the below URLs to switch the user and come back to the original admin user context:
http://localhost:8080/nestorurquiza-app/admin/switchUser?j_username=nurquiza@nestorurquiza.com
http://localhost:8080/nestorurquiza-app/admin/switchUserExit

Monday, October 31, 2011

Spring One 2011 Notes

It was quite a week of learning with the folks from Spring Framework in the SpringOne Conference in Chicago.

Not only I had the opportunity to hear Spring explained from the code committers but I managed to discuss some common architectural, managerial and leadership issues with other attendees.

The amount of work Spring helps to solve could be enough for four weeks of conference so there were mainly 4 slices (or tracks) of very related conferences from which you had to pick just one. Here are some comments about those that I attended.

Javascript application engineering

It is clear javascript is now stronger that ever. It was a nice look into how to do Object Oriented Programming with Javascript, Inheritance through prototypes (Object.create(Object.prototype)), closures and more. Between others websites like Javascript weekly and hacker news were recommended to be followed up.

Spring Integration. Practical Tips and tricks

The Spring Integration project is an attempt to implement the patterns described in the book Enterprise Integration Patterns Book by Gregor Hohpe and Bobby Wolf
It was a discussion about Consumers, Producers and the Publish/Subscribe pattern using AMQP channels and comparing them with JMS channels. Of course SOAP, REST and others are supported. Channel redundancy was presented Rabitt MQ and JSM. The project reminds me of ESB and indeed there was a mention to it. Basically Spring Integration is a different approach than ESB because there is no server needed but just patterns applied.

Messaging

Messaging was presented through the Rabbit MQ which is the reference implementation of the Advanced Messaging Queue Protocol (AMQP). It was even proposed to start sending messages between Controllers and Services to prepare the application to scale in the future. I have to say I won't buy into that even though for sure I would consider using it to implement future distributed needs in my projects. Messaging was presented as a way around problems inherent to the use of shared class loaders (OSGI). In reality that is what any distributed technique does after all, being more than anything a consequence of the paradigm in use.

Neo4J

This is a graph database and the presentation was useful especially to eliminate any of the ongoing discussions about what is the best database out there in the NoSQL domain. It is clear they all address different problems while of course there is more than one choice within each possible domain. Basically there is a trade between size and complexity and in that order we have different options like key-value, bigtable, document, graph.

Spring Insight Plugin Development

This is a project with a mission that sounds familiar: Inspect how the application is behaving performance wise. Classes are annotated with @InsightOperation and @InsightEndPoint to allow performance inspection: Basically how long the method gets to execute. In other words where the time is spent in the application. While this could be definitely useful in many scenarios we have tools doing this already so it looked to me like an effort to build a lightweight profiler focused on vFabric.

Spring Integration. Implementing Scalable Architectures with Spring Integration

A simple implementation based on a simple relational database locked row was presented to provide a messaging cluster implementation.

Spring Batch

In a glance this is a project to manipulate data. We learned how Spring Batch works while defining and instantiating jobs (Job and Job Instance) that basically act on data. I will not start a war about if spring is better or worst that etl for this task. It is just another way of doing the same and depending on your architecture and team one will be favored over the other. I tend to separate concerns as much as possible and I think data is better processed and tackle by data centric developers and existing tools. In my experience most of people dealing with Spring actually love to implement business rules that are not necessarily related to data intensive processing. But again this discussion should end here, there is not correct way just different ways.

Deliver Performance and scalability with ehcache and spring

JSR107 will standardize caching while some of us still use some caching api from Google code to cache data at method level. However terracotta ehCache and the new standard add more flexibility to caching. For simple caching you can use the Google code API but for more complicated situations you will use this approach which involves @Cacheable and @CacheEvict annotations. When the source code is not available @Cacheable can be used from XML which is great you will agree. AOP helps here to cache certain methods. Different providers like terracotta backed open source Ehcache offer different possibilities which go beyond the basis of the default ConcurrentMap implementation. Automatic Resource Control now in beta will allow ehacahe to use cache sizing in terms of bytes or percentage besides the existing object count. Then of course you can even go for enterprise class monitoring with "terracotta Developer Console".

Eventing Data with RabbitMQ and Riak

Riak was discussed as the unique NoSQL database that would allow hooking into the data insertion. This is of advantage to create a trigger for RabitMQ when new data gets stored. A very cool example was presented where WebSocket was used to notify the client directly from the server. Again the CAP theorem was discussed: You must pick one from Consistency, Availability and Performance triangle and that would ultimately drive your decision towards one database or another.

Where did my architecture go

It was disscused how to control the gap between architecture and codebase. Here you have code analysis with JDepend and Sonar, SonarGraph (formerly SonarJ). Suggestions were made about the usage of ServiceLoader or OSGI for externalizing class creation.

Tailoring Spring for Custom Usage

As we all know Spring is about DI, IOP and Enterprise Service Abstractions (libraries). This lecture went through each of the components giving real case utilization of them. We saw @Configuration and @Bean to provide configuration from Java code instead of XML, @Value to allow the injection of a system property as a string member of a class, Spring scopes to be used for example to eliminate the need to pass custom context objects from method to method (A technique that I still use in my projects for maintaining ControllerContext), even though Singleton and Prototype are the most common scopes there are several others like Request and Session, @Profile annotation to define production or development. The Spring expression language in general was introduced.

Improving Java with Groovy

There were several trcks of Groovy and Grails. I attended just this one from the latest slice because there was nothing for Spring Java available and even though it was basic stuff we saw examples that did work from an excellent demo showing how to use both Java and Groovy in your project. It was basically a practical presentation of the ideas behind the book "Making Java Groovy". Heavy use of closures showed how the code got smaller while avoiding Interfaces and Inner Classes.

Monday, October 24, 2011

Cron error email notification in Ubuntu

A member of the team realized one of the cron processes was failing but we were not getting any alerts.

While I favor the use of LogMonitor for absolutely everything there are simple commands you use in cron jobs for which ideally you would like to get notifications without the need of logging.

If you design your script carefully respecting the fact that error messages mut go to stderr this procedure should work:

  1. Be sure sSMTP is installed and properly configured:
    $ sudo apt-get install ssmtp
    $ sudo vi /etc/ssmtp/ssmtp.conf
    ...
    root=nurquiza@nestorurquiza.com
    ...
    mailhub=mail.nestorurquiza.com
    ...
    
  2. Add a line at the beginning of crontab stating who the error email should me sent to:
    $ crontab -e 
    MAILTO=alerts@nestorurquiza.com
    0 22 * * * /usr/sbin/sftp_backup.sh > /dev/null
    ...
    

Note the redirection of stdout to /dev/null. You want to receive I bet just errors right?

Saturday, October 22, 2011

Upgrading CouchDB in Ubuntu

I had to upgrade CouchDB to candidate release 1.1.1 and I did it just following the same instructions I followed to install it from scratch.

Installing CouchDB in Ubuntu

I have tested the below procedure in Ubuntu 10.10 (Maverick) to install CouchDB 1.1.1 candidate release.

$ ps -ef|grep couch|awk '{print $2}'|xargs sudo kill -9
$ sudo apt-get update
$ sudo apt-get autoremove
$ sudo apt-get remove couchdb
$ sudo apt-get build-dep couchdb
$ sudo apt-get install libtool zip
$ cd
$ curl -O http://ftp.mozilla.org/pub/mozilla.org/js/js185-1.0.0.tar.gz
$ tar xvzf js185-1.0.0.tar.gz 
$ cd js-1.8.5/js/src
$ ./configure
$ make
$ sudo make install
$ cd
$ curl -O http://www.erlang.org/download/otp_src_R14B04.tar.gz
$ tar xvzf otp_src_R14B04.tar.gz 
$ cd otp_src_R14B04
$ ./configure --enable-smp-support --enable-dynamic-ssl-lib --enable-kernel-poll
$ make
$ sudo make install
$ cd
$ svn co http://svn.apache.org/repos/asf/couchdb/branches/1.1.x/ couchdb1.1.x
$ cd couchdb1.1.x
$ ./bootstrap
$ prefix='/usr/local'
$ ./configure --prefix=${prefix} 
$ make
$ sudo make install
$ sudo useradd -d /var/lib/couchdb couchdb
$ sudo chown -R couchdb: ${prefix}/var/{lib,log,run}/couchdb ${prefix}/etc/couchdb
$ for dir in `whereis couchdb | sed 's/couchdb: //'`; do echo $dir | xargs sudo chown couchdb; done
$ export xulrunnerversion=`xulrunner -v 2>&1 >  /dev/null | egrep -o "([0-9]{1,2})(\.[0-9]{1,2})+"`
$ echo $xulrunnerversion
$ echo "/usr/lib/xulrunner-$xulrunnerversion" > /etc/ld.so.conf.d/xulrunner.conf
$ echo "/usr/lib/xulrunner-devel-$xulrunnerversion" >> /etc/ld.so.conf.d/xulrunner.conf
$ sudo ln -s /usr/local/etc/init.d/couchdb /etc/init.d/couchdb
$ update-rc.d couchdb defaults
$ /etc/init.d/couchdb start
$ curl -X GET http://localhost:5984
{"couchdb":"Welcome","version":"1.1.1a1187726"}

Friday, October 21, 2011

Monitoring CouchDB with Monit

Just add the below in monitrc and reload config. I am expecting you hardened CouchDB and so it is listening in an SSL port. Note that couchdb is already monitored by heart so you can probably just not monitor it as well ;-)

#################################################################
# couchdb
################################################################

#check process couchdb
#  with pidfile /usr/local/var/run/couchdb/couchdb.pid
#  start program = "/usr/local/etc/init.d/couchdb start"
#  stop program = "/usr/local/etc/init.d/couchdb stop"
#  if failed port 6984 then restart
#  if failed url https://localhost:6984/ and content == '"couchdb"' then restart

#couchdb does not save the parent pid of the starting process so the above would serve no purpose
check host couchdb with address localhost
  if failed port 6984 then alert
  if failed url https://localhost:6984/ and content == '"couchdb"' then alert
group couchdb

Hardening CouchDB: A more secure distributed database

Here are the steps to follow in order to harden CouchDB. I am currently using candidate release 1.1.1 and these are instructions for a local environment. For a production environment you have to locate the non dev config files local.ini and default.ini (Just run "ps -ef | grep couchdb" and look for the path, in my case /usr/local/etc/couchdb/local.ini and /usr/local/etc/couchdb/default.ini).

  1. Install a primary key and generated pem certificate:
    $ export DOMAIN=couchdb.nestorurquiza.com
    $ openssl genrsa -des3 -out ${DOMAIN}.pem 1024
    $ openssl req -new -key ${DOMAIN}.pem -out ${DOMAIN}.csr
    $ openssl x509 -req -days 3650 -in ${DOMAIN}.csr -signkey ${DOMAIN}.pem -out ${DOMAIN}.cert.pem
    $ openssl rsa -in ${DOMAIN}.pem -out ${DOMAIN}.pem
    $ sudo mkdir -p /opt/couchdb/certs/
    $ sudo cp ${DOMAIN}.pem /opt/couchdb/certs/
    $ sudo cp ${DOMAIN}.cert.pem /opt/couchdb/certs/
    

  2. Configure couchDB for *SSL only* and restart the server
    $ vi /Users/nestor/Downloads/couchdb1.1.x/etc/couchdb/local.ini
    [admins]
    ;admin = mysecretpassword
    nestorurquizaadmin = secret1
    ...
    [daemons]
    ; enable SSL support by uncommenting the following line and supply the PEM's below.
    ; the default ssl port CouchDB listens on is 6984
    httpsd = {couch_httpd, start_link, [https]}
    ...
    [ssl]
    cert_file = /opt/couchdb/certs/couchdb.nestorurquiza.com.cert.pem
    key_file = /opt/couchdb/certs/couchdb.nestorurquiza.com.pem
    $ vi /Users/nestor/Downloads/couchdb1.1.x/etc/couchdb/default.ini
    [httpd]
    bind_address = 0.0.0.0 #Use no loopback address only if the server will be exposed to a different machine
    ...
    [daemons]
    ...
    ; httpd={couch_httpd, start_link, []}
    ...
    
  3. Restart
    sudo couchdb #local OSX
    sudo /usr/local/etc/init.d/couchdb restart #Ubuntu
    

  4. Now we can access the secured CouchDB instance
    HOST="https://nestorurquizaadmin:mysecret@127.0.0.1:6984"
    curl -k -X GET $HOST
    

Note that at this time to remove the insecure http you have to use /usr/local/etc/couchdb/default.ini which gets overwritten after any upgrade.

Of course we should sign our certificate with a Certificate Authority (CA) to make this really secure but you can also add some security if you play with firewall rules and allow access to the SSL port to certain IP only. I can do so because I do use a middle tear between CouchDB and the browser. If you directly hitting CouchDB from the wild you better use a CA.

Thursday, October 20, 2011

CouchDB filtered replication

One of the greatest features of CouchDB is its replication which allows for great distributed computing. It reminds me when in 1999 I met Erlang language for the first time (Working for a Telco). Erlang is made for distributed computing and so CouchDB which of course is built in Erlang.

I have to say I have successfully tested this in upcoming version 1.1.1 (built from a branch) Do not try this in 1.1.0.

The example below is based on the document that I have been discussing in the three part tutorial about building a Document Management System (DMS) with CouchDB.

Filtered or selective replication is a two step process:
  1. First create a filter named for example "clientFilter" in a new document called "replicateFilter". This sample filter will reject any client not matching the clientId parameter (step 2 explains what this parameter is about). Any deleted documents will be deleted from the target as well.
    curl -H 'Content-Type: application/json' -X PUT http://127.0.0.1:5984/dms4/_design/replicateFilter -d \
    '{
      "filters":{
        "clientFilter":"function(doc, req) {
          if (doc._deleted) {
            return true;
          }
     
          if(!doc.clientId) {
            return false;
          }
     
          if(!req.query.clientId) {
            throw(\"Please provide a query parameter clientId.\");
          }
     
          if(doc.clientId == req.query.clientId) {
            return true;
          }
          return false;
        }"
      }
    }'
    
  2. Create a replication document called "by_clientId". This example passes clientId=1 as a parameter to the filter we created in step number 1 ("replicateFilter/clientFilter"). You figured we will end up replicating documents for that client.
    curl -H 'Content-Type: application/json' -X POST http://127.0.0.1:5984/_replicator -d \
    '{
      "_id":"by_clientId",
      "source":"dms4",
      "target":"http://couchdb.nestorurquiza.com:5984/dms4",
      "create_target":true,
      "continuous":true,
      "filter":"replicateFilter/clientFilter",
      "query_params":{"clientId":1}
    }'
    

Deleting a replication document is how you turn off that replication. This is not any different than deleting any other document:
nestor:~ nestor$ curl -X GET http://127.0.0.1:5984/_replicator/by_clientId
{"_id":"by_clientId","_rev":"5-e177ca7f79d9ba6f91b803a2cb2abc1e","source":"dms4","target":"http://couchdb.nestorurquiza.com:5984/dms4","create_target":true,"continuous":true,"filter":"replicateFilter/clientFilter","query_params":{"clientId":1},"_replication_state":"triggered","_replication_state_time":"2011-10-20T13:09:56-04:00","_replication_id":"d8dc09e97f4948de0294260dda19fc6f"}
nestor:~ nestor$ curl -X DELETE http://127.0.0.1:5984/_replicator/by_clientId?rev=5-e177ca7f79d9ba6f91b803a2cb2abc1e
{"ok":true,"id":"by_clientId","rev":"6-0d20d90cbed22837eb3233e2bd8dfb2c"}

The same applies for getting a list of the current defined "selective replicators". You can use a temporary view like I show here or create a permanent view to list all the replicators:
$ curl -X POST http://127.0.0.1:5984/_replicator/_temp_view -H "Content-Type: application/json" -d '{
  "map": "function(doc) {
            emit(null, doc);
          }"
}'

OSX homebrew update: unexpected token near HOMEBREW_BREW_FILE

Update: I have seen this issue with other packages as well. Here is a recipe to make sure a specific version of ruby is used:
cp -p /usr/local/bin/brew /usr/local/bin/brew.old
echo  '#!'/Users/`whoami`/.rvm/rubies/ruby-1.8.7-p160/bin/ruby > /tmp/f1
sed '1d' /usr/local/bin/brew > /tmp/f2
cat /tmp/f1 /tmp/f2 > /usr/local/bin/brew
Original Post: After upgrading homebrew:
$ brew update

I got errors:
$ brew install mozilla-js
/usr/local/bin/brew: line 4: syntax error near unexpected token `('
/usr/local/bin/brew: line 4: `HOMEBREW_BREW_FILE = ENV['HOMEBREW_BREW_FILE'] = File.expand_path(__FILE__)'

Here is a temporary solution which you will need to do everytime you update brew:
$ sudo vi /usr/local/bin/brew
#!/Users/nestor/.rvm/rubies/ruby-1.9.2-p290/bin/ruby
##!/usr/bin/ruby

Upgrading couchDB in OSX

While trying to use couchDB filtered replication I had some problems as documented in gist.

I had to upgrade then couchDB to a non released version. Here are the steps I followed.

Delete all files from previous installations:
$ sudo find /usr/local -name couchdb | sudo xargs rm -fR

Install ICU from http://download.icu-project.org
$ tar xvzf icu4c-4_8_1-src.tgz 
$ cd icu/source/
$ ./runConfigureICU MacOSX --with-library-bits=64 --disable-samples --enable-static # if this fails for you try: ./configure --enable-64bit-libs
$ make
$ sudo make install

Install Spidermonkey
$ curl -O http://ftp.mozilla.org/pub/mozilla.org/js/js185-1.0.0.tar.gz
$ tar xvzf js185-1.0.0.tar.gz 
$ cd js-1.8.5/js/src
./configure 
$ make
$ sudo make install

Then install latest Erlang:
$ curl -O http://www.erlang.org/download/otp_src_R14B04.tar.gz
$ tar xvzf otp_src_R14B04.tar.gz 
$ cd otp_src_R14B04
$ ./configure --enable-smp-support --enable-dynamic-ssl-lib --enable-kernel-poll --enable-darwin-64bit
$ make
$ sudo make install

Then checkout the needed version. I tried from git (http://git-wip-us.apache.org/repos/asf/couchdb.git) first but I had several problems with autoconf and beyond (.configure was not available so I needed to go with automake -a; autoconf; autoheader) so I then built from svn:
$ svn co http://svn.apache.org/repos/asf/couchdb/branches/1.1.x/ couchdb1.1.x
$ cd couchdb1.1.x
$ ./bootstrap
$ ./configure
$ make
$ sudo make install
$ sudo couchdb

Sunday, October 16, 2011

Document Management System with CouchDB - Third Part

This is the last part of my attempt to cover how to use CouchDB to build a DMS. We will be using Java and the Ektorp library for the implementation.

Using Ektorp

A good place to start is downloading the BlogPost application (org.ektorp.sample-1.7-project) that shows some of the most important concepts around the Ektorp API. Follow Ektorp Tutorial and of course do not miss the reference documentation.

I will show here the steps on how to make your existing spring project able to interact with CouchDB using the Ektorp project. The code presented here is an implementation of the ideas exposed in the second part of this project.

Here the dependencies for your project. Note that I am adding file upload dependencies because I am building a DMS and I will provide an upload module:
<!-- Ektorp for CouchDB -->
        <dependency>
            <groupId>org.ektorp</groupId>
            <artifactId>org.ektorp</artifactId>
            <version>${ektorp.version}</version>
            <exclusions>
             <exclusion>
              <artifactId>slf4j-api</artifactId>
              <groupId>org.slf4j</groupId>
             </exclusion>
            </exclusions>
        </dependency>
        <dependency>
            <groupId>org.ektorp</groupId>
            <artifactId>org.ektorp.spring</artifactId>
            <version>${ektorp.version}</version>
        </dependency>

        <!-- File Upload -->
        <dependency>
            <groupId>commons-fileupload</groupId>
            <artifactId>commons-fileupload</artifactId>
            <version>1.2.2</version>
        </dependency>

In spring application context xml (Note the upload component which is only needed because again we need it for the DMS upload functionality):
...
xmlns:util="http://www.springframework.org/schema/util"
xmlns:couchdb="http://www.ektorp.org/schema/couchdb"
...
http://www.springframework.org/schema/util http://www.springframework.org/schema/util/spring-util-3.0.xsd
http://www.ektorp.org/schema/couchdb http://www.ektorp.org/schema/couchdb/couchdb.xsd
...
<context:component-scan base-package="com.nu.dms.couchdb.ektorp.model"/>
<context:component-scan base-package="com.nu.dms.couchdb.ektorp.dao"/>
...
<util:properties id="couchdbProperties" location="classpath:/couchdb.properties"/>
<couchdb:instance id="dmsCouchdb" url="${couchdb.url}" properties="couchdbProperties" />
<couchdb:database id="dmsDatabase" name="${couchdb.db}" instance-ref="dmsCouchdb" />
...
<!-- File Upload -->
<bean id="multipartResolver" class="org.springframework.web.multipart.commons.CommonsMultipartResolver">
        <property name="maxUploadSize" value="2000000"/>
</bean>

In classpath environment properties file:
couchdb.url=http://localhost:5984
couchdb.db=dms4 #my database name

File classpath couchdb.properties file:
host=localhost
port=5984
maxConnections=20
connectionTimeout=1000
socketTimeout=10000
autoUpdateViewOnChange=true
caching=false

A Document POJO following the BlogPost POJO from the Ektorp example:
package com.nu.dms.couchdb.ektorp.model;

import org.ektorp.Attachment;
import org.ektorp.support.CouchDbDocument;

public class CustomCouchDbDocument extends CouchDbDocument {

    private static final long serialVersionUID = -9012014877538917152L;

    @Override
    public void addInlineAttachment(Attachment a) {
        super.addInlineAttachment(a);
    }   
}


package com.nu.dms.couchdb.ektorp.model;

import java.util.Date;

import javax.validation.constraints.NotNull;

import org.ektorp.support.TypeDiscriminator;

public class Document extends CustomCouchDbDocument {

    private static final long serialVersionUID = 59516215253102057L;
    
    public Document() {
        super();
    }
    
    public Document(String title) {
        this.title = title;
    }
    
    /**
     * @TypeDiscriminator is used to mark properties that makes this class's documents unique in the database. 
     */
    @TypeDiscriminator
    @NotNull
    private String title;
    
    private int clientId;
    private int createdByEmployeeId;
    private int reviewedByEmployeeId;
    private int approvedByManagerId;
    private Date dateEffective;
    private Date dateCreated;
    private Date dateReviewed;
    private Date dateApproved;
    private int investorId;
    private int categoryId;
    private int subCategoryId;
    private int statusId;
    
    public String getTitle() {
        return title;
    }
    public void setTitle(String title) {
        this.title = title;
    }
    public int getClientId() {
        return clientId;
    }
    public void setClientId(int clientId) {
        this.clientId = clientId;
    }
    public int getCreatedByEmployeeId() {
        return createdByEmployeeId;
    }
    public void setCreatedByEmployeeId(int createdByEmployeeId) {
        this.createdByEmployeeId = createdByEmployeeId;
    }
    public int getReviewedByEmployeeId() {
        return reviewedByEmployeeId;
    }
    public void setReviewedByEmployeeId(int reviewedByEmployeeId) {
        this.reviewedByEmployeeId = reviewedByEmployeeId;
    }
    public int getApprovedByManagerId() {
        return approvedByManagerId;
    }
    public void setApprovedByManagerId(int approvedByManagerId) {
        this.approvedByManagerId = approvedByManagerId;
    }
  
    public Date getDateEffective() {
        return dateEffective;
    }

    public void setDateEffective(Date dateEffective) {
        this.dateEffective = dateEffective;
    }

    public Date getDateCreated() {
        return dateCreated;
    }
    public void setDateCreated(Date dateCreated) {
        this.dateCreated = dateCreated;
    }
    public Date getDateReviewed() {
        return dateReviewed;
    }
    public void setDateReviewed(Date dateReviewed) {
        this.dateReviewed = dateReviewed;
    }
    public Date getDateApproved() {
        return dateApproved;
    }
    public void setDateApproved(Date dateApproved) {
        this.dateApproved = dateApproved;
    }
    public int getInvestorId() {
        return investorId;
    }
    public void setInvestorId(int investorId) {
        this.investorId = investorId;
    }
    public int getCategoryId() {
        return categoryId;
    }
    public void setCategoryId(int categoryId) {
        this.categoryId = categoryId;
    }
    public int getSubCategoryId() {
        return subCategoryId;
    }

    public void setSubCategoryId(int subCategoryId) {
        this.subCategoryId = subCategoryId;
    }
    public int getStatusId() {
        return statusId;
    }
    public void setStatusId(int statusId) {
        this.statusId = statusId;
    }
    @Override
    public void setRevision(String s) {
        // downstream code does not like revision set to emtpy string, which Spring does when binding
        if (s != null && !s.isEmpty()) super.setRevision(s);
    }
    
    public boolean isNew() {
        return getId() == null;
    }
}

A Document Repository:
package com.nu.dms.couchdb.ektorp.dao;

import org.ektorp.CouchDbConnector;
import org.ektorp.support.CouchDbRepositorySupport;

public class CustomCouchDbRepositorySupport<T> extends CouchDbRepositorySupport<T> {


    protected CustomCouchDbRepositorySupport(Class<T> type, CouchDbConnector db) {
        super(type, db);
    }

    public CouchDbConnector getDb() {
        return super.db;
    }   
}

package com.nu.dms.couchdb.ektorp.dao;

import java.io.InputStream;

@Component
public class DocumentRepository  extends CustomCouchDbRepositorySupport<Document> {
    
    private static final Logger log = LoggerFactory.getLogger(DocumentRepository.class);
    
    @Autowired
    public DocumentRepository(@Qualifier("dmsDatabase") CouchDbConnector db) {
        super(Document.class, db);
        initStandardDesignDocument();
    }

    @GenerateView @Override
    public List<Document> getAll() {
        ViewQuery q = createQuery("all")
                        .includeDocs(true);
        return db.queryView(q, Document.class);
    }
    
    public Page<Document> getAll(PageRequest pr) {
        ViewQuery q = createQuery("all")
                        .includeDocs(true);
        return db.queryForPage(q, pr, Document.class);
    }
    
    @View( name = "tree", map = "classpath:/couchdb/tree_map.js", reduce = "classpath:/couchdb/tree_reduce.js")
    public InputStream getTree(String startKey, String endKey, int groupLevel) {
        ViewQuery q = createQuery("tree")
        .startKey(startKey)
        .endKey(endKey)
        .groupLevel(groupLevel)
        .group(true);
        InputStream is = db.queryForStream(q);
        return is;
    }

}

Map and Reduce javascript functions in src/main/resources/couchdb in other words in the classpath:
//by_categoryId_map.js
function(doc) { 
 if(doc.title && doc.categoryId) {
  emit(doc.categoryId, doc._id)
 } 
}

//by_categoryId_reduce.js
_count

//tree_map.js
function(doc) {
  var tokens = doc.dateEffective.split("-");
  var year = null;
  var month = null;
  if(tokens.length == 3) {
    year = tokens[0];
    month = tokens[1];
  }
  var key = [doc.clientId, doc.categoryId, doc.subCategoryId, year, month].concat(doc.title);
  var value = null;
  emit(key, value);
}

//tree_reduce.js
_count

In web.xml we need to allow flash because we will use the swfUpload to upload multiple files:
<url-pattern>*.swf</url-pattern>
</servlet-mapping>

Keeping things short I am not using a Service layer so I provide a unique Controller that allows creating a document including the attachment and metadata, uploading multiple documents (using swfupload flash) which follow a strong naming convention to produce all necessary metadata our of their names, viewing the documents as explained in part two in a tree view (using jquery.treeview.async.js)

package com.nu.web.controller.dms;

import java.io.ByteArrayOutputStream;
import java.io.InputStream;
import java.net.URLConnection;
import java.text.SimpleDateFormat;
import java.util.Date;
import java.util.Iterator;
import java.util.Locale;
import java.util.Map;
import java.util.Scanner;

import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;

import org.apache.commons.codec.binary.Base64;
import org.apache.commons.fileupload.FileItemFactory;
import org.apache.commons.fileupload.FileUploadException;
import org.apache.commons.fileupload.disk.DiskFileItemFactory;
import org.apache.commons.fileupload.servlet.ServletFileUpload;
import org.apache.commons.lang.StringUtils;
import org.apache.poi.util.IOUtils;
import org.codehaus.jackson.JsonNode;
import org.codehaus.jackson.map.ObjectMapper;
import org.codehaus.jackson.node.ObjectNode;
import org.ektorp.Attachment;
import org.ektorp.AttachmentInputStream;
import org.ektorp.PageRequest;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Controller;
import org.springframework.ui.Model;
import org.springframework.validation.BindingResult;
import org.springframework.validation.ObjectError;
import org.springframework.web.bind.WebDataBinder;
import org.springframework.web.bind.annotation.InitBinder;
import org.springframework.web.bind.annotation.ModelAttribute;
import org.springframework.web.bind.annotation.PathVariable;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestParam;
import org.springframework.web.multipart.MultipartFile;
import org.springframework.web.multipart.support.ByteArrayMultipartFileEditor;
import org.springframework.web.servlet.ModelAndView;

import com.nu.dms.couchdb.ektorp.dao.DocumentRepository;
import com.nu.dms.couchdb.ektorp.model.Document;
import com.nu.web.ControllerContext;
import com.nu.web.RootController;
import com.nu.web.WebConstants;
import com.nu.web.validator.BeanValidator;
import com.windriver.gson.extension.GeneralObjectDeserializer;

@Controller
@RequestMapping("/dms/document/*")
public class DocumentController extends RootController {
    private static final Logger log = LoggerFactory.getLogger(DocumentController.class);
    
    @Autowired
    DocumentRepository documentRepository;
    
    @Autowired
    private BeanValidator validator;
    
    private static final String LIST_PATH = "/dms/document/list";
    private static final String FORM_PATH = "/dms/document/form";
    private static final String TREE_PATH = "/dms/document/tree";
    public static final long UPLOAD_MAX_FILE_SIZE = 20 * 1024 * 1024; //10 MB
    public static final long UPLOAD_MAX_TOTAL_FILES_SIZE = 1 * 1024 * 1024 * 1024; //1 GB
    private static final int DOCUMENT_LEVEL = 5;
    
    @RequestMapping("/list")
    public ModelAndView list(   HttpServletRequest request, 
                                HttpServletResponse response, 
                                Model m, 
                                @RequestParam(value = "p", required = false) String pageLink) {
        ControllerContext ctx = new ControllerContext(request, response);
        init(ctx);

        if (!isValidCsrfToken(ctx)) {
            return getModelAndView(ctx, LIST_PATH);
        }
        
        PageRequest pr = pageLink != null ? PageRequest.fromLink(pageLink) : PageRequest.firstPage(5);
        m.addAttribute(documentRepository.getAll(pr));
        return getModelAndView(ctx, LIST_PATH);
    }
    
    @RequestMapping("/add")
    public ModelAndView add(HttpServletRequest request,
            HttpServletResponse response,
            @RequestParam(value = "attachment", required = false) MultipartFile multipartFile,
            @ModelAttribute("document") Document document,
            BindingResult result) {

        //Store constants for JSP
        request.setAttribute("UPLOAD_MAX_FILE_SIZE", UPLOAD_MAX_FILE_SIZE); 
        
        ControllerContext ctx = new ControllerContext(request, response);
        init(ctx);

        if (!isValidCsrfToken(ctx)) {
            return getModelAndView(ctx, FORM_PATH);
        }

        if (!isSubmission(ctx)) {
            return getModelAndView(ctx, FORM_PATH);
        } else {
            
            validator.validate(document, result);
            
            if (result.hasErrors()) {
                return getModelAndView(ctx, FORM_PATH);
            } else {
                if(multipartFile == null) {
                    result.addError(new ObjectError("document", getMessage("error.add", new String[] {"document"})));
                    return getModelAndView(ctx, FORM_PATH);
                }
                
                try {
                    String title = document.getTitle();
                    if(StringUtils.isEmpty(title)) throw new Exception("Empty title");
                    document.setId(title);
                    String contentType = multipartFile.getContentType();
                    String base64 = new String (Base64.encodeBase64(multipartFile.getBytes()));
                    if(StringUtils.isEmpty(base64)) throw new Exception("Empty attachment");
                    Attachment a = new Attachment(title, base64, contentType);
                    document.addInlineAttachment(a);
                } catch (Exception ex) {
                    result.addError(new ObjectError("attachmentError", getMessage("error.attachingDocument")));
                    log.error(null, ex);
                    return getModelAndView(ctx, FORM_PATH);
                }   
                
                try {
                    document.setDateCreated(new Date());
                    documentRepository.add(document);
                } catch (Exception ex) {
                    result.addError(new ObjectError("document", getMessage("error.add", new String[] {"document"})));
                    log.error(null, ex);
                    return getModelAndView(ctx, FORM_PATH);
                }
                return getModelAndView(ctx, LIST_PATH, true, true);
            }
        }
    }
    
    
    @RequestMapping("/{id}/edit")
    public ModelAndView edit(HttpServletRequest request,
            HttpServletResponse response,
            @ModelAttribute("document") Document document,
            BindingResult result,
            @PathVariable("id") String id,
            Model model) {

        ControllerContext ctx = new ControllerContext(request, response);
        init(ctx);

        if (!isValidCsrfToken(ctx)) {
            return getModelAndView(ctx, FORM_PATH);
        }

        try {
            Document storedDocument = documentRepository.get(id);
            if(storedDocument == null) {
                throw new Exception("No document found with id '" + id + "'");
            }
           
            if (!isSubmission(ctx)) {
                model.addAttribute("document", storedDocument);
                return getModelAndView(ctx, FORM_PATH);
            } else {
                validator.validate(document, result);
                
                if (result.hasErrors()) {
                    return getModelAndView(ctx, FORM_PATH);
                } else {
                    String title = document.getTitle();
                    if(StringUtils.isEmpty(title)) throw new Exception("Empty title");
                    document.setId(title);
                    document.setDateCreated(storedDocument.getDateCreated());
                    document.setRevision(storedDocument.getRevision());
                    documentRepository.update(document);
                    return getModelAndView(ctx, LIST_PATH, true, true);
                }
            }
        } catch (Exception ex) {
            result.addError(new ObjectError("document", getMessage("error.edit", new String[] {"document"}) + "." + ex.getMessage()));
            log.error(null, ex);
            return getModelAndView(ctx, FORM_PATH);
        }
    }
    
    
    @RequestMapping("/{id}/delete")
    public ModelAndView delete(HttpServletRequest request,
            HttpServletResponse response,
            @ModelAttribute("document") Document document,
            BindingResult result,
            @PathVariable("id") String id,
            Model model) {
        ControllerContext ctx = new ControllerContext(request, response);
        init(ctx);

        if (!isValidCsrfToken(ctx)) {
            return getModelAndView(ctx, LIST_PATH);
        }

        try {
            document = documentRepository.get(id);
            if(document == null) {
                throw new Exception("No document found with id '" + id + "'");
            }
            documentRepository.remove(document);
        } catch (Exception ex) {
            result.addError(new ObjectError("document", getMessage("error.add", new String[] {"document"})));
            log.error(null, ex);
            return getModelAndView(ctx, LIST_PATH);
        }
        return getModelAndView(ctx, LIST_PATH, true, true);
    }
    
    @RequestMapping("/{id}/show")
    public ModelAndView show(HttpServletRequest request,
            HttpServletResponse response,
            @ModelAttribute("document") Document document,
            BindingResult result,
            @PathVariable("id") String id,
            Model model) {
        ControllerContext ctx = new ControllerContext(request, response);
        init(ctx);

        if (!isValidCsrfToken(ctx)) {
            return getModelAndView(ctx, LIST_PATH);
        }

        try {
            document = documentRepository.get(id);
            if(document == null) {
                throw new Exception("No document found with id '" + id + "'");
            }
            Map<String, Attachment> attachments = document.getAttachments();
            if(attachments == null || attachments.size() == 0) {
                throw new Exception("No attachment found for id '" + id + "'");
            }
            for(Map.Entry<String, Attachment> entry : attachments.entrySet()) {
                String attachmentId = entry.getKey();
                Attachment attachment = entry.getValue();
                //long contentLength = attachment.getContentLength();
                String contentType = attachment.getContentType();
                AttachmentInputStream ais = documentRepository.getDb().getAttachment(id, attachmentId);
                response.setHeader("Content-Disposition", "attachment; filename=\"" + document.getTitle() + "\"");
                response.setContentType(contentType);
                final ByteArrayOutputStream outputStream = new ByteArrayOutputStream();
                IOUtils.copy(ais, outputStream);
                render(response, outputStream);
            }
            return getModelAndView(ctx, LIST_PATH, true, true);
        } catch (Exception ex) {
            result.addError(new ObjectError("document", getMessage("error.internal", new String[] {"document"})));
            log.error(null, ex);
            return getModelAndView(ctx, LIST_PATH);
        }
        
    }
    
    @RequestMapping("/tree")
    public ModelAndView tree(HttpServletRequest request,
            HttpServletResponse response,
            @RequestParam(value = "root", required = false) String root) {
        ControllerContext ctx = new ControllerContext(request, response);
        init(ctx);

        if (!isValidCsrfToken(ctx)) {
            return getModelAndView(ctx, TREE_PATH);
        }
        if(root == null) {
            return getModelAndView(ctx, TREE_PATH);
        }
        try {
            Object objTree = getTreeObject(root);
            if(objTree == null) {
                ctx.setRequestAttribute("treeInfo", getMessage("noItemFound", new String[] {"record"}));
            }
            ctx.setRequestAttribute("tree", objTree);
            return getModelAndView(ctx, TREE_PATH);
        } catch (Exception ex) {
            ctx.setRequestViewAttribute("treeError", ex.getMessage());
            log.error(null, ex);
            return getModelAndView(ctx, TREE_PATH);
        }
        
    }
    
    /**
     * The needs for the current treeview plugin makes mandatory certain json structure so parsing the /document/tree service is not an option at the moment
     * @param request
     * @param response
     * @param root
     * @return
     */
    @RequestMapping("/ajaxTree")
    public ModelAndView ajaxTree(HttpServletRequest request,
            HttpServletResponse response,
            @RequestParam(value = "root", required = true) String root) {
        ControllerContext ctx = new ControllerContext(request, response);
        init(ctx);

        if (!isValidCsrfToken(ctx)) {
            return getModelAndView(ctx, LIST_PATH);
        }

        try {
            InputStream is = getTreeInputStream(root);
            ObjectMapper mapper = new ObjectMapper();
            JsonNode rootNode = mapper.readValue(is, JsonNode.class);
            JsonNode rowsNode = rootNode.get("rows");
            Iterator<JsonNode> iter = rowsNode.getElements();
            while (iter.hasNext()) {
                JsonNode row = iter.next();
                JsonNode keyNode = row.get("key");
                String[] key = mapper.readValue(keyNode, String[].class);
                if(key == null) {
                    continue;
                }
                String name = key[key.length - 1];
                String classes = null;
                if(key.length == DOCUMENT_LEVEL + 1) {
                    //Listing files
                    String extension = "unknownExtension";
                    String fileName = key[DOCUMENT_LEVEL];
                    String[] tokens = fileName.split("\\.");
                    if(tokens.length >= 2) {
                        extension = tokens[tokens.length - 1];
                    }
                    classes = "file " + extension;
                } else {
                    //Listing folders
                    classes = "folder";
                }
                
                ((ObjectNode)row).put("classes", classes);
                ((ObjectNode)row).put("name", name);
                boolean hasChildren = false;
                //To use the key as id for next request
                if(key.length != DOCUMENT_LEVEL + 1) {
                    int value = mapper.readValue(row.get("value"), Integer.class);
                    if(value > 0) {
                        hasChildren = true;
                    }
                } else {
                    ((ObjectNode)row).put("url", request.getContextPath() + "/dms/document/" + name + "/show?ctoken=" + ctx.getSessionAttribute(WebConstants.CSRF_TOKEN, ""));
                }
                ((ObjectNode)row).put("hasChildren", hasChildren);
                ((ObjectNode)row).put("id", keyNode.toString().replaceAll("[\\[\\]]", ""));
                
            }
            
            response.setContentType("application/json");
            final ByteArrayOutputStream outputStream = new ByteArrayOutputStream();
            mapper.writeValue(outputStream, rowsNode);
            render(response, outputStream);
            return null;
        } catch (Exception ex) {
            log.error(null, ex);
            return null;
        }
        
    }
    
    private Object getTreeObject(String root) {
        InputStream is = getTreeInputStream(root);
        String tree = new Scanner(is).useDelimiter("\\A").next();
        Object objTree = GeneralObjectDeserializer.fromJson(tree);
        return objTree;
    }
    
    private InputStream getTreeInputStream(String root) {
        //Making it compatible with the jquery tree view in use by Portal in Liferay
        String endKey = null;
        if("0".equals(root)) {
            endKey = "[{}]";
        } else {
            endKey = "[" + root.substring(0,root.length()) + ",{}]";
        }
         
        String[] tokens = endKey.replaceAll("[\\[\\]]", "").split(",");
        int groupLevel = tokens.length;
        String startKey = "[1]";
        if(!"{}".equals(tokens[0])) {
            startKey = "[" + root + "]";
        }
        InputStream is = documentRepository.getTree(startKey, endKey, groupLevel);
        return is;
    }
    
    @InitBinder
    public void initBinder(WebDataBinder binder) {
        binder.registerCustomEditor(byte[].class, new ByteArrayMultipartFileEditor());
    }

    
    /**
     * To be consumed by swfupload.swf which is in charge of uploading multiple files
     * 
     * @param request
     * @param response
     * @return
     */
    @RequestMapping("/addBatch")
    public void addBatch(HttpServletRequest request,
            HttpServletResponse response,
            @RequestParam(value = "Filedata", required = false) MultipartFile multipartFile,
            @ModelAttribute("document") Document document,
            BindingResult result) {

        String message = "Completed";
        ControllerContext ctx = new ControllerContext(request, response);
        init(ctx);
        
        try {
            if (!isValidCsrfToken(ctx)) {
                message = error("Invalid session token");
            } else {
                uploadFile(request, multipartFile);
            }
            
        } catch (FileUploadException e) {
            log.error("FileUploadException:", e);
            message = error(e.getMessage());
        } catch (Exception e) {
            String errorMessage = e.toString();
            log.error("FileUploadException:", e);
            if (errorMessage == null) {
                errorMessage = "Internal Error. Please look at server logs for more detail";
            }
            message = error(errorMessage);
        }
        
        try {
            render(response, message.getBytes());
        } catch (Exception ex) {
            log.error(null, ex);
        }
    }
    
    /**
     * File must follow this convention: clientId_categoryId_subCategoryId_year_month_investorId.extension
     * @param req
     * @param multipartFile
     * @throws Exception
     */
    private void uploadFile(HttpServletRequest req, MultipartFile multipartFile) throws Exception {

            // Create a new file upload handler
            FileItemFactory factory = new DiskFileItemFactory();
            ServletFileUpload upload = new ServletFileUpload(factory);
            upload.setFileSizeMax(UPLOAD_MAX_FILE_SIZE);
            upload.setSizeMax(UPLOAD_MAX_TOTAL_FILES_SIZE);

            if(multipartFile == null) {
                throw new FileUploadException(getMessage("error.add", new String[] {"document"}));
            }
            
            String fileName = multipartFile.getOriginalFilename();
            String[] tokens = fileName.split("_");
            if(tokens.length != 6) {
                throw new Exception("Filename must have 6 tokens");
            }
            int clientId = Integer.parseInt(tokens[0]);
            int categoryId = Integer.parseInt(tokens[1]);
            int subCategoryId = Integer.parseInt(tokens[2]);
            String year = tokens[3];
            String month = tokens[4];
            //Use whole filename as title
            int investorId = Integer.parseInt(tokens[5].split("\\.")[0]);
            String title = fileName;
            //Using swfupload we almost always get "application/octet-stream"
            String contentType = multipartFile.getContentType();
            String guessedContentType = URLConnection.guessContentTypeFromName(fileName);
            if (guessedContentType != null) {
                contentType = guessedContentType;
            }
            String base64 = new String (Base64.encodeBase64(multipartFile.getBytes()));
            if(StringUtils.isEmpty(base64)) throw new Exception("Empty attachment");
            Attachment a = new Attachment(fileName, base64, contentType);
            Document document = new Document(title);
            document.setId(fileName);
            document.addInlineAttachment(a);
            document.setDateCreated(new Date());
            document.setClientId(clientId);
            document.setCategoryId(categoryId);
            document.setSubCategoryId(subCategoryId);
            document.setDateEffective(new SimpleDateFormat("yyyy-MM-dd", Locale.ENGLISH).parse(year + "-" + month + "-" + 1));
            document.setInvestorId(investorId);
            documentRepository.add(document);
            
            //Within Spring the below does not work
            /*
            List<FileItem> items = (List<FileItem>) upload.parseRequest(req);
            Iterator<FileItem> iter = items.iterator();
            while (iter.hasNext()) {
                FileItem item = iter.next();
                if (item.isFormField()) {
                    String name = item.getFieldName();
                    String value = item.getString();
                    log.debug("Form field " + name + " with value " + value);
                } else {
                    String fileName = item.getName();
                    String contentType = item.getContentType();
                    Document document = new Document(fileName);
                    String base64 = new String (Base64.encodeBase64(item.get()));
                    if(StringUtils.isEmpty(base64)) throw new Exception("Empty attachment");
                    Attachment a = new Attachment(fileName, base64, contentType);
                    document.addInlineAttachment(a);
                    document.setDateCreated(new Date());
                    documentRepository.add(document);
                }
            }
            */
    }
    /*
     * To format the message so sfwupload understands it
     */
    private String error(String error) {
        return "ERROR: " + error;
    }
    
    
}

Below are some screenshots of our simple user interface:



Why CouchDB? and not a SQL database?

Relational Database Management Systems (RDBMS) are good to store tabular data, enforce relationship, remove duplicated information, ensure data consistency and the list goes on and on. There is one thing though that makes relational databases not ideal for distributed computing and that is locking. The need for an alternative comes from impediments related to replication but also from storing hierarchical structures in RDBMS which is not natural. Finally it is difficult to manage inheritance and schema changes can easily become a big problem when the system grows and new simple necessities emerge from business requirements. RDBMS are also slow if you want an index for all fields in a table or multiple complex indexes. If your content management system has a need for distributed computing, fast storage and you can afford the compromise about losing the ability for easy normalization (for example your documents once created are stamped with metadata available at that time and which does not change over time)

CouchDB is a noSql type database which stores data structures as documents (JSON Strings). Schema less, based on b-tree and with no locking (but Multi Version Concurrency Control) design it is a really fast alternative when you look for a solution to store hierarchical non-strict schema data like for example storing web pages or binary files. All this robustness is exposed with a HTTP REST API where JSON is used to send messages to the server as well as receive messages from it. This makes it really attractive for those looking for lightweight solutions. One of the "trade-offs" in couchDB is that the only way to query CouchDB without creating indexes is a temporary View and that is not an option for production as the mapped result will not be stored in a B-Tree hitting performance in your server.

I have no other option than considering CouchDB the logical pick for my BHUB Document Management functionality. CouchDB provides fast access to data specified by stored keys. Using Map functions in your Views there is no limit on the efficiency you can get out of the fact that you can create at any point a key composed of several document fields.

CouchDB has been engineered with replication in mind and that means you get distributed computing on top of the advantages I already discussed above. You just run a command specifying URLs for the source and the destination server and the replication is done. By default latest changes will be favored and if there are conflicts you will get the differences so you can update with changes that resolve the changes. Think about any versioning system like subversion for a comparison on how it works. You can replicate in both directions of course.

Followers