Monday, August 29, 2011

Transform XML with XSLT in Talend

Talend Open Studio is an excellent ETL tool that can be used beyond the typical database and CSV manipulation. XML processing for example is today all over the places in the Enterprise.

As a consequence XML transformations are a key skill for those folks doing data transformations.

Even though there are more efficient tools XSLT is a standard which is supported in Talend through the tXSLT component. You just need to provide your XML, XSL and output files and Talend will apply the transformation for you. Talend uses Saxon at the moment so you get the benefit of clear error messages when trying to build your XSL.

Of course XSLT might be a skill that even scares some people, however XSLT is not difficult at all and the more you work with it the better you get as with any other human skill. Do not try to avoid it, if you have XML to process and your ETL tool is Talend then do your homework and learn some XSL.

As there is no better way to teach than providing an example I decided to write this quick showcase that will pivot the data resulting from running an Advent Geneva RSL report (A SOAP service) which comes in the form of key value pairs into a tabular output. I will provide two responses: XML and HTML. The first is probably what you need in Talend while the second is probably what you need if you want to provide a quick add hoc HTML report page.

Here is the XML:
<?xml version='1.0' encoding='UTF-8'?>
<SOAP-ENV:Envelope xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/" xmlns:SOAP-ENC="http://schemas.xmlsoap.org/soap/encoding/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:nsg="http://geneva.advent.com">
	<SOAP-ENV:Body id="_0">
		<reportResults xmlns="http://geneva.advent.com">
			<return xsi:type="nsg:reportResultsPortfolioStruct">
				<results xsi:type="nsg:reportResultsStruct">
					<portfolioName xsi:type="xsd:string">Fund1</portfolioName>
					<header xsi:type="nsg:reportResultsRecordStruct">
						<field xsi:type="nsg:reportResultsVectorElement">
							<name xsi:type="xsd:string">Head</name>
							<value xsi:type="xsd:string">Fund 1 Example</value>
						</field>
						<field xsi:type="nsg:reportResultsVectorElement">
							<name xsi:type="xsd:string">Head</name>
							<value xsi:type="xsd:string">DIVIDENDS RECEIVABLE AND PAYABLE</value>
						</field>
						<field xsi:type="nsg:reportResultsVectorElement">
							<name xsi:type="xsd:string">Head</name>
							<value xsi:type="xsd:string">FOR THE PERIOD INCEPTION TO July 31, 2011</value>
						</field>
						<field xsi:type="nsg:reportResultsVectorElement">
							<name xsi:type="xsd:string">Head</name>
							<value xsi:type="xsd:string"/>
						</field>
					</header>
					<record xsi:type="nsg:reportResultsRecordStruct">
						<field xsi:type="nsg:reportResultsVectorElement">
							<name xsi:type="xsd:string">CcyCode</name>
							<value xsi:type="xsd:string">BRL</value>
						</field>
						<field xsi:type="nsg:reportResultsVectorElement">
							<name xsi:type="xsd:string">Custodian</name>
							<value xsi:type="xsd:string">My Custodian</value>
						</field>
						<field xsi:type="nsg:reportResultsVectorElement">
							<name xsi:type="xsd:string">IDesc</name>
							<value xsi:type="xsd:string">My IDesc</value>
						</field>
						<field xsi:type="nsg:reportResultsVectorElement">
							<name xsi:type="xsd:string">TransID</name>
							<value xsi:type="xsd:string">10145834</value>
						</field>
						<field xsi:type="nsg:reportResultsVectorElement">
							<name xsi:type="xsd:string">TDate</name>
							<value xsi:type="xsd:string">March 15, 2011 12:00:00 am</value>
						</field>
						<field xsi:type="nsg:reportResultsVectorElement">
							<name xsi:type="xsd:string">SDate</name>
							<value xsi:type="xsd:string">December 31, 9999 11:59:59 pm</value>
						</field>
						<field xsi:type="nsg:reportResultsVectorElement">
							<name xsi:type="xsd:string">Qty</name>
							<value xsi:type="xsd:string">13528.013</value>
						</field>
						<field xsi:type="nsg:reportResultsVectorElement">
							<name xsi:type="xsd:string">PerShare</name>
							<value xsi:type="xsd:string">0.15100000</value>
						</field>
						<field xsi:type="nsg:reportResultsVectorElement">
							<name xsi:type="xsd:string">TaxRate</name>
							<value xsi:type="xsd:string"/>
						</field>
						<field xsi:type="nsg:reportResultsVectorElement">
							<name xsi:type="xsd:string">DivLocal</name>
							<value xsi:type="xsd:string">2042.73</value>
						</field>
						<field xsi:type="nsg:reportResultsVectorElement">
							<name xsi:type="xsd:string">TaxLocal</name>
							<value xsi:type="xsd:string">0.00</value>
						</field>
						<field xsi:type="nsg:reportResultsVectorElement">
							<name xsi:type="xsd:string">DivLocalNet</name>
							<value xsi:type="xsd:string">2042.73</value>
						</field>
						<field xsi:type="nsg:reportResultsVectorElement">
							<name xsi:type="xsd:string">DivBook</name>
							<value xsi:type="xsd:string">1225.76</value>
						</field>
						<field xsi:type="nsg:reportResultsVectorElement">
							<name xsi:type="xsd:string">TaxBook</name>
							<value xsi:type="xsd:string">0.00</value>
						</field>
						<field xsi:type="nsg:reportResultsVectorElement">
							<name xsi:type="xsd:string">DivBookNet</name>
							<value xsi:type="xsd:string">1225.76</value>
						</field>
						<field xsi:type="nsg:reportResultsVectorElement">
							<name xsi:type="xsd:string">UnrealFXGL</name>
							<value xsi:type="xsd:string">0.00</value>
						</field>
						<field xsi:type="nsg:reportResultsVectorElement">
							<name xsi:type="xsd:string">BookBal</name>
							<value xsi:type="xsd:string">1225.76</value>
						</field>
					</record>
					<record xsi:type="nsg:reportResultsRecordStruct">
						<field xsi:type="nsg:reportResultsVectorElement">
							<name xsi:type="xsd:string">CcyCode</name>
							<value xsi:type="xsd:string">USD</value>
						</field>
						<field xsi:type="nsg:reportResultsVectorElement">
							<name xsi:type="xsd:string">Custodian</name>
							<value xsi:type="xsd:string">My Custodian</value>
						</field>
						<field xsi:type="nsg:reportResultsVectorElement">
							<name xsi:type="xsd:string">IDesc</name>
							<value xsi:type="xsd:string">My IDesc</value>
						</field>
						<field xsi:type="nsg:reportResultsVectorElement">
							<name xsi:type="xsd:string">TransID</name>
							<value xsi:type="xsd:string">10756740</value>
						</field>
						<field xsi:type="nsg:reportResultsVectorElement">
							<name xsi:type="xsd:string">TDate</name>
							<value xsi:type="xsd:string">April 27, 2011 12:00:00 am</value>
						</field>
						<field xsi:type="nsg:reportResultsVectorElement">
							<name xsi:type="xsd:string">SDate</name>
							<value xsi:type="xsd:string">July 1, 2011 12:00:00 am</value>
						</field>
						<field xsi:type="nsg:reportResultsVectorElement">
							<name xsi:type="xsd:string">Qty</name>
							<value xsi:type="xsd:string">205212.046</value>
						</field>
						<field xsi:type="nsg:reportResultsVectorElement">
							<name xsi:type="xsd:string">PerShare</name>
							<value xsi:type="xsd:string">0.16918500</value>
						</field>
						<field xsi:type="nsg:reportResultsVectorElement">
							<name xsi:type="xsd:string">TaxRate</name>
							<value xsi:type="xsd:string"/>
						</field>
						<field xsi:type="nsg:reportResultsVectorElement">
							<name xsi:type="xsd:string">DivLocal</name>
							<value xsi:type="xsd:string">34718.80</value>
						</field>
						<field xsi:type="nsg:reportResultsVectorElement">
							<name xsi:type="xsd:string">TaxLocal</name>
							<value xsi:type="xsd:string">0.00</value>
						</field>
						<field xsi:type="nsg:reportResultsVectorElement">
							<name xsi:type="xsd:string">DivLocalNet</name>
							<value xsi:type="xsd:string">34718.80</value>
						</field>
						<field xsi:type="nsg:reportResultsVectorElement">
							<name xsi:type="xsd:string">DivBook</name>
							<value xsi:type="xsd:string">22153.39</value>
						</field>
						<field xsi:type="nsg:reportResultsVectorElement">
							<name xsi:type="xsd:string">TaxBook</name>
							<value xsi:type="xsd:string">0.00</value>
						</field>
						<field xsi:type="nsg:reportResultsVectorElement">
							<name xsi:type="xsd:string">DivBookNet</name>
							<value xsi:type="xsd:string">22153.39</value>
						</field>
						<field xsi:type="nsg:reportResultsVectorElement">
							<name xsi:type="xsd:string">UnrealFXGL</name>
							<value xsi:type="xsd:string">0.00</value>
						</field>
						<field xsi:type="nsg:reportResultsVectorElement">
							<name xsi:type="xsd:string">BookBal</name>
							<value xsi:type="xsd:string">23379.15</value>
						</field>
					</record>
					<addendumErrors xsi:type="nsg:reportResultsRecordStruct">
						<field xsi:type="nsg:reportResultsVectorElement">
							<name xsi:type="xsd:string">BegDesc</name>
							<value xsi:type="xsd:string"/>
						</field>
						<field xsi:type="nsg:reportResultsVectorElement">
							<name xsi:type="xsd:string">EndDesc</name>
							<value xsi:type="xsd:string">DIVIDENDS RECEIVABLE - CLOSING BALANCE</value>
						</field>
						<field xsi:type="nsg:reportResultsVectorElement">
							<name xsi:type="xsd:string">BegBal</name>
							<value xsi:type="xsd:string"/>
						</field>
						<field xsi:type="nsg:reportResultsVectorElement">
							<name xsi:type="xsd:string">EndBal</name>
							<value xsi:type="xsd:string">71643.79</value>
						</field>
					</addendumErrors>
					<addendumErrors xsi:type="nsg:reportResultsRecordStruct">
						<field xsi:type="nsg:reportResultsVectorElement">
							<name xsi:type="xsd:string">BegDesc</name>
							<value xsi:type="xsd:string"/>
						</field>
						<field xsi:type="nsg:reportResultsVectorElement">
							<name xsi:type="xsd:string">EndDesc</name>
							<value xsi:type="xsd:string">DIVIDENDS PAYABLE - CLOSING BALANCE</value>
						</field>
						<field xsi:type="nsg:reportResultsVectorElement">
							<name xsi:type="xsd:string">BegBal</name>
							<value xsi:type="xsd:string"/>
						</field>
						<field xsi:type="nsg:reportResultsVectorElement">
							<name xsi:type="xsd:string">EndBal</name>
							<value xsi:type="xsd:string">-20200.35</value>
						</field>
					</addendumErrors>
				</results>
			</return>
		</reportResults>
	</SOAP-ENV:Body>
</SOAP-ENV:Envelope>

If you open it with Firefox and you choose "View XPath" from a right click on the body of the page (contextual menu) you could try several XPATH expresions. Together with XSLT skill it comes XPATH which is just a way to address a node, an attribute or textual content in the XML. See below how I tested one of the XPATH using this tool. Pay attention to the namespace definition, I use simple letters to abbreviate more verbose prefixes.


Now that you have a quick tool for finding nodes in the XML document let us see the desired document structure. Here is a screenshot of what we would like to see in HTML:

Here is in the XML we would like to obtain for further processing in Talend:

Here is the XSL that will output HTML:
<?xml version="1.0"?>
<xsl:stylesheet version="1.0"
  xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
  xmlns:e="http://schemas.xmlsoap.org/soap/envelope/"
  xmlns:g="http://geneva.advent.com">
    <xsl:output omit-xml-declaration="yes" indent="yes"/>

    <xsl:strip-space elements="*"/>

    <xsl:key name="kFieldNameByValue" match="/e:Envelope/e:Body/g:reportResults/g:return/g:results/g:record/g:field/g:name"
         use="."/>

    <xsl:variable name="vCols" select=
       "/e:Envelope/e:Body/g:reportResults/g:return/g:results/g:record/g:field/g:name
                [generate-id()
                =          
                 generate-id(key('kFieldNameByValue',.)[1])
                 ]"/>

    <xsl:template match="/">
             <table>
               <tr>
                 <xsl:apply-templates select="$vCols"/>
               </tr>

               <xsl:for-each select=
                 "/e:Envelope/e:Body/g:reportResults/g:return/g:results/g:record">                   
                 <tr>

                   <xsl:variable name="vPos" select="position()"/>

                   <xsl:for-each select="$vCols">
                     <td>
                       <xsl:value-of select=
                           "../../../g:record[$vPos]/g:field[g:name = current()]/g:value"/>
                     </td>
                   </xsl:for-each>

                 </tr>
              </xsl:for-each>
            </table>

    </xsl:template>

    <xsl:template match="/e:Envelope/e:Body/g:reportResults/g:return/g:results/g:record/g:field/g:name">
            <td>
              <xsl:value-of select="." />
            </td>   
    </xsl:template>
</xsl:stylesheet>

Here is the XSL that will output XML:
<?xml version="1.0"?>
<xsl:stylesheet version="1.0"
  xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
  xmlns:e="http://schemas.xmlsoap.org/soap/envelope/"
  xmlns:g="http://geneva.advent.com" 
>
    <xsl:output omit-xml-declaration="no" indent="yes"/>

    <xsl:strip-space elements="*"/>

    <xsl:key name="kFieldNameByValue" match="/e:Envelope/e:Body/g:reportResults/g:return/g:results/g:record/g:field/g:name"
         use="."/>

    <xsl:variable name="vCols" select=
       "/e:Envelope/e:Body/g:reportResults/g:return/g:results/g:record/g:field/g:name
                [generate-id()
                =          
                 generate-id(key('kFieldNameByValue',.)[1])
                 ]"/>

    <xsl:template match="/">
	  	<xsl:element name="root">
               	<xsl:element name="header">
				  <xsl:apply-templates select="$vCols"/>
				</xsl:element>  

               <xsl:for-each select="/e:Envelope/e:Body/g:reportResults/g:return/g:results/g:record">                   
                 <xsl:element name="record">
                   <xsl:variable name="vPos" select="position()"/>
                   <xsl:for-each select="$vCols">
                       <xsl:element name="value"><xsl:value-of select="../../../g:record[$vPos]/g:field[g:name = current()]/g:value"/></xsl:element>
                   </xsl:for-each>
                 </xsl:element>
              </xsl:for-each>

        </xsl:element>
    </xsl:template>

    <xsl:template match="/e:Envelope/e:Body/g:reportResults/g:return/g:results/g:record/g:field/g:name">
            <xsl:element name="name"><xsl:value-of select="."/></xsl:element>
    </xsl:template>

</xsl:stylesheet>

A quick explanation of what the XSL code does:
1-6 Is just stating it is an XML document which uses a namespace prefixed "xsl" for the transaformation instructions, a soap namespace with prefix "e" and an Advent Geneva namespace with prefix "g". Note that I abreviated the original namespace suffixes for the last two.

7-9 How the final output should look like.

11-19 We use the Muenchian grouping that allows us to have a list of all possible column names.

21-37 XSLT is a functional language which works matching nodes and applying transformations to them. The logic can be affected using xsl:apply-template. We use xsl:element to create our custom nodes: First the headers which come out of the Muenchian key and later the records for which we again use the keys while addressing the correct record through an xsl:foreach nested loop.

39-41 The template responsible for generating the name nodes.

Saturday, August 27, 2011

Twitter RSS: Follow silently

I blog about the work that I do just because "the palest ink is better than the best memory"

I tweet about most of the stuff that I do or I am interested in just because I think for some people is just easier to get an SMS alike message with a title that makes you decide if the article might be of your interest.

I consume most of the information I am able to digest via RSS.

These three statements probably make me not a good social network user. I would love to have time to follow 2000 people, interact with them, engage in interesting conversations and so on but I don't believe that is possible for a first generation immigrant, father of three kids and with the hope to remain competitive in the fast paced World of IT.

The way I live and work push me for automation even when consuming information. RSS, Atom or whatever format for syndicated news is then the answer to my needs.

Syndicated feeds should be a mandatory option for any website or service providing information. If you are like me you will love this trick which allows you to silently consolidate in your News reader (I use Google Reader BTW) tweets from people you would like to follow.

With an example here is how you can consume my tweets from your favorite reader:

http://api.twitter.com/1/statuses/user_timeline.rss?screen_name=nestorurquiza

I make my statements public because I do care about what others think. If you consume them via RSS, email, native program or any other means that is secondary for me.

Thursday, August 25, 2011

TSQL Stored Procedure faster from SSMS than the application

No, I have no quick fix for this because SQL Server is not about magic and there is a reason why there are guys that work 100% of the time as SQL developers.

You have to ensure your queries are optimized and you know what query plan will be used when they are run. This is not about Java or .NET problems because your SQL Server Management Studio (SSMS) is "apparently" running the query 10 or 20 times faster than the application. It is indeed about your SQL code optimizations.

So if you have this problem then run the stored procedure using something like:
SET ARITHABORT OFF
EXEC sp_custom_procedure 'param1', 2, 'param3'
GO

Then run it again like:
SET ARITHABORT ON
EXEC sp_custom_procedure 'param1', 2, 'param3'
GO

You should get a slower response for the first and that will only tell you that your stored procedure is using a non efficient query plan. Now it is time to do your homework reading the previous link and make your stored procedure run with a custom well predefined fast and efficient query plan (of course as fast and efficient as you can)

Error creating bean with name 'liferayTransactionManager' defined in class path resource [META-INF/hibernate-spring.xml]

Every once in a while when I install Liferay in different servers I get something like the below:
15:39:20,176 ERROR [ContextLoader:215] Context initialization failed
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'transactionAdvice' defined in class path resource [META-INF/base-spring.xml]: Cannot resolve reference to bean 'liferayTransactionManager' while setting bean property 'transactionManager'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'liferayTransactionManager' defined in class path resource [META-INF/hibernate-spring.xml]: Cannot resolve reference to bean 'liferayHibernateSessionFactory' while setting bean property 'sessionFactory'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'liferayHibernateSessionFactory' defined in class path resource [META-INF/hibernate-spring.xml]: Invocation of init method failed; nested exception is org.hibernate.HibernateException: Hibernate Dialect must be explicitly set

This is an error that relates to a connectivity problem with your database. Be sure you see the *whole* stacktrace and look for a clue about connection/permission problems. Commonly you should be able to get it right if you follow these directions (Using MySQL here):

  1. Check what IP MySQL is listening to. Below it shows as listening in 192.168.0.6
    $ netstat -an|grep 3306
    tcp        0      0 192.168.0.6:3306        0.0.0.0:*               LISTEN     
    
  2. Try connecting with the same host and credentials Liferay is using from command line mysql client:
    $ mysql -u liferay -p -h 192.168.0.6
    Enter password: 
    ERROR 1130 (HY000): Host '192.168.0.6' is not allowed to connect to this MySQL server
    
  3. As the example above shows there is a permision problem so go ahead and correct it. In MySQL password must be specified per each granted user@host combination:
    $ mysql -u root -p
    ...
    mysql>  GRANT ALL ON lportal.* TO 'liferay'@'192.168.0.6';
    Query OK, 0 rows affected (0.00 sec)
    
  4. Review your /etc/hosts in the client to make sure if you are using a host it maps to the specified IP:
    $ vi /etc/hosts
    192.168.0.6  sqlHost
    

Tuesday, August 23, 2011

nullmailer smtp Failed 550 5.7.1 Unable to relay for user@myhost.myhost big log files

One of our servers was reporting low HDD resources and as usual I took a look at log files. Yes, again /var/log/mail.* were above 1GB, in some cases even 5GB.

This is the error that I saw in the traces:
Aug 23 21:56:33 myhost nullmailer[782]: Starting delivery: protocol: smtp host: mail.nestorurquiza.com file: 1314082502.18221
Aug 23 21:56:33 myhost nullmailer[27145]: smtp: Failed: 550 5.7.1 Unable to relay for nestorurquizaadmin@myhost.myhost
Aug 23 21:56:33 myhost nullmailer[782]: Sending failed:  Permanent error in sending the message

This was originated because in etc/hosts I had a line like:
192.168.1.23 myhost

And the SMTP server was not configured to relay to such a thing like myhost.myhost. So just changing it to an authorized domain should fix it right?
192.168.1.23 myhost.nestorurquiza.com

Well I kept getting the same error!!!

The reason was queued emails. I was getting from logs also a line like below:
Aug 23 22:26:47 myhost nullmailer[828]: Delivery complete, 42489 message(s) remain.

So deleting the queue made finally the trick:
sudo rm -fR  /var/spool/nullmailer/queue/
sudo mkdir  /var/spool/nullmailer/queue

nullmailer smtp Failed 550 5.7.1 Unable to relay for user@myhost.myhost big log files

One of our servers was reporting low HDD resources and as usual I took a look at log files. Yes, again /var/log/mail.* were above 1GB, in some cases even 5GB.

This is the error that I saw in the traces:
Aug 23 21:56:33 myhost nullmailer[782]: Starting delivery: protocol: smtp host: mail.nestorurquiza.com file: 1314082502.18221
Aug 23 21:56:33 myhost nullmailer[27145]: smtp: Failed: 550 5.7.1 Unable to relay for nestorurquizaadmin@myhost.myhost
Aug 23 21:56:33 myhost nullmailer[782]: Sending failed:  Permanent error in sending the message

This was originated because in etc/hosts I had a line like:
192.168.1.23 myhost

And the SMTP server was not configured to relay to such a thing like myhost.myhost. So just changing it to an authorized domain should fix it right?
192.168.1.23 myhost.nestorurquiza.com

Well I kept getting the same error!!!

The reason was queued emails. I was getting from logs also a line like below:
Aug 23 22:26:47 myhost nullmailer[828]: Delivery complete, 42489 message(s) remain.

So deleting the queue made finally the trick:
sudo rm -fR  /var/spool/nullmailer/queue/
sudo mkdir  /var/spool/nullmailer/queue

Friday, August 12, 2011

apache2: Could not reliably determine the server's fully qualified domain name

This error is caused by an apache miss configuration. Below are those cases where I have seen it and how I have corrected them:

Sometimes it is just about duplicated directive ServerName. If you use Virtual hosts do not include a global directive but instead just define it in the virtual hosts files, for example:
...
<VirtualHost  nestorurquiza.com:443>
 SSLEngine on
 DocumentRoot "/var/nestorurquiza-app"
 ServerName nestorurquiza.com
...

Your server IP must be mapped to the server domain in /etc/hosts. It cannot be a loopback IP. Provide a separated line for that entry. Do not include any other mapping for the IP. If you need more then add them *below* the entry:
...
192.168.0.137   nestorurquiza.com
192.168.0.137   nestorurquiza
...

Thursday, August 11, 2011

Install openssl in Ubuntu

sudo apt-get install libcurl4-openssl-dev

Using more than one certificate from the JVM for SSL Web Services LDAPS and more

There is one time when your JVM hosting your servlets will demand connecting to more than one SSL Web Service, an LDAPS, passing in some cases private keys, using in some cases different keystore files and more.

In those cases the typical System properties to set up the keystore and password will apparently not work. You can of course pass the properties as part of the java command line that starts the JVM but you can do the same programatically of course.
System.setProperty("javax.net.ssl.keyStore", "keystore path");
System.setProperty("", "keystore password");
System.setProperty("javax.net.ssl.keyStoreType", keystore type);

You will be tempted to get into a really complicated solution.

However there is a simple approach around this problem. If you are contraint about having just one keystore to host multiple certificates that should not be a big issue. After all it is a key "store". All you need to do is import all your certificates and keys in just one keystore following a procedure like the one I described before.

Again what you need to have clear is how you deal with your keys which might be encrypted with a password and that password must be the same you use for your keystore. At that point if you have let us say just one case for a key using a password you can just use the same for your keystore. If not then negotiate a new common password with your service providers. This is after all your client key, yes it is private but it is private to you. A simple phone call and an agreement on the new password will make the trick. Do not use an email for that though!

We recently spent several days troubleshooting issues like this just because I was relying on "that was already tested and it did not work". It demanded a new developer to take over the task so he could start fresh and try the most basic stuff. Simpler is after all better.

As always log traces are your friend when Google is not. Be sure to activate them for SSL when you are dealing with certificate issues:
-Djavax.net.debug=ssl,handshake

Wednesday, August 10, 2011

Premature Optimization is not Agile or should I say it is Evil

The team was having an issue after introducing JPASecurity in the project. Some feature suddenly broke so it must be JPASecurity which is buggy, still in its early phase, ...

Come on, this was working before so ...

Well the fact that your code is working does not mean it is correct. An Abstract class cannot be instantiated but with so much Dependency Injection and Inversion Of Control sometimes we lose control LOL.

An Abstract class is not to be used by any other than implementing classes. But what happens if the developer declares the class abstract and after that uses it as a JPA Entity? Of course this is a mistake, a common mistake I would say where the engineer thinks he is comming up with a great design that will save our lifes in the future, some kind of generalization up front that will address all details in the future, the silver bullet or a synonym (not about performance but about feature implementation) of what the Industry knows as Premature Optimization.

The class is declared Abstract and there is not a single implementation of it but somehow JPA Hibernate EntityManager.merge(Entity) will not complain (Probably because after all it uses a proxy and not the actual class). All goes "good" until one day when you actually try to use reflection on the entity for example if you introduce JPASecurity to provide ACL in your JPA entities.

At this point you will end up with exceptions like:
java.lang.SecurityException: java.lang.InstantiationException
 at net.sf.jpasecurity.util.ReflectionUtils.throwThrowable(ReflectionUtils.java:109)
 at net.sf.jpasecurity.util.ReflectionUtils.newInstance(ReflectionUtils.java:38)
 at net.sf.jpasecurity.mapping.DefaultClassMappingInformation.newInstance(DefaultClassMappingInformation.java:216)
 at net.sf.jpasecurity.entity.EntityPersister.createUnsecureObject(EntityPersister.java:250)
 at net.sf.jpasecurity.entity.AbstractSecureObjectManager.getUnsecureObject(AbstractSecureObjectManager.java:140)
 at net.sf.jpasecurity.entity.EntityPersister.getUnsecureObject(EntityPersister.java:240)
 at net.sf.jpasecurity.entity.EntityPersister.merge(EntityPersister.java:73)
 at net.sf.jpasecurity.persistence.DefaultSecureEntityManager.merge(DefaultSecureEntityManager.java:130)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 at java.lang.reflect.Method.invoke(Method.java:597)
 at org.springframework.orm.jpa.ExtendedEntityManagerCreator$ExtendedEntityManagerInvocationHandler.invoke(ExtendedEntityManagerCreator.java:365)
 at $Proxy1122.merge(Unknown Source)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 at java.lang.reflect.Method.invoke(Method.java:597)
 at org.springframework.orm.jpa.SharedEntityManagerCreator$SharedEntityManagerInvocationHandler.invoke(SharedEntityManagerCreator.java:240)
 at $Proxy1013.merge(Unknown Source)
 at com.nestorurquiza.dao.CrudDaoImpl.update(CrudDaoImpl.java:105)
 at com.nestorurquiza.service.impl.CrudServiceImpl.update(CrudServiceImpl.java:45)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 at java.lang.reflect.Method.invoke(Method.java:597)
 at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:309)
 at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
 at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
 at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:110)
 at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
 at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
 at $Proxy1037.update(Unknown Source)

This is most likely because the proxy does not implement the default contructor in which case the Abstract class is the one attempted to be instantiated originating the Exception.

But the bottom line of this post is to serve as a little bit of education to those young fellas who are catching up with programming and those experienced ones that love over bloated Enterprise Design up front.

Please guys, try to keep it simple and do not fear refactoring. Embrace change and hard work. You will find yourself applying patterns as a result of pragmatism rather than as a result of that new cool design you learned from GOF alike book on Patterns.

Tuesday, August 09, 2011

BHUB: Allow user and password from any URL

The BHUB philosophy is a unique entry point for all the business logic. A Controller makes that happen and there is no question on the big savings. The logic can be used from a CLI script (even pure curl can do the job), it can be of course croned, it can be used from any device (handhelds, desktops, appliances). You get for free every single security you applied in your web tier and I better stop right here because I have written about this several times.

The common way to interact with BHUB would be to get a session, security token from an entry point let us say a login service and from there on keep requesting using that information.

It would be ideal though that the entry point could be any page. In other words allow the first request to be actually not just login but a service request where you provide login information.

In order to accomplish this we will need to hook into Spring once again. Just a custom filter will do the trick. Declare it and assign some permissions for the URLs. In this case I want to allow direct access to certain role (the API role) and I will be opening that functionality to just the pure CLI Controllers which of course does not mean that we cannot allow this functionality for the complete BHUB Service Suite:

<beans:bean id="customLoginFilter" class="com.nestorurquiza.web.filter.CustomLoginFilter"/>
<custom-filter  before="FORM_LOGIN_FILTER" ref="customLoginFilter" /> 
<http auto-config="true" use-expressions="true" access-decision-manager-ref="accessDecisionManager" disable-url-rewriting="true">
  <intercept-url pattern="/cli/**" access="hasRole('ROLE_API')" />


Here is the Filter code.
package com.nestorurquiza.web.filter;

import java.io.IOException;

import javax.servlet.Filter;
import javax.servlet.FilterChain;
import javax.servlet.FilterConfig;
import javax.servlet.ServletException;
import javax.servlet.ServletRequest;
import javax.servlet.ServletResponse;
import javax.servlet.http.Cookie;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.security.core.Authentication;
import org.springframework.security.core.context.SecurityContext;
import org.springframework.security.core.context.SecurityContextHolder;
import org.springframework.security.web.authentication.UsernamePasswordAuthenticationFilter;
import org.springframework.security.web.authentication.rememberme.AbstractRememberMeServices;
import org.springframework.security.web.util.TextEscapeUtils;

import com.nestorurquiza.utils.CsrfUtil;
import com.nestorurquiza.web.WebConstants;

/**
 * Allowing the creation of a session on the fly when requesting any URL
 * We authenticate the user if not authenticated in the case a user and password is provided 
 * @author nestor
 *
 */
public class CustomLoginFilter implements Filter 
{
    private static final Logger log = LoggerFactory.getLogger(CustomLoginFilter.class);
    
    @Autowired
    UsernamePasswordAuthenticationFilter usernamePasswordAuthenticationFilter;
    
    @Override
    public void doFilter(ServletRequest request, ServletResponse response, FilterChain chain) throws IOException, ServletException 
    {
        HttpServletRequest httpRequest = ((HttpServletRequest) request);
        String uri = httpRequest.getRequestURI();
        if(! uri.contains(".")) {
            HttpServletResponse httpResponse = ((HttpServletResponse) response);
            SecurityContext securityContext = SecurityContextHolder.getContext();
            String method = httpRequest.getMethod();
            String userName = request.getParameter("j_username");
            if(userName != null && "POST".equals(method) && (securityContext == null || securityContext.getAuthentication() == null || !securityContext.getAuthentication().isAuthenticated())) {
                try {
                    Authentication auth = usernamePasswordAuthenticationFilter.attemptAuthentication(httpRequest, httpResponse);
                    securityContext.setAuthentication(auth);
                    SecurityContextHolder.setContext(securityContext);
                    httpRequest.getSession().setAttribute("SPRING_SECURITY_CONTEXT", securityContext);
                    httpRequest.getSession().setAttribute(UsernamePasswordAuthenticationFilter.SPRING_SECURITY_LAST_USERNAME_KEY, TextEscapeUtils.escapeEntities(userName));
                    if(auth.isAuthenticated()) {
                        //Initialize CSRF token
                        CsrfUtil.initializeCsrfToken(httpRequest);
                        //Set the token as an attribute in the request to make it pass the security check
                        httpRequest.setAttribute(WebConstants.CSRF_TOKEN, httpRequest.getSession().getAttribute(WebConstants.CSRF_TOKEN));
                    }
                } catch (Exception e){
                    log.info("User could not be authenticated. We go on ...");
                }
                
            }
        }
        chain.doFilter(request, response);
    }

    @Override
    public void destroy() 
    {   

    }

    @Override
    public void init(FilterConfig config) throws ServletException 
    {
    
    }
    
    private Cookie getRememberMeCookie(Cookie[] cookies) {
        if (cookies != null)
          for (Cookie cookie : cookies) {
            if (AbstractRememberMeServices.SPRING_SECURITY_REMEMBER_ME_COOKIE_KEY.equals(cookie.getName())) {
              return cookie;
            }
          }
        return null;
    }
}

Tainting LDAP data with Talend

Even though I solved this through the use of a nodeJS script I wanted to get the problem solved from Talend.

Here I discuss how to do it from Talend. There are still some outstanding issues.

As you can see in the picture I am reusing the list of white listed emails from a hashmap (tHashOutput_9). Only one connection is done to the input LDAP server which needs normalization (tNormalize) for group/uniquemember. A tJavaRow is responsible for adding a column containing just the uniquemember email for filtering purposes executed by tMap_1. tLDAPOutput_3 inserts the root elements while tLDAPOutput_1 and tLDAPOutput_2 insert users and groups respectively. Note that the latest two are in a separate subjob to be able to create the root entries before. Again the use of hashes allow to communicate both subjobs.


Install gcc in Ubuntu

The Linux sysadmin must build from sources from time to time. It makes sense then to have the C environment ready. In Ubuntu all you need to do is to follow the below commands:
sudo apt-get update
sudo apt-get upgrade
sudo apt-get install build-essential
gcc --version

Saturday, August 06, 2011

Phishing Attack: Fake Twitter Email not marked as Spam

I got an email in my Gmail account from "Twitter Support" "with subject "Your account has been suspended" with no text content but an image (that I have disabled of course for security reasons). The image content was something like "We detected unusual activity ..."

This phishing email is nothing new but what came to my attention was that Gmail was not able to detect the spam even though the full headers from the message are showing how Google identified it as a candidate for Spam "Authentication-Results: mx.google.com; spf=hardfail (google.com: domain of support@twitter.com does not designate 203.115.131.123 as permitted sender) smtp.mail=support@twitter.com"

Any software is plenty of bugs. Even with the best developers on board you are still vulnerable. Good that "Report phishing" option is available albeit a little bit hidden behind an arrow close to the Reply link. User experience should be helping better here I would say but regardless the important lesson to learn is to be always suspicious up front. Do not trust any bad news (account hacked or compromised) or too good news (You just won a million dollar) you receive.

See below for the full headers of the message:
Delivered-To: nestor.urquiza@gmail.com
Received: by 10.236.179.100 with SMTP id g64cs68259yhm;
        Fri, 5 Aug 2011 22:37:44 -0700 (PDT)
Received: from mr.google.com ([10.142.187.15])
        by 10.142.187.15 with SMTP id k15mr3461337wff.111.1312609063904 (num_hops = 1);
        Fri, 05 Aug 2011 22:37:43 -0700 (PDT)
Received: by 10.142.187.15 with SMTP id k15mr2917056wff.111.1312609063502;
        Fri, 05 Aug 2011 22:37:43 -0700 (PDT)
Return-Path: <support@twitter.com>
Received: from vsfilter2.roc.bti.net.ph (vsf-mx4.bti.net.ph [203.115.131.123])
        by mx.google.com with ESMTP id w1si282094wfw.62.2011.08.05.22.37.42;
        Fri, 05 Aug 2011 22:37:43 -0700 (PDT)
Received-SPF: fail (google.com: domain of support@twitter.com does not designate 203.115.131.123 as permitted sender) client-ip=203.115.131.123;
Authentication-Results: mx.google.com; spf=hardfail (google.com: domain of support@twitter.com does not designate 203.115.131.123 as permitted sender) smtp.mail=support@twitter.com
X-IronPort-Anti-Spam-Filtered: true
X-IronPort-Anti-Spam-Result: AqA2AOHRPE7Lc4NugWdsb2JhbAAoEwcXgjgBD4NgjV+EQwGOLRNcAQEWJiVxSxISGQELCk0BAQECDQ4MJAJQh3oKIgGeN5I1jSaDLQyCLl8Eh1qYFoMBgQaCYTA
Received: from smtp4-roc.bti.net.ph (HELO smtp1.skyinet.net) ([203.115.131.110])
  by vsfilter2.roc.bti.net.ph with ESMTP; 06 Aug 2011 13:37:41 +0800
Received: from 110.55.232.159.BTI.NET.PH (unknown [110.55.236.20])
 by smtp4-roc.bti.net.ph (Postfix) with ESMTP id 55F8793DB3A
 for <nestor.urquiza@gmail.com>; Sat,  6 Aug 2011 13:37:41 +0800 (PHT)
From: "Twitter Support" <support@twitter.com>
Subject: Your account has been suspended
To: "nestor.urquiza" <nestor.urquiza@gmail.com>
Content-Type: multipart/alternative; charset="iso-8859-10"; boundary="LMRJGCZhTXlUeMlXLirvgZD=_SMWAE68zR"
MIME-Version: 1.0
Content-Transfer-Encoding: 8bit
Date: Sat, 6 Aug 2011 13:37:37 +0800
Message-Id: <20110806053741.55F8793DB3A@smtp4-roc.bti.net.ph>

This is a multi-part message in MIME format

--LMRJGCZhTXlUeMlXLirvgZD=_SMWAE68zR
Content-Type: text/plain ; charset="iso-8859-10"
Content-Transfer-Encoding: quoted-printable




--LMRJGCZhTXlUeMlXLirvgZD=_SMWAE68zR
Content-Type: text/html ; charset="iso-8859-10"
Content-Transfer-Encoding: quoted-printable

<HTML><HEAD>
<META name=3DGENERATOR content=3D"MSHTML 8.00.6001.23019"></HEAD>
<BODY>
<P><A href=3D"mexico.cnn.com/redirectComplete.php?url=3D//emailus%2Eit=
%2Etc/2ule3B"><IMG border=3D0 src=3D"http://3.bp.blogspot.com/-u_sWLHS=
Yjes/TjyqVZ73vrI/AAAAAAAAAEQ/hX5mKS-R7-g/s1600?2ule3B"></A> </P>
<P>&nbsp;</P></BODY></HTML>


--LMRJGCZhTXlUeMlXLirvgZD=_SMWAE68zR--                                                                                                                                                                                                                                                    

Thursday, August 04, 2011

Server scripting with NodeJs: Tainting LDAP data

This is a real world example of using nodejs to do server side scripting. It shows how to deal with command line arguments, invoke shell commands and parse responses, interact with mysql and how to use the library (when possible and makes sense) in a synchronous way.

Node (also knows and nodejs) is a javascript library that allows to run javascript code out of the browser. It favors event programming which can be difficult for backend developers used to languages like Java. At the same time it is straightforward for front end developers which are already used to javascript callback functions necessary for UI programming.

If you havent install it yet go ahead and do it now:

  1. I have tested all this with node-v0.4.10 as you see below.
    cd ~/Downloads
    wget http://nodejs.org/dist/node-v0.4.10.tar.gz
    tar -zxvf node-v0.4.10.tar.gz
    cd node-v0.4.10
    ./configure
    make
    sudo make install
    node --version
    
  2. Install npm (package manager for nodejs) in case you need extra modules (and you will when you get serious about nodejs development)
    curl http://npmjs.org/install.sh | sudo sh
    
  3. If you use MySQL here is how to install one library that just works (whole API: https://github.com/felixge/node-mysql)
    npm install mysql
    
  4. You should of course test with the simplest possible 'Hello World' script
    $ vi hello.js
    console.log('Hello World');
    $ node hello.js
    

The Node project is making a big effort to enforce the use of non blocking code (code that will be triggered but will not block the current thread). That is the reason why most of the code you will see is written with nested callback functions.

Event programming is good paradigm for certain problems. Responsiveness of a program can be improved a lot without having to deal with blocking in multithreading programming. Those of us that have worked with threads know how hard it can be. I am not nodejs is better than java or any other language to deal with multiprocessing. It is just a different paradigm and I would appreciate if you do not start ranting against or in favor of any programming language in this post.

Here is one extract from the process api: http://nodejs.org/docs/v0.3.1/api/process.html
process.argv.forEach(function (val, index, array) {
  console.log(index + ': ' + val);
});
console.log

The previous code is non blocking, so if you need to work with the arguments you will need to work inside the anonymous function. Look at the code below. After running it you will see the asynchronous code does not run before the console logs the message:
var asyncArg = 'Async Arguments:';
var syncArg = 'Sync Arguments:';

var argv = process.argv;
argv.forEach(function (val, index, array) {
 setTimeout(function() {asyncArg += ' ' + val; console.log( asyncArg ); }, 0);
 
});
console.log( asyncArg );

for( argIndex in argv ) {
 syncArg += ' ' + argv[argIndex];
}
console.log( syncArg );

So probably to work with command line arguments you are better off the asynchronous flow. One might expect that there is always a way to go synchronous with nodejs but that is not the case. As I said the nodejs library pushes for non blocking === asynchronous code. Look at the below code which comments should be self explanatory:
var sys = require('sys')
var exec = require('child_process').exec;

var child = exec("ldapsearch -x -v -H 'ldap://nestorurquiza:10389' -D 'uid=admin,ou=system' -w 'secret' -b 'o=nestorurquiza'", function (error, stdout, stderr) {
  console.log(stdout); //Will print the output of the command
  if (error !== null) {
    console.log('exec error: ' + error);
  }
});
console.log(child.stdout); //Will not print the output of the command but rather the object prototype

Here is a script that clones the data from one LDAP Server (tested with ApacheDS) and imports it into a second LDAP server. The script taints the emails to ensure we do not send messages to real production users while testing. It excludes certain domains and it shows how to interact with mysql to pull a white list of emails for which no tainting should be done. It also changes the passwords to a well known test password so the team can use it to debug what happens when different users interact with the application. BTW this task is better to be done from Talend but I needed a real problem to demonstrate nodejs is ready to work as server side scripting while I was figuring out how to make it happen from Talend:

#!/usr/local/bin/node
/*
** WARNING: This program will wipe out the specified BASE_DN from the target LDAP Server
**
** taintLdap.js A nodejs script to taint data from ldap. Use it to change passwords for all users to a known value and to change their emails to avoid sending messages from testing environment to real users
**
** Example: node taintLdap.js 'ldap://jnestorurquiza:10389' 'uid=admin,ou=system' 'secret' 'ldap://localhost:10389' 'uid=admin,ou=system' 'secret'
**
** @Author: Nestor Urquiza
** @Date: 08/02/2011
**
*/

/*
** Imports
*/
var sys = require('sys')
var exec = require('child_process').exec;
var Client = require('mysql').Client;
var client = new Client();

/*
** Constants
*/
var BASE_DN = "o=nestorurquiza";
var EXCLUSION_DOMAINS = ['nestorurquiza.com','nestoru.com'];
var APPEND_DOMAIN = 'nestorurquiza.com'
var COMMON_PASSWORD = 'e1NIQX1JcnFMUVdMT1o3ZXF0WHRBdUlFSFRlUnZkRFk9' //Testtest1 after SHA1. To generate a different password use: echo -n "mypassword" | openssl dgst -sha1

/*
** Arguments
*/
var argv = process.argv;
if( argv.length != 8 ) {
 usage();
 process.exit(1);
}
var fromUrl = argv[2];
var fromUser = argv[3];
var fromPasword = argv[4];
var toUrl = argv[5];
var toUser = argv[6];
var toPasword = argv[7];

client.user = 'root';
client.password = 'root';
client.host = 'localhost';
client.port = '3306';
client.database = 'nestorurquiza'

/*
** Main
*/
//console.log('Cloning and tainting from ' + fromUrl + ' to ' + toUrl);
var ldif;
var authorizedEmails = new Array();
var pattern = /[^\s=,]*@[^\s=,]*/g
var child = exec("ldapsearch -x -v -H '" + fromUrl + "' -D '" + fromUser + "' -w '" + fromPasword + "' -b '" + BASE_DN + "'", function (error, stdout, stderr) {
  if (error) {
    throw error;
  }
  ldif = stdout;
  client.connect();
  var authorizedEmailQuery = client.query(
  'SELECT name FROM authorized_test_email',
  function (error, results, fields) {
    if (error) {
      throw error;
    }
    for (var resultIndex in results){
      var result = results[resultIndex];
      authorizedEmails[resultIndex] = result.name;
    }
    client.end();
    //console.log('****************'  + authorizedEmails);
    ldif = taintLdifEmail();
 
 child = exec("ldapdelete -r -x -H '" + toUrl + "' -D '" + toUser + "' -w '" + toPasword + "' '" + BASE_DN + "'", function (error, stdout, stderr) {
   if (error) {
        //throw error;
        console.log("WARNING: Could not delete " + BASE_DN);
      }
      var command = "echo '" + ldif + "' | ldapmodify -x -c -a -H '" + toUrl + "' -D '" + toUser + "' -w '" + toPasword + "'";
      //console.log(command);
      child = exec(command, function (error, stdout, stderr) {
     
  if (error) {
   throw error;
  }
   });
    });   
 //ldapmodify -x -c -a -H ldap://localhost:10389 -D "uid=admin,ou=system" -w 'secret' < ~/Downloads/taintedLdap.ldif
  }
);
});

/*
** Functions
*/
function taintLdifEmail() {
 var matches = ldif.match(pattern);

 for ( var matchIndex in matches ) {
  var taint = true;
  var match = matches[matchIndex];
  for( var exclusionDomainIndex in EXCLUSION_DOMAINS ) {
   if( match.indexOf(EXCLUSION_DOMAINS[exclusionDomainIndex]) >= 0 ) {
    taint = false;
    break;
   }
  }
  if( !taint ) {
   continue;
  }
  for ( var authorizedEmailIndex in authorizedEmails ) {
   var authorizedEmail = authorizedEmails[authorizedEmailIndex];
   if( match == authorizedEmail ) {
    taint = false;
    continue;
   }
  }
  if( taint ) {
   var replacement = match.replace('.', '') + APPEND_DOMAIN;
   //console.log( match + " >>> " + replacement );
   ldif = ldif.replace(match, replacement);
  }
 }
 ldif = ldif.replace(/userPassword.*/g, 'userPassword:: ' + COMMON_PASSWORD);
 return ldif;
}

function usage() {
 console.log("Usage: " + "./taintLdap.js <fromUrl> <fromUser> <fromPasword> <toUrl> <toUser> <toPasword>");
}

Why I am trying to use nodejs if I already have bash, awk, perl, python, ruby and what not? I am building a team with strong separation of concerns. While we know we cannot be as good as the guy that is spending 100% of the time in just writing SQL stored procedures the whole team could cover for some days to be able to compensate vacation time for example, so yes SQL is a mandatory skill. Javascript is a mandatory skill as well and if I can script with it I could have some of those scripting needs done by anybody in the team as well. I am just trying to keep really low the amount of technologies and languages we use.

Isn't it better to use RhinoJs? Probably yes, but I am tempted to use something out of the JVM that runs faster and consume less resources. I have recently decomisioned a whole CLI project based in Java just because it was really resource intensive. I have favored the use of Controllers in our Business Hub which are called from simple CURL or WGET statements. I am not claiming RhinoJs is unnacceptable slow nor that NodeJs is better. I see value in both of them.

Why I am considering scripting after all if I promote the idea of a Business Hub? There are cases in which definitely using Unix Power Tools, existing CLIs etc do the job quickly and more reliable.

Do I think NodeJs is a better answer for Server side logic than Java? At the moment I am happy with plain java for my backend. The amount of existing code available for free is amazing. If that will be the case in the future it will depend on the open source community. At least for my current project I stick to Java.

PKCS12 to JKS keystore

Java uses a proprietary to Sun format (JKS) to store certificates in what is called a Keystore (A file containing entries of those certificates you trust)

When you get certificates included in a different keystore type like it is the case of PKCS12 (commonly using the *.p12 extension) you need to extract and add them to the JKS keystore. Failure to do so will make your program depending on individual keystore files rather than just one keystore where all certificates and private keys are kept.

Here is how you can do it (sample using OSX paths but applicable to other OS as well).
First find the key for the certificate to export. Basically the left first word for the specific entry from this command:
keytool -list -keystore /Users/nestor/Downloads/cert.p12 -storetype pkcs12
Then run something like the below. Note that alias.from.cert.p12 comes from the previous command.
keytool -importkeystore -srckeystore /Users/nestor/Downloads/cert.p12 -destkeystore /Library/Java/Home/lib/security/cacerts -srcstoretype PKCS12 -deststoretype JKS -srcstorepass cert.p12.password -deststorepass changeit.is.the.default.password -srcalias alias.from.cert.p12 -destalias alias.for.cacerts.new.certificate

Note that the private key inside PKCS12 might need a password and that password must be the same as the JKS where you are importing. Failure to do this will end up in Access Denied, Forbidden or any other error comming from the Server where the key is attempted to be used.

Wednesday, August 03, 2011

ldapsearch SSL with ApacheDS

Self signed certificates are treated different by the ldap cli tools.

The task was to connect with ldapsearch to a remote ApacheDS server serving SSL. Long story short the certificate is self signed and only certain IP range can access the server via LDAP over SSL (TLS).

Here are the steps showing how to configure ldapsearch and the rest of ldap tools to work with SSL (both signed and self signed):

  1. If not using self signed certificate then get the server certificate
    $ openssl s_client -connect ldap.nestorurquiza.com:636
    
  2. Non self signed certificate: Create a file with the contents from "-----BEGIN CERTIFICATE-----" up to "-----END CERTIFICATE-----") from the previous command
    $ sudo mkdir /etc/openldap/certs/
    $ vi /etc/openldap/certs/ldap.nestorurquiza.com.cert
    
  3. Be sure your certificate is not self signed. Basically check for a return code=0, not someting like "Verify return code: 18 (self signed certificate)"
    $ openssl s_client -connect ldap.nestorurquiza.com:636 -CAfile ldap.nestorurquiza.com.cert
    
  4. Edit the ldap configuration
    $ vi /etc/openldap/ldap.conf
    ...
    #Use the below if you want ldapsearch to work with self signed certificate. Probably a better option security wise is to buy a certificate right ;-) Note that the path is for OSX. For Ubuntu it is /etc/ldap/certs...
    #TLS_REQCERT    demand
    TLS_REQCERT     never
    #Use the below for non self signed certificates
    #TLS_CERT    /etc/openldap/certs/ldap.nestorurquiza.com.cert
    
  5. Run an ldapsearch command to be usre you get the ldif result
    ldapsearch -x -v -H ldaps://ldap.nestorurquiza.com:10636 -D "uid=admin,ou=system" -w 'secretPassword' -b "o=nestorurquiza"
    

Followers