Sunday, March 11, 2012

Serving Tapestry5 Assets As Static Resources

In Tapestry5 you use assets to reference *.js*.css or image files from your templates/code. The reference may look like:

    <link rel="stylesheet" type="text/css" href="${context:/css/all.css}" />

During the render phase Tapestry5 converts the ${context:/css/all.css} part to asset URL, which may look like the following (see Asset URLs section here):

    <link rel="stylesheet" type="text/css" href="/assets/stage-20120310/ctx/css/all.css" />

Here "stage-20120310" -- is an application version string, which Tapestry5 adds to asset URLs to manage assets versioning. When running in production Tapestry5 adds a far future expires header for the asset, which will encourage the client browser to cache it.

When you change one of your assets you have to change application version number in your AppModule.java, so that Tapestry5 generate new asset URLs and browser fetched new assets instead of using the ones from cache.

One disadvantage of such approach is that client browser will have to get all the assets once again, not just the one that was changed.

For the majority of assets the asset URL is generated by Tapestry5. Exceptions are assets, that are referenced from *.css files by the relative URL, like this (file all.css):

a.external {
background: transparent url(../images/external.png) no-repeat scroll right center;
display: inline-block;
margin-left: 2px;
height: 11px;
width: 11px;
zoom: 1;
}

In this case browser will form the URL itself relatively to "/assets/stage-20120310/ctx/css/all.css", and the resulting URL will be "/assets/stage-20120310/ctx/images/external.png".

So you have to change application version in AppModule.java if you provide new version of "external.png".

But, for the majority of assets it would be enough to append MD5/SHA1/... checksum as a GET-parameter to asset URL and make them look like:

    <link rel="stylesheet" type="text/css" href="/assets/stage-20120310/ctx/css/all.css?5ef25ac1ec38f119e283f338e6c120a4e53127b1" />

In Tapestry5 you have the ability to provide your own implementation of AssetPathConverter service and append this checksum manually. But, in this interface you only have original asset URL, and don't have the resource itself to calculate the checksum.

There are several ways this may be implemented. Ideally, I'd like this to be implemented in Tapestry5 core.

There's one thing I don't like about Tapestry5 assets handling, though, even if the above solution will be implemented -- is that assets are not static.

This means every asset URL is handled by the Java code, and in most cases assets handling is just streaming of existing files from filesystem to browser (with optional minimization and gzip-compression).

Once the asset was handled, Tapestry5 caches the response and uses it in further responses, but still this is all done in Java.

In Ping Service we've implemented "assets precompilation", and placed all the rendered assets as static files in the web app root folder.

This is done using custom implementation of org.apache.tapestry5.internal.services.ResourceStreamer, which is responsible for streaming every asset to client. During resource streaming we calculate asset checksum and store in a static.properties file, where we put asset URL as a key, and checksum as a value:

#Static Assets For Tapestry5 Application
#Sat Mar 10 19:42:38 UTC 2012
/assets/stage-20120310/ctx/css/all.css=5ef25ac1ec38f119e283f338e6c120a4e53127b1
/assets/stage-20120310/ctx/css/analytics.css=ee470432c344820e43995fb4632ab4bee3b92e38
/assets/stage-20120310/tapestry/t5-prototype.js=95e30b840a5654b82e6a0334a14a2766c57c4d99
...

Our implementation of AssetPathConverter uses this property file to modify asset URLs.

We run our implementation of ResourceStreamer only in production mode, since Google App Engine doesn't allow writing to the filesystem.

Also we've implemented it to work only if special HTTP-header passed with the request. To pass this header and to trigger every asset we have in our application, we use Selenium-powered integration test that queries every single page. We run this test before deploying new version to production.

Now Tapestry5 asset URLs and URLs of static files are the same in our application. So Google App Engine runtime won't even pass the request to Java. Also it uses its own facilities to serve static files, i.e. gzip-compression, etc.

Saturday, January 14, 2012

Simple Sorting Facade for Java (SSF4J)

In Java to sort two or more lists together you have to write a custom solution.

Say, you have list of names and corresponding list of weights. There is no API that allows you to sort names by weights (at least not that I know). However this is very common use case, especially when you analyzing data in your programs.

To achieve this, you, most likely, implement one of the sorting algorithms with a custom swap-logic.

Simple sorting facade is a pattern that already contains implementation of sorting algorithm(s) and only requires developers to specify source list, its bounds, and compare- and the swap-logic.

You can explore SSF4J on GitHub and contribute your implementations of sorting algorithms.

Here's an example of using SSF4J:


Monday, September 19, 2011

Mac OS X Lion HTTP Sniffer


Simple command-line sniffer:

sudo tcpdump -s 0 -A -i en1 port 80

Use ifconfig to lookup interface name (i.e. en1).

Monday, September 05, 2011

Running BIRT Reports in Tomcat

Context: You have database and you need to do data analysis: draw charts, build some tables, calculate totals, etc. You want it all to be available over the web and secured with a password.

Your database is any JDBC-supported database (I use MySQL 5.1.49).
Your server is running any OS where Java can run (I use Ubuntu Linux 10.10 available through SSH).

I will show how to implement this using Eclipse BIRT 3.7.

Developer Environment
  1. Download "Eclipse IDE for Java and Report Developers" package here.
    Unzip to install.
  2. Design new report (I created sales.rptdesign). This is really straightforward.
JDBC Driver. You will need MySQL JDBC driver to create Data Source for report. I got mysql-connector-java-5.1.17-bin.jar here.
Installing the driver in BIRT designer is easy using "Manage drivers..." dialog. But you will probably have problems deploying it to the runtime.

Fonts. You probably won't have any problems with fonts in BIRT designer. And again you will likely have problems with fonts in runtime.

Connection Profile Store. With BIRT 3.7 you can use Connection Profiles to hold database connections. After you've finished designing and testing your report, double click report Data Source to bring properties dialog and create new connection profile store in there. Save it to some file (I saved to planet33_v2.xml).
Here's what I have in there:
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<DataTools.ServerProfiles version="1.0">
    <profile autoconnect="No" desc=""
        id="ecc3bc60-d4fd-11e0-957a-e0e31b9a34ee" name="planet33_v2"
        providerID="org.eclipse.datatools.enablement.mysql.connectionProfile">
        <baseproperties>
            <property
                name="org.eclipse.datatools.connectivity.db.connectionProperties"
                value="" />
            <property name="org.eclipse.datatools.connectivity.db.savePWD"
                value="true" />
            <property name="org.eclipse.datatools.connectivity.drivers.defnType"
                value="org.eclipse.datatools.enablement.mysql.5_1.driverTemplate" />
            <property name="jarList"
                value="/usr/local/share/mysql-connector-java-5.1.17-bin.jar" />
            <property name="org.eclipse.datatools.connectivity.db.username"
                value="your_username" />
            <property name="org.eclipse.datatools.connectivity.db.driverClass"
                value="com.mysql.jdbc.Driver" />
            <property name="org.eclipse.datatools.connectivity.db.databaseName"
                value="planet33_v2" />
            <property name="org.eclipse.datatools.connectivity.db.password"
                value="your_password" />
            <property name="org.eclipse.datatools.connectivity.db.version"
                value="5.1" />
            <property name="org.eclipse.datatools.connectivity.db.URL"
                value="jdbc:mysql://127.0.0.1:3306/planet33_v2" />
            <property name="org.eclipse.datatools.connectivity.db.vendor"
                value="MySql" />
        </baseproperties>
        <org.eclipse.datatools.connectivity.versionInfo>
            <property name="server.version" value="5.1.49" />
            <property name="technology.name.jdbc" value="JDBC" />
            <property name="server.name" value="MySQL" />
            <property name="technology.version.jdbc" value="4.0.0" />
        </org.eclipse.datatools.connectivity.versionInfo>
        <driverreference>
            <property name="driverName" value="MySQL JDBC Driver" />
            <property name="driverTypeID"
                value="org.eclipse.datatools.enablement.mysql.5_1.driverTemplate" />
        </driverreference>
    </profile>
</DataTools.ServerProfiles>

Note: I bet you can use JDNI data sources here (and I suppose this is even preferable because of connection pooling, etc.). Please, drop a few lines in comments below with instructions how you do this.

You can now edit XML source of you report and replace /report/data-sources with something like this:

<data-sources>
    <oda-data-source extensionID="org.eclipse.birt.report.data.oda.jdbc.dbprofile"
        name="Planet33 V2 Data Source" id="359">
        <property name="OdaConnProfileName">planet33_v2</property>
        <property name="OdaConnProfileStorePath">../conf/planet33_v2.xml</property>
    </oda-data-source>
</data-sources>

Several things to mention here:
  • planet33_v2.xml (Connection Profile Store)
    • Check all properties and change them according to your connection.
    • Note the jarList property, there you should specify path(s) to where your JDBC drivers located (I copied driver that I've downloaded to /usr/local/share/mysql-connector-java-5.1.17-bin.jar).
    • When you create connection profile store file from designer it places property with name="org.eclipse.datatools.connectivity.driverDefinitionID". You should remove this property because of this issue.
  • sales.rptdesign (The Report)
    • You should keep value of ida-data-source@id attribute the same that was in your design.
    • Value of OdaConnProfileName should match value of DataTools.ServerProfiles/profile@name attribute from planet33_v2.xml.
    • Note that OdaConnProfileStorePath is relative path (see below). But you can keep it absolute if you want. 

Server Environment

(Note: I recommend to configure Tomcat instance on your developer machine first to make it easier to verify report settings, and then transfer the entire $CATALINA_HOME to production server. Of course, you can do all these steps on production server directly.)
  1. Download Apache Tomcat (any Java application server should be fine).
    Unzip to some folder (I used /usr/local/share/apache-tomcat-5.5.33/) -- this will be $CATALINA_HOME.
  2. Download BIRT "Runtime" package here.
    Copy birt.war (BIRT Web Viewer application) to $CATALINA_HOME/webapps.
  3. Edit $CATALINA_HOME/catalina.sh and paste these lines somewhere after JAVA_OPTS variable initialized (this prepares workspace for DTP plugin):
    
    
    java_io_tmpdir=$CATALINA_HOME/temp
    org_eclipse_datatools_workspacepath=$java_io_tmpdir/workspace_dtp
    mkdir -p $org_eclipse_datatools_workspacepath
    
    JAVA_OPTS="$JAVA_OPTS -Dorg.eclipse.datatools_workspacepath=$org_eclipse_datatools_workspacepath"
    
    
  4. Start Tomcat by running $CATALINA_HOME/startup.sh. After this BIRT Report Viewer application should be available by http://localhost:8080/birt. Also birt.war should be now extracted to $CATALINA_HOME/webapps/birt -- this will be $BIRT_HOME. You can now delete $CATALINA_HOME/webapps/birt.war.
  5. Copy planet33_v2.xml to $CATALINA_HOME/conf as (remember OdaConnProfileStorePath property in sales.rptdesign file?).
  6. Copy your sales.rtpdesign file to $BIRT_HOME.
At this point you should be able to execute the report by simply following the address http://localhost:8080/birt/frameset?__report=sales.rptdesign&__dpi=600.

Note the __dpi URL parameter -- it controls DPI of chart images rendered in HTML/PDF. You will probably want to modify like this it to increase image quality. Also note that if you set chart output format to SVG you will get vector graphics quality in PDF output.

Security

There are obvious reasons why you may want to keep your reports secure.

Besides that keep in mind that in BIRT Web Viewer application all reports sources (*.rptdesign files) available to user by request. Try navigating to http://localhost:8080/birt/sales.rptdesign and you'll see what I mean. I think this is a good reason, why you should use connection profile store (or at least JNDI data sources), because that files not available through HTTP.

Implementing simple (HTTP BASIC AUTH) security with Tomcat is pretty simple.

First, modify $BIRT_HOME/WEB-INF/web.xml, by adding this (you can change role names as you want):
<!-- Define a security constraint on this application -->
<security-constraint>
  <web-resource-collection>
    <web-resource-name>Entire Application</web-resource-name>
    <url-pattern>/*</url-pattern>
  </web-resource-collection>
  <auth-constraint>
    <!-- This role is not in the default user directory -->
    <role-name>manager</role-name>
  </auth-constraint>
</security-constraint>             
<!-- Define the login configuration for this application -->
<login-config>
  <auth-method>BASIC</auth-method>
  <realm-name>BIRT Report Viewer</realm-name>
</login-config>
<!-- Security roles referenced by this web application -->
<security-role>
  <description>
    The role that is required to log in to the BIRT Report Viewer
  </description>
  <role-name>manager</role-name>
</security-role>

You may also do the same for $CATALINA_HOME/conf/web.xml to secure all applications in this Tomcat instance.
Second, you should edit $CATALINA_HOME/conf/tomcat-users.xml to define user login and password.

Thats all, you're secured :) This is should be fine for most cases, but I would recommend you to read about HTTPS if your data is extremely secure.

Deploy to server

  1. Copy Tomcat to the server:
    Tip: Use scp command in terminal to transfer files from your machine to the server over SSH:
    scp /usr/local/share/apache-tomcat-5.5.33 dmitrygusev@planet33.ru:/usr/local/share/
  2. Copy JDBC Driver to the server:
    • Copy this driver to the same path as specified in the jarList property from planet33_v2.xml file.
    • DO NOT COPY this driver to $BIRT_HOME/WEB-INF/lib, because it may lead to ClassNotFoundException.
Fix file permissions. When you copy files over scp you may need to chmod them to grant read/execute access. This should fix it:

chmod a+r /usr/local/share/mysql-connector-java-5.1.17-bin.jar
chmod -R a+r /usr/local/share/apache-tomcat-5.5.33/
chmod a+x /usr/local/share/apache-tomcat-5.5.33/bin/*.sh

Now you should be able to start tomcat and run reports on the server.

Fonts. BIRT PDF output doesn't work good for Russian fonts out-of-the-box, because of licensing issues with fonts. One simple solution to fix this is:
  1. Get the *.ttf font files you need (you can copy them from any Windows installation, look in c:\Windows\Fonts). These 8 files should be enough in most cases (these are "Arial" and "Times New Roman" fonts):

    arialbd.ttf  arialbi.ttf  ariali.ttf  arial.ttf
    timesbd.ttf  timesbi.ttf  timesi.ttf  times.ttf
  2. Copy these files to /usr/share/fonts/truetype (or any other place that is referenced from fontsConfig.xml).
  3. Don't forget to fix file permissions:
    chmod a+r /usr/share/fonts/truetype/*.ttf
  4. Reference the fonts from *.rptdesign (or configure font-aliases):
    /report/styles
    <style name="report" id="4">
        <property name="fontFamily">"Arial"</property>
        <property name="fontSize">9pt</property>
    </style>
    
  5. Restart Tomcat:
    • $CATALINA_HOME/bin/shutdown.sh
    • $CATALINA_HOME/bin/startup.sh
Russian Localization. I used this method to do it. Although you may want to try BIRT Language Packs.


Troubleshooting.

  • Neither the JAVA_HOME nor the JRE_HOME environment variable is defined
    At least one of these environment variable is needed to run this program

    You should define JAVA_HOME variable. Execute this command before running Tomcat's *.sh files in terminal:

    export JAVA_HOME=/usr/bin/java
  • If you get OutOfMemoryError you may want to give JVM more memory. Edit $CATALINA_HOME/bin/catalina.sh to include this (see this thread on stackoverflow, and read more about JVM memory settings):

    JAVA_OPTS="$JAVA_OPTS -Xms512m -Xmx512m -XX:MaxPermSize=256m"
  • If you got OutOfMemoryError you most likely couldn't restart Tomcat using $CATALINA_HOME/bin/shutdown.sh script.

    To kill Tomcat instance use htop command in terminal. In htop interface select Tomcat process (this is /usr/lib/java), press 'k', select 9 SIGKILL in "Send signal" area, and press Enter. To exit htop press 'q'.
  • Executing report never stops. Tomcat process consumes all CPU resources.

    I've seen this situation when used charts in report and they were on page break. I fixed this by moving charts to other place (far from page break). Changing page size to avoid page breaks also fixes this issue.

Wednesday, December 01, 2010

Deploy SharePoint Designer 2010 Reusable Workflow As *.WSP File

SharePoint Designer 2010 makes workflow development really fast and simple. Much simpler than using Visual Studio. But unlike Visual Studio this tool has some limitations from developer's point of view.

This is due to SPD was created as a tool for end (SharePoint) users, but not for solution developers. As a result we have one serious limitation that prevents developers to use this tool: it doesn't allow deploy created workflows to another SharePoint servers. Which means you cannot develop and test workflows on a development SharePoint server and then move it to production. You forced to develop in production, which is not right.

In particular what I said is true for reusable workflows that work with custom list instances. The problem is once you reference some list in workflow, SPD will link workflow template (*.xoml) to this list using its ListId attribute which is unique identifier that is valid for that particular site. This ListId is a random value that SharePoint generates when the list deployed. Note that you deploy list instances not only when staging ready solution from development to production, but also repeatedly during development cycle.

There are two ways you can notice your workflow corrupted. The first is in SPD you will see GUIDs instead of list names, and if you click these GUIDs you'll see unbinded dialogs:




And if you deploy such workflow to another SharePoint site (with prior export to *.WSP) you will get the error like this (in SharePoint logs) when try to run it:

SOAP exception: System.Runtime.InteropServices.COMException (0x82000006): List does not exist. The page you selected contains a list that does not exist. It may have been deleted by another user.
at Microsoft.SharePoint.SoapServer.SPBaseImpl.GetSPListByTitle(SPWeb spWeb, String strListName)
at Microsoft.SharePoint.SoapServer.SPBaseImpl.GetSPList(SPWeb spWeb, String strListName, Boolean bGetMetaData, Boolean bGetSecurityData)
at Microsoft.SharePoint.SoapServer.SPBaseImpl.GetSPList(String strListName, Boolean bGetMetaData, Boolean bGetSecurityData)
at Microsoft.SharePoint.SoapServer.ListSchemaImpl.GetList(String strListName)
at Microsoft.SharePoint.SoapServer.ListSchemaValidatorImpl.GetList(String strListName)
at Microsoft.SharePoint.SoapServer.Lists.GetList(String listName)


The most common approach you may find on the Internet is to export SPD reusable workflow to *.WSP solution package, import that *.WSP to Visual Studio and continue development there. I don't like this approach for two reasons: first is once you do this you can't open that workflow in SharePoint designer again to made any changes (and you don't want to change it in Visual Studio because, like every auto-generated code, SPD generated *.xoml is not very human-friendly):


And the second (which I'm not sure, though)---you can't deploy such workflow as a sandboxed solution (correct me if I wrong).

Fortunately, there is nothing that prevents us from deploying SPD reusable workflows except ListIds. All we need to do is replace broken Ids with the new ones. Here's how you may do this:

  1. Export reusable workflows to *.WSP files using SPD 2010
  2. To fix ListIds change contents of the process*.xoml file contained in *.WSP file (which is *.CAB file that contain (inter alia) Feature.xml)
    To extract and package contents of *.WSP I recommend to use PowerShell + built-in expand command and WSPBuilder's CabLib.dll accordingly
  3. *.xoml is an regular text/xml file so we can simply find and replace GUID strings
  4. We know what GUIDs to be replaced by examining contents of the *.xoml file.
    Look for entries like this:
    <ns1:LookupActivity ListId="{}{909E9DFD-A30B-4E28-BF2E-5BA47095967D}"  
    x:Name="ID10" FieldName="ID" LookupFunction="LookupInt"
    __Context="{ActivityBind ROOT,Path=__context}"
    ListItem="{ActivityBind ID11,Path=ReturnValue}" />

  5. We know the replacement for old GUIDs by using PowerShell automation: get SPWeb object of site where we want to deploy the workflow, get list in that web by list title, get ID of that list and convert that ID to string. Note that GUID string should be in upper case, otherwize you won't be able to edit workflow in SPD, though it will run okay on site (thats probably SPD bug)
  6. After replacing GUIDs we create new *.WSP with relevant ListIds which may be deployed to SharePoint


Below is sample PowerShell script that you can use as a reference to implement steps described above. To run it save contents to file, say Deploy-Workflows.ps1, change values of $siteUrl, $wspDir and $listIds to match your environment. You will also have to place CabLib.dll and AnjLab-SharePoint.ps1 files to the same folder as Deploy-Workflows.ps1. After that open SharePoint 2010 Management Shell, CD to directory with Deploy-Workflows.ps1 and run the script with command .\Deploy-Workflows.ps1.

# Allow running *.ps1 scripts from network shares 
# Set-ExecutionPolicy Unrestricted

# Copy CabLib.dll to user's temp (to prevent security exceptions if your project files are on network share)
# Note: $cablibFullName is a global variable used in AnjLab-SharePoint.ps1
$cablibFullName = "$env:TEMP\CabLib.dll"
if ((test-path $cablibFullName) -eq $false)
{
cp "CabLib.dll" $env:TEMP
}

# Import AnjLab-SharePoint functions
. .\AnjLab-SharePoint.ps1

# Replace with yours

$siteUrl = "http://dev-en/gls/" # SharePoint site to deploy *.WSP workflows to
$wspDir = "bin\Debug\Workflows" # Directory with *.WSP files
# (all files from this folder will be updated and deployed)
$wspTempDir = "$wspDir\temp" # Temp directory
$wspFinalDir = "$wspDir\final" # Directory where final *.WSP files will be placed

###################################################################################################
# Define mapping for GUID replacement in the hashtable below.
# All GUIDs that match keys from this hashtable will be replaced with corresponding GUIDs of lists
# taken from $siteUrl by specified list titles.
###################################################################################################

$listIds = @{ # ListId in workflow's *.WSP List Title on SharePoint Site
# ------------------------------------- -----------------------------
"909E9DFD-A30B-4E28-BF2E-5BA47095967D" = "Consumers";
"2F311150-7360-45DA-A4B1-C64339F3B931" = "Warehouses";
"435E8D1B-FC3F-42A9-B761-1958A31D9BDE" = "Leads";
}

# Replace List Ids

$wspFiles = (Get-ChildItem "$wspDir\*.wsp")
Update-WspListIds $siteUrl $wspFiles $wspTempDir $wspFinalDir $listIds

# Deploy Packages

$wspFiles = (Get-ChildItem "$wspFinalDir\*.wsp")
Deploy-Wsp $siteUrl $wspFiles $wspTempDir

Write-Host "Done"