Tuesday, September 14, 2010

Add Interactivity to ASP.NET ReportViewer

In one of our projects I had a task to add interactivity to charts located on (MS Report Server) reports that were viewed in ASP.NET web application using ReportViewer control.

Charts I worked with contained date series and user should had have the ability to add notes to data in series by clicking on that data on chart.

In the final solution I used jQuery to find images (charts) in rendered report HTML that had special value encoded in image's alt tag. Client javascript (which was on the same page as ReportViewer control) made AJAX requests to server to calibrate chart axes. Then calibration result was used on client side to build HTML MAP with AREAs to which jQuery onclick handlers were attached.

Here's how working example looked in browser:



Client side implementation was written in pure jQuery with help of jQuery qTip plugin and takes around 450 lines of code. Nothing interesting.

The most interesting part in this approach was getting chart image from generated report on server side.

To get image on server side I used the same mechanism that ReportViewer web control uses. I must to say that I couldn't do this without Red Gate's .Net Reflector (I used free version, it was enough here). This is really helpful tool that makes (nearly all) .Net libraries open sourced. I highly recommend it.

Browser obtains chart image by URL, so I had to pass that URL to server side to grab that image from there.

            function calibrate($image) {
$.ajax({
type: "POST",
contentType: "application/json; charset=utf-8",
url: options.calibrateAxesUrl,
data: "{imageUrl: '" + $image.attr("src") + "'}",
dataType: "json",
success: function(msg) {
calibrationResult = msg.d;

// process calibration result...
}
});
}


As you see I used image's src tag to get image URL and invoked web method CalibrateAxes (which is a public static method with [WebMethod] annotation) of my ASP.NET page (the full URL looks like '<%= ResolveClientUrl("~/Reporting/ReportViewer.aspx") + "/CalibrateAxes" %>').

CalibrateAxes method gets chart image as bitmap and does some bitmap analysis to form the result:


        [WebMethod]
public static CalibrationResult CalibrateAxes(string imageUrl)
{
byte[] image = RenderImage(imageUrl);

using (var stream = new MemoryStream(image))
using (var bmp = new Bitmap(stream))
{
var result = new CalibrationResult();

// analyze bitmap...

return result;
}
}

private static byte[] RenderImage(string imageUrl)
{
var imageUri = new Uri(HttpContext.Current.Request.Url.GetLeftPart(UriPartial.Authority) + imageUrl);

var parameters = HttpUtility.ParseQueryString(imageUri.Query);

var reportSession = parameters["ReportSession"];
var controlID = parameters["ControlID"];
var culture = parameters["Culture"];
var uiCulture = parameters["UICulture"];
var reportStack = parameters["ReportStack"];
var streamID = parameters["StreamID"];

var rs = new ReportExecutionService
{
Url = ConfigurationManager.AppSettings["ReportExecutionServiceUrl"],
Credentials = new NetworkCredential(
ConfigurationManager.AppSettings["ReportServerUserName"],
ConfigurationManager.AppSettings["ReportServerUserPassword"],
ConfigurationManager.AppSettings["ReportServerUserDomain"]),
ExecutionHeaderValue = new ExecutionHeader {ExecutionID = reportSession}
};

string deviceInfo = GetDeviceInfo(reportSession, controlID, culture, uiCulture, reportStack);

string encoding;
string mimetype;

return rs.RenderStream("HTML4.0", streamID, deviceInfo, out encoding, out mimetype);
}

private static string GetDeviceInfo(string reportSession, string controlID, string culture, string uiCulture, string reportStack)
{
var writer = new StringWriter();
var xmlWriter = new XmlTextWriter(writer);
xmlWriter.WriteStartElement("DeviceInfo");
var url = CreateUrl(reportSession, controlID, culture, uiCulture, reportStack);
xmlWriter.WriteElementString("StreamRoot", url);
xmlWriter.WriteEndElement();
return writer.ToString();
}

private static string CreateUrl(string reportSession,
string controlID,
string culture,
string uiCulture,
string reportStack)
{
var uriBuilder = new UriBuilder(HttpContext.Current.Request.Url.GetLeftPart(UriPartial.Authority));
var applicationPath = HttpContext.Current.Request.ApplicationPath;
if (!applicationPath.EndsWith("/", true, CultureInfo.InvariantCulture))
{
applicationPath = applicationPath + "/";
}
applicationPath = applicationPath + "Reserved.ReportViewerWebControl.axd";
applicationPath = HttpContext.Current.Response.ApplyAppPathModifier(applicationPath);
uriBuilder.Path = applicationPath;

var builder = new StringBuilder();
builder.AppendFormat("{0}={1}", "ReportSession", reportSession);
builder.AppendFormat("&{0}={1}", "ControlID", HttpUtility.UrlEncode(controlID));
builder.AppendFormat("&{0}={1}", "Culture", culture);
builder.AppendFormat("&{0}={1}", "UICulture", uiCulture);
builder.AppendFormat("&{0}={1}", "ReportStack", reportStack);
builder.Append("&OpType=ReportImage&StreamID=");
uriBuilder.Query = builder.ToString();

return uriBuilder.Uri.PathAndQuery;
}


This code of getting rendered image from report turned to be very generic, so you may also use it in your web applications.

Tapestry5: Caching Method Results

Assume you have methods that (almost) always return the same result for the same input arguments. If preparing method result is a heavy operation and/or it consumes time, it is reasonable to cache these results.

One way of building method cache in Tapestry5 is by implementing MethodAdvice interface like this:

public class CacheMethodResultAdvice implements MethodAdvice {

private static final Logger logger = LoggerFactory.getLogger(CacheMethodResultAdvice.class);

private final Cache cache;
private final Class<?> advisedClass;
private final Object nullObject = new Object();

public CacheMethodResultAdvice(Class<?> advisedClass, Cache cache) {
this.advisedClass = advisedClass;
this.cache = cache;
}

@Override
public void advise(Invocation invocation) {
String invocationSignature = getInvocationSignature(invocation);

String entityCacheKey = String.valueOf(invocationSignature.hashCode());

Object result;

if (cache.containsKey(entityCacheKey))
{
result = cache.get(entityCacheKey);

logger.debug("Using invocation result ({}) from cache '{}'", invocationSignature, result);

invocation.overrideResult(result);
}
else
{
invocation.proceed();

if (!invocation.isFail())
{
result = invocation.getResult();

cache.put(entityCacheKey, result);
}
}
}

private String getInvocationSignature(Invocation invocation) {
StringBuilder builder = new StringBuilder(150);
builder.append(advisedClass.getName());
builder.append('.');
builder.append(invocation.getMethodName());
builder.append('(');
for (int i = 0; i < invocation.getParameterCount(); i++) {
if (i > 0) {
builder.append(',');
}
Class<?> type = invocation.getParameterType(i);
builder.append(type.getName());
builder.append(' ');

Object param = invocation.getParameter(i);
builder.append(param != null ? param : nullObject);
}
builder.append(')');

return builder.toString();
}

}


Implementation of getInvocationSignature(...) is not ideal, but you may improve it to match your requirements. One issue I see here is building invocation signature for null-value parameters in a clustered environment (which is GAE). In this implementation method nullObject.toString() will return something like java.lang.Object@33aa9b. And this value will vary in different instances of your application. You may replace nullObject with just "null" string. Just keep in mind that "null" != null.

To make this advice working you should declare it in your AppModule.java:

    @SuppressWarnings("unchecked")
@Match("IPResolver")
public static void adviseCacheIPResolverMethods(final MethodAdviceReceiver receiver, Logger logger, PerthreadManager perthreadManager) {
try {
Map props = new HashMap();

// IP address of URL may change, keep it in cache for one day
props.put(GCacheFactory.EXPIRATION_DELTA, 60 * 60 * 24);

CacheFactory cacheFactory = CacheManager.getInstance().getCacheFactory();
Cache cache = cacheFactory.createCache(props);

LocalMemorySoftCache cache2 = new LocalMemorySoftCache(cache);

// We don't want local memory cache live longer than memcache
// Since we don't have any mechanism to set local cache expiration
// we will just reset this cache after each request
perthreadManager.addThreadCleanupListener(cache2);

receiver.adviseAllMethods(new CacheMethodResultAdvice(IPResolver.class, cache2));
} catch (CacheException e) {
logger.error("Error instantiating cache", e);
}
}

@Match("LocationResolver")
public static void adviseCacheLocationResolverMethods(final MethodAdviceReceiver receiver, Cache cache) {
// Assume that location of IP address will never change,
// so we don't have to set any custom cache expiration parameters
receiver.adviseAllMethods(new CacheMethodResultAdvice(LocationResolver.class, cache));
}


These declarations tell Tapestry5 to add our advice to all methods of services that implement IPResolver and LocationResolver interfaces.

Note that we able to use caches with different settings for different methods/services like in example above (see comments in code).

See also:

Monday, September 13, 2010

How To Determine Client TimeZone In A Web Application

In web applications when client and server located in different timezones we need a way to determine client timezone to display date/time sensitive information. This is almost always true for Google Appengine, where default server time zone is UTC.

There are several ways to determine client timezone.

One of them is resolving client IP address to location:

  1. Get client IP

  2. Get client location (latitude, longitude) by the IP-address

  3. Get information about timezone by the location coordinates


Every web framework provides API to get client IP. For instance, in java there is a method ServletRequest.getRemoteAddr() for this purpose.

To resolve IP and location information you can use one of the numerous web services available online.

For instance, to resolve IP to location Ping Service uses IP-whois.net service.

Another service, Geonames.org provides web service API to get timezone information by latitude/longitude pair.

Here's an implementation of described approach in java:


    private TimeZone getTimeZoneByClientIP() {
        TimeZone timeZone = UTC_TIME_ZONE;
        
        try {
            String clientIP = globals.getHTTPServletRequest().getRemoteAddr();
            
            if (!Utils.isNullOrEmpty(clientIP)) {
                Location location = locationResolver.resolveLocation(clientIP);
                
                if (!location.isEmpty()) {
                    timeZone = timeZoneResolver.resolveTimeZone(location.getLatitude(), location.getLongitude());
                }
                
                if (timeZone == null) {
                    timeZone = UTC_TIME_ZONE;
                }
            }
            
            logger.debug("Resolved timeZoneId is {}", timeZone.getID());
        } catch (Exception e) {
            logger.error("Error resolving client timezone by ip " 
                    + globals.getHTTPServletRequest().getRemoteAddr(), e);
        }
        
        return timeZone;
    }


The disadvantages using this approach are:

  • Your code becomes dependent on third party online services that are not 100% reliable

  • Requesting third party services online will take time (up to several seconds) which may result in long response time

Note: according to Ping Service statistics IP-Whois.net availability is close to 100% with average response time ~270 ms, while Geonames.org availability is only around 80% with average response time ~1100 ms. Geonames.org low level availability is due to GAE hosting: Geonames.org restricts free access to its API to 3000 requests per IP per hour.

On the other hand you have really simple solution to implement that allows to determine client timezone at the very first client request so you can display all date/time sensitive data using client local time.

See also:
Update: GAE 1.6.5 introduces some request headers which already contains Lat/Lng pair for incoming request: https://developers.google.com/appengine/docs/java/runtime#Request_Headers

Thursday, September 02, 2010

Profiling GAE API calls

While optimizing performance of GAE application its convenient to measure GAE API calls.

I'm using the following implementation of com.google.apphosting.api.ApiProxy.Delegate to do this:

public class ProfilingDelegate implements Delegate<Environment> {

private static final Logger logger = LoggerFactory.getLogger(ProfilingDelegate.class);

private final Delegate<Environment> parent;
private final String appPackage;

public ProfilingDelegate(Delegate<Environment> parent, String appPackage) {
this.parent = parent;
this.appPackage = appPackage;
}

public void log(Environment env, LogRecord logRec) {
parent.log(env, logRec);
}

@Override
public byte[] makeSyncCall(Environment env, String pkg, String method, byte[] request) throws ApiProxyException {
long start = System.currentTimeMillis();
byte[] result = parent.makeSyncCall(env, pkg, method, request);
StringBuilder builder = buildStackTrace(appPackage);
logger.info("GAE/S {}.{}: ->{} ms<-\n{}", new Object[] { pkg, method, System.currentTimeMillis() - start, builder });
return result;
}

/**
*
* @param appPackage
* Only classes from this package would be included in trace.
* @return
*/
public static StringBuilder buildStackTrace(String appPackage) {
StackTraceElement[] traces = Thread.currentThread().getStackTrace();
StringBuilder builder = new StringBuilder();
int length = traces.length;
StackTraceElement traceElement;
String className;
for (int i = 3; i < length; i++) {
traceElement = traces[i];
className = traceElement.getClassName();
if (className.startsWith(appPackage)) {
if (builder.length() > 0) {
builder.append('\n');
}
builder.append("..");
builder.append(className.substring(className.lastIndexOf('.')));
builder.append('.');
builder.append(traceElement.getMethodName());
builder.append(':');
builder.append(traceElement.getLineNumber());
}
}
if (builder.length() == 0) {
for (int i = 1; i < length; i++) {
traceElement = traces[i];
className = traceElement.getClassName();
if (builder.length() > 0) {
builder.append('\n');
}
builder.append(className);
builder.append('.');
builder.append(traceElement.getMethodName());
builder.append(':');
builder.append(traceElement.getLineNumber());
}
}
return builder;
}

@Override
public Future<byte[]> makeAsyncCall(Environment env, String pkg, String method, byte[] request, ApiConfig config) {
long start = System.currentTimeMillis();
Future<byte[]> result = parent.makeAsyncCall(env, pkg, method, request, config);
StringBuilder builder = buildStackTrace(appPackage);
logger.info("GAE/A {}.{}: ->{} ms<-\n{}", new Object[] { pkg, method, System.currentTimeMillis() - start, builder });
return result;
}
}


To register this delegate add the following code to prior to any API calls, i.e. to filter init() method:

    public void init(FilterConfig config) throws ServletException
{
this.config = config;
// Note: Comment this off to profile Google API requests
ApiProxy.setDelegate(new ProfilingDelegate(ApiProxy.getDelegate(), "dmitrygusev"));
}


Here's an example of log output:

02.09.2010 0:22:19 dmitrygusev.tapestry5.gae.ProfilingDelegate makeSyncCall
INFO: GAE/S datastore_v3.BeginTransaction: ->1076 ms<-
...LazyJPATransactionManager$1.assureTxBegin:48
...LazyJPATransactionManager$1.createQuery:137
...AccountDAOImpl.findByEmail:36
...AccountDAOImpl.getAccount:26
...AccountDAOImplCache.getAccount:36
...Application.getUserAccount:395
...Application.trackUserActivity:400
...AppModule$1.service:229
...AppModule$2.service:291
...LazyTapestryFilter.doFilter:62
02.09.2010 0:22:19 dmitrygusev.tapestry5.gae.LazyJPATransactionManager$1 assureTxBegin
INFO: Transaction created (1200 ms) for context ...AccountDAOImpl.findByEmail:36
...AccountDAOImpl.getAccount:26
...AccountDAOImplCache.getAccount:36
...Application.getUserAccount:395
...Application.trackUserActivity:400
...AppModule$1.service:229
...AppModule$2.service:291


See also GAE and Tapestry5 Data Access Layer

Wednesday, September 01, 2010

GAE and Tapestry5 Data Access Layer

GAE provides two ways communicating with its datastore from Java:


  1. Using low-level API

  2. Using JDO/JPA (with DataNucleus appengine edition)

In this post I will try to explain some performance improvements of JPA usage. Of course, there's always some overhead using high-level API. But I use JPA in Ping Service and think it worth it.

Update (17.09.2010): There is another way to communicate with GAE datastore from Java: Objectify

Spring vs. Tapestry-JPA



Its a good practice using JPA in conjunction with IoC-container to inject EntityManager into your services. At the very beginning of development I used Spring 3.0 as IoC and for transaction management. It worked, but it takes too much time to initialize during load requests, and every time user opens its first web page, he ended with DeadlineExceededException.

Then I tried tapestry-jpa from Tynamo and it fits perfectly. It runs pretty fast and allows to:

  • inject EntityManager to DAO classes (as regular T5 services)

  • manage transactions using @CommitAfter annotation


DAO and Caching



Since GAE datastore can't operate with multiple entities in a single transaction I've added @CommitAfter annotation to every method of each DAO class.

Datastore access is a an expensive operation in GAE, so I've implemented DAO-level caching:

DAO interface

public interface JobDAO {

// ...

@CommitAfter
public abstract Job find(Key jobKey);
@CommitAfter
public abstract void update(Job job, boolean commitAfter);

DAO implementation

public class JobDAOImpl implements JobDAO {

// ...

@Override
public Job find(Key jobKey) {
return em.find(Job.class, jobKey);
}

public void update(Job job, boolean commitAfter) {
if (!em.getTransaction().isActive()){
// see Application#internalUpdateJob(Job)
logger.debug("Transaction is not active. Begin new one...");

// XXX Rewrite this to handle transactions more gracefully
em.getTransaction().begin();
}
em.merge(job);

if (commitAfter) {
em.getTransaction().commit();
}
}

DAO cache

public class JobDAOImplCache extends JobDAOImpl {

// ...

@Override
public Job find(Key jobKey) {
Object entityCacheKey = getEntityCacheKey(Job.class, getJobWideUniqueData(jobKey));
Job result = (Job) cache.get(entityCacheKey);
if (result != null) {
return result;
}
result = super.find(jobKey);
if (result != null) {
cache.put(entityCacheKey, result);
}
return result;
}

@Override
public void update(Job job, boolean commitAfter) {
super.update(job, commitAfter);
Object entityCacheKey = getEntityCacheKey(Job.class, getJobWideUniqueData(job.getKey()));

Job cachedJob = (Job)cache.get(entityCacheKey);

if (cachedJob != null) {

if (!cachedJob.getCronString().equals(job.getCronString())) {
abandonJobsByCronStringCache(cachedJob.getCronString());
abandonJobsByCronStringCache(job.getCronString());
}

cache.put(entityCacheKey, job);
} else {
abandonJobsByCronStringCache();
}

updateJobInScheduleCache(job);
}


Notice how update method implemented in JobDAOImplCache. If DAO method changes object in database it is responsible for updating all cached object instances in the entire cache. It may be difficult to support such implementation, on the other hand it may be very effective because you have full control over cache.

Each *DAOImplCache class uses two-level JSR-107 based cache:

  • Level-1: Local memory (appserver instance, request scoped)

    provides quick access to objects that were "touched" during current request


  • Level-2: Memcache (cluster wide)

    allows application instances to share cached objects across entire appengine cluster



Note that local memory cache should be request scoped, or it may lead to stale data across appserver instances. To reset local cache after each request it should be registered as ThreadCleanupListener:

    public static Cache buildCache(Logger logger, PerthreadManager perthreadManager) {
try {
CacheFactory cacheFactory = CacheManager.getInstance().getCacheFactory();
Cache cache = cacheFactory.createCache(Collections.emptyMap());

LocalMemorySoftCache cache2 = new LocalMemorySoftCache(cache);

// perthreadManager may be null if we creating cache from AbstractFilter
if (perthreadManager != null) {
perthreadManager.addThreadCleanupListener(cache2);
}

return cache2;
} catch (CacheException e) {
logger.error("Error instantiating cache", e);
return null;
}
}


Here's how LocalMemorySoftCache implementation looks like:

public class LocalMemorySoftCache implements Cache, ThreadCleanupListener {

private final Cache cache;

private final Map<Object, Object> map;

@SuppressWarnings("unchecked")
public LocalMemorySoftCache(Cache cache) {
this.map = new SoftValueMap(100);
this.cache = cache;
}

@Override
public void clear() {
map.clear();
cache.clear();
}

@Override
public boolean containsKey(Object key) {
return map.containsKey(key)
|| cache.containsKey(key);
}

@Override
public Object get(Object key) {
Object value = map.get(key);
if (value == null) {
value = cache.get(key);
map.put(key, value);
}
return value;
}

@Override
public Object put(Object key, Object value) {
map.put(key, value);
return cache.put(key, value);
}

@Override
public Object remove(Object key) {
map.remove(key);
return cache.remove(key);
}

// ...

/**
* Reset in-memory cache but leave original cache untouched.
*/
public void reset() {
map.clear();
}

@Override
public void threadDidCleanup() {
reset();
}
}


Make Tapestry-JPA Lazy



On every request Tapestry-JPA creates new EntityManager and starts new transaction on it. And at the end of request if current transaction is still active it gets rolled back.

But if all data were taken from cache, there won't be any interaction to database. In this case EntityManager creation and transaction begin/rollback were not required. But they consumed time and another resources.

Moreover Tapestry-JPA creates EntityManagerFactory instance on application load which is very expensive, though you might not need it (because of DAO cache or simply because request isn't using datastore at all).

To avoid this I created lazy implementations of JPAEntityManagerSource, JPATransactionManager and EntityManager, you can find them here: LazyJPAEntityManagerSource and LazyJPATransactionManager.