One-off Webpages That Make Life Easier

Below is a list of one-off webpages that are extremely useful in my day-to-day life as a developer. I’d love to hear what other pages I am missing out on or if you have better versions of any of these tools.

1. JSON Beautifier: http://jsonviewer.stack.hu

Simple and clean UI. Handles broken/incomplete JSON very well.

2. Convert Java Date to Millis:  http://www.fileformat.info/tip/java/date2millis.htm

Converts both ways. Nothing special. When I don’t want to load Eclipse.

3. Amazon EC2 Instance Comparison: http://www.ec2instances.info

Not sure how Amazon doesn’t have something like this. Chart form. Easy to compare. Keeps up to date as best as I can tell.

4. Ruby Regex Tester: http://www.rubular.com

Clean UI. Legend at bottom. Able to easy test many inputs.

5. Git Cheatsheet: http://cheat.errtheblog.com/s/git

Commonly used commands with descriptions.

6. AWS Service Health Dashboard: http://status.aws.amazon.com

Quickly see if a service you depend on is having trouble.

7. AWS Service Pricing: http://www.cloudomix.com/pricing

Cost of all services and comparison. A lot of information.

8. Apache Hive Functions:  https://cwiki.apache.org/confluence/display/Hive/LanguageManual+UDF

List of all built-in functions with descriptions.

9. TCP Variables:https://www.frozentux.net/ipsysctl-tutorial/chunkyhtml/tcpvariables.html

Simple list with descriptions all in one place.

10. Find Ruby Gems: https://www.ruby-toolbox.com

Quick way to find and compare gems with same purposes.

11. IP Address Lookup: http://www.maxmind.com/en/geoip_demo

Most reliable and matches most accurately.

12. Colour Palette Search: http://www.colourlovers.com/palettes/search

User generated palettes that you can search by color.

13. Online Diagram Creator: https://www.draw.io

Great for architecture diagrams.

14. Nerd jokes: http://devopsreactions.tumblr.com/archive

When you are having a bad day.

That’s all I have. These help me tremendously through my work day and help make my life more productive. What are yours?

Discuss on Hacker News.

Advertisement

Setting up EclipseLink MOXy

I wrote earlier how I found that the EclipseLink MOXy library performed great in deserialization of JSON.  I wanted to share a workaround that worked for me that I didn’t find anywhere else.

1. First here is some sample code for doing manual deserialization of JSON using MOXy.

2. My root element got annotated with @XmlRootElement.

3. All my objects were annotated with @XmlAccessorType(XmlAccessType.FIELD). Other types here.

4. All my fields with names that didn’t match how they came in over the wire, I annotated with: @XmlElement(name = “<json key>”). For example, if my POJO attribute name is “personId” but it comes in as “id”, I would annotate like:

@XmlElement(name = "id")
public Integer personId;


5. I removed my annotations and my paramters from my servlets.

6. Many sites tell you to create a jaxb.properties file in the directory where deserialization POJO’s live and add the following text. This tells JAXB which deserialization to use.

javax.xml.bind.context.factory=org.eclipse.persistence.jaxb.JAXBContextFactory


But this didn’t work for me. Instead of creating JAXBContext objects statically using

JAXBContext jc = JAXBContext.newInstance(Request.class);


I decided to generate them using the JAXBContextFactory in code:

JAXBContext jc = org.eclipse.persistence.jaxb.JAXBContextFactory.createContext(new Class[] { Request.class }, null);


And this gave me access to my POJO’s I annotated. Once I figured that out, everything worked.

Some links that I found helpful:

• Sample Code
• EclipseLink Wiki
• Specifying EclipseLink MOXy as Your JAXB Provider

JSON Deserialization and CPU

At Scout Advertising we have built an ad impression buying engine written in Java and running on Apache Tomcat 7.  At peak, our server handles ~10-15K external requests / second.  We are in the process of some major re-architecting to help us scale to 5-50x our current volume.  As part of that effort, we decided to take steps to remove our JSON deserialization away from using Jersey 1.12, which uses Jettison 1.1.

Our long-term goal is to remove our dependency on Jersey so we can explore different web architectures for handling requests.  I was tasked with removing the JSON deserialization step from Jersey and into our own module.

Criteria for new deserialization library

• deserialize to POJOs (plain old Java object) without too much custom code
• comparable speed to Jettison 1.1
• comparable CPU performance to Jettison 1.1

After researching libraries online, the general consensus is that the best JSON libraries for speed are GSON and Jackson:

http://stackoverflow.com/questions/2378402/jackson-vs-gson

http://www.linkedin.com/groups/Can-anyone-recommend-good-Java-50472.S.226644043

There is also a great benchmark for JSON library performance.  It gives you a good sense of the libraries available.  But you should always run benchmarks for your own use case, which I did below.

Try 1 – Jackson is the right man for the job

I decided to go with Jackson 1.x with data-bind.  There is a lot of good documentation and it is widely used.  We already used the library elsewhere in our codebase, so this approach wouldn’t add any more dependencies.  It also has many supporters.  The amount of effort to switch to using Jackson 1.x was minimal.  Mainly involved changing the class annotations on our POJOs.  After a good amount of testing, we released the code and everything was working fine.  We host our bidding engines on AWS and after about a week we realized our servers were running hot (CPU) and we were using ~20% more servers on average (we employ a scaling policy based on CPU).  The increase coincided with the release of the new deserialization code.

After digging through our commits, I was able to prove that the extra processing was coming from using the Jackson 1.x serialization vs Jersey’s Jettison library.

I was able to reproduce the results in our load testing environment.  Perfect!  My load tests showed Jackson was using ~15% more CPU and more memory as well.  Here are the graphs of CPU and memory from VisualVM.

Jersey 1.12 w\ Jettison 1.1

CPU is hovering just below 80%.

Jackson 1.x (with databind)

CPU is hovering between 90-95%.

Try 2: Sequels are always better?

Now that I could reproduce the behaviour, the goal was to try other libraries that would perform better.  I chose Jackson 2.x (with databind) over GSON since the only thing I had to do to switch was including a different library.

But still no luck.  CPU was just as high.

Try 3: EclipseLink MOXy

I stumbled upon MOXy, which also uses the JAXB annotations to build objects.  Getting the code up and running took a little bit of time.  But once I got it working, I proved that MOXy used much less CPU than Jackson, and slightly less than Jettison.  It also didn’t noticeably change our latencies, which was also a requirement.

 

I will be writing another post on how I used MOXy, since I had some trouble and no other tutorials that I found had worked for me.

In conclusion, we will be trying MOXy in production.  It provides the speed without blowing out our CPU.  I could’t find anything else on the web that compared CPU performance.  Most benchmarks I found, compared speed.

JUL vs Logback vs Blitz4j

Over the last little while I have been trying to improve the performance of our java web app running on Tomcat.  Over the past 2 days, I decided to test our current logging framework against a few other competitors.  There are many blog posts about which is the fastest but nothing beats your own testing!

We current use SLF4J which is great because it makes exercises like this one much easier.  It basically allows you to plug and play many java logging libraries without changing any code (you just need to add certain JARs to the classpath).  Our current implementation uses java.util.logging (JUL) a.k.a JDK14 logging.

Our goal: Find the fastest java logging framework that can be switched to in < 2 days of effort.  We operate in a very low latency environment where speed is everything.

Contenders: JUL, Logback, Bllitz4j (not log4j b/c of reasons below)

If you poke around “common knowledge” on the internet is:

– Most people use log4j

– Logback is 10x faster than log4j

– Blitz4j (Netflix’s improvement to log4j) is 3x faster than log4j

So going into the experiments I was expecting to see Logback or Blitz4j to come out on top.

The real problem was that I couldn’t find any articles that compared (and the reason why I wanted to post this article) JUL logger to other frameworks.

So I setup a dummy endpoint that would flex all the logger muscles, nothing too complicated as below. (the idea is the same, some syntax for the loggers had to be changed a bit)

logger.debug("Debug Message");logger.info("Info Message");logger.warn("Warn Message");logger.error("Error Message");logger.trace("Trace message");logger.error(aShortString);if (logger.isDebugEnabled())  logger.debug("Debug Message With Check");if (logger.isInfoEnabled())  logger.info("Info Message With Check");if (logger.isWarnEnabled())  logger.warn("Warn Message With Check");if (logger.isErrorEnabled())  logger.error("Error Message With Check");if (logger.isTraceEnabled())  logger.trace("Trace message With Check");logger.error(aLongLongString);

I deployed my app to an EC2 host (m1.large) running on Apache Tomcat 7.  Versions were:

– JUL 1.4, LogBack 1.0.9, Blitz4J 1.18, SLF4J 1.7.2

For our performance testing we use JMeter.  I used both JMeter and Cloudwatch to gather the timing metrics.  I used 4 JMeter hosts which each had 3 threads making requests, so I could make sure that the logging framework could handle concurrent requests well.

Below are the results and as you can see, JUL took the cake!  I definitely didn’t expect that to happen and it was by a large margint the victor!  Since I didn’t test just plain old Log4J, I didn’t test the claims by Logback or Blitz4j.

A. Average request time (single iteration per request)

• JUL 1.4 – 5 ms
• Logback 1.0.9 – 15 ms
• Blitz4j 1.18 – 18 ms

B. Average request time (100 iterations per request)

• JUL 1.4 – 761 ms
• Logback 1.0.9 – 1412 ms
• Blitz4j 1.18 – 1267 ms

 

Anyways, thats all.  It was just a small test but I was very surprised to say the least.  I know Netflix’s library can probably take a lot more load than this, and might not have been a fair comparison.  Looks like we will be sticking with JUL for the short-term unless something comes up that warrants a switch.

Remote Java Profiling on EC2

Had to do some profiling and there are a bunch of articles strewn across everywhere, so thought I would help the cause for people trying to find them. 

Props to Mriddle: http://rukuro-blog.heroku.com/2011/07/26/running-visualvm-through-an-ssh-tunnel

Another good article by Neil Figg: http://neilfigg.blogspot.com/2011/07/remote-jvm-profiling-on-amazon-ec2.html

I didn’t use Neils because I didn’t need to go deeper in my profiling at this stage.  I ran into a bunch of gotchas which Mriddle helped with and some comments:

1.  Create an SSH tunnel and make visual VM proxy through that:

– sudo ssh -i .ssh/yourkeyfileifyougotone.pem -D 9696 your.ip.goes.here

(options to add to jvisualvm)

-J-Dnetbeans.system_socks_proxy=localhost:9696 -J-Djava.net.useSystemProxies=true)

2.  Command to check if jstatd is listening:

sudo netstat -nlp | grep jstatd

3.  Look at startup command for tomcat

ps aux | grep tomcat

4.  VisualVM looks at temp directories for process info, so make sure that you start jvisualvm and specify the proper temp dir (you can see it from step #3):

(option to add to jvisualvm)

 -J-Djava.io.tmpdir=/opt/tomcat7/temp

 

Things that I read but didn’t work/didn’t try:

1.  Make sure the java bin you are using matches that on the remote server.  Apparently, there is a difference between the jre java bin and jdk java bin.

 

2.  Open the proper ports after your instance comes up on EC2.  Since jstatd picks a random port, you need to add the port to your security policy.

 

ec2-authorize <security_group> -p <port> -s <your ip>

 

My env on EC2:

OpenJDK Runtime Environment (IcedTea6 1.11.4) (amazon-52.1.11.4.46.amzn1-x86_64)

OpenJDK 64-Bit Server VM (build 20.0-b12, mixed mode)

Linux version 3.2.12-3.2.4.amzn1.x86_64 (mockbuild@gobi-build-31003) (gcc version 4.4.6 20110731 (Red Hat 4.4.6-3) (GCC) )

Apache Tomcat/7.0.23

Locally:

VisualVM: 1.6.0_33 (Build 110613); platform 110613-unknown-revn

Mac OS X (10.7.4) , x86_64 64bit

1.6.0_33; Java HotSpot(TM) 64-Bit Server VM (20.8-b03-424, mixed mode)