What else is running on my EC2 instance?…

At Thinkear we use a lot of Amazon Webservices. We use Elastic Compute Cloud (EC2) to host our Apache Tomcat server running Java 6. We recently had some major performance issues with our service, which lead us to analyze our EC2 hosts and figure out what was running on it.

At peak hours our servers handle ~35K requests/second and we have a 100 ms SLA to maintain with our partners. In this type of environment performance is the top priority and we were surprised by some of the things we found running on our EC2 instances. I thought I would share what we found. Some of it was surprising.  Some of it was documented after you knew what to look for – but I find the Amazon docs hard to navigate. Throughout our interactions with Amazon support, we found that not all representatives were aware of some of these points below.

For an AMI we have the Amazon Linux  X86 64-bit (ami-e8249881) running a Tomcat 7 Linux Configuration.

1. Apache runs as a proxy.

Our load balancer (AWS Elastic Load Balancing), directs port 80 traffic to port 80 on the hosts. Then Apache runs a proxy that forwards requests from port 80 to port 8080 (default Tomcat port).

Config files are located in /etc/httpd/conf.d and /etc/httpd/conf. We had to tweak settings in /etc/httpd/conf/httpd.conf based on our use case. These settings were the root cause of our issues. We had never looked into them because everything seemed to work.

We tried by-passing Apache because we didn’t need the features it brings. Unfortunately, we had issues with our servers on deployment when we by-passed apache. We haven’t found the root cause of this as of yet.

2. Logrotate Elastic Beanstalk Log Files

elasticbeanstalk.conf in /etc/httpd/conf.d/ defines the ErrorLog and AccessLog properties for Apache. These files are then rotated out by /etc/cron.hourly/logrotate-elasticbeanstalk-httpd. The problem was that we didn’t know these log files existed and we felt the settings were too aggressive for us.

These are our current settings: https://gist.github.com/KamilMroczek/7296477. We changed the size parameter to be 50 MB and to only keep 1 rotated file. Smaller files take less time to compress. We didn’t need all those extra copies.

3. Logrotate Tomcat Log Files

logrotate-elasticbeanstalk in /etc/cron.hourly defines rotating catalina.out and localhost_access_log.txt out of the Tomcat logs directory! As nice as it is for them to do that, we had no idea. It didn’t have a large impact on us, since we handled the rotating of our log files ourselves already at shorter intervals. We ended up removing this unnecessary step anyway.

Original Log rotate script: https://gist.github.com/KamilMroczek/7296539

4. Publishing logs to S3

We noticed that we had CPU spikes at 20 minute intervals on our hosts at 10, 30 and 50 minutes passed the hour. We couldn’t explain these. When we looked at our CPU usage through top we found the culprit.

/etc/cron.d/publish_logs is a python script that publishes:

• /var/log/httpd/*.gz (#2 above)
• /var/log/tomcat7/*.gz (#3 above)

I originally thought that we were uploading the same files a ton since the logrotate only rotated every hour and kept the last 9 copies, but the publishing happened 3 times an hour. But we found out that the code has de-duplicating logic.

We removed this cron task because we didn’t need the data uploaded. We already uploaded our tomcat logs separately and the beanstalk logs were of no use to us at the time. Nor have we ever used them to troubleshoot issues.

5. Amazon Host Configuration

The entire configuration for your environment can be found through Amazon Cloud Formation. There is a describe-stacks (or cfn-describe-stacks depending on the CLI version) call that allows you to pull the entire configuration for an environment. We are in the process of auditing ours. More complete instructions are here:

http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/using-cfn-describing-stacks.html

As will all tough problems, after troubleshooting them you inevitably get a deeper understanding of your system and its architecture. When you own and provision your own servers you understand everything on them because you are responsible for creating the template. When you use a hosted solution such as Amazon Webservices, you can run into the problem of not knowing everything about your image. But we learned that you need to take the time to understand what you are getting.

Advertisement

Setting up EclipseLink MOXy

I wrote earlier how I found that the EclipseLink MOXy library performed great in deserialization of JSON.  I wanted to share a workaround that worked for me that I didn’t find anywhere else.

1. First here is some sample code for doing manual deserialization of JSON using MOXy.

2. My root element got annotated with @XmlRootElement.

3. All my objects were annotated with @XmlAccessorType(XmlAccessType.FIELD). Other types here.

4. All my fields with names that didn’t match how they came in over the wire, I annotated with: @XmlElement(name = “<json key>”). For example, if my POJO attribute name is “personId” but it comes in as “id”, I would annotate like:

@XmlElement(name = "id")
public Integer personId;


5. I removed my annotations and my paramters from my servlets.

6. Many sites tell you to create a jaxb.properties file in the directory where deserialization POJO’s live and add the following text. This tells JAXB which deserialization to use.

javax.xml.bind.context.factory=org.eclipse.persistence.jaxb.JAXBContextFactory


But this didn’t work for me. Instead of creating JAXBContext objects statically using

JAXBContext jc = JAXBContext.newInstance(Request.class);


I decided to generate them using the JAXBContextFactory in code:

JAXBContext jc = org.eclipse.persistence.jaxb.JAXBContextFactory.createContext(new Class[] { Request.class }, null);


And this gave me access to my POJO’s I annotated. Once I figured that out, everything worked.

Some links that I found helpful:

• Sample Code
• EclipseLink Wiki
• Specifying EclipseLink MOXy as Your JAXB Provider

JSON Deserialization and CPU

At Scout Advertising we have built an ad impression buying engine written in Java and running on Apache Tomcat 7.  At peak, our server handles ~10-15K external requests / second.  We are in the process of some major re-architecting to help us scale to 5-50x our current volume.  As part of that effort, we decided to take steps to remove our JSON deserialization away from using Jersey 1.12, which uses Jettison 1.1.

Our long-term goal is to remove our dependency on Jersey so we can explore different web architectures for handling requests.  I was tasked with removing the JSON deserialization step from Jersey and into our own module.

Criteria for new deserialization library

• deserialize to POJOs (plain old Java object) without too much custom code
• comparable speed to Jettison 1.1
• comparable CPU performance to Jettison 1.1

After researching libraries online, the general consensus is that the best JSON libraries for speed are GSON and Jackson:

http://stackoverflow.com/questions/2378402/jackson-vs-gson

http://www.linkedin.com/groups/Can-anyone-recommend-good-Java-50472.S.226644043

There is also a great benchmark for JSON library performance.  It gives you a good sense of the libraries available.  But you should always run benchmarks for your own use case, which I did below.

Try 1 – Jackson is the right man for the job

I decided to go with Jackson 1.x with data-bind.  There is a lot of good documentation and it is widely used.  We already used the library elsewhere in our codebase, so this approach wouldn’t add any more dependencies.  It also has many supporters.  The amount of effort to switch to using Jackson 1.x was minimal.  Mainly involved changing the class annotations on our POJOs.  After a good amount of testing, we released the code and everything was working fine.  We host our bidding engines on AWS and after about a week we realized our servers were running hot (CPU) and we were using ~20% more servers on average (we employ a scaling policy based on CPU).  The increase coincided with the release of the new deserialization code.

After digging through our commits, I was able to prove that the extra processing was coming from using the Jackson 1.x serialization vs Jersey’s Jettison library.

I was able to reproduce the results in our load testing environment.  Perfect!  My load tests showed Jackson was using ~15% more CPU and more memory as well.  Here are the graphs of CPU and memory from VisualVM.

Jersey 1.12 w\ Jettison 1.1

CPU is hovering just below 80%.

Jackson 1.x (with databind)

CPU is hovering between 90-95%.

Try 2: Sequels are always better?

Now that I could reproduce the behaviour, the goal was to try other libraries that would perform better.  I chose Jackson 2.x (with databind) over GSON since the only thing I had to do to switch was including a different library.

But still no luck.  CPU was just as high.

Try 3: EclipseLink MOXy

I stumbled upon MOXy, which also uses the JAXB annotations to build objects.  Getting the code up and running took a little bit of time.  But once I got it working, I proved that MOXy used much less CPU than Jackson, and slightly less than Jettison.  It also didn’t noticeably change our latencies, which was also a requirement.

 

I will be writing another post on how I used MOXy, since I had some trouble and no other tutorials that I found had worked for me.

In conclusion, we will be trying MOXy in production.  It provides the speed without blowing out our CPU.  I could’t find anything else on the web that compared CPU performance.  Most benchmarks I found, compared speed.

Why does your login page suck?

Everyone has their pet peeves.  I am not a usability expert and for a software engineer I am not particularly adept at creating dazzling user interfaces (UI).  But I do appreciate well designed UIs.  I wanted to call out some of the poor login experiences I know of.  I understand that login screens are probably one of the first screens designed on a site.  And it could be forgotten amongst the other parts of a site.  But it will be one of your most trafficked pages.  I don’t think I have ever stopped using a site because it had a poor login experience.  But a part of me dies inside each time I logon to those sites.

1. Usernames

Why can’t I just login with my email address?!?  It guarantees uniqueness and you ask for it anyway, so you can email me.  Now I have to make up some funky unique name like MarkStevens1234, which god forbid if I ever have to remember it.  Well Mark Stevens might have to.  But you get the point.

I am okay if the login accepts both email and username.  But please state that.

Many banking sites do this and I guess they use this as an extra layer of security.  I am not 100% if I buy that argument.  A username could probably be guessed pretty easily by knowing the email address of a person and/or other usernames they use.

Don’t believe me?  Try taking mark@gmail.com.  Probably won’t be available.  Neither will mark5 nor mark100.

Examples: www.401k.com, www.wellsfargo.com

2. Arbitrary Numbers as a Login ID

Even better!  Now I have to remember a number that I couldn’t guess if I my life depended on it.  I just hate this.  It’s like 2 passwords.  How about a longer password?

Examples: www.arbonne.com, www.capitalone.com

3. Short Password Lengths Limits

It is absolutely baffling that some sites limit passwords to 8-12 characters.  Longer passwords are more secure.  So why would they limit them?  This is the opposite direction we are supposed to moving.  The only excuse I can see there being is that it is a legacy system.  I can’t think of any other reason out of pure laziness.  It is not hard to widen a database column from 12 characters to 24/32/64 characters.

TD Canada Trust finally fixed this on their site. Here is an image of their site a little while ago.

 

(EDIT: July 28, 2013)

Pink Dot @ 247waiter.com.  A piece of me died inside.

4.  Improper redirecting post login

If you are clicking on a link from an email and you need to sign in.  You expect that you will be taken to the page you wanted.  I hate sites that always take you to a standard page.  Now you need to hit back a few times or re-click the link.  Premier League website does this.

5. Plaintext passwords

After signing up for a site, some sites send you an email with your password in plaintext.  Nice.

Many people believe email is a lot more secure than it is.  And it isn’t insecure because people break into accounts.  Anyone reading traffic from the millions of internet nodes could be reading emails that are being sent through them.  I have signed up for a few sites that send passwords in plain text. plaintextoffenders.com has a great list though!

6.  Login with Facebook

I hate the way they do this.  This will not be widely accepted until you can use the mechanism without giving the app/site permissions to your Facebook account.  I almost never use it unless I want the app interacting with my account.  They should allow you to choose what permissions you allow.  Many times these apps ask for a lot of permissions.  If they do these things, they will be used much more.

Login experiences are getting better.  Most follow a pretty standard template.  But there is still this big problem of remembering passwords and following the rules of secure passwords.  It takes too much time and effort.  I like ideas like OpenID.  Password saving plugins are good too, like LastPass.

We Can Do Better Than Capitalism

I am sure I have read somewhere that you shouldn’t write publicly when your emotions are out of whack.  Oops.  I caught up on some of the news this week about the bill (H.R. 933) that passed Congress and it makes me sad.  I don’t like being sad.  Just to be frank, I am not super connected to the political world.  I get 95% of my political news from John Stewart – I frankly don’t have enough time in my life to wade through most of the garbage that comes out of politicians mouths – I can’t blame them, the media scares them into groupthink.  John Stewart hits the tip of the iceberg and if I find something interesting I will read some articles.

So when I heard about the provision that was added to H.R. 933 I got really angry.  If you haven’t read the text it is below.  From section 735:

“In the event that a determination of non-regulated status made pursuant to section 411 of the Plant Protection Act is or has been invalidated or vacated, the Secretary of Agriculture shall, notwithstanding any other provision of law, upon request by a farmer, grower, farm operator, or producer, immediately grant temporary permit(s) or temporary deregulation in part, subject to necessary and appropriate conditions consistent with section 411(a) or 412(c) of the Plant Protection Act, which interim conditions shall authorize the movement, introduction, continued cultivation, commercialization and other specifically enumerated activities and requirements, including measures designed to mitigate or minimize potential adverse environmental effects, if any, relevant to the Secretary’s evaluation of the petition for non-regulated status, while ensuring that growers or other users are able to move, plant, cultivate, introduce into commerce and carry out other authorized activities in a timely manner: Provided, That all such conditions shall be applicable only for the interim period necessary for the Secretary to complete any required analyses or consultations related to the petition for non-regulated status: Provided further, That nothing in this section shall be construed as limiting the Secretary’s authority under section 411, 412 and 414 of the Plant Protection Act.”

Anyways the gist is, if the government finds out that an approval for a genetically modified crop was acquired illegally, the USDA is required to ignore a court’s decision finding the agency approved a crop illegally until it can investigate more thoroughly”.  Isn’t this backwards thinking?  Shouldn’t it be, “lets see if it is healthy before distributing it” instead of “let’s distribute it until we find it is not healthy”.  The US has routinely made big blunders in terms of public health – compare the number of substances banned the US by the FDA in cosmetic products vs in Europe.

Too many of the critics and supporters of the bill are worrying about the wrong parts of the issue. Details that the item was added anonymously and the provision was in there for more than a year.  WHO CARES.  The important part is why did it get into the bill?  For critics, focusing on this insignificant pieces of information gives the supporters rebuttal power that detracts from the actual issues.

I read an article by Jon Entine on Forbes.  His logic could be cut down by anyone with just a little bit of reasoning.  For example,

To date, no court has ever held that a biotechnology crop presents a risk to health, safety or the environment

Since he doesn’t state it, I assume he means that we shouldn’t be worried that any crop ever will harm humans.  Or that they have never done anything wrong, thus they could never possibly in the future.  Both would be ridiculous claims.  Take this scenario: What if a biotechnology company influenced the court’s decision by using money or power…just like Monsanto did by increasing the amount of money it has donated to Roy Blunt, senator from Missouri and the man who helped insert the provision.  I am not saying that is true, but rather it is not that large of a stretch.

Too many people get stuck in the weeds and can’t see the overall big picture.  Why would we subvert the judicial system’s power for big business?  The more rulings I hear congress/supreme court, the more I am saddened.  This reminds me of the “Corporations are people” debate.  How could anyone not logically conclude that if you do not cap campaign donations by corporations, then politicians will be controlled by people with the most money (e.g. corporations).

Many people in the US believe capitalism is the perfect system.  They are wrong.  It is the best we have so far.  One major downfall is that money is power and it rules all.  Money has more influence than government (hence why senators take money and vote “on behalf” of companies).

South By…A First Timers Perspective

Last weekend I got to attend SXSW for the Interactive portion of the conference/festival (SXSWi).  It was also my first time staying at an AirBnB.  I can say both experiences were very memorable and awesome.

Every programmer/software engineer/coder/developer should go to South By at least once because it is truly a unique experience.  I haven’t been to many conferences but this one straddles the line very well between party/festival and conference.  There are probably 100 sessions a day, including lectures, panels, meetups, hackathons, Q&A and competitions.  For the interactive portion, there is a healthy mix of developers, graphic designers, UX people and app developers.  There are so many sessions that looking at the schedule feels like drinking from a fire hose.  Since there are so many sessions, you are free to choose your own adventure.  Don’t like the talk you are at, leave for another session.  Met some cool people after a talk, join them for a drink or at their next session!  There will be many sessions that you will want to attend that are going on concurrently.  Most venues are within a 10 minute walk of the Austin Convention Center which makes it very walkable and easy to get around.

Everyone from the speakers to the volunteers to the organizers, preach about the “hallway magic” that occurs at South By.  They encourage you to leave your friends and try to meet new people.  They encourage you to note the sessions you’d like to attend but to embrace the serendipity of South By – you never know who you will meet and when you will meet them.  At the start of the festival I was a bit timid in meeting people, but by the last day I was arriving at the convention center with nothing to do for 2 hours and just heading to a phone/laptop recharge lounge to chat with random people.  I was even going to the parties at night by myself (first time going to the bar by myself) and trying to meet as many new people as I could.

SXSW is really what you make of it.  You can party from 11am to 1am every day.  Or you can attend sessions from 9am – 6pm and not party at all.  You could even arrive without a official badge and enjoy all the free stuff and network.

My favourite sessions I attended:

• Hackathon for Social Good – got to meet some cool people and do some work!
• Elon Musk Keynote – loved his no bullshit attitude, his humility and bravery
• 100 Mistakes from contributor to CEO – Meebo cofounder Elaine Wherry – so engaging and learned a ton!
• Next Gen Entertainment Entrepreneurs – Vhx.tv, slated.com, blcklst.com – Panel discussion on breaking through Hollywoods barriers

I know I did it right because I wouldn’t really change anything about my trip.  I had a great time.  Now for a list of tips:

1. Don’t hang out with your friends – put yourself in a situation where you have to meet new people
2. Know that you can’t get to everything and be okay with that
3. Drink lots of water because you will be consuming more alcohol than you think
4. For most parties RSVP doesn’t get you anything – I used rsvpster but I didn’t use it once to get in….I still don’t get why they make you rsvp. (I still will probably RSVP next year just in case)
5. You will be disappointed by how bad (or not what you expected) some sessions are and surprised by how good some other ones are – be flexible with your schedule

See you in SXSW next year!

Every Software Engineer Needs Rules To Code By

As software engineers, we make lots of decisions on a daily basis.  It is definitely a skill that you can develop and improve upon.  From my experience, you get much more practice at this skill in a startup.  You get to build a lot of new software with not much guidance.  Many times you need to pick the technology/library/etc before you even start developing a solution.  I feel my decision making has improved tremendously in the year that I have worked for a start-up.  There are many ways you can improve your decision making ability.  One helpful mechanism for decision making, is having a set of rules that you live (design/code/test/implement) by.

Every software engineer should have a set of rules they live by.  Most developers have 0 rules when they finish university and start their first job.  The main reason for this is that we “gather” new rules through experience of screwing things up.  And I am referring to those bugs or time wasters that really cost you or your company time and money.  In school, you usually aren’t working on large pieces of software or mission critical features.  You also don’t own code for a substantial period of time usually.  So you don’t carry the burden of bad code or see the result of bad decisions.

The typical scenario goes something like:

1. Create and release some feature
2. Something very bad happens. (e.g. bug costing thousands/millions, spend 3 days debugging a problem, spend 3 extra days refactoring, etc)
3. You fix it.
4. You think to yourself, how do I not let this happen again.

There are a ton of other ways it happens, but from personal experience this is the most common scenario.  So people amass rules depending on the experiences they have had.  Very rarely do we learn from someone else’s mistakes.  One of my favourite quotes:

“Smart people learn from their own mistakes.  Smarter people learn from other peoples mistake.”

I think these rules are important to have because they help prevent mistakes and make decisions easier.  The less questions you have to ask yourself, the less decision fatigue you will experience.  Leave your decision making power to the more difficult decisions.

Without further adieu, some rules off the top of my head.

1. No code change should go untested.  No matter how small or trivial.  I don’t mean unit test every single line of code, but see it work.  Too many times I decided not test something because it was too small or trivial.  Then subsequently it broke in testing or production and I looked really dumb for not performing a basic test.  I have seen this happen to other engineers as well.  It usually is painfully obvious that no testing was done.
2. Don’t try to predict the future.  I have spent too much time thinking of what might come next when developing certain features.  I have been burned implementing extra features (feature creep) and then throwing it all away anyway because the system was going to be used in the way I foresaw.  This is especially true for marketing oriented organizations as they tend to change their mind a lot.
3. When you can’t decide, just pick a route and go from there.  I used to spend a lot of time comparing 2 different approaches to a problem.  Often there wasn’t an obvious winner and I would spend too much time splitting hairs.  Now, when I realize that I am doing this, I pick a solution and go from there.
4. Have more than one solution.  I read somewhere that having only one idea is dangerous.  Too many times I have implemented a feature only to have a co-worker suggest a different (more logical) approach which had me scrap half my code.  Now I try to think of these before I start designing.
5. Behave as if things will go wrong.  I know this is on the “glass half-empty” side, but this sort of thinking has helped me.  How many times have you finished writing a piece of code and went to run/test it and it worked first try?  You will start to ask yourself valuable questions such as; how can this break? If it breaks, how will I know?  how will I debug it?

These are a few that I have learned and wanted to share.  I am still early in my career and I am sure this list will grow!

Make your RSpec tests better

Below is a (non-exhaustive) list of good practices that I developed when writing my RSpec tests.

1. Use context blocks a lot.  TIP: Write your context blocks first and then fill in the tests.  It helps re-use and organization.

GOOD:

context "campaign has locations" do
  context "when locations < 10" do
    ...
  end

  context "when locations >= 10" do
    ...
  end
end

BAD: (notice double context block, doesn’t promote code sharing because you need to setup the same scenario multiple times. Also if someone modifies the code, they will have to modify in multiple places)

context "campaign has locations" do
  context "when locations < 10" do
    ...
  end
end

context "campaign has locations" do
  context "when locations =< 10" do
    ...
  end
end

2. If you have too many context blocks, that MIGHT mean you need to refactor your method to smaller methods.

3. Make sure that cases belong to their proper context block:

BAD:

context "campaign has locations" do
  context "when locations < 10" do
    ...
  end
  context "when locations >= 10" do
    ...
  end
  context "when locations = 0" do #<--- BAD (should be outside context)
    ...
  end
end

4. Write unit tests for single methods first.  You can skip for small “trivial” methods and include tests for small methods in the calling methods.  Worry about integration last.

5. I usually write unit tests will full stubs to verify input and output.  I might throw in a few tests to test the interaction and method chain.

6. Make sure you DON’T writes tests that test other components. Assume dependent components are written correctly and stub them out.  Good example of this is for external APIs.

7. Make sure your test breaks if you make it false.  I can’t count how many times it hasn’t for myself.

8.  For loops, I usually write a test case to make sure everything happens correctly for 1 iteration.  Then I have a test case to test for n (usually between 2-5) iterations.  TIP: extract the entire contents of the loop into its own sub method.

9. In your before block, if you are testing a certain condition on an object.  Explicitly set that, and don’t rely on Factory defaults:

BAD:

context "when campaign budget is NULL" do
  before do
    @campaign = Factory :campaign   #factory sets campaign budget = nil
  end
end

GOOD:

context "when campaign budget is NULL" do
  before do
    @campaign = Factory :campaign, :budget => nil
  end
end

The Ruby Way == Idiomatic Ruby

Almost 1 year ago, I joined a great start-up Thinknear (acquired by Telenav).  From a software engineer’s standpoint I did a complete 180 in the direction I was heading (outside of the fact that I was dabbling in Project Management at my old company Vistaprint).  

My previous company, Vistaprint, was a .NET shop for the most part.  I worked with C#, VB.NET, ASP.NET, IIS and MS SQL Server.  When I joined Thinknear, it was completely different.  We used Java, Ruby on Rails, Heroku, Apache Tomcat, Memcached, etc (the list could go on for awhile).

Anyways, I took learning Ruby to heart.  It had be awhile since I learned a new language and it was quite different than anything I had done before.  As I mentioned in earlier posts, I started from the ground up with tryruby.org.  I had about 3 weeks between knowing I got the job and starting the new gig (mainly because I had to transfer my visa).

I remember my boss (if you can call him that on a 3 person dev team), told me you can always spot a new Ruby person who came from an OO language.  It was hard for me to understand at the time, but now I get it.  I feel I am in a position to help the Ruby new comers.  So I want to introduce you to the “The Ruby Way” (a.k.a “Idiomatic Ruby”).

When I first started at Thinknear my code reviews were painstaking.  My teammates didn’t let up in learning Ruby and they made sure I would do everything properly.

The one thing that helped me the most was writing done every non-idiomatic mistake I made on code reviews and made sure I looked over the list before I submitted to GitHub.  I made very few mistakes more than 3 times, which believe me…was a HUGE step up from where I was coming from.

I am only 1 year into Ruby, but I am loving it!  It is a very natural and concise language.  It also helps you focus on the core algorithms and problems instead of a lot of other nuances in other languages.  I will be soon be ready to read my Metaprogramming in Ruby book, which will be fun and probably mind bending.  But here are my list of Ruby/Rails Idioms that I learned early on (low hanging fruit):

1. Ruby Style Guide – essential if you want to contribute to projects
2. Your best friend which you will be using a lot: http://ruby-doc.com/
3. Your best friend which you can try anything: irb
4. Do not use for i = 0; i < array.size, i++ etc…. use iterators!
   • Iterate over each element array.each { |element| puts element }
   • Iterate over each element, but transform each item: array.map { |element| element + 1 }
   • So many cool iterators: select, reject, inject
5. Do not use array.length > 0 or array == nil
   • array.present? is the bomb! It checks for nil AND to see if there are any items in the list (Rails only) – this also works for hashes
   • Checking for nil on any object? Use object.nil?
   • Checking for emptiness? array.empty?
   • Opposite of empty? = array.any?
6. Worried that something might be nil? try is your friend! Especially useful for hashes: Instead of hash.has_key?('missing-key') && hash['missing-key'] == value you can use hash.try[:[], 'missingkey') == value). Try will return “nil” if it can’t return anything.
7. Need the opposite of a condition? Do not write if !condition ... end , but instead use unless! unless condition ... end
   • Bonus points for writing conditionals on single line: do_something if condition

That is all I can think of, but I wanted this to be short and sweet and introduce some glaring Ruby newbie mistakes.  There is so much to offer in the language and I hope you have as much fun as I do!

JUL vs Logback vs Blitz4j

Over the last little while I have been trying to improve the performance of our java web app running on Tomcat.  Over the past 2 days, I decided to test our current logging framework against a few other competitors.  There are many blog posts about which is the fastest but nothing beats your own testing!

We current use SLF4J which is great because it makes exercises like this one much easier.  It basically allows you to plug and play many java logging libraries without changing any code (you just need to add certain JARs to the classpath).  Our current implementation uses java.util.logging (JUL) a.k.a JDK14 logging.

Our goal: Find the fastest java logging framework that can be switched to in < 2 days of effort.  We operate in a very low latency environment where speed is everything.

Contenders: JUL, Logback, Bllitz4j (not log4j b/c of reasons below)

If you poke around “common knowledge” on the internet is:

– Most people use log4j

– Logback is 10x faster than log4j

– Blitz4j (Netflix’s improvement to log4j) is 3x faster than log4j

So going into the experiments I was expecting to see Logback or Blitz4j to come out on top.

The real problem was that I couldn’t find any articles that compared (and the reason why I wanted to post this article) JUL logger to other frameworks.

So I setup a dummy endpoint that would flex all the logger muscles, nothing too complicated as below. (the idea is the same, some syntax for the loggers had to be changed a bit)

logger.debug("Debug Message");logger.info("Info Message");logger.warn("Warn Message");logger.error("Error Message");logger.trace("Trace message");logger.error(aShortString);if (logger.isDebugEnabled())  logger.debug("Debug Message With Check");if (logger.isInfoEnabled())  logger.info("Info Message With Check");if (logger.isWarnEnabled())  logger.warn("Warn Message With Check");if (logger.isErrorEnabled())  logger.error("Error Message With Check");if (logger.isTraceEnabled())  logger.trace("Trace message With Check");logger.error(aLongLongString);

I deployed my app to an EC2 host (m1.large) running on Apache Tomcat 7.  Versions were:

– JUL 1.4, LogBack 1.0.9, Blitz4J 1.18, SLF4J 1.7.2

For our performance testing we use JMeter.  I used both JMeter and Cloudwatch to gather the timing metrics.  I used 4 JMeter hosts which each had 3 threads making requests, so I could make sure that the logging framework could handle concurrent requests well.

Below are the results and as you can see, JUL took the cake!  I definitely didn’t expect that to happen and it was by a large margint the victor!  Since I didn’t test just plain old Log4J, I didn’t test the claims by Logback or Blitz4j.

A. Average request time (single iteration per request)

• JUL 1.4 – 5 ms
• Logback 1.0.9 – 15 ms
• Blitz4j 1.18 – 18 ms

B. Average request time (100 iterations per request)

• JUL 1.4 – 761 ms
• Logback 1.0.9 – 1412 ms
• Blitz4j 1.18 – 1267 ms

 

Anyways, thats all.  It was just a small test but I was very surprised to say the least.  I know Netflix’s library can probably take a lot more load than this, and might not have been a fair comparison.  Looks like we will be sticking with JUL for the short-term unless something comes up that warrants a switch.