Cue dreamy effect and go back in time…

About 3 years ago, we bought a product as the basis for our learning management system. The learning system coordinator didn’t want the users accessing it directly (having to learn a whole new UI) so we wrote a few custom portlets to browse the catalog and track your progress using the APIs of the product. The APIs seemed kinda clunky, but they worked.

Fast forward back to this morning

After trying all of our known ‘fixes’ to take care of the problem without success I had to dig. I found some funniness in the logs. Our app was getting a java.rmi.RemoteException from the web service when trying to execute the API calls. Basically the error message was zero help. We had made no changes to the code. We had not updated the product in any way. So why is this happening? We determined is was definitely an API issue only – we could use the out of the box client and do all functions just fine. We called in a support ticket to the vendor. Luckily, they could tell us that the exception we were seeing would only be thrown in a scenario where we were providing either a null value or a zero length string as a parameter to the API method call. I knew that we hadn’t changed any of the code or the product for a looooong time so I was confident that we would prove to the support tech that she was wrong and we were sending in a value so I added some logging statements to the app and redeployed it.

Shock and dismay

Immediately, the logs proved that we were sending in a null value.

“What the #$%!! This has worked for 6 months with no changes. The only thing we did was change the logging level 2 days ago from DEBUG to ERROR and that should have no affect on this kind of thing. Its worked since then… right?”

Never trust other people

Just to be sure, I checked with the originator of the error (the guy who gets the calls when something is wrong or a change is needed – and a good all around guy):

Me: “Hey… this has been working OK since Monday, right?”.

Him: “Yeah. It was fine yesterday”.

…another 15 minutes pass trying to figure out how it could have ever worked since we’re passing a null value as a parameter.

Finally I came out of my funk and decided I just had to fix it with what I know now rather than trying to figure out what changed.

The realization – Not crazy, just stupid

As I looked into what the problem was, I found that a value for an attribute that we get from calling a utility library was null rather than a value pertaining to the logged in user. “How could this be null? Its been working forever!”. I seriously thought logic was not prevailing, that something had gone awry in not one, but three servers and was conspiring to make me insane. After another 5 minutes, I found that there was a VERY KEY line of code inside of an if block where the condition was ‘logger.isDebugEnabled’. So changing the log level from DEBUG to ERROR caused a major issue in the program since it skipped that key line of code.

Uhhhhhh

Immediately I tried to understand why it took me so long to figure out the problem. I reckoned if I’d have known right after the change that something was wrong I could have easily tracked it to the logging issue (and still kicked myself in the arse for such a silly bug). But since I heard from someone that it just broke ‘this morning’ I went on a wild goose chase.

This bug, had ’slept’ for nearly 3 years without notice since the logging level was left at DEBUG on the QA and production servers.

Lessons learned

- Test with your production logging level
- Systems that aren’t mission critical can have an issue for a good long time before someone actually complains about it
- Dont trust what anyone else says when it comes to observation of a software problem (This is why when you call support for any product you start at step 1, even though you are a seasoned computer person and dont need to get instructions on what alt-ctrl-delete means – apparently they’ve already learned the lesson I learned AGAIN today)
- Accept that some bugs are just plain silly and no matter how experienced you are, you can create them.

As excited as I am about the new position, I am still sad to leave Perficient. They have been nothing but good to me. I really like the atmosphere they provide. Some of the smartest people I’ve ever met are in key positions at Perficient so I am sure they’ll be at the top of my list when we need outside expertise.  I don’t have anything bad to say about Perficient.  But when consulting, one of the things that can happen to you (especially when you are a ’systems integrator’ focused on an area like portals) is you can get stuck working on the same thing over and over again.  I can’t say that I am burned out on Portal because its definitely a fun and challenging area, but I’d rather be more of a jack of all trades and get to do development more often than not. 

In my new position I think I’ll be able to do the things that make me love this profession.  I know the environment well since I have been consulting there for so long and I understand the challenges and opportunities that I will be faced with in the coming years.  It’s an exciting time for me.

There are blog posts out the wazoo talking about FaceBook burying sites like myspace and LinkedIn.  I am certainly not bashing FaceBook because I’m not much of a user – it seems like it has a nicely designed web interface and certainly is getting a lot of pub so they must be doing something right.  I just don’t get the hype from the perspective of saying it will ‘replace’ sites like LinkedIn.  When I created my FaceBook account and started trying to connect to contacts I felt like I was in a dating service.  Maybe its more of a general issue with society, but the pictures of people are almost always in a party mode and most of the people seem to me to be in more of a ’social mood’ in their profiles and comments.  Maybe I’m just turning into an old guy, but it just seems like the purpose is different, whether the creators intended it to be different or not.  Maybe the new wave is to totally mashup your personal and professional life.  I’m not sure I’m down with that though… that’ll take some getting used to. 

So, again I’m not saying use LinkedIn  over Facebook.  I’m looking for someone to convince me that I should invest the time into FaceBook for professional endeavors.  So… let me have it.

For those of you who aren’t sure what bank account phishing is, have you ever gotten one of those emails from a bank, probably one that you don’t even have an account with, asking you to click a link in the email and provide some information for update purposes? Most of them use really official looking HTML emails with valid logos and images from the site they’re spoofing. They have usually hacked the site of some innocent company or individual and are using it to sucker you into entering information. A couple days ago I got one from ‘Bank of America’. The URL was http://www.24hourinkjet.com/www.bankofamerica.com/onlineid-sessionload/cgi-bin/sso.login.controllernoscript=true/sessiondid=2335454893_…. So it looks sort of official. Unfortunately when I went to http://www.24hourinkjet.com to alert them, I was greeted with an explitive laced web page indicating that the hacker was proud of what he had done.

If you are traversing the ‘blogosphere’ then you are probably smart enough to avoid phishing attacks, but a huge amount of the online population probably is not. I think of my grandparents. They don’t do a whole lot online. I only signed them up for a webTV account a couple of years ago so they could keep up with the family abroad using emails. If they happened to get a phishing attack email from a bank they actually deal with they’d probably go right to the link and enter their info. Its unfortunate, but the phishers do this for a reason – they obviously land some ‘phish’. So it’s important to come up with countermeasures. I like general idea that Mikko offers, but you can’t take the ignorance out of the internet user.

I’m no internet security expert, but I have opinions on the matter. The first problem with this is the ‘ignorant human’ one. For instance, my grandparents wouldn’t know the ‘rule’ about only giving your bank information to a site at a *.bank domain name. Second, as was apparent with the 24hourinkjet.com example I posted above is that the *.bank sites have to be hackproof. Certainly today the major banks are very secure, but I’m guessing that if this idea went through then every small bank and credit union would need to get a .bank domain name. The fact that they’d be spending a large part of their normal annual software/IT budget on a domain name means they have to spend less on stuff like paying a decent hosting service (i.e. one that is actually secure) or paying an admin that knows how to secure against such attacks. Plus I’d hate to see the small banks and credit unions forced to spend that kind of dough on a domain name effectively hurting their ability to compete with ‘the big boys’.

I definitely think this is a step in the right direction, but I would say that the ‘cost’ of the .bank domain name cannot be the barrier to entry. To me, I think the barrier to entry should be some form of proof that the applicant is actually acting in the interest of a bonified financial institution. I can’t offer a solution other than some governing body (warning – possible ruination of the entire point) that decides what applicants are worthy of the .bank domain name.

Personally, this is disturbing. In the documentary one thing he said transcends the so-called facts thrown around… and that is something along the lines of when your grandchildren ask you how you could have ignored this issue how will you answer them. That resonates in me for some reason. I’m not a big wasteful turd, but I’m certainly not the most responsible steward of human resources. Based on this thought… and whether it has any truth to it or not, I’m going to make an effort to be more responsible to the earth. I will recycle as much as I can… I will print less… I will turn lights out when they aren’t necessary… etc. So, Mr. Gore, whether I would vote for you or not, and whether you speak the truth or not, the concept has touched me and I will reduce my carbon signature.

I wonder if clock cycles could matter. In twenty years will there be so many lines of code running inefficiently that there is some stat that says “if all the code in the world were 50% more efficient we wouldn’t be facing the dilemma we’re facing now”. I don’t normally have to deal with detailed code level issues, let alone clock cycle issues, but I wonder if this is possible. I’m not starting an efficient code crusade, I’m just wondering.