nodejs + express with jsonp example

I’ve been working more and more with nodejs and have to say I am really loving how easy it’s been to get to grips with. I’ll be posting up more about how I’m using node and the problems I’m using it to solve over the coming weeks. However I wanted to illustrate just how simple it is do something that would be more work in other languages such as PHP.

So here’s a quick example I created that illustrates how to make a simple JSONP call to a nodejs + express server.

Here’s the server side code:

and here’s the client side Html code:

To enable jsonp callback support in an ExpressJS application you just have to include the line:

app.enable(“jsonp callback”);

Once you’ve done this you can use the .json() method on the Response object to handle everything for you. So in my example above any HTTP GET request to /foo?cb=myfunction would return myfunction(“hello world”); with the Content-Type header set to text/javascript.

Lifes too short – write fast code!

ABSTRACT

This is the second talk that follows-up on the 14 best practices from YSlow and “High Performance Web Sites”. The first talk presented three new best practices: Split the Initial Payload, Load Scripts Without Blocking, and Don’t Scatter Inline Scripts.

The most important of these is loading external scripts without blocking other downloads and preventing page rendering. One complication is this may introduce undefined symbol errors if inlined code uses symbols from the external scripts. Luckily, there are several techniques to workaround this problem. That and other topics will be covered in this presentation of three more best practices:

* Coupling Asynchronous Scripts
* Use Iframes Sparingly
* Flush the Document Early

Much of this talk discusses material from Steve’s book, High Performance Websites: Essential Knowledge for Front-End Engineers. The talk is full of great advice, I found the discussion around loading scripts both synchronously and asynchronously and the performance gains that can be achieved. However this has to be combing with understanding that you also have to couple scripts together in order to preserver the order they are loaded in, as well as understanding that by default loading external scripts blocks download of other elements on the page. Steve discusses a number of techniques that can address these issues as well as the pros and cons associated with each. His discussion around John Resigs idea of using degrading script tags is extremely useful.

This is a hugely useful tech talk and a must for anyone doing serious Javascript development.

Why PHP Won

An excellent article by Eric Reis over on his blog in which he talks about “why PHP won” in his web application development over other (web scripting) languages:

As a language, it’s inelegant. Its object-orientation support is “much improved” – which is another way of saying it’s been horrendous for a long time. Writing unit tests or mock objects in PHP is an exercise in constant frustration. And yet I keep returning to PHP as a development platform, as have most of my fellow startup CTOs. This post is about why.

Its an interesting piece in which Eric chooses to describe PHP’s success in terms of what a new language might have to do better in order to challenge PHP’s popularity/success, in short he suggests the following:

  • Speed of iteration (a good write/test/debug cycle)
  • Better mapping of outputs to inputs
  • A similar standard library
  • A better OOP implementation

I have to confess I found myself agreeing with Eric. His piece is well worth reading!

Geppeto: Consumer’s Approach to Programming

ABSTRACT

Contemporary society is experiencing a steady stream of new electronic gadgets, software products, and web applications. In this flood of functionality, users have adapted to rely less on manuals (if they are present at all) and shift their learning to trial and error, common paradigms, and experimentation. To accommodate this style of use — or perhaps driving this behavior – developers have successfully abstracted much of the technological complexity and transformed it into intuitive user interfaces often avoiding the need for reading lengthy manuals and formal training. Is it possible to adopt the same trial-and-error experimentation habit not only for using gadgets, but also for application development? We claim that intuitive aggregation and combination of software gadgets makes this possible.

In this talk, we will show the use of current technology in building a consumer oriented development tool appropriate for individuals not formally trained in programming. We demonstrate that the complexity of existing system and scripting languages i.e.; syntax, semantics, control and data flow, data structures, data types, and programming components can be successfully replaced with analogies intuitively accessible to a much wider consumer population based exclusively on their use and understanding of user interfaces in popular web applications. We present a demo of Geppeto — a consumer tool for gadget-based application development. Composing gadgets with Geppeto does not require programming experience or reading of convoluted manuals. The presented research is sponsored by Google Inc. and the Croatian Ministry of Science.

Test-Driven JavaScript Development with JsUnit

The last time I used JsUnit was when I first joined Talis. At the time my colleague Ian Davis asked me to write a JavaScript client library for one of our platform API’s to make it easy for developers to perform bibliographic searches. It wasn’t a particularly difficult task and I did it relatively easily. It was around the same time that Rob was extolling the virtues of Test Driven Development to me, and to try to prove his point we agreed to do an experiment: he asked me to set aside the library I had written and to see if I could develop the library again using test driven development. It meant I had to figure out how to unit test JavaScript, and thats when I found JsUnit. I did the exercise again and even I was impressed with the results. By having to think about the tests first, and design the interface to the library as I wrote each test it evolved very differently to my original solution. Consequently it was also far superior.

Anyway fast forward two and half years and I find myself in a similar situation. We have only just begun to start writing bits of JavaScript code based around prototype.js to help us create richer user experiences in our products if we detect that JavaScript is enabled in the browser. This now means I want to ensure that we are using the same rigour when writing these bits of code as we do in all other parts of the application – just because its JavaScript and executed inside the browser this doesn’t mean it shouldn’t be tested.

I’ve just spent the morning getting JsUnit installed and figuring out how to get it to run as part of a continuous integration process, as well as thinking about how to write tests for some slightly different scenarios. Here’s what I’ve discovered today:

Installing JsUnit

Couldn’t be easier … go to www.jsunit.net and download the latest distribution, and extract into a folder on your system somewhere, lets say
/jsunit for now. The distribution contains both the standard test runner as well as jsunit server which you will need it if you want to hook it into an ant build.

Writing Tests

In JsUnit we place our tests in a HTML Test Page which is the equivalent of a Test Class, this test page must have a script reference to the jsUnitCore.js so the test runner knows its a test. So lets work through a simple example. Let’s say we want to write a function that returns the result of adding two parameters together. The Test Page for this might look like this:

  1.  
  2. <html>
  3.  <head>
  4.   <title>Test Page for add(value1, value2)</title>
  5.   <script language="javascript" src="/jsunit/app/jsUnitCore.js"></script>
  6.   <script language="javascript" src="scripts/addValues.js"></script>
  7.  </head>
  8.  <body>
  9.     <script language="javascript">
  10.     function testAddWithTwoValidArguments() {
  11.         assertEquals("2 add 3 is 5", 5, add(2,3) );
  12.     }
  13.   </script>
  14.  </body>
  15. </html>
  16.  

For now lets save this file to /my-jsunit-tests/addTest.html

To run the test you need to point your browser at the following local url:

file:///jsunit/testRunner.html?testPage=/my-jsunit-tests/addTest.html

The test will not run since we haven’t defined the add function. Let’s do that (very crudely):

  1.  
  2. function add(param1, param2)
  3. {
  4.     return param1 + param2;
  5. }
  6.  

Now if you go to that URL it will run the test and report that it passed. Excellent, we’ve written a simple test in JavaScript. Now lets extend this a little, lets say I want to write something more complicated like a piece of JavaScript that uses Prototype.js to update the DOM of a page. Is this possible? Can I do that test first? It turns out that you can …

Lets say we have a div on the page called ‘tableOfContents’ and we want to use Prototype.js to dynamically inject a link onto the page that says [show] and lets say we want to write a function that will toggle this link to say [hide] when the user clicks on it, this link will also set the visible state of the table of contents itself which for now we’ll say is just an ordered list (OL). Our test page is going to be slightly more complex …

  1.  
  2. <html>
  3.  <head>
  4.   <title>Test Page for multiplyAndAddFive(value1, value2)</title>
  5.   <script language="javascript" src="/jsunit/app/jsUnitCore.js"></script>
  6.   <script language="javascript" src="scripts/prototype/prototype-1.6.0.2.js"></script>
  7.   <script language="javascript" src="scripts/tableOfContents.js"></script>
  8.  </head>
  9.  <body>
  10.     <div id="tableOfContents">
  11.     <h2 id="tableOfContentsHeader">Table of contents</h2>
  12.     <ol id="list-toc">
  13.     </ol>
  14.     </div>    
  15.     <script language="javascript">
  16.     function testTOC()
  17.     {
  18.         var title = $(‘lnkToggleTOC’).title;
  19.         assertEquals("should be Show the table of contents", "Show the table of contents", title);
  20.        
  21.         toggleTableOfContents();
  22.        
  23.         var title = $(‘lnkToggleTOC’).title;
  24.         assertEquals("should be Hide the table of contents", "Hide the table of contents", title);
  25.                    
  26.     }
  27.   </script>
  28.  </body>
  29. </html>
  30.  

There are some differences in this test. Firstly the html contains some markup, that I’m using as the containers for my table of contents. The table of contents has a header and the contents in the form of an empty ordered list. Now I know that I want the javascript to execute when the page is loaded, so I’ve written this test to assume that the script will run and will inject and element called ‘linkToggleToc’ which is the show/hide link next to the heading. Therefore the first line of the test uses prototype.js element selector notation to set a local variable called title to the value of the title of the element that has the id ‘linkToggleToc’. If the script failes to execute then this element will not be present and the subsequent assert will fail. If the assert succeeds, then we call the toggleTableOfContents function and then repeat the same evaluation only now we are checking to see if the link has been changed.

The code for tableOfContents.js is as follows:

  1. Event.observe(window, ‘load’, injectTOCLinks);
  2.  
  3. var titleShowTOC = ‘Show the table of contents’;
  4. var titleHideTOC = ‘Hide the table of contents’;
  5.  
  6. function injectTOCLinks()
  7. {
  8.     $(‘list-toc’).hide();
  9.     $(‘tableOfContentsHeader’).style.display = ‘inline’;
  10.    
  11.        
  12.     var elShowHideLink = new Element(‘a’, { ‘id’: ‘lnkToggleTOC’, ‘href’: ‘javascript:toggleTableOfContents()’, ‘title’: titleShowTOC, ‘class’: ‘ctr’ }).update("[show]");
  13.    
  14.     $(‘tableOfContentsHeader’).insert({‘after’:elShowHideLink});
  15. }
  16.  
  17. function toggleTableOfContents()
  18. {
  19.     var toc = $(‘list-toc’);
  20.    
  21.     if (toc.visible())
  22.     {
  23.         toc.hide();
  24.         $(‘lnkToggleTOC’).update(‘[show]’);
  25.         $(‘lnkToggleTOC’).title = titleShowTOC;
  26.     }
  27.     else
  28.     {
  29.         toc.show();
  30.         $(‘lnkToggleTOC’).update(‘[hide]’);
  31.         $(‘lnkToggleTOC’).title = titleHideTOC;
  32.     }
  33. }

Now if we run this test in the same way we executed the previous test it will pass. I accept that this example is a bit contrived since I know it already works and I’ve skimmed over some of the details around it. The point I’m trying to make though is that you can write unit tests for pretty much any kind of JavaScript you need to write, even tests for scripts that do dom manipulation, or make AjaxRequests etc.

Setting up the JsUnit server so you can run it in a build

JsUnit ships with its own ant build file that requires some additional configuration before you can run the server. The top of the build file contains a number of properties that need to be set, here’s what you set them to ( using the paths that I’ve been using in the above example)

  1.  
  2. <project name="JsUnit" default="create_distribution" basedir=".">
  3.  
  4.   <property
  5.     name="browserFileNames"
  6.     value="/usr/bin/firefox-2" />
  7.  
  8.   <property
  9.     id="closeBrowsersAfterTestRuns"
  10.     name="closeBrowsersAfterTestRuns"
  11.     value="false" />
  12.  
  13.   <property
  14.     id="ignoreUnresponsiveRemoteMachines"
  15.     name="ignoreUnresponsiveRemoteMachines"
  16.     value="true" />
  17.  
  18.   <property
  19.     id="logsDirectory"
  20.     name="logsDirectory"
  21.     value="/my-jsunit-tests/results/" />
  22.  
  23.   <property
  24.     id="port"
  25.     name="port"
  26.     value="9001"  />
  27.  
  28.   <property
  29.     id="remoteMachineURLs"
  30.     name="remoteMachineURLs"
  31.     value="" />
  32.  
  33.   <property
  34.     id="timeoutSeconds"
  35.     name="timeoutSeconds"
  36.     value="60" />
  37.  
  38.   <property
  39.     id="url"
  40.     name="url"
  41.     value="file:///jsunit/testRunner.html?testPage=/my-jsunit-tests/tocTest.html" />
  42. </project>
  43.  

You can then type the following command in the root of the jsunit distribution to launch the jsunit server, executes the test, and outputs a test results log file, formatted just like JUnit, and reports that the build was either successful or not if the test fails.

  ant standalone_test

Remember that in this example I’ve used a simple Test Page, however JsUnit, like any XUnit framework allows you to specify Test Suites, which is how you would run multiple Test Pages. Also the parameters in the build file woudn’t be hardcoded in you continuous integration process but would rather be passed in, and you would want to call it from your projects main ant build file … all of which is pretty simple to configure, once you know what is you want to do and what’s possible.

A possible future of Software Development

This talk begins with an overview of software development at Adobe and a look at industry trends towards systems built around object oriented frameworks; why they “work”, and why they ultimately fail to deliver quality, scalable, software. We’ll look at a possible alternative to this future, combining generic programming with declarative programming to build high quality, scalable systems.

… a very interesting talk, that raises some important questions, about the very nature of software development.

Processing.js

Processing is a Open Source data visualization programming language. I first played around with it about a year ago. I was recently reminded of it by Rob, and have started playing with it again. However, I just discovered that earlier in the week John Resig released his JavaScript Port, Processing.js. So far it looks amazing, virtually all the demo/example applications that are shipped with Processing are running using the CanvasElement in JavaScript. I’m going to have a lot of fun with this.

John deserves a huge amount of credit for this contribution.

Flare: Visualization on the Web

Came across Flare today and am thinking it might be very useful for some data visualisation work I’m doing, here’s the blurb:

Flare is a collection of ActionScript 3 classes for building a wide variety of interactive visualizations. For example, flare can be used to build basic charts, complex animations, network diagrams, treemaps, and more. Flare is written in the ActionScript 3 programming language and can be used to build visualizations that run on the web in the Adobe Flash Player. Flare applications can be built using the free Adobe Flex SDK or Adobe’s Flex Builder IDE. Flare is based on prefuse, a full-featured visualization toolkit written in Java. Flare is open source software licensed under the terms of the BSD license, and can be freely used for both commercial and non-commercial purposes.

Try out the online demo here.

Holding a Program in One’s Head

from an excellent new essay by Paul Graham

Good programmers manage to get a lot done anyway. But often it requires practically an act of rebellion against the organizations that employ them. Perhaps it will help to understand that the way programmers behave is driven by the demands of the work they do. It’s not because they’re irresponsible that they work in long binges during which they blow off all other obligations, plunge straight into programming instead of writing specs first, and rewrite code that already works. It’s not because they’re unfriendly that they prefer to work alone, or growl at people who pop their head in the door to say hello. This apparently random collection of annoying habits has a single explanation: the power of holding a program in one’s head.

Whether or not understanding this can help large organizations, it can certainly help their competitors. The weakest point in big companies is that they don’t let individual programmers do great work. So if you’re a little startup, this is the place to attack them. Take on the kind of problems that have to be solved in one big brain.

so very true … !

Best practises in JavaScript library design

This is a very useful tech talk by John Resig, that explores a number of techniques used to build robust, reusable cross-platform JavaScript libraries.

John offers some excellent advise and whilst some if it might seem obvious it’s worrying how many existing API’s fall into some of the common pitfalls he describes.

John argues that part of writing good solid API’s is to keep the code orthogonal by ensuring that whenever you perform an action on an object that action should be consistent across all objects. In other words each object should expose the same methods, i.e. add(), remove(), all(), … etc. this creates familiarity and developers using the API know that different objects that are responsible for different things can all be used consistently.

John also makes the obvious and profound point that when creating an API you should Fear adding methods, the reason being that every method that you write is one that you will have to maintain. In fact you should try to embrace the idea that removing unused code is a good thing. It reduces the size of your API, makes it easier to learn and easier to maintain.

Going back to consistency its imperative that we use good naming conventions and naming schemes and stick with them, this also means you have to be very diligent about argument position in method calls … I know how frustrating it is when you use some of the string processing methods in PHP but the argument order changes it’s annoying!

John goes onto offer much more advice on encapsulation, functional programming, compression of libraries using Dojo. He advocates Test Driven Development for API design which generally results in better API design.

It’s an excellent talk and well worth watching for anyone who working on building JavaScript, or indeed any kind of API.