This extension is very simple: in one click you create a receipt from a webpage and send it to your Expensify account.

The extension itself is cool, but the process of creating it and its implication for Expensify is even better.

Birth of the Extension

Back in september, a user send an email to help@expensify.com, asking us if we have a Chrome Extension. At this time we didn’t, and we had no intention to have one. But it was a friday afternoon, and I was kind of bored so I decided to create the smallest extension possible: an icon in the new tab that open a new tab to the website when you click on it. My background of Toolbar developer help me to do that in 5 min and I send an email to everybody: “Yes we have an extension”. The pandora box was open. Immediately I tried to do more: what about a popup with an iframe ? Kind of cool but also useless compare to the experience of the real website. We start to brainstorm and the idea of importing a receipt to Expensify directly from the page I’m seeing was awesome.

Problem, how to solve the technical problem ?

First idea: getting the source of the page and send that to Expensify to convert it into an image. Some program allow you to convert HTML into something else, so we could use them. But what about the auth, and the infrastructure work that it is implying ? Also getting the source is not that easy.

So while I was researching for this solution, I went back into the Google Chrome Extension docs, and I discover a new function, that was not available few months ago when I was working for APN.

The extension is mainly build around this method  captureVisibleTab.

It returns a data image URL of the visible part of the current tab. Magical! In one call we get a perfect image, for no cost and very fast. Associate with some scrolling logic, content scripts, canvas, ajax, localStorage and our extension is there!

Backend

Now that we have our receipt, we still need to send it to Expensify. As you can imagine, we already had some kind of internal private API to upload an image from “outside”. Our mobile app need it!

But we still need to send the receipt to the account of the user using the extension. And this is where it is hard. We can not trust the Chrome Extension the same way we trust our mobile app. Any body can change the code and ship its own version. So we need to provide some kind of public API to authenticate a user and send a receipt to his account. And that’s how we decided to create the “Scan anywhere” API.

Public API

We receive often request from people to get access to our API or if we have a public API. These people can be individual or working for a company that want to integrate with us.  Expensify used to have a public API, but for some reason, this API has been shutdown. So we decide to reopen a portion of it but with some refactoring in order to make reliable, scalable and available in the long run. We build a great security model and a cool UI/UX to allow people to sign in or register from the partner page.

I’m really proud of the result. It is working well, it is clear and fast. I’m sure there is few bugs ( send me an email if you found one ). If you like it, please leave a review on the Chrome Webstore page.

People start to worry about the performance implication of relying heavily on localStorage to save network request ( especially caching files ).

• DOM localStorage considered harmful by Taras Glek
• There is no simple solution for localStorage by Chris Heilmann

localStorage is a synchronous API doing file I/O. So if the computer is busy, the page is going to freeze, making the user experience not that great. This is a reality.

Some other blog post start to pop-up to defend localStorage, and do some benchmark to really measure what kind of “bad performance” we should expect.

• localStorage perhaps not so harmful by John Allsopp
• In defense of localStorage by Nicholas Zakas

Both side are right, Taras and Chris were just trying to highlight the fact that localStorage is not the solution for everything and can have some limitation that a Web developer has to know before choosing to use the API.

As I explained in a previous post, the Expensify website is now using heavily localStorage. When I design this feature, I haven’t thought of the problem that file I/O can creating bottleneck in this situation. For us, the problem was to reduce the number of API call has some of them are very slow. And it works. The average page load of the site has decrease. Expensify is still not a really fast website, but this solution has help to improve the situation.

Now, if I should rewrite this feature knowing its limitations, I would almost redo the same thing. As a first iteration, the code is simple and it helps a lot. If we have the opportunity to iterate on it to make it better, I will probably use the solution of Nicholas Zakas: using Web Workers to communicate with the localStorage. This solution will “restore” the full asynchronous of our “ajax request” and again improve performance.

But Web Workers are not available in the same amount of browser than localStorage. And it would be a fairly complex piece of code in order to avoid communication with the localStorage engine as much as possible.

Our site is not a single page site. So for every page we load the same data again and again. And that’s not find. First it is stupid, second it is making the website slower and finally it increase the load on our backend for no valid reason.

localStorage for cache

To store the data we will use the HTML5 localStorage. It is available almost on every browser ( IE8 included ). It is available across tab and you can store up to 5MB ( but far less in practice ). Also the API is very simple to use, it is fast, and you get easily debug the content of the cache with the Chrome Developer tools ( under the resources tab ).

localStorage content

Some jQuery plugins have be written using localStorage to store ajax results. https://github.com/paulirish/jquery-ajax-localstorage-cache But we decided to not used them, not because we want to recreate the wheel but because the model used to invalidate the cache doesn’t work for us.

Cache invalidation

There are only two hard things in Computer Science: cache invalidation and naming things — Phil Karlton

For every cache system the most sensitive part if the algorithm which determined if the cache content is valid. With Expensify data, this task is very complex. Our data are heavily linked with each others ( receipts link to expenses on a report ). So in order to keep a simple code for our prototype, we decided that if  the user change ANYTHING, we will clear the whole cache. This is very inefficient in a “cache hit” point of view, but that way we are sure that what we are displaying is the correct version of the data.

So we know how to invalidate the cache if the user is doing an operation on the website. But what if he is using an other browser, computer or mobile device ?

For example, if you are on the receipt page with your computer and you use your Expensify mobile app to take a picture of a receipt, the web page on the browser is refreshing itself automatically and will display the receipts after few seconds. We want to keep this functionality. So we need some way to know when the cache is becoming invalid because of the action of an “external” actor.

So we store on our server the date of the latest modification on an account. If anything happened which impact the account of the user, we update this value. So before we make any request, we ask the server to return this value. If the date is more recent that the one we have stored, our cache is not valid anymore, we need to clear it.

Clear and performance

To clear our cache, we use localStorage.clear().  So we delete everything. If the user add an expense, we clear the cache for receipts, reports, etc. Everything. It is dumb and not efficient for cache hits, but we are sure that we are not going to display out of date data. Also keep in mind that this system has been designed has a proof of concept to know if this kind of strategy could help our front end performance.

With the cache, we build a statistic system which will log on the server how many request has been done, how many cache hit, number of polling, etc. And we got the good surprised to see that around 48% of the request were hitting the cache. For a stupid cache system like that it is a surprisingly good number.

In the future, we will improve these performances by creating a key-based cache expiration system. That we could delete only part of the cache instead of everything.

Ghostscript Logo

At my work I had to correct a bug related to the conversion of a PDF to a jpeg. To do that we use the convert command line tools from ImageMagick.

The server which is doing the conversion run an Ubuntu with the latest version of the package ghostscript ( 8.71).

Here is the error:

11:04 1442 ~/Desktop : convert corrup1.pdf toto.jpg
Error: /undefined in findresource
Operand stack:
–dict:8/17(L)– C2_1 1 –dict:5/5(L)– –dict:5/5(L)– Arial –dict:9/12(ro)(G)– –nostringval– CIDFontObject –dict:7/7(L)– –dict:7/7(L)– Adobe-Identity
Execution stack:
%interp_exit .runexec2 –nostringval– –nostringval– –nostringval– 2 %stopped_push –nostringval– –nostringval– –nostringval– false 1 %stopped_push 1878 1 3 %oparray_pop 1877 1 3 %oparray_pop 1861 1 3 %oparray_pop –nostringval– –nostringval– 2 1 1 –nostringval– %for_pos_int_continue –nostringval– –nostringval– –nostringval– –nostringval– %array_continue –nostringval– false 1 %stopped_push –nostringval– %loop_continue –nostringval– –nostringval– –nostringval– –nostringval– –nostringval– –nostringval– %array_continue –nostringval– –nostringval– –nostringval– –nostringval– –nostringval– %loop_continue
Dictionary stack:
–dict:1161/1684(ro)(G)– –dict:1/20(G)– –dict:75/200(L)– –dict:75/200(L)– –dict:108/127(ro)(G)– –dict:288/300(ro)(G)– –dict:22/25(L)– –dict:6/8(L)– –dict:26/40(L)–
Current allocation mode is local
Last OS error: 2
GPL Ghostscript 8.71: Unrecoverable error, exit code 1
convert: Postscript delegate failed `corrup1.pdf’: No such file or directory @ pdf.c/ReadPDFImage/634.
convert: missing an image filename `toto.jpg’ @ convert.c/ConvertImageCommand/2838.

Weird. A google of the Error: /undefined in findresource will lead you to this bug.
To sum up the installer of ghostscript is fucked up, and no new version of the package is deploy to solve it. Awesome.

ghostscript is not able to find a font ( Adobe-Identity ) and try to find a substitution font ( Arial ) but can’t find it.
So let’s correct the configuration manually.

First make sur that you have install the microsoft webfont on your system.

Then we need to edit our cidmap which is the configuration file which tell ghostscript substitution font. Mine looks like after adding the last line:
/var/lib/defoma/gs.d/dirs/fonts/cidfmap

1 /Dotum-Bold << /FileType /TrueType /Path (/usr/share/fonts/truetype/unfonts/UnDotumBold.ttf) /SubfontID 0 /CSI [(Korea1) 0] >> ;
2 /Batang-Regular << /FileType /TrueType /Path (/usr/share/fonts/truetype/unfonts/UnBatang.ttf) /SubfontID 0 /CSI [(Korea1) 0] >> ;
3 /Batang-Bold << /FileType /TrueType /Path (/usr/share/fonts/truetype/unfonts/UnBatangBold.ttf) /SubfontID 0 /CSI [(Korea1) 0] >> ;
4 /Dotum-Regular << /FileType /TrueType /Path (/usr/share/fonts/truetype/unfonts/UnDotum.ttf) /SubfontID 0 /CSI [(Korea1) 0] >> ;
5 /Arial << /FileType /TrueType /Path (/usr/share/fonts/truetype/msttcorefonts/arial.ttf) /SubfontID 0 /CSI [(Artifex) (Unicode) 0] >> ;

Make sur to use the correct path for the arial.ttf file.

Doing that produce a second error:

1:16 1442 ~/Desktop : convert corrup1.pdf toto.jpg
Can’t find CMap Identity-UTF16-H building a CIDDecoding resource.
Error: /undefinedresource in findresource
Operand stack:
–dict:8/17(L)– C2_1 1 –dict:5/5(L)– –dict:5/5(L)– Arial –dict:9/12(ro)(G)– –nostringval– CIDFontObject –dict:7/7(L)– –dict:7/7(L)– Arial CIDFont true Arial false Arial –dict:23/26(G)– Unicode.Unicode CIDDecoding true Unicode.Unicode –dict:28/48(ro)(G)– Identity-UTF16-H
Execution stack:
%interp_exit .runexec2 –nostringval– –nostringval– –nostringval– 2 %stopped_push –nostringval– –nostringval– –nostringval– false 1 %stopped_push 1878 1 3 %oparray_pop 1877 1 3 %oparray_pop 1861 1 3 %oparray_pop –nostringval– –nostringval– 2 1 1 –nostringval– %for_pos_int_continue –nostringval– –nostringval– –nostringval– –nostringval– %array_continue –nostringval– false 1 %stopped_push –nostringval– %loop_continue –nostringval– –nostringval– –nostringval– –nostringval– –nostringval– –nostringval– %array_continue –nostringval– –nostringval– –nostringval– –nostringval– –nostringval– %loop_continue –nostringval– –nostringval– 1829 13 9 %oparray_pop findresource %errorexec_pop –nostringval– –nostringval– –nostringval– –nostringval– –nostringval– –nostringval– –nostringval– –nostringval– %loop_continue –nostringval– –nostringval– –nostringval– 1829 20 10 %oparray_pop findresource %errorexec_pop –nostringval– –nostringval– –nostringval– –nostringval– –nostringval– –nostringval– %array_continue –nostringval–
Dictionary stack:
–dict:1161/1684(ro)(G)– –dict:1/20(G)– –dict:75/200(L)– –dict:75/200(L)– –dict:108/127(ro)(G)– –dict:288/300(ro)(G)– –dict:22/25(L)– –dict:6/8(L)– –dict:26/40(L)– –dict:20/26(ro)(G)– –dict:18/25(ro)(G)– –dict:16/25(ro)(G)–
Current allocation mode is global
Last OS error: 2
GPL Ghostscript 8.71: Unrecoverable error, exit code 1
convert: Postscript delegate failed `corrup1.pdf’: No such file or directory @ pdf.c/ReadPDFImage/634.
convert: missing an image filename `toto.jpg’ @ convert.c/ConvertImageCommand/2838.

The error is Can’t find CMap Identity-UTF16-H. Ghostscript can’t find this, so let’s create it !

Mine need to be created in /usr/share/ghostscript/8.71/Resource/CMap/Identity-UTF16-H

The content of this file can be find here. Or copy it from here:

% Copyright (C) 2003 Artifex Software. All rights reserved.
%
% This software is provided AS-IS with no warranty, either express or
% implied.
%
% This software is distributed under license and may not be copied,
% modified or distributed except as expressly authorized under the terms
% of the license contained in the file LICENSE in this distribution.
%
% For more information about licensing, please refer to
% http://www.ghostscript.com/licensing/. For information on
% commercial licensing, go to http://www.artifex.com/licensing/ or
% contact Artifex Software, Inc., 101 Lucas Valley Road #110,
% San Rafael, CA 94903, U.S.A., +1(415)492-9861.
%
% $Id: Identity-UTF16-H 7095 2006-10-10 10:03:37Z leonardo $
% Identity-UTF16-H CMap
% An identity mapping of UTF16 codes to CIDs.

/CIDInit /ProcSet findresource begin

12 dict begin

begincmap

/CIDSystemInfo 3 dict dup begin
/Registry (Artifex) def
/Ordering (Unicode) def
/Supplement 0 def
end def

/CMapName /Identity-UTF16-H def
/CMapVersion 1.000 def
/CMapType 1 def

/UIDOffset 0 def
% No XUID yet obtained.

/WMode 0 def

1 begincodespacerange

endcodespacerange

1 begincidrange
0
endcidrange
endcmap
CMapName currentdict /CMap defineresource pop
end
end

%%EndResource
%%EOF

And it is working !!! Woot !

Google Chrome logo

Google Chrome has the same feature since version 8 if I’m not wrong. you can see a thumbnail of your current tab and easily move to one with a smooth animation. But this feature is hidden and you need to turn a flag to enable it.

To do that, write about:flags in your address bar.

Google Chrome about flags

Then find and enable the Tab overview experiment.

Scroll down and click Relaunch Now.

That’s it ! Now use 3 fingers on your trackpad to use it !

Google Chrome Tab Thumbnail

1. download Rhino
2. download qunit-cli
3. download qunit
4. Unzip every thing
5. Go in your qunit directory, and copy the directory qunit and test
6. Paste them in qunit-cli/qunit


qunit directory structure

7. Now open your terminal / console and move into qunit-cli/rhino
8. Run the command to test that it is working

java -cp ../../rhino1_7R2/js.jar org.mozilla.javascript.tools.shell.Main -opt -1 suite.js

9. TADAAAA


qunit rhino output

10. Edit qunit-cli/rhino/suite.js to load your source files ( at the top ) and your test files ( at the bottom )

You can also change the formating of the output by editing the suite.js file and using the right functions.

Note:
You should consider the creation of an alias to run the command :

export RHINO_HOME="~/development/rhino1_7R2"
alias js="java -cp $RHINO_HOME/js.jar org.mozilla.javascript.tools.shell.Main -opt -1"
alias jsd="java -cp $RHINO_HOME/js.jar org.mozilla.javascript.tools.debugger.Main"

Links:
http://twoguysarguing.wordpress.com/2010/11/02/make-javascript-tests-part-of-your-build-qunit-rhino/
https://github.com/asynchrony/qunit
https://github.com/benjaminplee/QUnit-CLI
http://www.mozilla.org/rhino/download.html
http://docs.jquery.com/Qunit

jQuery conference San Francisco 2011 badge and Tshirt

I was there. It was ok. Some great talks and some other less good ( no offense ).

The browser action on the top right corner of the browser.

Browser Action icon

Or the page action at the right of the address bar

Page Action icon

So how can you create a toolbar ?

Well Google Chrome provide you “content scripts“. These scripts are running in the parallel of the page. So this script can manipulate the DOM of the page. So we can inject something in the page. And if this something is an iframe, we could inject a “page” without worrying about JavaScript or CSS collision. And this iframe can be host on a server or in the extension. In this case you can use the chrome.extension.getURL method to get the path of the file.

Et voilà !

jQuery UI logo

I did in the past a jQuery plugin to do that. Here is the problem I tried to solve: when you have an HTML table with thousand of lines, it is very long to display, instable, impossible to search or to sort element. Even plugins like http://tablesorter.com/docs/ can’t do th job when you have 10 000 lines. Or they use a pager which give a bad user experience.

So tried to solve this problem and I succeed. Instead of displaying every thing, I keep the data in an array in the memory. Every operation will be done on this array : sorting, filtering, search, selection, …
And I just display the number of row that the screen can display.
I check every scroll event or action on the fake scrollbar I have create to update the display.

And it works well with 90 000 lines !!! \o/

My plugin also support plugins to change the behavior or to customize the display.

But I’m sur the jQuery team is going to do something awesome ! Be sur that I will look at their progress very closely and try to be involve into this.

Terminal icon

Look at this example: I want to delete a file, but it own by an other user.

rm delete a file

I’m sur that I can delete the file, so I have to use the sudo command before the rm command.

So I have 2 solutions:

• write sudo rm mysql.sock = equal rewrite evereything
• write sudo !! = use a shortcut !!

Of course you choose the second solution: it ‘s quicker and you don’t make any mistake by retyping the command. So !! contain the last command you wrote. Let’s see in action:

!! contain the last command

And I works also if you want to add a parameter. I will use the example of the command ls which list the content of a directoy.

ls list the content of a directory

I the screenshot, I used ls then !! -a = ls -a to list all the file of the directory, Then

ls -l -a list all file of a directory with more information

!! -l = ls -a -l = ls -al

So this shortcut is very handy when you have long command to repeat.