Homepage of Michael Goerz

Floating Points Numbers, Down to the Bit

2011-05-28T00:00:00+02:00

Floating Point Memory Format

The general idea behind floating point numbers is the same as behind scientific notation: write a mantissa, combined with an exponent to move the decimal point. E.g. $1.093 \times 10^5 = 10^5 \sum_{k=0}^3 a_k 10^{-k}$ with $a_k = 1,0,9,3$. The exact same thing can be done in binary:

$$ v = 2^e \sum_{k=0}^{n-1} a_k 2^{-k} $$

In memory, floats can be stored by reserving a section of the total available bits to store the exponent, while the mantissa is stored in the remaining bits. For a double precision float, the total size is 64 bit. One bit is used to encode the sign of the number, 11 bits for the exponent, and the remaining 52 bits for the mantissa. In order to only have to store positive integers as exponents, as fixed bias exponent of 1023 is subtracted from the stored exponent. Also, the digit $a_0$ is assumed to be 1, and is not stored explicitly in the mantissa; the effective number of mantissa digits is therefore 53. For details, see the Wikipedia article for the Double precision floating-point format. The Single precision floating-point format is very similar, but only uses 32 total bits.

Parsing Double Precision Floats in Python

It is relatively easy to parse a binary representation of a double precision float. I’ve written a python script to do so (both for double and single precision), available on Github.

Given a representation (as a hex string) or a float value (as a string like “1.2” or “-2.3456e-192”) the scripts decompose the binary representation into sign, exponent, and mantissa, and print out this information along with the exact decimal the binary representation corresponds to according to the floating point model.

A similar online tool for double precision floats is available at binaryconvert.com (also single precision and others).

The key routine:

    BIAS = 1023                # constant to be subtracted from the stored exponent
    SIGN_BIT = 63              # bit index at which the sign is stored
    EXP_BIT = 52               # bit index at which the exponent starts
    EXP_MASK = 0x000fffffffffffff  # bit-mask to remove exponent, leaving mantissa
    NAN_EXP = 0x7ff            # special exponent which encodes NaN/Infinity
    
    def parse_hex(hexstring, float_format='%.15e', no_decimal=False):
        """ Take a 8-byte hex string (16 digits) representing a double precision,
            parse it, and print detailed information about the represented float
            value.
        """
        bits = int('0x%s' % hexstring, 16)
        sign = '+1'
        if test_bit(bits, SIGN_BIT) > 0:
            sign = '-1'
        bits = clear_bit(bits, SIGN_BIT)
        stored_exp = bits >> EXP_BIT
        mantissa = bits & EXP_MASK # mask the exponent bits
    
        print ""
        print "Bytes         = 0x%s" % hexstring
        print "Float         = "+ float_format \
                               % struct.unpack('!d', hexstring.decode('hex'))[0]
        print "Sign          = %s" % sign
        if stored_exp == 0:
            print "Exponent      = 0x%x (Special: Zero/Subnormal)" % stored_exp
            print "Mantissa      = 0x%x" % mantissa
            if not no_decimal:
                if mantissa == 0:
                    print "Exact Decimal = %s0" % sign[0]
                else:
                    print "Exact Decimal = %s (subnormal)" \
                        % float2decimal(hex2float(hexstring))
        elif stored_exp == NAN_EXP:
            print "Exponent      = 0x%x (Special: NaN/Infinity)" % stored_exp
            print "Mantissa      = 0x%x" % mantissa
            if not no_decimal:
                if mantissa == 0:
                    print "Exact Decimal = %sInfinity" % sign[0]
                else:
                    print "Exact Decimal = NaN"
        else:
            print "Exponent      = 0x%x = %i (bias %i)" % (stored_exp,
                                                           stored_exp, BIAS)
            print "Mantissa      = 0x%x" % mantissa
            if not no_decimal:
                mantissa = set_bit(mantissa, EXP_BIT) # set the implicit bit
                print "Exact Decimal = %s 2^(%i) * [0x%x * 2^(-52)]" \
                                    % (sign[0], stored_exp-BIAS, mantissa)
                print "              = %s" % float2decimal(hex2float(hexstring))

The procedure is quite straightforward:

Extract the sign from the left-most bit, and set it to 0
Extract the stored exponent by shifting out the mantissa; to get the actual exponent the bias exponent (1023) has to be subtracted
Extract the mantissa by setting all bits of the exponential to 0 with a suitable bitmask

Any float can be represented as an exact decimal, without loss of information. The calculating of the exact decimal is not completely trivial to implement. The basic idea is to double the mantissa until it is a (possibly very large) integer. A routine float2decimal that does this is available in the FAQ for the Decimal module.

Connection to the Fortran Numerical Model

In Fortran, the numerical model is described as follows (see the Fortran Handbook section 13.2.3):

$$ x = s b^e \sum_{k=1}^p f_k b^{-k} $$

Note that in good Fortran fashion, the index $k$ counts from 1 as opposed to 0. This means that the exponential $e$ is shifted by one compared to the more traditional formula used in the “Floating Point Memory Format” section above.

The specific parameters for single and double precision for a given compiler/system can be found using the very useful kindfinder utility. On my system, I get the following:

 FLOATINGPOINT MODEL (Real/Complex):
  Name:                     Single              Double              Extnd1
  KIND:                          4                   8                  16
  DIGITS:                       24                  53                 113
  RADIX:                         2                   2                   2
  MINEXPONENT:                -125               -1021              -16381
  MAXEXPONENT:                 128                1024               16384
  PRECISION:                     6                  15                  33
  RANGE:                        37                 307                4931
  EPSILON:               1.192E-07           2.220E-16           1.926E-34
  HUGE:                  3.403E+38           1.798+308          1.190+4932
  TINY:                  1.175E-38           2.225-308          3.362-4932

Looking at double precision floats, the DIGITS corresponds to the total number of mantissa digits, $p$ in the model. Note that DIGITS is 53, whereas only 52 bits are used to store the mantissa in memory format: the 53rd bit for $2^0$ is implicit, as discussed earlier.

TINY and HUGE are the smallest and largest representable numbers, respectively. The parse_dp_float.py script gives the following information:

For TINY:

Bytes         = 0x0010000000000000
Float         = 2.225073858507201e-308
Sign          = +1
Exponent      = 0x1 = 1 (bias 1023)
Mantissa      = 0x0
Exact Decimal = + 2^(-1022) * [0x10000000000000 * 2^(-52)]
              = 2.225073858507201383090232717332404064219215980462331830553327
416887204434813918195854283159012511020564067339731035811005152434161553460108
856012385377718821130777993532002330479610147442583636071921565046942503734208
375250806650616658158948720491179968591639648500635908770118304874799780887753
749949451580451605050915399856582470818645113537935804992115981085766051992433
352114352390148795699609591288891602992641511063466313393663477586513029371762
047325631781485664350872122828637642044846811407613911477062801689853244110024
161447421618567166150540154285084716752901903161322778896729707373123334086988
983175067838846926092773977972858659654941091369095406136467568702398678315290
680984617210924625396728515625E-308

For HUGE:

Bytes         = 0x7fefffffffffffff
Float         = 1.797693134862316e+308
Sign          = +1
Exponent      = 0x7fe = 2046 (bias 1023)
Mantissa      = 0xfffffffffffff
Exact Decimal = + 2^(1023) * [0x1fffffffffffff * 2^(-52)]
              = 17976931348623157081452742373170435679807056752584499659891747
680315726078002853876058955863276687817154045895351438246423432132688946418276
846754670353751698604991057655128207624549009038932894407586850845513394230458
323690322294816580855933212334827479782620414472316873817718091929988125040402
6184124858368

Note the MINEXPONENT = EXPONENT(TINY) and MAXEXPONENT = EXPONENT(HUGE), where EXPONENT is the Fortran function that returns the exponent in the Fortran numerical model. These values are shifted by one compared to the stored exponent in the binary representation, due to the sum in the numerical model starting at one, as discussed above.

The PRECISION is the number of significant digits (the digits after the decimal point) that are guaranteed to be represented accurately; That is, when you assign a number to a double precision float, the number actually stored will always match the assigned number within at least 15 significant digits.

The RANGE is defined as INT(MIN( LOG10(HUGE), -LOG10(TINY) )), in this case RANGE = INT(-LOG10(2.225e-308)) = INT(- (308 + 0.347)) = INT(307.653) = 307.

Finally, EPSILON is the difference between one and the next closest representable number, given as $2^{-52}$. This is the smallest difference for any two neighboring numbers greater than one. 0.5*EPSILON provides an upper bound for the relative error due to rounding.

Some final observations:

Two distinct double precision numbers can look the same when printed to 15 significant digits. The best example for this is one + EPSILON(one). However, they will always differ in the 16th significant digit in that case.
When printing a float to an arbitrary precision (greater the machine precision), there are never any random digits printed. The exact decimal encoded by the float determines the result. However, the digits beyond the machine precision resulting from a computation involving two floats may depend and the compiler and the environment so that in general, results obtained on different systems cannot be expected to match beyond the machine precision.
To test whether two floats r1 and r2 are the same within some relative error delta_r, the following expression can be used:
```
 approx_eq = (ABS(r1 - r2) <= 0.5d0*ABS(r1+r2) * delta_r)
```
It makes no sense to use a delta_r < 1.0d-16; the test would be equivalent to a direct comparison on the bit-level.
To write out the hex representation of a double precision float r in Fortran, use write(*,'(Z16)') transfer(r, 1_8).
Read What Every Computer Scientist Should Know about Floating Numbers

Advanced Array-Passing in Fortran

2011-05-19T00:00:00+02:00

There are instances where one wishes to change the rank (the number of dimensions) of an array as it is passed to a subroutine. Maybe you want to call a library routine that expects to get a vector, but you have the data to be passed stored in a matrix. There should be no problem in principle, as the data for an array – no matter its rank – is stored sequentially in memory (array element order of subscripts along the first dimension varying more quickly). Thus, it should be possible to simply re-interpret the data to a different shape. In practice, one can change the rank of the array by declaring the dummy array with an explicit shape specification. As soon as the dummy array has an explicit shape, one can pass anything that contains enough elements to fill the dummy array, no matter what the rank or shape. The feature is known as array element sequence association and is illustrated in the following example:

    program test
      implicit none
      integer :: test_array1(10)
      integer :: test_array2(10,2)
      test_array1 = (/1, 2, 3, 4, 5, 6, 7, 8, 9, 10/)
      test_array2(:,1) = (/1,   2,  3,  4,  5,  6,  7,  8,  9, 10/)
      test_array2(:,2) = (/11, 12, 13, 14, 15, 16, 17, 18, 19, 20/)
      write(*,*) "print_a(test_array1, 10)"
      call print_a(test_array1, 10)
      write(*,*) "print_a(test_array2, 10)"
      call print_a(test_array2, 10)
      write(*,*) "print_a(test_array2, 20)"
      call print_a(test_array2, 20)
      write(*,*) "print_a(test_array2(:,2), 10)"
      call print_a(test_array2(:,2), 10)
      write(*,*) "print_m(test_array1, 2,5)"
      call print_m(test_array1, 2, 5)
    
    contains
    
      subroutine print_a(a, n)
        integer, intent(in) :: n
        integer, intent(in) :: a(n)
        integer :: i
        do i = 1, n
          write(*,*) a(i)
        end do
      end subroutine
    
      subroutine print_m(a, n, m)
        integer, intent(in) :: n
        integer, intent(in) :: m
        integer, intent(in) :: a(n,m)
        integer :: i, j
        do i = 1, n
          do j = 1, m
            write(*,*) a(i,j)
          end do
        end do
      end subroutine
    
    end program test

There are two alternatives to do basically the same thing, one standard, one not. The first one is known as assumed-size, and looks like this:

      subroutine print_a(a, n)
        integer, intent(in) :: a(*)
        integer, intent(in) :: n
        integer :: i
        do i = 1, n
          write(*,*) a(i)
        end do
      end subroutine

Here, a has rank one, and a size large enough to fit whatever is passed to the routine. However, the routine does not actually know the size of a (size(a) is illegal in this context). There certainly is nothing that ensures that the size of a is n, and the compiler has no way of perfoming any checks on the use of a. In that sense, assumed-size is not very useful compared to explicit-shape, except maybe in some rare instances where the array size can somehow be deduced from the data stored in the array, so that the array size n does not have to be passed along.

The second alternative is a non-standard trick that behaves identically to assumed-size, but was used before the assumed-size feature was introduced to Fortran. It looks like this:

      subroutine print_a(a, n)
        integer, intent(in) :: a(1)
        integer, intent(in) :: n
        integer :: i
        do i = 1, n
          write(*,*) a(i)
        end do
      end subroutine

In principle, this is also an example of array element sequence association. In the subroutine, a is called with subscripts outside of the bounds of its declared size 1, which is more or less guaranteed to work since the array is passed by reference. Again, the compiler has not idea about the actual size of a. Most compilers will recognize this trick and not complain about going out of bounds, even with bound-checking enabled, understanding a(1) to be identical to a(*), with the only difference being that size(a) is legal, but of course returns 1, i.e. not the size of the actual passed array. Declaring a as a(2) will not work, however.

Any of these definitions should only be used if the rank of the array needs to be changed. For just passing arrays of identical rank, but unknown shape, one should always use the assumed-shape syntax (e.g. a(:,:))

A few pointers about where to find the discussed concepts in the Fortran 90 Handbook: array element order is discussed in section 6.4.7, the definition of array element sequence association appears in section 12.5.2.1, and explicit-shape, assumed-shape, deferred-shape, and assumed-size specification are explained in section 5.3.1.

Thanks to the people on comp.lang.fortran for illuminating some of these concepts to me.

PDF Bookmarks with LaTeX

2011-04-29T00:00:00+02:00

The combination of pdfpages, hyperref, and bookmark allows for a very neat way of adding an outline to an existing pdf file. For example, we can use the following tex file to add a (partial) outline to my diploma thesis.

    \documentclass{article}
    \usepackage[utf8]{inputenc}
    \usepackage{pdfpages}
    \usepackage[
      pdfpagelabels=true,
      pdftitle={Optimization of a Controlled Phasegate for Ultracold Calcium Atoms in an Optical Lattice},
      pdfauthor={Michael Goerz},
    ]{hyperref}
    \usepackage{bookmark}

    \begin{document}

    \pagenumbering{arabic}
    \setcounter{page}{1}
    \includepdf[pages=1-2]{diploma_thesis.pdf}

    \pagenumbering{roman}
    \setcounter{page}{1}
    \includepdf[pages=3-8]{diploma_thesis.pdf}

    \pagenumbering{arabic}
    \setcounter{page}{1}
    \includepdf[pages=9-]{diploma_thesis.pdf}

    \bookmark[page=1,level=0]{Title Page}
    \bookmark[page=9,level=0]{1 Introduction}
    \bookmark[page=17,level=0]{2 Quantum Computation with Calcium Atoms}
        \bookmark[page=18,level=1]{2.1 Trapping of Calcium Atoms}
        \bookmark[page=20,level=1]{2.2 Internal Degrees of Freedom and Description of a Single Qubit}
        \bookmark[page=21,level=1]{2.3 Description of Two Qubits}
            \bookmark[page=22,level=2]{2.3.1 Qubit-Qubit Interaction}
    \bookmark[page=23,level=2]{2.3.2 Harmonic Approximation of the Trap Potential}
            \bookmark[page=24,level=2]{2.3.3 Summary of Two-Qubit Description}
        \bookmark[page=26,level=1]{2.4 Asymptotic Description of Two Qubits}
        \bookmark[page=27,level=1]{2.5 Quantum Information Processing with Calcium}
    \bookmark[page=31,level=0]{3 Numerical Tools}
    \bookmark[page=47,level=0]{4 Phasegate Optimization Schemes}
    \bookmark[page=65,level=0]{5 Optimization Results for the Controlled Phasegate}
    \bookmark[page=87,level=0]{6 Summary and Outlook}
    \bookmark[page=93,level=0]{Appendices}

    \end{document}

Assuming that the above is stored as thesis_with_bm.tex, simply running pdflatex thesis_with_bm.tex will create a copy thesis_with_bm.pdf of diploma_thesis.pdf that contains the defined outline. Also, the title and author in the pdf meta data are set, and the pages are labeled correctly. A great way to post-process scanned documents!

Creating Combined tikz/png Plots

2010-01-24T00:00:00+01:00

In order to create truly high quality plots, you can have gnuplot write tikz files. Wheres the latest CVS development version of gnuplot have the tikz terminal included, I’m still using the patch by Peter Hedwig for gnuplot 4.2.

Generating tikz code from gnuplot works very nicely for “standard” plots, but can be problematic for others. If the plot contains a lot of points, the resulting tikz file can be hundreds of megabyte in side, and the resulting pdf will constipate your pdf reader, if you get it to compile at all. In fact, you run into the same problem if you use the pdf or postscript terminal in gnuplot directly.

Another type of plot where this tends to happen is 3D color maps. The result in tikz is thousands of small colored squares, each of to be rendered as vector graphics.

Is there any way to make these plots manageable, while keeping the high quality of tikz output in place? A possible solution is to combine a bitmapped plot with vector-graphics decorations (axes, legend, etc.). To do this, we let gnuplot generate the same plot twice: once with the tikz terminal, and once with the png terminal, but without any axes, labels, etc. We then modify the tikz output and delete all the parts for the actual plot, and insert the png image in their place. This means we have the png image, and everything else drawn on top of it with tikz.

Let’s look at this in practice.

Suppose I have a huge data file of a Wigner plot that I want to visualize as a color map. I use the following gnuplot file:

set term lua plotsize 8cm,6cm font " \\tiny "
set out "wigner.tikz"

set pm3d map
set palette defined (-0.0015 "blue", 0 "white", 0.0015 "red")

splot "wig" u 1:2:3

set term png size 800,600
set out "wigner.png"

set lmargin at screen 0
set rmargin at screen 1.0
set bmargin at screen 0
set tmargin at screen 1.0

unset tics
unset border

splot "wig" u 1:2:3

We can see that the tikz plot is create first, then the png plot. With those settings, the png file contains just the actual plot area without any margins whatsoever. It looks like this: wigner.png

The tikz file that comes out of this has some 150 MB. However, if we open it up in a text editor, we can easily identify its structure, and write a small script that replaces the actual plot with the png file:

combine_pm3dmap.pl

The second thing this script does is to make the plot legend look quite a bit nicer. In the original, the legend was just a bunch of colored boxes stacked on top of each other to create a fake gradient, which looks horrible. We replace this with two true gradients, which works because we set the colors in the gnuplot script to go from blue to white to red. So we can have one true gradient from blue to white, and one from white to red.

Of course, the script is not really general, but I think it should work for any pm3d map plot that uses a three-color gradient centered around white as the palette. Also, if it’s just for a single plot, you could just as easily do this by hand.

After processing the tikz file by calling ./combine_pm3dmap.pl wigner.tikz wigner.png, we get this modified tikz file that is only 4.8 KB in size:

wigner.tikz

It can be easily compiled (e.g with the tikz2pdf script, and don’t forget to include gnuplot-lua-tikz.sty), resulting in the following pdf:

wigner.pdf

Accessing GMail through Python

2009-11-29T00:00:00+01:00

Over the last couple of years, I’ve made several attempts at efficiently interacting with Gmail through Python. There were a number of motivations:

Filtering mail: Putting emails into folder based on more elaborate rules than the server-side filters Gmail provides.
Processing mail: Modifying emails, for example decrypting gpg-encrypted mails, or fixing missing header fields.
Import and export: You want to keep local backups of your emails, and restore them.
New mail notification, distinguishing between senders.

How Does Gmail Work?

Gmail follows a somewhat new paradigm compared to traditional mail systems.

Traditionally, a mailbox contains a ordered list of messages. These messages can sometimes have a limited number of tags (e.g. ‘replied’), and conversation threads can be constructed from the ‘references’ field in the email’s header. In Python, there is a class describing the traditional mailbox interface. There tends to be a collection of mailboxes (folders) associated with one email account / server, usually at least an Inbox and a Sent mailbox.

In contrast, a Gmail mailbox does not center around messages, but instead around conversation threads: the mailbox is an unordered list of conversation threads. Each thread can have a number of tags or labels associated with it. Note that theses labels are not associated with the individual messages, but always with the thread as a whole. Threads cannot be split or combined, there is no guaranteed relationship between the ‘references’ header field in the messages and the membership to a thread.

Also, even though the ‘Inbox’, ‘Sent’, ‘All Mail’, etc. look very similar to the structure of multiple folders on, say, an IMAP server, this can be misleading. You should look at your Gmail account as a single mailbox, with ‘Inbox’, ‘Sent’, ‘All Mail’, etc. being labels to threads. The difference is that each thread can have multiple labels, compared to a message always being in a single folder. If you put a message into multiple folders on your IMAP server, these messages would be completely independent: They’d take up more space, and deleting the message from one folder would leave it entirely unaffected in the other. In Gmail, applying additional labels to a thread does not use more space, and putting a thread into the Trash removes the entire thread globally. Google explains this in their help section.

How Does Gmail Work through IMAP?

In addition to its web interface, which is based on the paradigm I just explained, Google also provides access via IMAP.

To do this, it is necessary to map Gmail features somehow into IMAP. Google decided to break the focus on threads (in the web interface, you’ll always have entire threads in your Inbox, for example, while in IMAP, not all of the messages belonging to the same thread have to be in the Inbox at the same time). Also, they mapped labels to folders. This means that from the IMAP client’s point of view identical messages have no immediate connection between them, they have to be downloaded multiple times, and any changes to one do not affect the other copies. To keep the consistency, Google behaves as if there constantly was a second IMAP client connected to the account, reacting to anything you do. If you move a message to the Trash, the shadow client will also delete the copies of the message from all other folders. In this way, Google conforms to the IMAP standard, while keeping the consistency with their paradigm.

Nonetheless, in many situations the way in which the Gmail system was mapped to IMAP is not so well chosen. For one thing, it would be nice to (at least optionally) keep the messages of a thread together even in the IMAP interface. Of course, this would mean even more intrusion by the shadow client, but at least you’d be able to look at the entire thread without having to switch to the ‘All Mail’ folder. It is also not always clear when a message will show up in a given folder.

Secondly, at least from the programmer’s point of view, it would be nice to have access to the labels of a message not just through different folders. It turns out the the IMAP standard allows for arbitrary labels (‘flags’) to messages (RFC3501. I am not aware of any email client that makes full use of this possibility, a few (like Thunderbird) use quasi-standard labels ‘$1’, ‘$2’, etc and internally map them to labels like ‘personal’, ‘work’, etc. This lack of support is probably what suggested Google to use folders instead. Nonetheless, there is no reason why the Gmail IMAP server should not provide access to the label-flags directly.

Gmail Mailbox Class

It would be nice to have a Python class that provides a high level interface to Gmail through IMAP. Of course, one can access the Gmail account just on the normal IMAP level, as explained in the previous section. For this, I’ve written a package called ProcImap. Just using that, one can already do a lot. However, one would really like to move slightly more in the direction of the unique Gmail paradigm. Specifically, we want two additional methods:

Return all the labels of a given message
Return all messages that belong to the same thread as a given message.

It turns out that both of these demands are extremely difficult to fulfill. I experimented with some implementations for this in a Gmail module as part of the ProcImap package, which was meanwhile removed again due to the difficulties involved. Obviously, if Google decided to give access to the labels via the IMAP flags feature, the first problem would be solved. For the second problem, Google would have to keep a strict correlation between the ‘references’ headers in the messages and the thread status.

So, what makes this problem so hard? Essentially, it comes down to the fact that you have to deal with multiple copies of the same messages in different folders. We could just go through all folders, find out which of them contain a copy of the given message, and thus reconstruct a list of labels. Likewise, we could reconstruct the threads (to the limit of the information we can get from the ‘references’ headers) by identifying copies of the same message.

Unfortunately, this is something IMAP was absolutely not made for. There simply is no efficient way to decide whether two email messages are copies of each other. In theory, the ‘message-id’ header field should fulfill this purpose, but its proper use has never been enforced. Messages may have missing IDs, or different messages may have the same ID. Comparing the full header of two messages is also not enough: one could be a copy of the other, stripped of attachments, for example. The only thing that would work is to compare a hash of the full messages. This is highly inefficient and also very sensitive to any sort of corruption.

There is also one last issue: Gmail sometimes considers different mails as identical, and refuses to add them to the account. It is completely unclear under which conditions this happens. For example, I’ve had instances where Gmail would refuse to add (i.e. silently ignore) the decrypted version of an email while the encrypted version was still in the mailbox. Also, trying to upload a copy of an email with some modified header fields often fails.

The severity of these problems becomes clear when we look at the decryption problem again in detail. We would have to follow the following steps:

Save all the labels of the message. As discussed above, this is very difficult.
Download the encrypted email and move the original to the Trash (direct deletion of a message is not possible). This will remove all copies of the message from all folders.
Identify the message in the Trash, and delete it from there. As discussed above, the identification is hard. If we neglect this step we may run into the problem that the upload of the decrypted message will fail. Alternatively, we can delete the entire trash, this however may be undesirable.
Decrypt the message and upload it again.
Store copies in all the original labels.

So far, there is no satisfactory way to go through this process.

Brute Forcing Gmail

There is one possible but inefficient way out of this: Download every message in every folder (that means downloading multiple copies of the same message), and analyze the structure of the messages. We can then keep the information of which message has which labels, and which message belongs to which thread in a local cache. We would have to update that cache in regular intervals (which could be done at a sufficiently small cost since we only need to process changes since the last update). Using that cache, we could implement an interface that provides access to most aspects of the Gmail paradigm. The no longer existing Gmail class in the ProcImap package experimented with that approach, but the entire concept turned out to be very inelegant in its implementation.

Nonetheless, the brute force approach can be useful for certain applications. For example, I have a collection of python scripts (gmailbkp) that work on top of ProcImap and provide incremental backup of your gmail mailbox.

libgmail

There is one available alternative to the IMAP approach: The libgmail package. This package uses Java Script to access Gmail (basically, it connects like a browser to the Gmail website). The package provides a full abstraction of the Gmail paradigm, and it does absolutely everything we want exactly how we want it. There is one big drawback, however: Google does not support the library and monitors their web interface for ‘suspicious activity’. If you access your account too intensely using libgmail, Google will lock you out for 24 hours. This makes it impossible to use the library for backup purposes. I did try: gmail_archive.py. Be warned that using this script will very likely get you locked out of your account.

For smaller tasks, libgmail can be extremely valuable. It probably could solve the decryption example. However, even with such small task the danger of being locked out is always present.

In conclusion, there is no satisfactory way to interface Python with Gmail. You either have to limit yourself to IMAP, or work with the dangers of libgmail. The ProcImap package will probably not provide any explicit support for Gmail in the future.

Three little git scripts

2009-10-24T00:00:00+02:00

I have three scripts that help me with my daily use of git

git-cat

I like to have a quick way to look at the contents of a file in an old revision. In SVN, that was svn cat. With git, you could use git show, as in git show rev:<relative/to/repo/root/file>, but that requires to give the full path of the file you’re trying to cat. Not very practical. My git-cat script solves this. It’s just one line:

git cat-file -p $(git ls-tree $1 "$2" | cut -d " " -f 3 | cut -f 1)

You use it as git cat rev file. I suppose it should be possible to define this as an alias, but somehow I wasn’t able to get that to work.

git-vimdiff

While the colored diffs that git provides are extremely nice, I sometimes like to compare the modified and the unmodified version of a file side-by-side in vim. That’s what git-vimdiff does. It relies on the above git-cat script. All it really does is to cat the HEAD version of the file to a temporary file, and call vimdiff.

git-graph

Lastly, I like to look at graphs. git-graph helps with that. It’s not that different from doing git log --graph --decorate --all, but first of all, I don’t like to remember all of that (sure, I can define an alias). git-graph also shows the references in a nicer way (shorter, color coded: green for local, blue for remote). Lastly, it can also show SVN revision numbers, which is extremely useful when you’re using git-svn.

When you download the git-graph script, it’s best to save it without the extension (like the other two scripts). The thing is, when you have one of the newer versions of git installed where you can use both the dashed (git-clone) and the non-dashed (git clone) version of commands, this extends to anything on your system. When you call git graph, git will automatically execute git-graph.

Installing a CVS version of Gnuplot in MacPorts

2009-10-24T00:00:00+02:00

I wanted to have a bleeding edge version of gnuplot, mostly for the new TikZ terminal. Here’s what to do:

Download the CVS version of gnuplot with

cvs -z3 -d:pserver:anonymous@gnuplot.cvs.sourceforge.net:/cvsroot/gnuplot export -D now gnuplot
Run ./prepare inside of the gnuplot folder, in order to generate a configure script
Rename the gnuplot folder to gnuplot-4.5.0
Create a tarball gnuplot-4.5.0.tar.gz and put it up on some server
Create a local portfile repository, as explained in the MacPorts Guide. I created mine in

~/.macports/portrepo
Take a copy of the original gnuplot Portfile and put it in

~/.macports/portrepo/science/gnuplot
Modify the Portfile to adapt it to your self-created tarball. I end up with Portfile
- Update the version to 4.5.0
- Set your url at the master site
- Update the checksums (use md5sum and openssl as described in the the MacPorts Guide)
- Take out the references to gnuplot.pdf
Make sure that your local portfile repository is listed in

/opt/local/etc/macports/sources.conf

and do portindex inside of the repository
You can now do

sudo port install gnuplot

in order to install your CVS version

Lastly, if you hate the Aquaterm terminal as much as I do, you can change the default by putting export GNUTERM=x11 in your .bashrc.

Annotating LaTeX Lists with Flowchart symbols

2009-10-06T00:00:00+02:00

There’s a neat little trick of putting TikZ code in the place of list bullets in LaTeX. Together with TikZ’s overlay feature, I was able to make a small graphical flowchart out of of list items: You just put a small tikzpicture in the place of every bullet, defining some globally accessible node with the ‘remember picture’ option. This has to be done by defining a command for inserting the picture, like this:

    \newcommand\fcInstr[1]{%
      \begin{tikzpicture}[x=10pt, y=10pt, remember picture]%
        \node[coordinate] (#1) at (1,0.5) {};
        \draw (0,0) rectangle (2,1);%
      \end{tikzpicture}%
    }

Not that the node is named after the first parameter: every node needs to have a unique name.

You then write your list

    \begin{itemize}
      \item[\fcInstr{a}] Set $t_n$: time grid for global $[0,T]$
      ...

Lastly, you can add add another tikzpicture to your document in which the defined nodes are referenced. In my example, I want to draw arrows between the ‘bullet’ picture.

Alltogether, it comes out like this:

Command Line Interface for InsomniaX

2009-09-03T00:00:00+02:00

For the brief period of time that InsomniaX did not work on Snow Leopard, I tried to work with the legacy (0.5) version, but it was a mess. @4lex pointed out to me that you can just load the kernel extension that Insomnia provides from the command line.

Meanwhile, Insomnia has been updated for Snow Leopard, but nontheless it is very convenient to have a command line interface. So, here’s the script: