Hiding data in permutations

The latest issue of Paged Out! has an article by Stephen Hewitt “An off-line backup of your cryptographic key using playing cards.” The idea is to use a deck of 52 to store a 128-bit cryptographic key. To erase the key, shuffle the deck. Hewitt gives his algorithm for embedding a key, one that can be carried out manually but isn’t maximally efficient.

You could store a 225-bit key as a permutation of 52 cards because

log2(52!) = 225.581.

But then how would you number permutations so you could go from a number to a particular permutation and later decode the permutation to a number? Is this even practical? For a small number n, you could encode a number k < n by enumerating the first k permutations of a set of n items, and you could decode by enumerating permutations until you find the one you have. But this is completely impractical for large n, such as n = 52.

The process of mapping permutation to an integer is called ranking, and the mapping from an integer to a permutation is called unranking. How efficiently can rankings and unrankings be calculated?

Let n be the number of symbols being permuted. Then there are simple algorithms for ranking and unranking with respect to lexicographical order that have complexity O(n²) and more sophisticated algorithms that have complexity O(n log n). There are also O(n) algorithms that do not preserve lexicographical order.

The Permutations class in SymPy has methods unrank_lex and rank to unrank and rank permutations according to lexicographical order.

The notation the Permutations class uses requires a little explanation. For example, suppose we unrank 2026.

>>> from sympy.combinatorics import Permutation
>>> Permutation.unrank_lex(52, 2026)
Permutation(45, 47, 51, 48, 46, 50)

The output is not a full list of 52 numbers in permuted order; it is only a cycle. The notation refers to the permutation that sends 45 to 47, 47 to 51, …, 50 to 45 and leaves everything else fixed.

If we rank the permutation given above, we get 2026 back.

>>> Permutation.rank(Permutation(45, 47, 51, 48, 46, 50))
2026

Note that we didn’t say how many elements (45, 47, 51, 48, 46, 50) is a permutation of. Because of lexicographical order, the rank would be the same whether we viewed this as a permutation of 52 objects or of more objects.

Now let’s do something larger. Let’s generate a 220-bit number and encode it as a permutation.

>>> n = random.getrandbits(225)
>>> a = Permutation.unrank_lex(52, n)
>>> n
40234719030664563684489051530416964877785781669439875437823431388841
>>> a
Permutation(0, 25, 32, 15, 8, 28)(1, 48, 34, 14, 10, 51, 38, 31, 21, 5, 42, 47, 29, 26, 46, 30, 50, 49, 37, 22, 18, 23)(2, 45, 17, 20, 36, 40, 11, 4, 7, 41, 33, 3, 43, 44, 19, 16, 35, 39, 12, 6, 9)
>>> Permutation.rank(a) == n
True

Now just for fun, let’s display the permutation above applied to a standard (French) deck of 52 cards. As explained here, symbols associated with these cards have a range of Unicode values. By printing these values, we can visualize the permuted deck.

Here’s the code that made the image above.

spades = list(range(0x1F0A1, 0x1F0AF))
spades.remove(0x1F0AC) # take out the knight
cards = [s + 16*i for s in spades for i in range(4)]

a = Permutation.unrank_lex(52, n)
p = a(cards)

for i in range(4):
    for j in range(13):
        print(chr(p[13*i + j]), end="")
    print()

The code above is plenty fast, but Permutation has methods rank_nonlex and unrank_nonlex that run in O(n) time, which could be useful for n much larger than 52.

Counting permutations with roots

My post from yesterday on permutation roots ends with a Mathematica code for finding the probability that a permutation of n elements has a kth root. This is done by finding the coefficient of xn in the generating function

\prod_{m=1}^\infty \exp_{\text{gcd}(m, k)} \left\frac{x^m}{m}\right)

I wanted to say more about this, and look at implementing the same code in SymPy. I was curious how well SymPy would do because I’ve noticed that LLMs often generate SymPy code since it’s an open source CAS.

Wilf [1] describes the infinite product above as the exponential generating function (egf) of f(n, k), the number of permutations of n objects that have a kth root. Since egfs have a n! term in the denominator, this is also the ordinary generating function (ogf) of the probability that a randomly chosen permutation on n objects has a kth root.

My first attempt at using Mathematica to probe the generating function was

expq[x_, q_] := MittagLefflerE[q, x^q]	 
p[n_, k_] :=  SeriesCoefficient[	 
    Product[expq[x^m/m, GCD[m, k]], {m, 1, Infinity}], {x, 0, n}]

This hung forever when I tried to use it on a small example. I realized, but apparently Mathematica did not, that Infinity could be replaced by n since terms higher than n do not contribute to the coefficient of xn. With that change, the code ran quickly.

This morning I tried converting the Mathematica code to Sympy; Claude did this in one shot. I also reproduced the table of f(n, k) values on page 150 of [1] to test the code. Since Wilf tabulated f(n, k), not f(n, k)/n!, I multiplied the results by n!.

Here is the output:

k = 2 [1, 1, 3, 12, 60, 270, 1890, 14280, 128520, 1096200]
k = 3 [1, 2, 4, 16, 80, 400, 2800, 22400, 181440, 1814400]
k = 4 [1, 1, 3, 12, 60, 270, 1890, 13020, 117180, 1039500]
k = 5 [1, 2, 6, 24, 96, 576, 4032, 32256, 290304, 2612736]
k = 6 [1, 1, 1, 4, 40, 190, 1330, 8680, 52920, 340200]
k = 7 [1, 2, 6, 24, 120, 720, 4320, 34560, 311040, 3110400]

and here is the SymPy code. I edited the main but the rest is verbatim from Claude.

from sympy import symbols, gcd, factorial, Rational, S

x = symbols('x')

def expq_coeffs(m, q, n):
    """
    Truncated (degree <= n) series coefficients of
        expq(x**m/m, q) = MittagLefflerE(q, (x**m/m)**q)
    Since q is a positive integer:
        E_q(y^q) = sum_j y^(q*j) / (q*j)!
    with y = x**m/m, so the term of degree m*q*j has coefficient
        1 / ( m**(q*j) * (q*j)! ).
    Returns a list c[0..n] of coefficients.
    """
    c = [S.Zero] * (n + 1)
    j = 0
    while m * q * j <= n:
        deg = m * q * j
        c[deg] += Rational(1, m**(q * j) * factorial(q * j))
        j += 1
    return c

def poly_mult_trunc(a, b, n):
    """Multiply two series (lists of coeffs, index = degree) truncated to degree n."""
    c = [S.Zero] * (n + 1)
    for i, ai in enumerate(a):
        if ai == 0:
            continue
        max_j = n - i
        for j2 in range(max_j + 1):
            bj = b[j2]
            if bj != 0:
                c[i + j2] += ai * bj
    return c

def p(n, k):
    """
    SymPy equivalent of:
        expq[x_, q_] := MittagLefflerE[q, x^q]
        p[n_, k_] := SeriesCoefficient[
            Product[expq[x^m/m, GCD[m, k]], {m, 1, n}], {x, 0, n}]
    """
    result = [S.Zero] * (n + 1)
    result[0] = S.One
    for m in range(1, n + 1):
        q = gcd(m, k)
        factor = expq_coeffs(m, q, n)
        result = poly_mult_trunc(result, factor, n)
    return result[n]

# example
if __name__ == "__main__":
    for k in range(2, 8):
        print("k =", k, [factorial(n)*p(n, k) for n in range(1,11)])

[1] Herbert Wilf. Generatingfunctionology. Available online here.

Printing floating point numbers in binary

It’s well known that you can convert the base 16 (hex) representation of an integer to the base 2 (binary) representation by simply converting each digit from hex to binary. For example,

CAFEhex = 1100 1010 1111 1110two

I imagine it’s less well known that you can do the same thing with floating point numbers.

I wanted to find the binary representation of a floating point number using Python, and discovered that it has no function to do this. However, there is a method on floats to show a hex representation. For example, here’s the hex representation of π.

>>> import math
>>> (math.pi).hex()
'0x1.921fb54442d18p+1'

Curiously, the p+k part at the end is an exponent of 2, not an exponent of 16. So after we convert 1.921fb54442d18 to binary, we’ll need to multiply by 2, i.e. move the fractional point one space to the right.

So first we convert 1.921fb54442d18hex to binary by converting 1, 9, 2, etc. each to binary.

1.1001 0010 0001 1111 1011 0101 0100 0100 0100 0010 1101 0001 1000two

Then after shifting the fraction point to account for the p+1 part we have

π = 11.001001000011111101101010100010001000010110100011000two

You could use Python’s bin() function to convert the fractional part, interpreted as an integer, to hex, though you may need to pad with 0 bits. For example,

>>> (1.03).hex()
'0x1.07ae147ae147bp+0

>>> bin(0x7ae147ae147)
'0b1111010111000010100011110101110000101000111'

The binary representation of 1.03ten is

1.000001111010111000010100011110101110000101000111two

We added a total of five zero bits, four for the 0 after the fractional point and one for converting 7 to 0111two.

Permutation roots

Let σ be a permutation on n elements. If there is a permutation τ such that applying τ twice has the same effect on the list of elements as applying σ once, we say σ = τ² and τ is a square root of σ.

If we let our n elements be the integers 0 through n − 1, then we can represent permutations by what they do to this list of numbers. In Python as a tuple of length n and compose permutations with the following function:

import itertools

def compose(sigma, tau):
    "Return the composition σ ∘ τ (apply τ first, then σ)."
    return tuple(sigma[j] for j in tau)

We can always construct permutations that have square roots by squaring a permutation. If we run the following code

tau = (3, 1, 4, 5, 2, 0)
sigma = compose(tau, tau)

we find σ = (5, 1, 2, 0, 4, 3), and by construction (3, 1, 4, 5, 2, 0) is a square root of &sigma, though it’s not the only one.

The following code shows that σ has four roots.

import itertools

def numroots(sigma):
    n = len(sigma)
    c = 0
    for tau in itertools.permutations(range(n)):
        if sigma == compose(tau, tau):
            c += 1
    return c

print( numroots(sigma) )
print( numroots( (1, 2, 3, 4, 5, 0) ) )

It also shows that the rotation (1, 2, 3, 4, 5. 0) has no roots.

The function numroots has runtime proportional to n! and so it’s not practical for large permutations. There is a theorem that says a permutation σ has a square root if and only if the number of cycles it has of every even length is even. See [1].

We can also define cubes and cube roots of permutations, and higher powers and roots.

How common is it for permutations to have square roots, or cube roots, etc.? If you pick a random permutation on n elements, what is the probability that it has a kth root?

This is a hard question in general, but it is equivalent to finding the coefficient of xk in the infinite product

\prod_{m=1}^\infty \exp_{\text{gcd}(m, k)} \left(\frac{x^m}{m}\right)

This is theorem 4.8.3 in [1]. This theorem was the motivation for writing about expq in the previous post.

Although the product is infinite, there’s no need to compute terms in the product that only contribute powers of x higher than you’re interested in. The following Mathematica code will compute the probability that a permutation on n elements has a kth root.

expq[x_, q_] := MittagLefflerE[q, x^q]	 
p[n_, k_] :=  SeriesCoefficient[	 
    Product[expq[x^m/m, GCD[m, k]], {m, 1, n}], {x, 0, n}]

So, for example, the probability that a permutation of 10 elements has a square root is 29/96.

[1] Herbert Wilf. Generatingfunctionology. Available online here.

exp_q

The function expq(x) is defined by taking the power series for exp(x) and keeping only the terms whose index is a multiple of q. For example, exp2(x) keeps only the even-numbered terms in the exponential power series and so equals cosh(x).

\exp_2(x) = 1 + \frac{x^2}{2!} + \frac{x^4}{4!} + \frac{x^6}{6!} + \cdots = \cosh(x)

In general,

\exp_q(x) = \sum_{n=0}^\infty [q \mid n] \frac{x^n}{n!} = \sum_{n=0}^\infty \frac{x^{nq}}{(nq)!}

The first sum uses Iverson’s bracket notation: a Boolean expression in brackets denotes the function that returns 1 when the expression is true and zero when it is false. Here the bracket equals 1 when q divides n and is zero otherwise.

Closed forms

Let ω = exp(2πi / q). Then

\exp_q(x) = \frac{1}{q}\sum_{k=0}^{q-1} \exp(\omega^k x)

This lets us find closed-form expressions for expq(x). For example, when q = 4, ω = i and

\exp_4(x) = \frac{1}{2}\left( \cosh(x) + \cos(x) \right)

Here’s a proof of the identity above:

\begin{align*} \frac{1}{q} \sum_{k=0}^{q-1} \exp(\omega^k x) &= \frac{1}{q} \sum_{k=0}^{q-1} \sum_{n=0}^\infty \frac{\omega^{kn}x^n}{n!} \\ &= \sum_{n=0}^\infty \left( \frac{1}{q} \sum_{k=0}^{q-1} \omega^{kn}\right) \frac{x^n}{n!} \\ &= \sum_{n=0}^\infty [q \mid n] \frac{x^n}{n!} \\ &= \exp_q(x) \end{align*}

In the proof we used the identity

\frac{1}{q} \sum_{k=0}^{q-1} \omega^{kn} = [q \mid n]

which is important in deriving the properties of the discrete Fourier transform.

Differential equations

The first time I saw the function expq(x) was in differential equations, though I didn’t know at the time the function had a name.

When a course in differential equations gets to power series solutions, a common example or homework problem is to solve

y^{(k)}(x) = y(x)

for k = 3 or 4, i.e. to find a function that equals its third or fourth derivative.

If the initial conditions are

y(0) = 0

and

y^\prime(0) = y^{\prime\prime}(0) = \cdots = y^{(k-1)}(0) = 0

the unique solution to

y^{(k)}(x) = y(x)

is y(x) = expk(x).

Mathematica and Mittag-Leffler

Mathematica does not have a built-in function implementing expq(x), but it does have an implementation of the Mittag-Leffler function, and so thanks to a relation between this function and expq(x) you can implement the latter as

expq[x_, q_] := MittagLefflerE[q, x^q]

Combinatorics

The first time I saw the notation expq(x) was in combinatorics. I had intended to include an application from that book here, but I make that the topic for the next post.

Excel column numbering

I was working with a wide spreadsheet from a client the other day and I had to convert between Excel column labels and column numbers. I had never paid attention to how Excel labels columns and implicitly thought it was base 26 using letters rather than digits. But then I realized that’s not right.

Excel labels columns A through Z, then AA through AZ, then BA through BZ, etc. If this is base 26, then does A correspond to 0? That could work for A through Z, but then what about AA? Then you’d have to say the first A corresponds to 26 but the second A corresponds to 0.

Does Z correspond to 0? If so then the column numbers would be 1 through 25, followed by 0, then 27. And it would mean that columns ZA through ZZ are the same as A through Z.

In fact nothing in Excel column labeling corresponds to 0. The labels cannot be interpreted as a positional number system.

There’s a name for this kind of number system: bijective base 26. The concept extends generally to bijective base b for any positive integer b. The idea is ancient, but the name was coined recently. It has also been called k-adic numbering. For most of history it didn’t have a name.

The motivation behind the name bijective base b is that there is a bijection (a one-to-one correspondence) between these symbols and positive integers; there’s no possibility of leading zeros that would keep the mapping from being a bijection, unlike say 7 and 07 representing the same number.

Excel limits

Before 2007, an Excel file could have a maximum of 28 = 256 columns, and so the largest column label was IV.

Then in 2007 the column limit was increased to 214 = 16,384 and the largest column label is XFD.

Conversion code

Converting from column labels to integers is easy; going the other way is a little more complicated.

letter_to_ordinal = lambda c: ord(c) - ord('A') + 1
ordinal_to_letter = lambda n: chr(ord('A') + n - 1)

def label_to_num(label):
    label = label.upper()
    n = 0
    for c in label:
        n = n*26 + letter_to_ordinal(c)
    return n

def num_to_label(n):
    letters = []
    while n > 0:
        n, remainder = divmod(n - 1, 26)
        letters.append(ordinal_to_letter(remainder + 1))
    return ''.join(reversed(letters))

Here’s an online calculator based on the code above.

Tests

The following code verifies the assertions above about the maximum number of Excel columns over time.

assert(num_to_label(256) == "IV")
assert(label_to_num("IV") == 256)

assert(num_to_label(2**14) == "XFD")
assert(label_to_num("XFD") == 2**14)

The conversion routines are not limited to actual Excel labels but work for arbitrarily large integers and bijective base 26 representations. For example, the following code shows that the bijective base 26 representation of Avogadro’s number is MUAEKAUDYDXEWOSDD.

avogadro = 602_214_076_000_000_000_000_000
assert(label_to_num(num_to_label(avogadro)) == avogadro)
print(num_to_label(avogadro))

Related posts

An almost periodic function

This post takes a more abstract view of the previous post. That post looked at the concrete question of whether a number ever has the same sine in radians as in degrees. The relation between radians and degrees is irrelevant except that π/180 is an irrational number.

Suppose α and β are two positive numbers such that α/β is irrational. In the previous post, α = 1 and β = π/180. Then the function

f(x) = sin(αx) − sin(βx)

is almost periodic: it is not periodic, but it comes close to being periodic, as close as you’d like provided you’re willing to look over a sufficiently long range of x‘s.

The identity

sin(αx) − sin(βx) = 2 cos((α + β)x/2) sin((α − β)x/2)

shows that f(x) is the product of two periodic functions but is not periodic itself. The periods of the cosine and sine above never coincide because the ratio of their frequencies is irrational.

The zeros of f are not periodic, though they can be divided into two subsequences that are periodic.

When sine of x degrees equals sine of x radians

Ordinarily the sine of x radians and the sine of x degrees are very different numbers. Having your calculator in radian mode when it should be in degree mode, or vice versa, results in a major error.

But sometimes it doesn’t matter. A trivial example is when x = 0. A more interesting example is

x = 180π/(180 + π) = 3.08770208….

For that value of x,

sin(x) = sin(x°).

In this article I’ll use the common convention of using radians by default and denoting degrees with ° as above.

Note that

x = πx°/180

and so we are interested in solutions to the equation

sin(x) = sin(πx/180)

Now two angles A and B have the same sine if they differ by a multiple of 2π, or if they’re supplementary (i.e. A = π − B), or both. To put it another way, if A and B have the same sine, they are either equal mod 2π or supplementary mod 2π. This means that

sin(x) = sin(πx/180)

if and only if

x = πx/180 + 2πk

or

x = π − πx/180 + 2πk

for some integer k.

Therefore all solutions have the form

x = 360πk/(180 − π)

or

x = 180π(2k + 1)/(180 + π).

Alternative solution

The derivation above is correct, but it occurred to me later that a simpler argument would be to use the identity

sin(A) − sin(B) = 2 cos((AB)/2) sin((AB)/2).

Thus A and B have the same sine if

cos((AB)/2) = 0

or if

sin((AB)/2) = 0.

These two possibilities correspond to the two families of solutions above.

Density

When reduced modulo 2π, both families are dense in [0, 2π]. This means that for every y in [−1, 1], there is a number x such that

sin(x) = sin(x°) ≈ y

and we can make the approximation as good as we’d like.

Example 1

For example, today is July 22, so let’s set y = 0.722. We’d like to find a value of x such that the sine of x radians and the sine of x degrees both approximately equal 0.722. And let’s say our approximation tolerance is ε = 0.0001.

We can search for a value of x in the first family of solutions by looking for a value of k with

| sin(360πk/(180 − π)) − 0.722 | < 0.0001

and the smallest such k is 96343 and so

x = 360×96343 π/(180 − π) = 616093.78713621…

will do, and sin(x) = 0.72191…

Example 2

Now let’s set y = 0.2026 and look for a solution in the other family of solutions, and this time let’s set ε = 10−6. The smallest value of k such that

| sin(180π(2k + 1)/(180 + π)) − 0.2026 | < 10−6

is k = 741141. Then

sin( 4576848.310950611 ) = sin( 4576848.310950611° ) = 0.202600139…

Forensic accounting in Python

I recently had a project in which I had to reverse engineer a data analysis. There was some ambiguity regarding which of several possibilities someone chose for several of the variables, something analogous to the following example.

Suppose you have three numbers with uncertain values with a known, or at least purported, sum. The first number could be 31, 41, or 59; the second could be either 26 or 53; the last could be 58, 97, 93, or 23.

The following code enumerates all 3 × 2 × 4 = 24 possibilities and prints their sums.

from itertools import product

# Example input
possibilities = [(31, 41, 59), (26, 53), (58, 97, 93, 23)]

for combo in product(*possibilities):
    total = sum(combo) 
    print(f"Combination {combo} sums to: {total}")

In this example all the sums are unique, though of course that might not happen in practice. If, for example, you know the sum is 187, you know the three numbers were 41, 53, and 93. If the reported sum is 200, you know some assumption has been violated because none of the possible choices add up to 200.

More forensics posts

Locally everywhere does not imply everywhere

A couple days ago, Levent Alpöge, a mathematician working at Anthropic, discovered a counterexample to the Jacobian conjecture using Claude Fable 5.

I was curious whether most mathematicians were trying to prove or disprove the conjecture, so I asked Claude.

Before a counterexample to the Jacobian conjecture was found, did most mathematicians believe it was true or false?

Claude’s response was

The premise of this question isn’t quite right — no counterexample to the Jacobian conjecture has been found. It remains an open problem in mathematics: no one has proven it true, and no one has found a counterexample disproving it. … If you encountered a claim that a counterexample was found, do you have a source for that? I’d be happy to look into it, since that would actually be a major result in algebraic geometry if true.

Of course Claude doesn’t know that it solved the conjecture. It didn’t even solve the conjecture. It was an inanimate tool in the hand of a mathematician, just like a piece of chalk or a dry erase marker.

The middle part of Claude’s response was that mathematicians are (were) divided on whether the conjecture is true. So it was not like the Riemann hypothesis, which most people believe to be true, or the P = NP conjecture, which most people believe to be false.

Now what is the Jacobian conjecture? It says that a polynomial function from ℝn to ℝn with constant, non-zero Jacobian determinant has a polynomial inverse. (The conjecture was stated more generally for fields of characteristic 0, in which the derivatives defining the Jacobian would have to be defined algebraically, not in terms of limits.)

Alpöge came up with a counterexample, a polynomial function from ℝ³ to ℝ³ with constant Jacobian determinant −2. The function is

\begin{align*}F(x,y,z)={}\bigl(~\!& z (1+xy)^3 + y^2 (1+xy) (4+3xy),\\ &y + 3x(1+xy)^2 z + 3xy^2 (4+3xy), \\ &2x - 3x^2 y - x^3 z ~\!\bigr).\end{align*}

It’s a tedious but simple calculus exercise to show that the determinant equals −2 everywhere. The inverse function theorem says that a function is locally invertible at any point where the Jacobian determinant is non-zero, so Alpöge’s function is locally invertible everywhere.

However, the function takes on some values more than once. For example, (0, 0, −1/4) and (1, −3/2, 13/2) both map to (−1/4, 0, 0). Therefore the function is not invertible globally. So not only does the function not have a polynomial inverse, it doesn’t have an inverse even if you allow non-polynomial functions.

Alpöge’s counterexample disproves the Jacobian conjecture for n = 3. It can trivially be extended to all n > 3 by defining the function to be Alpöge’s function for three variables and the identity for the rest. The conjecture remains open for n = 2.