Taxis and hash functions

In 2014 New York City released dump of trip and fare logs from NYC taxis in the previous year. In an attempt to protect privacy, the taxi license numbers and medallion numbers were hashed using the MD5 hash function.

If you’re familiar with secure has functions, your first reaction might be that the problem with the scenario above is that NYC used a hash function that had been shown to be broken a decade earlier. But that was not the problem.

MD5 was considered broken in 2004 because someone showed how to create a hash collision, not because one can calculate a preimage. It’s easy to create two messages that have the same MD5 hash value. However, if you have a message m with hash MD5(m) = h, it is still not practical to infer m from h in general. However, if you know that your message m comes from a fairly small set, such as the set of all NYC taxi license numbers, you can hash them all and see which one hashes to h. This would be true for a more secure hash function like SHA-256 just as much as for MD5.

The root of the problem was that the hash function was known. If the city had used a keyed hash, this would have prevented anyone from simply hashing all the license and medallion numbers. If they had chosen a random number and concatenated it to each value before hashing, this would have prevented an attacker from creating a lookup table, even using MD5. There may have been other privacy problems with the data, but this particular line of attack would have been blocked.

Related posts

Orbital footnote

A couple days ago I wrote a post about the orbit of a new dwarf planet candidate. The article says the newly discovered object, ED138, has an orbital period of 281.1 years and is “46 times farther from the sun than Earth is.”

The earlier post showed that we can infer from Kepler’s third law that the orbit must be elliptical, and that therefore the phrase “46 times farther from the sun” is ambiguous. The post worked out the orbit of ED138 under the assumption that the object is 46 AU from the sun at its aphelion, the point in its orbit furthest from the sun.

But what if is “46 times farther from the sun” refers to the average distance from the sun? I added a footnote to the post to explore this possibility. I wanted to expand that footnote into its own post because there’s something I find interesting about it.

If the maximum distance of ED138 from the sun is 46 AU, then the orbit is an ellipse with eccentricity e =  0.072. My intuition was that if we assumed the average distance from ED138 to the sun is 46 AU it wouldn’t make much difference. But it does. In this case the eccentricity becomes e = 0.38, about five times larger.

The outer ellipse, plotted in blue, is the orbit assuming 46 AU is the maximum distance. It’s nearly circular. The blue dot is the position of the sun, near the center of the ellipse.

The inner ellipse, plotted in orange, is the orbit assuming 46 AU is the mean distance. It’s noticeably more elliptical. The orange dot is the position of the sun, further away from the center of the ellipse.

Update: Someone pointed out an error in my original post, and I updated the plot. Now I’m back to the difference between the two orbits not being so big. As always, the change in eccentricity makes more of a difference to the position of the sun in the orbit than to the shape of the orbit.

Related posts

The longest polynomial level set

Let p(z) be a polynomial with leading coefficient 1, i.e. a monic polynomial. What is the maximum length of the curve

{z : |p(z)| = 1}

for each degree n?

Erdős conjectured that the maximum length was obtained by the polynomial p(z) = zn − 1. He was right, as Teng Zhang proved recently.

Examples

Here’s an example for degree five. The maximum length is obtained by p(z) = z5 − 1.

This curve has length 15.02.

Here’s the corresponding curve for p(z) = z5 − z

which has length 6.909.

In general the level set {z : |p(z)| = 1} need not be connected, in which case the length is the sum of the lengths of the components.

Here’s an example that tweaks the constant parameter of the optimal curve p(z) = z5 − 1.1.

This curve has (total) length 14.59.

Details

The length of the maximum curve for given n is the length for the polynomial p(z) = zn − 1. The only curves obtaining the maximum length are translations and rotations of this curve.

Note that for n = 1 the maximum curve is a circle and for n = 2 the curve is the lemniscate of Bernoulli. The general curve is called the Erdős lemniscate of order n, generalizing Bernoulli’s lemniscate.

The maximum length for given n is

21/n B(1/n, 1/2n)

where B is the beta function.

The Erdős lemniscates look similar to the “rose” curves r = cos(nθ), but these curves are not polynomial level sets.

Related posts

Peeking inside the bash rabbit hole

I have much bigger issues to be concerned with than saving a few seconds of typing with obscure bash features. But precisely because I have bigger issues on my plate, it’s fun to take a moment and solve some little tangible problem.

The shell !$ saves the last part of the latest command. (Mnemonic: bang often refers to commands, such as !! to redo the previous command, and cash often refers to the last of something, such as the end of a line in a regular expression.)

This comes in handy, for example, when you copy a file to a directory and then want to cd into that directory.

cp myfile /big/long/path
cd !$

The !$ variable is fragile, however. The sequence of commands

cp myfile /big/long/path
ls
cd !$

won’t work because after running ls the variable !$ now contains “ls” rather than “/big/long/path”.

There are two ways to fix this. In the previous example you could change the last command to

cd !-2:$

This says go back two commands (!-2) and extract (:) the last argument ($).

Another possibility is

cd !cp:$

This says to pull up the most recent command that started with “cp” (!cp) and extract the last argument.

This only scratches the surface of conveniences bash offers. To learn about similar features, search on “history expansion” or “event designators.”

Because bash offers so many shortcuts, it’s hard to learn any of them from reading the manual: nothing will stick. It reminds me of when I was in Paris and had one hour to see the Louvre. I should have meandered around a room or two and been content with that. Instead, I blitzed through the museum. I wanted to “see the Louvre” but I didn’t really see anything.

Kepler and the dwarf

ED138

Kepler’s third law

A recent article from Smithsonian magazine says a new dwarf planet candidate is 46 times further from the sun as earth is, and it takes 281.1 years to orbit the sun.

I did a back-of-the-envelope calculation and thought something was wrong. Kepler’s third law says

T² = c a³.

Here T is the orbital period and a is the semimajor axis. If T is measured in years and a is measured in astronomical units (AU) then the proportionality constant c equals 1.

If T = 281.1 years, then a = 42.91 AU, not 46 AU.

Eccentricity

The difference between 46 AU and 42.91 AU is only a contradiction if we assume the dwarf planet is in a circular orbit. The semimajor axis a is the the maximum distance to the center of the orbit, not the distance to the sun. In an elliptical orbit, the focus of the ellipse where the sun sits is not at the center. The orbit is eccentric, meaning off-center.

Assume the dwarf planet candidate ED138 is in an orbit with semimajor axis 42.91 AU. What does the article mean when it says ED138 is “46 times farther from the sun than Earth is”? Does is mean that it is currently 46 AU from the sun, or that its maximum distance from the sun at aphelion is 46 AU? Presumably the latter [1].

If the distance to the dwarf at aphelion is 46 AU and the semimajor axis is 42.91 AU, then the distance between the sun and the center of the dwarf’s elliptical orbit is c = 3.09 AU. The eccentricity is then

e = c/a = 3.09/42.91 = 0.072

making the orbit more eccentric than that of Saturn but less than that of Mars. This would make the semiminor axis, the minimum distance to the center of the orbit,

b = a √(1 – e²) = 42.8.

The orbit isn’t far from circular, but far enough to account for the substantial difference between the distance to the sun and the distance to the center of the ellipse.

This touches on something that has come up before, and that is eccentricity doesn’t affect the shape of an orbit so much as the position of the sun in that orbit. The orbit of ED138 isn’t far from being circular, but the sun is significantly off-center in that orbit.

Related posts

[1] Another possibility is that the article has in mind the mean distance to the sun, in which case

a(1 + ½e²) = 46

as explained here. This would make the eccentricity much larger, e = 0.38, and the semimajor axis much smaller, b = 39.7.

Image credit: NSF-DOE Vera C. Rubin Observatory / NOIRLab / SLAC /AURA / R. Proctor

Long permutations and long cycles

A permutation rearranges the elements of a list. The number of arrangements, say of a deck of cards, is finite, so eventually you have to arrive at an arrangement you’ve produced before, and history will repeat itself. Along the way, some cards may move in smaller cycles than others.

Example

For example, consider the permutation

p = [5, 6, 2, 1, 0, 8, 3, 4, 9, 7]

that sends i to p(i). Here indices start with 0.

The number 2 is a fixed point: p(2) = 2. So it has a cycle length of 1.

The number 1 is part of a cycle of length 3: p(1) = 6, p(6) = 3, and p(3) = 1.

The longest cycle is [0, 5, 8, 9, 7, 4] which has length 6.

Longest cycles

What happens to the length of cycles as you permute longer lists? Obviously the longest possible cycle increases with the length of the list, but what happens to the average length of the longest cycle?

Let Ln be the length of the longest cycle in a random permutation of n items. You’d expect the average value of Ln, written E[Ln], would increase when n increases. But do they increase at the same rate? Conceivably E[Ln] goes to infinity as n goes to infinity, but at a slower rate, say proportional to √n rather than proportional to n. Let’s make a plot to see.

There’s a lot of variance, but the central tendency looks approximately linear in n. Let’s repeat our experiment, this time creating 100 random permutations of each size and averaging their maximum cycle lengths.

As we’d expect from the central limit theorem, averaging over 100 samples reduced the scatter by about a factor of 10.

It appears that the expected value of the maximum cycle length grows in proportion to n, and in fact the expected value is somewhere around 0.6n. In the example at the top of the post, the longest cycle is exactly 0.6n. I didn’t plan that: the permutation example was chosen at random.

Asymptotic value

The limit of the ratio E[Ln]/n is known to be

\lim_{n\to\infty} \frac{\text{E}[L_n]}{n} = \int_0^1 \exp(\text{li}(t)) \, dt = 0.6243\ldots

Shepp and Lloyd [1] proved the result above. The constant on the right hand side is known as the Golomb-Dickman constant. I think Golomb investigated E[Ln]/n and had found that there was a limiting value, but Shepp and Lloyd were the first to give a closed-form expression for the limit. Dickman had run into the same constant in a theorem in number theory.

[1] L. A. Shepp and S. P. Lloyd, “Ordered Cycle Lengths in a Random Permutation,” Transactions of the American Mathematical Society, vol. 121, no. 2, pp. 340–357, February 1966.

Balanced Incomplete Block Designs

Suppose you have n different fertilizers you want to test. The simplest experiment would be to take a big plot of land, divide it into n sections, and apply each fertilizer to a section. That’s fine if you have one big enough plot of land, and the land itself doesn’t matter.

If the fertilizer supplies a nutrient that is already abundant in a particular soil, you won’t see as much benefit from that fertilizer on that soil. Plots of land will vary in other ways besides soil composition—some may receive more sun than others, some may have better drainage, etc.—that may interact with the effect of the fertilizer. So you’d like to test your fertilizers on several different plots of land. Call the number of plots b.

Suppose each of your plots is large enough to test q < n fertilizers, i.e. the plots are not large enough to test every fertilizer on every plot. You would like to design your experiment so that each fertilizer is tested the same number of times, and so that every pair of fertilizers is tested somewhere on the same plot of land for a direct comparison.

The resulting experimental design is called a balanced incomplete block design or BIBD. The blocks are the plots of land. The design is incomplete because you cannot test everything on every plot. The design is balanced because every pair of fertilizers is tested together the same number of times. BIBDs are important in the design of experiments.

A BIBD is a special case of the combinatorial designs discussed in the previous post with parameters (n, q, 2, λ), i.e. the parameter r is set to 2 because you want every pair to appear in some number of blocks.

A design with parameters (n, q, r, λ) is sometimes called an r-(n, q, λ) design [1]. Steiner systems correspond to λ = 1 and BIBDs correspond to r = 2.

While the number of blocks b is of interest, it is not an independent parameter. For combinatorial designs in general the number of blocks is determined by the other parameters:

b = \lambda \frac{\binom{n}{r}}{\binom{q}{r}}

Similarly, you may see the number of blocks each treatment appears in given as a parameter, but this count is also determined by the other parameters, namely

\lambda \frac{\binom{n-1}{r-1}}{\binom{q-1}{r-1}}

Examples

Here is a design to test n = 6 treatments on b = 10 plots, testing q = 3 treatments on each plot, with each pair of treatments (i.e. r = 2) appearing in λ = 2 plots, i.e. a (6, 3, 2) design.

    0000011122
    1123423433
    2345554545

The way to read this is that each treatment is a number 0 through 5, and each column indicates the treatments tested in a particular block (plot).

Here is another example, an (8, 4, 3) design: n = 8 treatments, blocks of size q = 4, with each pair of treatments being tested in λ = 3 blocks.

    00000001111222
    11123342334334
    22554666455455
    34675777677766

[1] Combinatorial designs are sometimes called t-designs, which makes sense if you denote the size of the set you’re looking for in each block by t but not if you denote it by r. You’ll see both conventions. And while we’re mentioning notational variations, the number of treatments is sometimes denoted v for variety, going back to agricultural applications, testing crop varieties.

Combinatorial designs

The last several posts have been building up to this one, moving from tangible to more abstract.

The posts on finite geometry gave examples of Steiner triples and more generally Steiner systems, introduced in the previous post. Combinatorial designs take this one step further.

The previous post gave the following definition:

A Steiner triple system, written STS(n), is a collection of three-element subsets, called blocks, of a set X with n elements, such that every two-element subset of X belongs to exactly one block.

The post pointed out that the smallest projective plane is STS(7) and the smallest affine plane is STS(9).

The post then defined a general Steiner system:

A Steiner system S(t, k, n) is a collection of k-element subsets, called blocks, of a set X with n elements, such that every t-element subset of X belongs to exactly one block.

Moving from Steiner triples to Steiner systems we generalized the size of the blocks from 3 to k, and generalized the size of sets contained in a unique block from 2 to t.

Combinatorial designs take this one step further by generalizing the number of blocks that a set of t elements falls in from 1 to λ.

Unfortunately when we go from Steiner systems to combinatorial designs the notation changes and the order of the parameters changes.

A design with parameters (n, q, r, λ) is a collection of q-element subsets of an n-element set X such that every r-element subset of X is contained in exactly λ blocks.

So a Steiner system S(t, k, n) corresponds to a (n, k, t, 1) design.

Existence of designs

Given a set of parameters (n, q, r, λ) there may or may not exist a design with these parameters. It’s easy to show that a few consistency conditions are necessary. It was long conjectured that these conditions were sufficient, except possibly for a finite number of values of n. Peter Keevash proved in 2024 that the conjecture is correct.

Theorem [1]: Fix q > r ≥ 1 and λ ≥ 1. There is an N, depending on q, r, and λ, such that for every n > N with

\left. \binom{q-i}{r-i} \;\middle|\; \lambda\binom{n-i}{r-i} \right.
for all 0 ≤ i ≤ r, a design with parameters (n, q, r, λ) exists.

[1] Peter Keevash. The existence of designs. https://arxiv.org/abs/1401.3665v4

Steiner Systems

Steiner triples

Several examples from the recent posts on finite geometry are special cases of Steiner triples.

A Steiner triple system, written STS(n), is a collection of three-element subsets, called blocks, of a set X with n elements, such that every two-element subset of X belongs to exactly one block.

STS(7) is the smallest finite projective plane, the Fano plane, and STS(9) is the smallest finite affine plane. In both cases you interpret the elements of S as points and the blocks as lines.

Several questions come up naturally. First, for which values of n does an STS(n) exist? A necessary and sufficient condition for existence is that n = 1 or 3 mod 6.

For a given n, can there be more than one STS(n), i.e. are there Steiner triple systems of the same size that are not isomorphic? Yes indeed. The number of non-isomorphic instances grows rapidly with n.

When does (some instance of) STS(n) correspond to a projective or affine space? When n = 2k − 1 for k ≥ 3. For n ≥ 13 there are multiple isomorphism classes of STS(n), so for k > 3 there will be an instance of STS(2k − 1) that corresponds to a projective space and other instances that do not. The instance that does correspond to a projective plane will be isomorphic to a finite field analog of the real projective plane.

Steiner systems

Steiner triple systems are a special case of Steiner systems.

A Steiner system S(t, k, n) is a collection of k-element subsets, called blocks, of a set X with n elements, such that every t-element subset of X belongs to exactly one block. Steiner triples are S(2, 3, n) systems.

Steiner systems often correspond to finite geometries with the blocks interpreted as lines. As with the special case of triples, there can be non-isomorphic instances of Steiner systems, and so when I say below that S(t, k, n) corresponds to a certain geometry, I mean that at least one instance corresponds to that geometry.

Projective

The Steiner system

S(2, q + 1, (qd + 1 − 1)/(q − 1))

with q a prime power greater than 2 is a d-dimensional projective space.

Affine

The space

S(2, q, qd)

is an affine space. Again q > 2 is a prime power.

Hyperbolic

The finite hyperbolic plane given as an example in an earlier post corresponds to S(2, 2, 6). Many other Steiner systems are hyperbolic, but I’m not aware of a tidy classification similar to the ones above.

Steiner systems outside geometry

Steiner systems were named after Swiss mathematician Jakob Steiner because he published a highly cited article asking questions about what we now call Steiner triples. His interest in these systems was combinatorial. Others developed the connections to geometry later.

Steiner systems arise in several areas of mathematics outside of geometry, including coding theory, design of experiments, and graph theory.

Finite hyperbolic planes

Euclid’s fifth postulate says that given a line ℓ and a point P not on ℓ , there is a unique line through P parallel to ℓ. In the projective plane there are no parallel lines, and in the hyperbolic plane there are at least two lines through P parallel to ℓ.

Recent posts have looked at finite projective and affine planes. This post will look at finite hyperbolic planes.

Finite projective planes come up far more often in application than finite hyperbolic planes. I’ve seen several resources on finite geometries that don’t mention the hyperbolic case. Finite hyperbolic geometries have less structure than their projective or affine counterparts and so are less useful.

The definition of a finite hyperbolic plane isn’t entirely standardized. We will use the following axioms for this post.

  1. Any two distinct points lie on exactly one line.
  2. Given a line ℓ and a point P not on ℓ, there exist at least two lines through P parallel to ℓ.
  3. There exist four points, no three collinear.
  4. Every line contains at least two points.

An example of a system satisfying these axioms is the complete graph of six vertices. The nodes are the points and the edges are the lines. In set-theoretic terms, the points are the one-element subsets of {1, 2, 3, 4, 5, 6} and the lines are all possible two-element subsets.

Complete graph K6

Note that lines in the finite hyperbolic plane are two element sets. The lines {5, 6} and {1, 3}, for example, are parallel because they do not intersect, even though the representations of {5, 6} and {1, 3} in the visualization above are not parallel in the Euclidean sense.

There are many other examples of system satisfying the hyperbolic plane axioms. Too many in some sense. You could say finite hyperbolic systems are less common in application because they are more common in count.

Discrete geometry posts