In high dimensions, randomly chosen vectors are very likely nearly orthogonal. I’ll unpack this a little bit then demonstrate it by simulation. Then I’ll look at what happens when we restrict our attention to points with positive coordinates.
***
The lengths of vectors don’t contribute to the angles between them, so we may as well focus our attention on spheres.
Suppose we pick two points on a familiar sphere in 3 dimensions, our home planet. By symmetry, we can assume the first vector runs from the center of the earth to the north pole. Otherwise we can imagine picking the first point, then rotating the earth so that the chosen point becomes the north pole.
What vectors are orthogonal (perpendicular) to the north pole? The vectors from the center to a point on the equator. The vectors that are nearly orthogonal to the north pole are the vectors that are near the equator.
On a high-dimensional sphere, nearly all the area is near the equator (more on that here). This means that if we choose two vectors, they are very likely to be nearly orthogonal.
Simulating points on sphere
I’ll illustrate this by choosing pairs of points at random on a unit sphere in 100 dimensions and computing their dot product. This gives the cosine similarity of the points, the cosine of the angle between them. I’ll repeat this process 1,000 times and look at the average cosine similarity.
from scipy.stats import norm
import numpy as np
np.random.seed(20230809)
numruns = 1000
avg = 0
dim = 100
for _ in range(numruns):
x = norm(0,1).rvs(size = dim)
y = norm(0,1).rvs(size = dim)
avg += np.dot(x, y)/(np.dot(x,x)*np.dot(y,y))**0.5
print(avg/numruns)
Here we use the standard method of generating random points on a sphere: generate each component as a normal (Gaussian) random variable, then divide by the length. The code above combines the normalization with finding the dot product.
The code above prints 0.00043. The average dot product between vectors is very small, i.e. the vectors are nearly orthogonal.
Simulating points in orthant
Next let’s look at one orthant of the sphere. (“Orthant” is the generalization to higher dimensions of quadrant in two dimensions and octant in three dimensions.) We’ll generate random samples from the first orthant by first generating random samples from the whole sphere, then taking the absolute value to fold all the points from the other orthants into the first orthant.
The only necessary change to the code is to put abs() around the code generating the random points.
x = abs(norm(0,1).rvs(size = dim))
y = abs(norm(0,1).rvs(size = dim))
When we run the code again it prints 0.639. This is the average value of the cosine of the angle between points, so the average angle is about 0.877 radians or about 50°.
This isn’t quite the right thing to do. We should convert to angles first, then take the average. But that leads to essentially the same result.
If my pencil-and-paper calculations are correct, the exact expected dot product value approaches 2/π = 0.6366 as dimension goes to infinity, and the value of 0.639 above would be consistent with this.