A More Realistic Example

There aren't just three dimensions

It's easy for us as humans to understand things that happen in two or three dimensions, but vectors in the real world can contain many, many dimensions. I was intentionally misleading us in the beginning. What's interesting is no matter how many dimensions or "traits" we choose, the math expands out to fit them. For example, the dot product for a three dimensional vector may be $\mathbf a \cdot \mathbf b = a_1 b_1 + a_2 b_2 + a_3 b_3$, but it actually expands out to n dimensions: $$a_1 b_1 + a_2 b_2 + a_3 b_3 + \cdots + a_{n-1} b_{n-1} + a_n b_n$$ neat, right?

A larger example

Using the exact same math from our simple video game vector example, we are going to scale this to 50 dimensions. Select a word from the list below and you'll get back the nearest neighbors using our similarity function!

defmodule AlgorithmsForOptimization.CosineSimilarity do
  def dot_product(vector1, vector2) do
    # take both lists, start at index 0, perform x*y
    # This is actually the whole dot product between the two vectors
    Enum.zip_reduce(vector1, vector2, 0, fn x, y, acc -> acc + x * y end)
  end

  def magnitude(vector) do
    ## squrt a1^2 + a2^2 ...
    sum = Enum.reduce(vector, 0, fn x, acc -> acc + x * x end)
    :math.sqrt(sum)
  end

  @spec cosine_similarity(list(), list()) :: float()
  ## the exact same formula from our example
  def cosine_similarity(a, b) do
    dotp = dot_product(a, b)
    magA = magnitude(a)
    magB = magnitude(b)
    dotp / (magA * magB)
  end
end

Here is an example of what "king" looks like with our example embeddings:

And that's still nowhere close to what an actual vector would look like!

All the training and inference that AI models use nowadays is exactly what we just walked through in these few pages, just an enormous amount of dot products across more data than we can imagine. No wonder GPU prices are so high.

That's all I wanted to share, thanks for reading if you made it this far, or if you didn't. Please use the suggest a topic button up above for anything. There is no topic too small.

You also got your first taste of linear algebra. See, it isn't that bad.