The primary goal of training to to pick the best parameters, with secondary goals to to find these parameters fast and with minimal resources. All three of these goals compete with one another. This is a hard problem. Ultimately, you’re using convex optimization techniques to solve a non-convex optimization problem. The loss surface you’re trying to find a minimum for might look kind of crazy; imagine something like this:

Note, this is just a fake objective function. Pavel, et al.3 created some visualizations of real objective functions in deep neural nets that make it clear this visualization isn’t particularly far off from objective functions you may encounter in the wild. Keep in mind though, that this objective function has only two variables. A neural net with millions, or even billions4, of parameters can have types of complexity that are impossible to really visualize like this.

Just looking at this, it’s clear that the risk of converging to a local minimum is quite real. Converging to a local minimum isn’t necessarily a huge problem in and of itself. If a local minimum is close to a global minimum for example, the cost of getting it wrong might not be that high on inference, assuming the training data is fairly representative of the test. That said, even “good” local minimum can slow down training, since techniques like mini-batch gradient descent5 are used in part to avoid getting stuck in local minimum. When the algorithm gets near a local minimum, it may hang out around there for a while before jumping out, since the gradient is flat their.

Saddle points are another problem. Like local minimum, they can slow down optimizers, due to their flat gradients. In the high-dimensional loss functions common in neural networks, saddle points are much more common than local minimum6.

Yet another problem is “sharp” minima. Informally, a sharp minimum is one where a small change in parameters yields a large change in the loss function near the minimum. More formally, a sharp minimum is one where ∇²f(x_min), the second-derivative matrix of the loss function at the minimum, has large positive eigenvalues7. Because of this sensitivity, and because training data will never perfectly represent inference data, sharp minima degrade generalization quality.

In the next few posts, I’m going to explore techniques that can be used to mitigate some of these issues. As you’ll see, much of the standard deep learning toolkit is really focused on this problem. I’ll look at the role of batch size in mini-batch gradient descent, advanced optimizer techniques, normalization techniques, and other optimization innovations.

Subscribe now

The idea of entropy was originally created in physics, defined as a measure of disorder and irreversibility in thermodynamic systems. Intuitively, entropy quantifies how dispersed or random the energy in a system is. The Second Law of Thermodynamics states that in an isolated system, one with no energy exchange with the environment, entropy tends to increase or remain constant. In other words, natural processes lead to greater disorder over time1. A classic example is heat flowing from a hot object to a cold one: the energy spreads out and becomes less available to do work, increasing the overall entropy of the combined system.

In statistical mechanics, if a system can be in microstate i with probability p_i​, then the Gibbs entropy is:

where k_B is the Boltzmann constant2. Here a microstate is the specific arrangement of the particles in the system: their positions and momentums. This Gibbs entropy formula is deeply connected to the concept of information entropy.

Entropy in information theory

In 1948, Claude Shannon introduced entropy in the context of information theory to quantify uncertainty in a data source3. Information entropy measures the average information content or “surprise” of a random variable. For a discrete random variable X with possible outcomes {x₀, x₁, …} having corresponding probability {p₀, p₁, …}. The entropy H is defined as:

To get some intuition around this formula, consider the entropy of a coin toss, where the probability of flipping heads varies from 0 to 1:

Note that the entropy is greatest when heads and tails are equally likely. The entropy converges to zero whenever there is no randomness to the coin toss. Intuitively, there are more “surprises” in flipping an fair coin, than when flipping an unfair one.

Note that entropy provides a lower bound for the average number of bits required to transmit or store a sequence of events drawn from a probability distribution. Also, you can find an encoding that gets arbitrarily close to this lower bound. Consider the case where P(heads) = 2/3 and P(tails) = 1/3. The entropy in this case is roughly 0.918. If you encode pairs of heads/tails sub-sequences using this encoding:

• heads, heads => 0

• heads, tails => 10

• tails, heads => 110

• tails, tails => 111

then average number of bits required to encode two symbols is 1.889, and the average number of bits per symbol is 0.944. You can reduce this number closer to the lower bound by increasing the size of the subsequence used in the encoding.

A related idea to entropy is cross entropy4. Let’s say the true probability distribution is p, but your estimate of the distribution is q. Then the cross entropy (also written with H) is defined as:

In information theory, cross entropy provides a lower bound for the average number of bits required to transmit or store a sequence of events drawn from a probability distribution p when you estimate the distribution is q. H(q, p) will always be greater than or equal to H(p), so there’s a cost to bad estimation. The difference between H(q, p) and H(p) is called the Kullback–Leibler (KL) divergence:

Subscribe now

Entropy in machine learning

In machine learning, cross entropy is commonly used as a loss function for classification problems5. In a classification problem with c classes, each output y is a vector of length c, where y_i is 1 when the label belongs to class i, and 0 for all other entries. For example, an output might be y = (0, 0, 1, 0). An estimate of the output ŷ might be something like (0.1, 0.05, 0.8, 0.05). The cross-entropy loss function is:

This is the same as the cross-entropy function above replacing (p, q) with (y, ŷ). Just like in the information theory case, this function is minimized when y = ŷ.

To see an example of what this loss function looks like, let y = (1, 0), and ŷ = (a, 1-a), for 0 < a ≤ 1:

The loss equals zero when a =1.

There are other uses of entropy in machine learning, in addition to cross-entropy loss. KL divergence often appears in scenarios where you explicitly want to push one probability distribution toward another. For example, in variational inference and variational autoencoders (VAEs), the training objective includes a KL divergence term to make an approximate posterior close to a prior6. In reinforcement learning, algorithms like Trust Region Policy Optimization7 and Proximal Policy Optimization8 constrain or penalize the KL divergence between the new and old policy to ensure gradual updates.

The key idea is that applying the linear transformation A to the eigenvector v scales the vector’s magnitude, but otherwise leaves it untransformed. I recently dug into how eigenvectors and eigenvalues get computed in practice, and I wanted to share it, because it was fairly unintuitive, and actually kind of cool.

First, it’s worth sharing how I learned to compute eigenvalues back in school, since you might have seen this in a linear algebra class somewhere. Note that:

Assuming v is a non-zero eigenvector, this then implies that

For an n×n matrix A, this determinant expression translates into a n^th order polynomial of λ. The roots of this equation are the eigenvalues of A. For a given eigenvalue λ_i of A, you can find the corresponding eigenvector by plugging this into the equation and solving for v_i:

That’s all great, but that doesn’t tell you how to actually compute the roots of the polynomial. I did some digging and discovered that standard root-finding algorithms like Matlab’s4 find the roots by finding eigenvalues of the polynomial’s companion matrix5, so this seems circular:

1. To find the eigenvalues of a matrix, just find the roots of a corresponding polynomial.

2. To find the roots of a polynomial, just find the eigenvalues of a corresponding matrix.

I decided to go down the rabbit hole and get to the bottom of this. Here’s a quick summary of how modern libraries actually find eigenvalues.

Triangular Matrices

Note that if you have an upper-triangular matrix, like:

then the eigenvalues are just the diagonal entries of the matrix, since:

This is key, because the general algorithm for finding eigenvalues transforms the matrix into a triangular matrix with the same eigenvalues. Once you’ve done this, your done, since you can just read the diagonal entries as the eigenvalues.

Matrix Similarity

Two matrices A and B are similar6 if there exists an invertible matrix P, such that

A key property of similar matrices is that they have the same eigenvalues. So the basic idea of finding eigenvalues for A is to find a similar matrix to A that’s also upper triangular.

Subscribe now

Schur decomposition

For any square matrix A, there exists an orthogonal matrix Q_* and an upper triangular matrix H such that

Q_* being orthogonal is just a fancy way of saying:

Since H is upper triangular, once you have the Schur decomposition, the eigenvalues are just the diagonal entries of H.

The tricky part is computing the Schur decomposition. To get that, modern eigenvalue solvers use a simpler matrix decomposition: the QR decomposition7. The QR decomposition is just a breakdown of A into an orthogonal matrix Q and an upper triangular matrix R:

QR decomposition is straightforward. A simple implementation using the Gram-Schmidt8 process is shown here:

A more popular implementation uses Householder reflections9. Householder reflections are a bit less straightforward than Gram-Schmidt, but they are the de facto implementation, because they have better numerical stability.

There’s a neat property of QR decomposition that lets you use it to find the Schur decomposition. First, note that for an orthogonal Q:

Q^TAQ is a similar matrix to A. If you set:

you then get a sequence (A₀, A₁, …) of similar matrices to A. This sequence has the nifty property10 that

where Q* is the orthogonal matrix from the Schur decomposition, and H is the upper-triangular matrix Schur matrix. This gives you your eigenvalues, from it’s diagonals. This means you can use code like this to find the eigenvalues:

This is not too different than what production eigenvalue solvers do. There are a couple of key changes they make, for better performance, however:

1. Before the first step, they transform A into an upper Hessenberg11 form. This is a one-time transformation that costs O( n³ ) time, but it then reduces the time of each QR step in the loop from O( n³ ) to O( n² ).

2. At each step, they perform a Wilkinson shift12. This is a transformation to the matrix that makes the overall loop go from having linear convergence to quadratic convergence, for the general case, and cubic convergence when A is symmetric.

1. A Dataset is an object used to access samples from your data set.

2. A Dataloader is an object used to batch samples together.

I probably would have named these classes DataAccessor and DataBatcher, to make it more clear what they do, but the key thing to remember is that you need a Dataset to tell PyTorch how to access your data, and a Dataloader to tell PyTorch how to convert samples into batches for training and inference.

To show how this works in practice, I downloaded the Intel Image Classification dataset from Kaggle3 to my local machine. I created ImageDateset to read the images from my local directory:

The key methods are __len__, which tells PyTorch how many images I have, and __getitem__, which returns an image and its label. For this data set, there are only six labels, which are derived from the subdirectory that contained the image. While not something you’d typically do in PyTorch, you can work with this ImageSet object directly:

Number of images: 14034
First image shape: torch.Size([3, 150, 150])

Note that each image dimension is 3 ⨉ 150 ⨉ 150. The first dimension corresponds to color channel: red, green, and blue. The second two dimensions are a 150 ⨉ 150 intensity grid for each color channel in the image.

I can now wrap this in a Dataloader to produce image batches:

Note that the first time I ran this, I ran into a problem:

RuntimeError: stack expects each tensor to be equal size, but got [3, 150, 150] at entry 0 and [3, 103, 150] at entry 37

The problem is that there’s at least one image which is smaller than the others. I can make a simple transformation to pad all images to have width and height exactly 150:

Now instead of an error I get:

Input shape: torch.Size([64, 3, 150, 150])
Label shape: torch.Size([64])
Input shape: torch.Size([64, 3, 150, 150])
Label shape: torch.Size([64])
...
Input shape: torch.Size([64, 3, 150, 150])
Label shape: torch.Size([64])
Input shape: torch.Size([18, 3, 150, 150])
Label shape: torch.Size([18])

Note that the input shape for all but the last batch is 64 ⨉ 3 ⨉ 150 ⨉ 150. Here, 64 is the batch dimension. I’m batching 64 examples at a time. The very last batch is smaller, with just 18 samples, because the total number of samples wasn’t evenly divisible by 64. If this is undesirable I can set the drop_last argument of Dataloader to True to drop the last batch.

Subscribe now

And DataFrames, Oh My!

DataFrames aren’t a thing in PyTorch, but come instead from the Pandas4 data analysis library. A DataFrame5 is an in-memory representation of tabular data, like what you might read from a database or CSV file. I often find myself starting with a DataFrame at the beginning of my AI pipeline. Because I do this so often, I’ve created a helper class called RowAccessor, which converts a DataFrame to a Dataset:

It takes a DataFrame and a list of label columns as input:

Features: tensor([2., 5.], dtype=torch.float64)
Labels: tensor([1])

If I need to preprocess a DataFrame to get to it ready for PyTorch, I do this before creating the RowAccessor. So if I start with a DataFrame like this:

And I want to make one-hot encoded6 features for the day of the week in the date column and drop the x2 column, I would do something like this:

   mon  tue  wed  thu  fri  sat  sun   x1  label
0  0.0  0.0  1.0  0.0  0.0  0.0  0.0  4.0      0
1  0.0  0.0  0.0  1.0  0.0  0.0  0.0  5.0      1
2  0.0  0.0  0.0  0.0  1.0  0.0  0.0  6.0      0

Finally, I just pass the transformed object to RowAccessor.

In this post, I walk through an “easy case” of getting data: downloading security price data from Yahoo! Finance1. In this case, there’s a good library for the task at hand: yfinance2. yfinance makes API calls to Yahoo! to get CSV price data, and returns them as Pandas DataFrames3. As you’ll see, even this easy case has several gotchas you need to be aware of.

The main method for price data is the download method:

Gotcha 1: Yahoo! broke yfinance

Note, that as I ran this method for the first time for this post, I ran into this issue:

I’ve seen a similar issue before, with a previous version of the library. I did some searching, and uncovered this recent bug, that explained why this happened to catch me. In a nutshell, Yahoo! stopped accepting requests with the user agent string that yfinance was providing. The error says I’ve been rate limited, but that’s a lie; instead it’s just that my requests were rejected by Yahoo!’s servers. With this workaround:

I now get some reasonable-looking data:

Note: the yfinance maintainers will likely fix this bug in a future release, but I’ve seen similar issues working with this API in the past. It’s a bit of a game of Whac-A-Mole, since Yahoo! makes frequent changes, some of which cause the yfinance library to break. There are paid services for getting price data that are more reliable and don’t depend on Yahoo! Finance, like marketstack4, Alpha Vantage5, and Polygon.io6. That said, yfinance is free, and these libraries can be a bit pricey, especially if you don’t use them frequently.

Gotcha 2: Actual rate limiting

Allegedly, there actually is rate limiting associated with Yahoo! Finance. yfinance documents how to reduce the risk of being rate limited by using local caches.7 Unfortunately, there’s no documentation about what rate limits are actually applied. Yahoo! does document that rate limiting exists in its legal documents,8 but it’s not clear if this is just to scare people away from abusing the service, or if there is indeed rate limiting.

Gotcha 3: Adjusted prices

The next issue is which data gets served. By default, all prices served by this library are adjusted to account for dividends and splits. For example, if a $21 stock pays a $1 dividend, and there was no other price movement, the stock price should go down to $20. The investor didn’t lose any money though, since they could just reinvest their dividend back in the stock. Adjusted prices account for this, so you don’t see big discontinuities in the stock price when dividends or splits happen. In this example, if the $20 price was today’s price after the $1 dividend was paid, prices before the dividend date would get multiplied by (20/21). These stack up, to account for multiple dividends and other corporate actions like splits that change the price. For many analyses, this is desirable, but it’s not desirable if you want to look at a signal like the dividend to price ratio, since the adjusted historic price may be different than the actual historic price.

On that note, the library can return dividends and splits, and there’s an argument for getting the price data unadjusted. I personally think it’s best to get all the data, and then I can decide which columns I care about later. There’s no way to get all unadjusted and adjusted data in a single call, but I found a workaround, which is to download the unadjusted data, which already includes the adjusted close prices, and and then use the adjusted close prices to adjust the open, high, and low prices. Putting it all together, I end up with:

Data transformation

Once you download the data, you need some format to store the data and work with it later. As I mentioned in my last post, I store the data using Avro. Avro is a serialization format, so that’s not very helpful for working with the data. DataFrames are easier to work with than the dicts returned from reading Avro files, so I have one step that converts the DataFrame I downloaded from yfinance into an Avro-writable dict:

And I have another method that converts the data I cached in the Avro file into a DataFrame:

Note that the final data frame is slightly easier to work with than the one I got from yfinance, because the column names are all proper variable names, so I can use accessors like df.adj_close to access the adjusted close prices. Also, I made the dates be a column, rather than an index, so df.date returns a column of Python date9 objects. Finally, I picked the format of my Avro schema to make it easy to extract a DataFrame when reading the input.

Subscribe now

My conclusions

As far as data downloads go, this was a fairly mild amount of pain. Still, even in this simple case, I hit several snafus, and it wasn’t the most fun programming exercise. There are several other common data collection problems I was lucky to avoid, like I didn’t need to do any web scraping, which is typically a ton of work to build, and a ton of work to maintain as the underlying website evolves. Also, it’s common that APIs are poorly documented, and it’s not always clear what they return.

This basic idea of a materialized view1 is old, and it comes with some complications. There are many ways to achieve this, and a good place to start with is to think through the requirements. In may case, I’m the only person accessing this data, so for me:

1. I don’t need to access the data frequently. I may go days or longer between running a given pipeline.

2. I’m only using this data for myself, in a single process, so I don’t need to worry about concurrency problems like reads happening before writes complete.

3. After I’ve stored the data, I want to be able to access it even if my machine dies.

4. I want the materialized format to be consistent for all the data I save. By that, I mean, I don’t want to store the data in CSV for some cases, JSON for others, and so on.

5. I’d like the materialized data to be compact, to keep my storage costs low.

6. I want to “remember” what parameters I used to pull the data with, in case I want different variations of the same data for different pipelines.

7. I’d like to be able to survive losing my machine without losing the data.

8. If for some reason I lost the materialized data, I want to be able to rematerialize it as long as the underlying data source is still available, because it’s better than nothing.

9. I’d like some ability to evolve my approach, if I did start bringing in other collaborators.

Because of the infrequent access, files are a nice way to store the data. I could consider a more “proper” database solution, but that adds unnecessary overhead, like the need to have a process running that does nothing most of the time. Also, versioning is non-concern, since I really want the data to be immutable.

Avro2 makes a nice choice of file format. Like CSV it can be used to store tabular data, and like JSON, it can also be used to store hierarchical data. Avro’s binary format is more compact than JSON and CSV, which is helpful for keeping the data compact. Protocol buffers3 or Thrift4 could be used in place of Avro. I chose Avro, since I’m working with Python, and since Avro doesn’t require code generation, which is fairly clunky when working in a dynamically-typed language like Python.

It’s important that the stored data is safe against a crash on my local machine. Because I have an iCloud account and I’m running on a Mac, I’m currently just storing the files in iCloud5. I could just as easily store the files in another distributed file storage system, like Amazon S36. The astute reader might notice that storing my files with a 3rd party violates the whole motivation I had in the first place, which was to reduce the 3rd-party dependency. While this is technically true, my data is realistically safer in a cloud file system than on my local machine, since these systems implement redundancy. This is a risk I’m willing to accept.

One thing I need is a way to keep track of all the files I’ve stored locally, and what parameters I used to grab the data. For this I create a “registry” to track:

1. The data source. I use a unique key to index the type of data I’m loading. If I’m downloading temperature data from the National Weather Service, this type might be something like temperature.nws.

2. Parameters. The parameters will be data-source dependent. For the temperature data for instance, the parameters might be start_date, end_date, and city.

3. File location. This is where the data is actually stored.

I store the entire registry in a JSON file. JSON affords me the same flexibility as Avro, and is easier to debug if something goes wrong. Since I expect the meta-data in the registry to be small relative to the data stored in Avro files, I’m not worried about the extra storage overhead it adds on top of a more compressed format.

Implementation

I created a DataRegistry class that’s responsible for managing the registry. It has two main methods, register and mark_written. The register method is used to register a file for storage, or find a file if one has already been written:

The mark_written method is used to notify the registry that the file has been successfully written to the given location:

The reason for separating these two steps is to handle failures when trying to write data. A client of the registry:

1. Calls register to see if the file has already been written for the given key, schema, and data-provider-specific parameters.

2. If the file has not been written, the client now owns the right to write the file.

3. The client writes the file, and updates the registry to mark the file as written. This way later clients can know that they can safely read the data from the cache or not.

This logic is a bit subtle. To avoid having to think about it more than once, I created an abstract helper class DataProvider, which has a concrete method called cached_read that retrieves the records from the local cache if they exist, or adds them to the local cache if they don’t exist. Concrete subclasses only need to specify how to retrieve records for the first time. This is an example DataProvider subclass:

get_parameters would return something non-trivial if the same data could be retrieved with different parameters, like different time stamps. get_records can return an iterator, and in particular not necessarily a list. This can be useful for retrieving large data sets, where it may not be possible to store the entire data set in memory at once.

The full code for this post can be found in my GitHub repo.

Scaling up to multiple users

As I mentioned, this approach was built just for me. There are some elements of this design which can scale up fairly easy to multiple users. First, as long as the “local” storage is a distributed file system shared with multiple users, there’s no problem having multiple users read from the stored data.

Scaling the registry to multiple users is a bit harder. Rather than storing the entire registry in a single JSON file, it would scale better to have one registry document per key, to limit access collisions. A document database like MongoDB7 can work well for this scenario, since it can provide read-write locking on a per-key basis out of the box. At the cost of running a document database, the solution generalizes fairly well to a multi-user scenario. There’s also a need for a distributed lock, to handle the case when two writers are trying to write to the same file at the same time.

Finally, because the underlying data is stored in Avro, I have flexibility to work with teammates that are using languages other than Python. Most major programming languages have a library to read and write Avro files.

Subscribe now