The 'HEAD' call that is done to resolve package revisions from
unpinned versions is already quite cheap, but it would still be better
to avoid overloading Github with such calls; especially for users of a
language-server that would compile on-the-fly very often. Upstream
packages don't change often so there's no need to constantly check the
etag.
So we now keep a local version of etags that we fetched, as well as a
timestamp from the last time we fetched them so that we only re-fetch
them if more than an hour has elapsed. This should be fairly resilient
while still massively improving the UX for people showing up after a
day and trying to use latest 'main' features.
This means that we now effectively have two caching levels:
- In the manifest, we store previously fetched etags.
- In the filesystem, we have a cache of already downloaded zip archives.
The first cache is basically invalidated every hour, while the second
cache is only invalidated when a etag changes. For pinned versions,
nothing is invalidated as they are considered immutable.
And so, even for unpinned package. In this case, we can't do a HEAD request. So we fallback by looking at what's available in the cache and using the most recently downloaded version from the cache. This is only a best effort as the most recently downloaded one may not be the actual latest. But common, this is a case where (a) someone didn't pin any version, (b) is trying to build on in an offline setup. We could possibly make that edge-case better but, let's see if anyone ever complains about it first.
Best-effort to assert whether a version refers is a git sha digest or a tag. When it is, we
avoid re-downloading it if it's already fetched. But when it isn't, and thus refer to a branch,
we always re-download it. Note however that the download might be short-circuited by the
system-wide package cache, so a download doesn't actually mean a network request.
The package cache is however smart-enough to assert whether a package in the cache must be
re-downloaded (using HTTP ETag). So this is mostly about delegating the re-downloading logic to
the global packages cache.
This is a bit cleaner, as the 'cmd/new' had many on-the-fly functions
which are better scoped inside this module.
Plus, it plays nicely with the std::str::FromStr trait definition.
Aiken's build system uses an internal global cache system to avoid
downloading the same packages over and over across projects. However,
prior to this commit, the cache key would be based of the dependency
version which can be either:
- A commit hash
- A branch or tag name
However, in the latter case, it means that the very first time we end
up fetching a dependency will lock its version forever (or until the
cache is cleared). This was inconvenient.
This commit changes that so that we use not only a branch name as
cache key, but additionally, the etag returned by the GitHub API
server. The etag is part of the HTTP headers, so it can be fetched
quickly using a simple HEAD request. It changes whenever the content
behind the endpoint changes -- which happens to be exactly what we
want. With this, we can quickly check whether an upstream package has
been updated and download the latest version should users have
specified a branch name as a version number.
For example, my current cache now looks as follow:
```
/Users/ktorz/Library/Caches/aiken/packages/
├── aiken-lang-stdlib-1cedbe85b7c7e9c4036d63d45cad4ced27b0d50b.zip
├── aiken-lang-stdlib-6b482fa00ec37fe936c93155e8c670f32288a686.zip
├── aiken-lang-stdlib-7ca9e659688ea88e1cfdc439b6c20c4c7fae9985.zip
└── aiken-lang-stdlib-main@04eb45df3c77f6611bbdff842a0e311be2c56390f0fa01f020d69c93ff567fe5.zip
```
KtorZ