The 'HEAD' call that is done to resolve package revisions from
unpinned versions is already quite cheap, but it would still be better
to avoid overloading Github with such calls; especially for users of a
language-server that would compile on-the-fly very often. Upstream
packages don't change often so there's no need to constantly check the
etag.
So we now keep a local version of etags that we fetched, as well as a
timestamp from the last time we fetched them so that we only re-fetch
them if more than an hour has elapsed. This should be fairly resilient
while still massively improving the UX for people showing up after a
day and trying to use latest 'main' features.
This means that we now effectively have two caching levels:
- In the manifest, we store previously fetched etags.
- In the filesystem, we have a cache of already downloaded zip archives.
The first cache is basically invalidated every hour, while the second
cache is only invalidated when a etag changes. For pinned versions,
nothing is invalidated as they are considered immutable.
And so, even for unpinned package. In this case, we can't do a HEAD request. So we fallback by looking at what's available in the cache and using the most recently downloaded version from the cache. This is only a best effort as the most recently downloaded one may not be the actual latest. But common, this is a case where (a) someone didn't pin any version, (b) is trying to build on in an offline setup. We could possibly make that edge-case better but, let's see if anyone ever complains about it first.
When the version isn't a git sha or a tag, we always check that we got
the last version of a particular dependency before building. This is
to avoid those awkward moments where someone try to use something from
the stdlib that is brand new, and despite using 'main' they get a
strange build failure regarding how it's not available.
An important note is that we don't actually re-download the package
when the case occurs; we merely check an HTTP ETag from a (cheap) 'HEAD'
request on the package registry. If the tag hasn't changed then that
means the local version is correct.
The behavior is completely bypassed if the version is specified using
a git sha or a tag, as here, we can assume that fetching it once it
enough (and that it can change). If a package maintainer force-pushed
a tag however, there may be discrepency and the only way around that
is to `rm -r ./build`.
This is a bit cleaner, as the 'cmd/new' had many on-the-fly functions
which are better scoped inside this module.
Plus, it plays nicely with the std::str::FromStr trait definition.
Aiken's build system uses an internal global cache system to avoid
downloading the same packages over and over across projects. However,
prior to this commit, the cache key would be based of the dependency
version which can be either:
- A commit hash
- A branch or tag name
However, in the latter case, it means that the very first time we end
up fetching a dependency will lock its version forever (or until the
cache is cleared). This was inconvenient.
This commit changes that so that we use not only a branch name as
cache key, but additionally, the etag returned by the GitHub API
server. The etag is part of the HTTP headers, so it can be fetched
quickly using a simple HEAD request. It changes whenever the content
behind the endpoint changes -- which happens to be exactly what we
want. With this, we can quickly check whether an upstream package has
been updated and download the latest version should users have
specified a branch name as a version number.
For example, my current cache now looks as follow:
```
/Users/ktorz/Library/Caches/aiken/packages/
├── aiken-lang-stdlib-1cedbe85b7c7e9c4036d63d45cad4ced27b0d50b.zip
├── aiken-lang-stdlib-6b482fa00ec37fe936c93155e8c670f32288a686.zip
├── aiken-lang-stdlib-7ca9e659688ea88e1cfdc439b6c20c4c7fae9985.zip
└── aiken-lang-stdlib-main@04eb45df3c77f6611bbdff842a0e311be2c56390f0fa01f020d69c93ff567fe5.zip
```
KtorZ