The 'HEAD' call that is done to resolve package revisions from
unpinned versions is already quite cheap, but it would still be better
to avoid overloading Github with such calls; especially for users of a
language-server that would compile on-the-fly very often. Upstream
packages don't change often so there's no need to constantly check the
etag.
So we now keep a local version of etags that we fetched, as well as a
timestamp from the last time we fetched them so that we only re-fetch
them if more than an hour has elapsed. This should be fairly resilient
while still massively improving the UX for people showing up after a
day and trying to use latest 'main' features.
This means that we now effectively have two caching levels:
- In the manifest, we store previously fetched etags.
- In the filesystem, we have a cache of already downloaded zip archives.
The first cache is basically invalidated every hour, while the second
cache is only invalidated when a etag changes. For pinned versions,
nothing is invalidated as they are considered immutable.
And so, even for unpinned package. In this case, we can't do a HEAD request. So we fallback by looking at what's available in the cache and using the most recently downloaded version from the cache. This is only a best effort as the most recently downloaded one may not be the actual latest. But common, this is a case where (a) someone didn't pin any version, (b) is trying to build on in an offline setup. We could possibly make that edge-case better but, let's see if anyone ever complains about it first.
When the version isn't a git sha or a tag, we always check that we got
the last version of a particular dependency before building. This is
to avoid those awkward moments where someone try to use something from
the stdlib that is brand new, and despite using 'main' they get a
strange build failure regarding how it's not available.
An important note is that we don't actually re-download the package
when the case occurs; we merely check an HTTP ETag from a (cheap) 'HEAD'
request on the package registry. If the tag hasn't changed then that
means the local version is correct.
The behavior is completely bypassed if the version is specified using
a git sha or a tag, as here, we can assume that fetching it once it
enough (and that it can change). If a package maintainer force-pushed
a tag however, there may be discrepency and the only way around that
is to `rm -r ./build`.
Best-effort to assert whether a version refers is a git sha digest or a tag. When it is, we
avoid re-downloading it if it's already fetched. But when it isn't, and thus refer to a branch,
we always re-download it. Note however that the download might be short-circuited by the
system-wide package cache, so a download doesn't actually mean a network request.
The package cache is however smart-enough to assert whether a package in the cache must be
re-downloaded (using HTTP ETag). So this is mostly about delegating the re-downloading logic to
the global packages cache.
fix: Opaque types are now properly handled in code gen (i.e. code gen functions, in datums/redeemers, in from data casts)
chore: add specific nested opaque type tests to code gen
microproofs