Narrow deps.edn research to git-only on jpm
Drop Maven. Verified the piggyback: jpm's download-bundle clones a git dep into its cache without building (the build half, bundle-install, needs a project.janet and is separable). Plan is git resolution via jpm/pm, a loader that searches the clones' src dirs, and compiling used namespaces into the image at build. Pure clj/cljc only.
This commit is contained in:
Yogthos
parent
5e20ff21ff
commit
47bcc743b4
1 changed files with 98 additions and 130 deletions
|
|
@ -1,154 +1,122 @@
|
||||||
# Loading Clojure libraries via deps.edn
|
# Loading Clojure libraries via deps.edn (git deps, on jpm)
|
||||||
|
|
||||||
Research notes on letting Jolt consume `deps.edn` so a project can pull real
|
Research notes for letting a Jolt project pull pure-Clojure libraries through a
|
||||||
Clojure libraries and `(require ...)` them. This documents what works today, what
|
`deps.edn` and `(require ...)` them. Scope is deliberately narrow:
|
||||||
it would take, and a recommended path. Nothing here is implemented yet.
|
|
||||||
|
|
||||||
## Goal
|
- **git deps only** — no Maven/`~/.m2` resolution.
|
||||||
|
- **pure `clj`/`cljc`** — anything needing the JVM (Java interop, host classes)
|
||||||
|
won't load or run, and that's expected.
|
||||||
|
- **no classpath machinery** — we just need `require` to find a dep's namespaces
|
||||||
|
at dev time (REPL) so they can be compiled into the image at build time.
|
||||||
|
- **piggyback on jpm** — reuse jpm's existing git fetch + cache; don't write a
|
||||||
|
package manager.
|
||||||
|
|
||||||
Given a `deps.edn` like
|
Nothing here is implemented yet. Everything below was verified against the
|
||||||
|
installed jpm and a real lib (medley).
|
||||||
|
|
||||||
```clojure
|
## How jpm handles dependencies today
|
||||||
{:paths ["src"]
|
|
||||||
:deps {medley/medley {:mvn/version "1.0.0"}
|
|
||||||
some/gitlib {:git/url "https://..." :git/sha "..."}}}
|
|
||||||
```
|
|
||||||
|
|
||||||
run `jolt` in that directory and have `(require '[medley.core :as m])` find and
|
jpm's package code lives in `/opt/homebrew/lib/janet/jpm/pm.janet`. The relevant
|
||||||
load the library's source from the resolved dependency, the same way the stdlib
|
pieces:
|
||||||
is loaded today.
|
|
||||||
|
|
||||||
## What works today
|
- **`resolve-bundle`** normalizes a dep spec to `{:url :tag :type :shallow}`. It
|
||||||
|
accepts a table (`:url`/`:repo`, and `:tag`/`:sha`/`:commit`/`:ref`) or a
|
||||||
|
`"url::type::tag"` string. So a deps.edn `{:git/url … :git/sha …}` maps onto it
|
||||||
|
directly.
|
||||||
|
- **`download-bundle url :git tag shallow`** clones into a cache dir and returns
|
||||||
|
the path. The cache is `(find-cache)` = `<modpath>/.cache`, and the per-dep
|
||||||
|
directory is a deterministic id: `git_<tag>_<sanitized-url>`. Under the hood
|
||||||
|
`download-git-bundle` does `git init` + `remote add origin` + fetch + reset to
|
||||||
|
the tag/sha (plus submodules). Pure git — **no build step**.
|
||||||
|
- **`bundle-install`** is the part we *don't* want: after downloading it does
|
||||||
|
`(require-jpm "./project.janet")` and runs build/install rules. A Clojure lib
|
||||||
|
has no `project.janet`, so this would fail. The clone/cache half
|
||||||
|
(`download-bundle`) is cleanly separable from this build half.
|
||||||
|
|
||||||
- **Jolt reads EDN.** `(read-string (slurp "deps.edn"))` parses a deps.edn into a
|
So jpm already gives us git resolution + a content-addressed cache for free; we
|
||||||
Jolt map — no extra parser needed.
|
just skip its build phase.
|
||||||
- **Library source ships in the jars.** Maven Clojure jars contain the `.clj` /
|
|
||||||
`.cljc` source at namespace-matching paths (e.g. `medley/core.cljc`,
|
|
||||||
`msgpack/core.clj`), not just compiled `.class` files. So we never need a JVM
|
|
||||||
to *run* the code — only to fetch/resolve it, and even that is optional (below).
|
|
||||||
- **Jolt can run real library source.** Loading `medley/core.cljc` straight from
|
|
||||||
its jar works: `(medley.core/abs -5)` → `5`, `(medley.core/find-first odd? …)`
|
|
||||||
→ `5`. Some functions hit features Jolt doesn't fully support yet — coverage is
|
|
||||||
per-function, not all-or-nothing.
|
|
||||||
- **The real resolver is on the box.** `clojure`, `clj`, and `mvn` are installed,
|
|
||||||
with `~/.m2` and `~/.gitlibs` populated. `clojure -Spath` already prints the
|
|
||||||
fully-resolved, transitive classpath (dirs + jars).
|
|
||||||
|
|
||||||
## What's missing
|
### Verified
|
||||||
|
|
||||||
The loader is single-rooted. `evaluator.janet/ns->path` hardcodes:
|
|
||||||
|
|
||||||
```janet
|
```janet
|
||||||
(string "src/jolt/" (dots->slashes (dashes->underscores ns)) ".clj")
|
(import jpm/config :as cfg)
|
||||||
|
(import jpm/pm :as pm)
|
||||||
|
(cfg/load-default) ; sets gitpath, cache defaults, etc.
|
||||||
|
(setdyn :modpath "<project>/jpm_tree") ; where the cache lives
|
||||||
|
(def s (pm/resolve-bundle {:url "https://github.com/weavejester/medley"
|
||||||
|
:tag "1.0.0" :shallow true}))
|
||||||
|
(pm/download-bundle (s :url) (s :type) (s :tag) (s :shallow))
|
||||||
|
;; => "<modpath>/.cache/git_1.0.0_https___github.com_weavejester_medley"
|
||||||
|
;; with source at <that>/src/medley/core.cljc
|
||||||
```
|
```
|
||||||
|
|
||||||
and `maybe-require-ns` loads exactly that one path if it exists. To load deps we
|
`cfg/load-default` is required — calling `download-bundle` without jpm's config
|
||||||
need:
|
dyns set fails in its `shell` helper.
|
||||||
|
|
||||||
1. **A classpath** — a list of source roots searched in order, not one fixed
|
And the source loads and runs in Jolt: `(medley.core/abs -5)` → `5`,
|
||||||
prefix. Roots = `:paths` from deps.edn + each resolved dependency's source.
|
`(medley.core/find-first odd? [2 4 5])` → `5` (coverage is per-function; a few
|
||||||
2. **`.cljc` support** — try `foo/bar.cljc` as well as `foo/bar.clj` (most libs
|
hit interpreter gaps, which is fine).
|
||||||
ship `.cljc` or `.clj`; the loader only tries `.clj` today).
|
|
||||||
3. **`ns`-form handling on load.** Stdlib files have no `ns` form, so
|
|
||||||
`maybe-require-ns` sets the current ns manually before loading. Library files
|
|
||||||
*do* have `(ns ...)`. Both already work in practice (the `ns` form re-asserts
|
|
||||||
the namespace), but the loader should not assume "no ns form."
|
|
||||||
|
|
||||||
## Resolving dependencies — three options
|
## The shape
|
||||||
|
|
||||||
The hard part is turning coordinates into local source roots. Maven resolution
|
1. **Resolve.** Read `deps.edn`, take the `:deps` whose specs are git
|
||||||
(transitive deps, version conflict resolution, POM parsing) is real work; git
|
(`:git/url` + `:git/sha`/`:git/tag`). For each, `resolve-bundle` +
|
||||||
deps are comparatively easy.
|
`download-bundle` into the project's `jpm_tree/.cache`. Read each cloned
|
||||||
|
dep's own `deps.edn` and recurse for transitive git deps. (`:local/root`
|
||||||
|
deps are even simpler — just a path, no fetch.)
|
||||||
|
2. **Collect source roots.** A dep's source dirs are its deps.edn `:paths`
|
||||||
|
(default `["src"]`), so a root is `<clone-dir>/<path>`. The result is just an
|
||||||
|
ordered list of directories — not a classpath abstraction, a list of roots.
|
||||||
|
3. **Teach the loader the roots.** `evaluator.janet/ns->path` currently hardcodes
|
||||||
|
`src/jolt/<ns>.clj`. Generalize `maybe-require-ns` to, after the stdlib path,
|
||||||
|
search each dep root for `<ns>.clj` then `<ns>.cljc`. `src/jolt/` stays first
|
||||||
|
so the stdlib always wins. (Two small changes: a root list in the ctx, and
|
||||||
|
trying `.cljc`.)
|
||||||
|
4. **Dev vs build.**
|
||||||
|
- *Dev:* on REPL/CLI start, if `deps.edn` is present, resolve once (cache keyed
|
||||||
|
on a hash of deps.edn so it's a no-op when unchanged) and register the roots.
|
||||||
|
`(require '[medley.core])` then just works.
|
||||||
|
- *Build:* the dep namespaces a project actually uses get compiled into the
|
||||||
|
image the same way the embedded `jolt.nrepl` source is today — load them at
|
||||||
|
build time so the shipped binary needs neither the deps nor jpm.
|
||||||
|
|
||||||
### A. Shell out to the Clojure CLI (recommended first cut)
|
## Why this fits "no classpath"
|
||||||
|
|
||||||
Run `clojure -Spath` (optionally `-Sdeps`/aliases) in the project dir, capture
|
We never construct a Java-style classpath or deal with jar extraction, ordering
|
||||||
the `:`-separated classpath, then:
|
semantics, or version conflict resolution. Resolution is a tree walk over git
|
||||||
|
deps; "the classpath" is just the list of `src` dirs of the clones. The loader
|
||||||
|
already does path-based namespace lookup — we widen it from one root to a few.
|
||||||
|
|
||||||
- directory entries → add directly as source roots;
|
## Integration with jpm
|
||||||
- jar entries → extract `*.clj` / `*.cljc` into a cache dir
|
|
||||||
(`.jolt/classpath/<sha>/`) with `unzip`/`jar` (both present; Janet has no
|
|
||||||
built-in zip) and add the cache dir as a root.
|
|
||||||
|
|
||||||
Pros: reuses the canonical resolver, so transitive deps, exclusions, aliases, and
|
jpm stays the build tool for the Jolt binary; this lives beside it and *calls
|
||||||
version conflict resolution are all correct and match what JVM Clojure sees. Tiny
|
into* `jpm/pm` for the git/cache work (import the module at dev/build time — jpm
|
||||||
amount of code.
|
is always present then; the shipped binary doesn't need it). Open question:
|
||||||
|
whether to depend on jpm's internal `pm` functions (not a stable public API) or
|
||||||
|
shell `git` ourselves into the same cache layout. Reusing `pm` is less code and
|
||||||
|
shares jpm's cache; shelling git is ~15 lines and avoids the internal-API risk.
|
||||||
|
Leaning toward reusing `pm` first.
|
||||||
|
|
||||||
Cons: requires the Clojure CLI (hence a JVM) *at resolve time*. Runtime stays
|
## Limitations
|
||||||
JVM-free. We'd cache the result so resolution only reruns when `deps.edn`
|
|
||||||
changes.
|
|
||||||
|
|
||||||
### B. Jolt-native resolver
|
- Pure `clj`/`cljc` only. JVM interop, host classes, and unimplemented
|
||||||
|
`clojure.core` corners fail — expected, not a goal.
|
||||||
|
- Per-function coverage: a namespace can load with most functions working and a
|
||||||
|
few not.
|
||||||
|
- Source only; compiled `.class` files (if any in a git dep) are ignored.
|
||||||
|
|
||||||
Parse `deps.edn` ourselves and resolve:
|
## Plan
|
||||||
|
|
||||||
- `:git/url` + `:git/sha` → `git clone`/checkout into a cache (this is roughly
|
1. **Loader roots.** Generalize `maybe-require-ns` to search an ordered root list
|
||||||
what jpm already does for Janet git deps — see "jpm" below);
|
and try `.clj` + `.cljc`; add a `JOLT_PATH`/`--path` to set roots by hand.
|
||||||
- `:mvn/version` → download the POM + jar from Maven Central over HTTP, parse the
|
Point it at a checked-out lib's `src` and load it. Independently testable,
|
||||||
POM for transitive deps, resolve versions.
|
unblocks the rest.
|
||||||
|
2. **Resolve git deps via jpm.** Read `deps.edn`, resolve `:git/*` (+
|
||||||
Pros: no JVM dependency at all; self-contained.
|
`:local/root`) through `jpm/pm` into `jpm_tree/.cache`, recurse for transitive
|
||||||
|
git deps, register the roots. `jolt deps` to resolve/print; auto on startup
|
||||||
Cons: reimplementing Maven resolution (POM transitive graph, `:exclusions`,
|
when `deps.edn` exists.
|
||||||
nearest-wins version selection) is the bulk of tools.deps and easy to get subtly
|
3. **Build-time compile-in.** Fold the used dep namespaces into the image at
|
||||||
wrong. Large effort.
|
build (as with embedded `jolt.nrepl`).
|
||||||
|
4. **Conformance.** Pull a few popular pure-`cljc` git libs, see what loads/runs,
|
||||||
### C. Hybrid
|
and drive interpreter gaps from the failures — same loop as the
|
||||||
|
|
||||||
Native path for `:paths` and `:git/*` deps (cheap, no JVM); shell to `clojure
|
|
||||||
-Spath` only when `:mvn/*` deps are present. Gives a JVM-free experience for
|
|
||||||
git-only / local projects and correct Maven resolution when needed.
|
|
||||||
|
|
||||||
## Where jpm fits
|
|
||||||
|
|
||||||
jpm builds the Jolt binary and manages *Janet* packages; it has no Maven/Clojure
|
|
||||||
notion, so deps.edn support sits beside jpm rather than inside it. Two useful
|
|
||||||
touch points:
|
|
||||||
|
|
||||||
- jpm already fetches and caches **git** repositories for Janet deps — the same
|
|
||||||
machinery (or `~/.gitlibs`) can back option B/C's git-dep handling, so we don't
|
|
||||||
write a git cache from scratch.
|
|
||||||
- A project-level `jpm` rule (in `project.janet`) could run resolution as a build
|
|
||||||
step and write a classpath file, for projects that want deps resolved at build
|
|
||||||
time rather than first run.
|
|
||||||
|
|
||||||
But the primary integration is at the Jolt runtime/CLI, not jpm: see below.
|
|
||||||
|
|
||||||
## Proposed shape
|
|
||||||
|
|
||||||
- **Classpath in the context.** Add a `:classpath` (ordered list of roots) to the
|
|
||||||
ctx env. `ns->path` becomes "search each root for `foo/bar.clj` then
|
|
||||||
`foo/bar.cljc`", with `src/jolt/` always first so the stdlib wins.
|
|
||||||
- **Resolution step.** On startup (or via `jolt deps`), if `deps.edn` exists,
|
|
||||||
resolve it to roots (option A to start) and set `:classpath`. Cache keyed on a
|
|
||||||
hash of `deps.edn` so it's a no-op when unchanged.
|
|
||||||
- **Config knobs.** `JOLT_CLASSPATH` env / `--classpath` flag to set roots
|
|
||||||
directly (bypassing resolution), mirroring how `JOLT_MUTABLE` works.
|
|
||||||
|
|
||||||
## Limitations (set expectations)
|
|
||||||
|
|
||||||
- **JVM-only libraries don't run.** Anything depending on Java interop, host
|
|
||||||
classes, or `clojure.core` features Jolt lacks will fail to load or fail at a
|
|
||||||
call. Target audience is pure-`clj`/`cljc` libraries.
|
|
||||||
- **Coverage is per-function.** As the medley probe showed, a namespace can load
|
|
||||||
and have most functions work while a few hit unimplemented core behavior.
|
|
||||||
- **No AOT/`.class` execution** — ever. We only consume source from the
|
|
||||||
classpath; compiled classes in jars are ignored.
|
|
||||||
- Macro/protocol/reader-conditional support is whatever the Jolt interpreter
|
|
||||||
already provides (reader conditionals `#?` are supported, which is why `.cljc`
|
|
||||||
loads).
|
|
||||||
|
|
||||||
## Recommended plan (phased)
|
|
||||||
|
|
||||||
1. **Loader classpath.** Generalize `ns->path`/`maybe-require-ns` to search an
|
|
||||||
ordered root list and try `.clj` + `.cljc`. Add `JOLT_CLASSPATH`/`--classpath`.
|
|
||||||
No resolution yet — point it at a directory of source by hand and load a lib.
|
|
||||||
(Unblocks everything; independently testable.)
|
|
||||||
2. **deps.edn → classpath via `clojure -Spath` (option A).** Resolve, extract
|
|
||||||
jar source to a cache, set the classpath. `jolt deps` to resolve/print;
|
|
||||||
auto-resolve on startup when `deps.edn` is present.
|
|
||||||
3. **Native git deps (toward option C).** Resolve `:git/*` (and `:local/root`)
|
|
||||||
without the JVM, falling back to the CLI only for `:mvn/*`.
|
|
||||||
4. **Conformance pass.** Pull a handful of popular pure-`cljc` libs, see what
|
|
||||||
loads/runs, and use the failures to drive interpreter gaps — same loop as the
|
|
||||||
clojure-test-suite battery.
|
clojure-test-suite battery.
|
||||||
|
|
|
||||||
Loading…
Add table
Add a link
Reference in a new issue