From 5e20ff21ff0c461367f9891eda5b05ac4092b65a Mon Sep 17 00:00:00 2001 From: Yogthos Date: Fri, 5 Jun 2026 22:28:22 -0400 Subject: [PATCH 1/6] Research notes: loading Clojure libs via deps.edn Findings and a phased plan for consuming deps.edn so projects can require real Clojure libraries. Jolt already reads EDN, jars ship .clj/.cljc source, and a real lib (medley) loads and runs. The gaps are a single-rooted loader (needs a classpath + .cljc) and dependency resolution. Recommends reusing 'clojure -Spath' to start, then native git-dep resolution; jpm stays the Janet build tool alongside this. See doc/tools-deps.md. --- doc/tools-deps.md | 154 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 154 insertions(+) create mode 100644 doc/tools-deps.md diff --git a/doc/tools-deps.md b/doc/tools-deps.md new file mode 100644 index 0000000..08a26b0 --- /dev/null +++ b/doc/tools-deps.md @@ -0,0 +1,154 @@ +# Loading Clojure libraries via deps.edn + +Research notes on letting Jolt consume `deps.edn` so a project can pull real +Clojure libraries and `(require ...)` them. This documents what works today, what +it would take, and a recommended path. Nothing here is implemented yet. + +## Goal + +Given a `deps.edn` like + +```clojure +{:paths ["src"] + :deps {medley/medley {:mvn/version "1.0.0"} + some/gitlib {:git/url "https://..." :git/sha "..."}}} +``` + +run `jolt` in that directory and have `(require '[medley.core :as m])` find and +load the library's source from the resolved dependency, the same way the stdlib +is loaded today. + +## What works today + +- **Jolt reads EDN.** `(read-string (slurp "deps.edn"))` parses a deps.edn into a + Jolt map — no extra parser needed. +- **Library source ships in the jars.** Maven Clojure jars contain the `.clj` / + `.cljc` source at namespace-matching paths (e.g. `medley/core.cljc`, + `msgpack/core.clj`), not just compiled `.class` files. So we never need a JVM + to *run* the code — only to fetch/resolve it, and even that is optional (below). +- **Jolt can run real library source.** Loading `medley/core.cljc` straight from + its jar works: `(medley.core/abs -5)` → `5`, `(medley.core/find-first odd? …)` + → `5`. Some functions hit features Jolt doesn't fully support yet — coverage is + per-function, not all-or-nothing. +- **The real resolver is on the box.** `clojure`, `clj`, and `mvn` are installed, + with `~/.m2` and `~/.gitlibs` populated. `clojure -Spath` already prints the + fully-resolved, transitive classpath (dirs + jars). + +## What's missing + +The loader is single-rooted. `evaluator.janet/ns->path` hardcodes: + +```janet +(string "src/jolt/" (dots->slashes (dashes->underscores ns)) ".clj") +``` + +and `maybe-require-ns` loads exactly that one path if it exists. To load deps we +need: + +1. **A classpath** — a list of source roots searched in order, not one fixed + prefix. Roots = `:paths` from deps.edn + each resolved dependency's source. +2. **`.cljc` support** — try `foo/bar.cljc` as well as `foo/bar.clj` (most libs + ship `.cljc` or `.clj`; the loader only tries `.clj` today). +3. **`ns`-form handling on load.** Stdlib files have no `ns` form, so + `maybe-require-ns` sets the current ns manually before loading. Library files + *do* have `(ns ...)`. Both already work in practice (the `ns` form re-asserts + the namespace), but the loader should not assume "no ns form." + +## Resolving dependencies — three options + +The hard part is turning coordinates into local source roots. Maven resolution +(transitive deps, version conflict resolution, POM parsing) is real work; git +deps are comparatively easy. + +### A. Shell out to the Clojure CLI (recommended first cut) + +Run `clojure -Spath` (optionally `-Sdeps`/aliases) in the project dir, capture +the `:`-separated classpath, then: + +- directory entries → add directly as source roots; +- jar entries → extract `*.clj` / `*.cljc` into a cache dir + (`.jolt/classpath//`) with `unzip`/`jar` (both present; Janet has no + built-in zip) and add the cache dir as a root. + +Pros: reuses the canonical resolver, so transitive deps, exclusions, aliases, and +version conflict resolution are all correct and match what JVM Clojure sees. Tiny +amount of code. + +Cons: requires the Clojure CLI (hence a JVM) *at resolve time*. Runtime stays +JVM-free. We'd cache the result so resolution only reruns when `deps.edn` +changes. + +### B. Jolt-native resolver + +Parse `deps.edn` ourselves and resolve: + +- `:git/url` + `:git/sha` → `git clone`/checkout into a cache (this is roughly + what jpm already does for Janet git deps — see "jpm" below); +- `:mvn/version` → download the POM + jar from Maven Central over HTTP, parse the + POM for transitive deps, resolve versions. + +Pros: no JVM dependency at all; self-contained. + +Cons: reimplementing Maven resolution (POM transitive graph, `:exclusions`, +nearest-wins version selection) is the bulk of tools.deps and easy to get subtly +wrong. Large effort. + +### C. Hybrid + +Native path for `:paths` and `:git/*` deps (cheap, no JVM); shell to `clojure +-Spath` only when `:mvn/*` deps are present. Gives a JVM-free experience for +git-only / local projects and correct Maven resolution when needed. + +## Where jpm fits + +jpm builds the Jolt binary and manages *Janet* packages; it has no Maven/Clojure +notion, so deps.edn support sits beside jpm rather than inside it. Two useful +touch points: + +- jpm already fetches and caches **git** repositories for Janet deps — the same + machinery (or `~/.gitlibs`) can back option B/C's git-dep handling, so we don't + write a git cache from scratch. +- A project-level `jpm` rule (in `project.janet`) could run resolution as a build + step and write a classpath file, for projects that want deps resolved at build + time rather than first run. + +But the primary integration is at the Jolt runtime/CLI, not jpm: see below. + +## Proposed shape + +- **Classpath in the context.** Add a `:classpath` (ordered list of roots) to the + ctx env. `ns->path` becomes "search each root for `foo/bar.clj` then + `foo/bar.cljc`", with `src/jolt/` always first so the stdlib wins. +- **Resolution step.** On startup (or via `jolt deps`), if `deps.edn` exists, + resolve it to roots (option A to start) and set `:classpath`. Cache keyed on a + hash of `deps.edn` so it's a no-op when unchanged. +- **Config knobs.** `JOLT_CLASSPATH` env / `--classpath` flag to set roots + directly (bypassing resolution), mirroring how `JOLT_MUTABLE` works. + +## Limitations (set expectations) + +- **JVM-only libraries don't run.** Anything depending on Java interop, host + classes, or `clojure.core` features Jolt lacks will fail to load or fail at a + call. Target audience is pure-`clj`/`cljc` libraries. +- **Coverage is per-function.** As the medley probe showed, a namespace can load + and have most functions work while a few hit unimplemented core behavior. +- **No AOT/`.class` execution** — ever. We only consume source from the + classpath; compiled classes in jars are ignored. +- Macro/protocol/reader-conditional support is whatever the Jolt interpreter + already provides (reader conditionals `#?` are supported, which is why `.cljc` + loads). + +## Recommended plan (phased) + +1. **Loader classpath.** Generalize `ns->path`/`maybe-require-ns` to search an + ordered root list and try `.clj` + `.cljc`. Add `JOLT_CLASSPATH`/`--classpath`. + No resolution yet — point it at a directory of source by hand and load a lib. + (Unblocks everything; independently testable.) +2. **deps.edn → classpath via `clojure -Spath` (option A).** Resolve, extract + jar source to a cache, set the classpath. `jolt deps` to resolve/print; + auto-resolve on startup when `deps.edn` is present. +3. **Native git deps (toward option C).** Resolve `:git/*` (and `:local/root`) + without the JVM, falling back to the CLI only for `:mvn/*`. +4. **Conformance pass.** Pull a handful of popular pure-`cljc` libs, see what + loads/runs, and use the failures to drive interpreter gaps — same loop as the + clojure-test-suite battery. From 47bcc743b46f48ffe551af5d6da465cacaafd0cd Mon Sep 17 00:00:00 2001 From: Yogthos Date: Fri, 5 Jun 2026 22:42:08 -0400 Subject: [PATCH 2/6] Narrow deps.edn research to git-only on jpm Drop Maven. Verified the piggyback: jpm's download-bundle clones a git dep into its cache without building (the build half, bundle-install, needs a project.janet and is separable). Plan is git resolution via jpm/pm, a loader that searches the clones' src dirs, and compiling used namespaces into the image at build. Pure clj/cljc only. --- doc/tools-deps.md | 228 ++++++++++++++++++++-------------------------- 1 file changed, 98 insertions(+), 130 deletions(-) diff --git a/doc/tools-deps.md b/doc/tools-deps.md index 08a26b0..2b05aba 100644 --- a/doc/tools-deps.md +++ b/doc/tools-deps.md @@ -1,154 +1,122 @@ -# Loading Clojure libraries via deps.edn +# Loading Clojure libraries via deps.edn (git deps, on jpm) -Research notes on letting Jolt consume `deps.edn` so a project can pull real -Clojure libraries and `(require ...)` them. This documents what works today, what -it would take, and a recommended path. Nothing here is implemented yet. +Research notes for letting a Jolt project pull pure-Clojure libraries through a +`deps.edn` and `(require ...)` them. Scope is deliberately narrow: -## Goal +- **git deps only** — no Maven/`~/.m2` resolution. +- **pure `clj`/`cljc`** — anything needing the JVM (Java interop, host classes) + won't load or run, and that's expected. +- **no classpath machinery** — we just need `require` to find a dep's namespaces + at dev time (REPL) so they can be compiled into the image at build time. +- **piggyback on jpm** — reuse jpm's existing git fetch + cache; don't write a + package manager. -Given a `deps.edn` like +Nothing here is implemented yet. Everything below was verified against the +installed jpm and a real lib (medley). -```clojure -{:paths ["src"] - :deps {medley/medley {:mvn/version "1.0.0"} - some/gitlib {:git/url "https://..." :git/sha "..."}}} -``` +## How jpm handles dependencies today -run `jolt` in that directory and have `(require '[medley.core :as m])` find and -load the library's source from the resolved dependency, the same way the stdlib -is loaded today. +jpm's package code lives in `/opt/homebrew/lib/janet/jpm/pm.janet`. The relevant +pieces: -## What works today +- **`resolve-bundle`** normalizes a dep spec to `{:url :tag :type :shallow}`. It + accepts a table (`:url`/`:repo`, and `:tag`/`:sha`/`:commit`/`:ref`) or a + `"url::type::tag"` string. So a deps.edn `{:git/url … :git/sha …}` maps onto it + directly. +- **`download-bundle url :git tag shallow`** clones into a cache dir and returns + the path. The cache is `(find-cache)` = `/.cache`, and the per-dep + directory is a deterministic id: `git__`. Under the hood + `download-git-bundle` does `git init` + `remote add origin` + fetch + reset to + the tag/sha (plus submodules). Pure git — **no build step**. +- **`bundle-install`** is the part we *don't* want: after downloading it does + `(require-jpm "./project.janet")` and runs build/install rules. A Clojure lib + has no `project.janet`, so this would fail. The clone/cache half + (`download-bundle`) is cleanly separable from this build half. -- **Jolt reads EDN.** `(read-string (slurp "deps.edn"))` parses a deps.edn into a - Jolt map — no extra parser needed. -- **Library source ships in the jars.** Maven Clojure jars contain the `.clj` / - `.cljc` source at namespace-matching paths (e.g. `medley/core.cljc`, - `msgpack/core.clj`), not just compiled `.class` files. So we never need a JVM - to *run* the code — only to fetch/resolve it, and even that is optional (below). -- **Jolt can run real library source.** Loading `medley/core.cljc` straight from - its jar works: `(medley.core/abs -5)` → `5`, `(medley.core/find-first odd? …)` - → `5`. Some functions hit features Jolt doesn't fully support yet — coverage is - per-function, not all-or-nothing. -- **The real resolver is on the box.** `clojure`, `clj`, and `mvn` are installed, - with `~/.m2` and `~/.gitlibs` populated. `clojure -Spath` already prints the - fully-resolved, transitive classpath (dirs + jars). +So jpm already gives us git resolution + a content-addressed cache for free; we +just skip its build phase. -## What's missing - -The loader is single-rooted. `evaluator.janet/ns->path` hardcodes: +### Verified ```janet -(string "src/jolt/" (dots->slashes (dashes->underscores ns)) ".clj") +(import jpm/config :as cfg) +(import jpm/pm :as pm) +(cfg/load-default) ; sets gitpath, cache defaults, etc. +(setdyn :modpath "/jpm_tree") ; where the cache lives +(def s (pm/resolve-bundle {:url "https://github.com/weavejester/medley" + :tag "1.0.0" :shallow true})) +(pm/download-bundle (s :url) (s :type) (s :tag) (s :shallow)) +;; => "/.cache/git_1.0.0_https___github.com_weavejester_medley" +;; with source at /src/medley/core.cljc ``` -and `maybe-require-ns` loads exactly that one path if it exists. To load deps we -need: +`cfg/load-default` is required — calling `download-bundle` without jpm's config +dyns set fails in its `shell` helper. -1. **A classpath** — a list of source roots searched in order, not one fixed - prefix. Roots = `:paths` from deps.edn + each resolved dependency's source. -2. **`.cljc` support** — try `foo/bar.cljc` as well as `foo/bar.clj` (most libs - ship `.cljc` or `.clj`; the loader only tries `.clj` today). -3. **`ns`-form handling on load.** Stdlib files have no `ns` form, so - `maybe-require-ns` sets the current ns manually before loading. Library files - *do* have `(ns ...)`. Both already work in practice (the `ns` form re-asserts - the namespace), but the loader should not assume "no ns form." +And the source loads and runs in Jolt: `(medley.core/abs -5)` → `5`, +`(medley.core/find-first odd? [2 4 5])` → `5` (coverage is per-function; a few +hit interpreter gaps, which is fine). -## Resolving dependencies — three options +## The shape -The hard part is turning coordinates into local source roots. Maven resolution -(transitive deps, version conflict resolution, POM parsing) is real work; git -deps are comparatively easy. +1. **Resolve.** Read `deps.edn`, take the `:deps` whose specs are git + (`:git/url` + `:git/sha`/`:git/tag`). For each, `resolve-bundle` + + `download-bundle` into the project's `jpm_tree/.cache`. Read each cloned + dep's own `deps.edn` and recurse for transitive git deps. (`:local/root` + deps are even simpler — just a path, no fetch.) +2. **Collect source roots.** A dep's source dirs are its deps.edn `:paths` + (default `["src"]`), so a root is `/`. The result is just an + ordered list of directories — not a classpath abstraction, a list of roots. +3. **Teach the loader the roots.** `evaluator.janet/ns->path` currently hardcodes + `src/jolt/.clj`. Generalize `maybe-require-ns` to, after the stdlib path, + search each dep root for `.clj` then `.cljc`. `src/jolt/` stays first + so the stdlib always wins. (Two small changes: a root list in the ctx, and + trying `.cljc`.) +4. **Dev vs build.** + - *Dev:* on REPL/CLI start, if `deps.edn` is present, resolve once (cache keyed + on a hash of deps.edn so it's a no-op when unchanged) and register the roots. + `(require '[medley.core])` then just works. + - *Build:* the dep namespaces a project actually uses get compiled into the + image the same way the embedded `jolt.nrepl` source is today — load them at + build time so the shipped binary needs neither the deps nor jpm. -### A. Shell out to the Clojure CLI (recommended first cut) +## Why this fits "no classpath" -Run `clojure -Spath` (optionally `-Sdeps`/aliases) in the project dir, capture -the `:`-separated classpath, then: +We never construct a Java-style classpath or deal with jar extraction, ordering +semantics, or version conflict resolution. Resolution is a tree walk over git +deps; "the classpath" is just the list of `src` dirs of the clones. The loader +already does path-based namespace lookup — we widen it from one root to a few. -- directory entries → add directly as source roots; -- jar entries → extract `*.clj` / `*.cljc` into a cache dir - (`.jolt/classpath//`) with `unzip`/`jar` (both present; Janet has no - built-in zip) and add the cache dir as a root. +## Integration with jpm -Pros: reuses the canonical resolver, so transitive deps, exclusions, aliases, and -version conflict resolution are all correct and match what JVM Clojure sees. Tiny -amount of code. +jpm stays the build tool for the Jolt binary; this lives beside it and *calls +into* `jpm/pm` for the git/cache work (import the module at dev/build time — jpm +is always present then; the shipped binary doesn't need it). Open question: +whether to depend on jpm's internal `pm` functions (not a stable public API) or +shell `git` ourselves into the same cache layout. Reusing `pm` is less code and +shares jpm's cache; shelling git is ~15 lines and avoids the internal-API risk. +Leaning toward reusing `pm` first. -Cons: requires the Clojure CLI (hence a JVM) *at resolve time*. Runtime stays -JVM-free. We'd cache the result so resolution only reruns when `deps.edn` -changes. +## Limitations -### B. Jolt-native resolver +- Pure `clj`/`cljc` only. JVM interop, host classes, and unimplemented + `clojure.core` corners fail — expected, not a goal. +- Per-function coverage: a namespace can load with most functions working and a + few not. +- Source only; compiled `.class` files (if any in a git dep) are ignored. -Parse `deps.edn` ourselves and resolve: +## Plan -- `:git/url` + `:git/sha` → `git clone`/checkout into a cache (this is roughly - what jpm already does for Janet git deps — see "jpm" below); -- `:mvn/version` → download the POM + jar from Maven Central over HTTP, parse the - POM for transitive deps, resolve versions. - -Pros: no JVM dependency at all; self-contained. - -Cons: reimplementing Maven resolution (POM transitive graph, `:exclusions`, -nearest-wins version selection) is the bulk of tools.deps and easy to get subtly -wrong. Large effort. - -### C. Hybrid - -Native path for `:paths` and `:git/*` deps (cheap, no JVM); shell to `clojure --Spath` only when `:mvn/*` deps are present. Gives a JVM-free experience for -git-only / local projects and correct Maven resolution when needed. - -## Where jpm fits - -jpm builds the Jolt binary and manages *Janet* packages; it has no Maven/Clojure -notion, so deps.edn support sits beside jpm rather than inside it. Two useful -touch points: - -- jpm already fetches and caches **git** repositories for Janet deps — the same - machinery (or `~/.gitlibs`) can back option B/C's git-dep handling, so we don't - write a git cache from scratch. -- A project-level `jpm` rule (in `project.janet`) could run resolution as a build - step and write a classpath file, for projects that want deps resolved at build - time rather than first run. - -But the primary integration is at the Jolt runtime/CLI, not jpm: see below. - -## Proposed shape - -- **Classpath in the context.** Add a `:classpath` (ordered list of roots) to the - ctx env. `ns->path` becomes "search each root for `foo/bar.clj` then - `foo/bar.cljc`", with `src/jolt/` always first so the stdlib wins. -- **Resolution step.** On startup (or via `jolt deps`), if `deps.edn` exists, - resolve it to roots (option A to start) and set `:classpath`. Cache keyed on a - hash of `deps.edn` so it's a no-op when unchanged. -- **Config knobs.** `JOLT_CLASSPATH` env / `--classpath` flag to set roots - directly (bypassing resolution), mirroring how `JOLT_MUTABLE` works. - -## Limitations (set expectations) - -- **JVM-only libraries don't run.** Anything depending on Java interop, host - classes, or `clojure.core` features Jolt lacks will fail to load or fail at a - call. Target audience is pure-`clj`/`cljc` libraries. -- **Coverage is per-function.** As the medley probe showed, a namespace can load - and have most functions work while a few hit unimplemented core behavior. -- **No AOT/`.class` execution** — ever. We only consume source from the - classpath; compiled classes in jars are ignored. -- Macro/protocol/reader-conditional support is whatever the Jolt interpreter - already provides (reader conditionals `#?` are supported, which is why `.cljc` - loads). - -## Recommended plan (phased) - -1. **Loader classpath.** Generalize `ns->path`/`maybe-require-ns` to search an - ordered root list and try `.clj` + `.cljc`. Add `JOLT_CLASSPATH`/`--classpath`. - No resolution yet — point it at a directory of source by hand and load a lib. - (Unblocks everything; independently testable.) -2. **deps.edn → classpath via `clojure -Spath` (option A).** Resolve, extract - jar source to a cache, set the classpath. `jolt deps` to resolve/print; - auto-resolve on startup when `deps.edn` is present. -3. **Native git deps (toward option C).** Resolve `:git/*` (and `:local/root`) - without the JVM, falling back to the CLI only for `:mvn/*`. -4. **Conformance pass.** Pull a handful of popular pure-`cljc` libs, see what - loads/runs, and use the failures to drive interpreter gaps — same loop as the +1. **Loader roots.** Generalize `maybe-require-ns` to search an ordered root list + and try `.clj` + `.cljc`; add a `JOLT_PATH`/`--path` to set roots by hand. + Point it at a checked-out lib's `src` and load it. Independently testable, + unblocks the rest. +2. **Resolve git deps via jpm.** Read `deps.edn`, resolve `:git/*` (+ + `:local/root`) through `jpm/pm` into `jpm_tree/.cache`, recurse for transitive + git deps, register the roots. `jolt deps` to resolve/print; auto on startup + when `deps.edn` exists. +3. **Build-time compile-in.** Fold the used dep namespaces into the image at + build (as with embedded `jolt.nrepl`). +4. **Conformance.** Pull a few popular pure-`cljc` git libs, see what loads/runs, + and drive interpreter gaps from the failures — same loop as the clojure-test-suite battery. From 6ea43fb6237536ac5ab874eeebb9c0d3a6eb5eae Mon Sep 17 00:00:00 2001 From: Yogthos Date: Fri, 5 Jun 2026 22:53:20 -0400 Subject: [PATCH 3/6] deps phase 1: loader searches multiple source roots The loader resolved a namespace against one hardcoded root (src/jolt, .clj only). Now it searches an ordered list (ctx env :source-paths, stdlib first) and tries .clj then .cljc. init adds roots from the :paths opt and JOLT_PATH. This is the foundation for loading deps.edn libraries; verified loading medley from an added root. No change to stdlib loading (3913 suite baseline held). --- doc/tools-deps.md | 18 +++++----- src/jolt/api.janet | 9 ++++- src/jolt/evaluator.janet | 35 ++++++++++++------ src/jolt/types.janet | 3 ++ test/integration/deps-loader-test.janet | 48 +++++++++++++++++++++++++ 5 files changed, 92 insertions(+), 21 deletions(-) create mode 100644 test/integration/deps-loader-test.janet diff --git a/doc/tools-deps.md b/doc/tools-deps.md index 2b05aba..8f41ea7 100644 --- a/doc/tools-deps.md +++ b/doc/tools-deps.md @@ -91,11 +91,10 @@ already does path-based namespace lookup — we widen it from one root to a few. jpm stays the build tool for the Jolt binary; this lives beside it and *calls into* `jpm/pm` for the git/cache work (import the module at dev/build time — jpm -is always present then; the shipped binary doesn't need it). Open question: -whether to depend on jpm's internal `pm` functions (not a stable public API) or -shell `git` ourselves into the same cache layout. Reusing `pm` is less code and -shares jpm's cache; shelling git is ~15 lines and avoids the internal-API risk. -Leaning toward reusing `pm` first. +is always present then; the shipped binary doesn't need it). **Decision: reuse +`jpm/pm`** (`resolve-bundle` + `download-bundle`, after `jpm/config/load-default`) +rather than shelling `git` ourselves — less code, and it shares jpm's cache. The +internal-API risk is acceptable since it's only used at dev/build time. ## Limitations @@ -107,10 +106,11 @@ Leaning toward reusing `pm` first. ## Plan -1. **Loader roots.** Generalize `maybe-require-ns` to search an ordered root list - and try `.clj` + `.cljc`; add a `JOLT_PATH`/`--path` to set roots by hand. - Point it at a checked-out lib's `src` and load it. Independently testable, - unblocks the rest. +1. **Loader roots.** *(done)* `maybe-require-ns` now searches an ordered root + list (`ctx.env :source-paths`, stdlib first) trying `.clj` then `.cljc`; + `init` adds roots from the `:paths` opt and `JOLT_PATH` (colon-separated). + Verified loading a real lib (medley) from an added root. See + `test/integration/deps-loader-test.janet`. 2. **Resolve git deps via jpm.** Read `deps.edn`, resolve `:git/*` (+ `:local/root`) through `jpm/pm` into `jpm_tree/.cache`, recurse for transitive git deps, register the roots. `jolt deps` to resolve/print; auto on startup diff --git a/src/jolt/api.janet b/src/jolt/api.janet index 4e59165..215ec6b 100644 --- a/src/jolt/api.janet +++ b/src/jolt/api.janet @@ -30,10 +30,17 @@ opts may contain: :namespaces — map of {ns-name → {sym → value, ...}, ...} :mutable? — use Janet mutable data structures instead of persistent - :compile? — enable compilation of Clojure forms to Janet" + :compile? — enable compilation of Clojure forms to Janet + :paths — extra source roots to search for namespaces (after the stdlib)" [&opt opts] (default opts {}) (let [ctx (make-ctx opts)] + # Extra source roots: opts :paths, then JOLT_PATH (colon-separated). These are + # searched after the stdlib so (require ...) finds deps.edn-resolved libs. + (let [roots (get (ctx :env) :source-paths)] + (each p (get opts :paths []) (array/push roots p)) + (when-let [jp (os/getenv "JOLT_PATH")] + (each p (string/split ":" jp) (when (> (length p) 0) (array/push roots p))))) # Collection representation (persistent vs mutable) is selected at BUILD time # via JOLT_MUTABLE (see config.janet); init-core! registers vec/vector/conj/ # etc. that produce the mode-appropriate values, so nothing extra to load. diff --git a/src/jolt/evaluator.janet b/src/jolt/evaluator.janet index 7213f7a..c3b5a5b 100644 --- a/src/jolt/evaluator.janet +++ b/src/jolt/evaluator.janet @@ -185,12 +185,24 @@ [sym-s] (if (sym-s :ns) (string (sym-s :ns) "/" (sym-s :name)) (sym-s :name))) -(defn- ns->path - "Map a namespace name to its Jolt stdlib source path (dots->dirs, dashes->_)." +(defn- ns->relpath + "Namespace name to its file-relative path (dots->dirs, dashes->_), no extension." [ns-name] - (string "src/jolt/" - (string/replace-all "." "/" (string/replace-all "-" "_" ns-name)) - ".clj")) + (string/replace-all "." "/" (string/replace-all "-" "_" ns-name))) + +(defn- find-ns-file + "Search the context's source roots (stdlib first, then deps.edn dirs) for the + namespace's source, trying .clj then .cljc. Returns the path or nil." + [ctx ns-name] + (let [rel (ns->relpath ns-name) + roots (or (get (ctx :env) :source-paths) @["src/jolt"])] + (var found nil) + (each root roots + (each ext [".clj" ".cljc"] + (when (nil? found) + (let [p (string root "/" rel ext)] + (when (os/stat p) (set found p)))))) + found)) (defn- load-ns-file "Parse and evaluate every form in a .clj file in the given context." @@ -202,17 +214,18 @@ (when (not (nil? f)) (eval-form ctx @{} f)))) (defn- maybe-require-ns - "If namespace ns-name isn't populated yet and a stdlib source file exists, - load it. Restores the current namespace afterwards (the file's `ns` form + "If namespace ns-name isn't populated yet and source for it exists on the + context's source roots, load it. Restores the current namespace afterwards (a + library's own `ns` form, or our manual switch for ns-form-less stdlib files, changes it). No-op for already-loaded namespaces." [ctx ns-name] (let [ns (ctx-find-ns ctx ns-name)] (when (and (= 0 (length (ns :mappings))) (not= ns-name "clojure.core")) - (let [path (ns->path ns-name)] - (when (os/stat path) + (let [path (find-ns-file ctx ns-name)] + (when path (let [saved (ctx-current-ns ctx)] - # Jolt stdlib files have no `ns` form; switch into the target ns so - # their defs intern there, then restore. + # Stdlib files have no `ns` form, so switch into the target ns first + # (their defs intern there); a library's own `ns` form overrides this. (ctx-set-current-ns ctx ns-name) (load-ns-file ctx path) (ctx-set-current-ns ctx saved))))))) diff --git a/src/jolt/types.janet b/src/jolt/types.janet index 96cb1d9..41ab0fe 100644 --- a/src/jolt/types.janet +++ b/src/jolt/types.janet @@ -366,6 +366,9 @@ :class->opts @{} :current-ns "user" :compile? compile? + # Ordered roots searched (after the stdlib) to resolve a namespace + # to a .clj/.cljc file. deps.edn resolution appends dep src dirs. + :source-paths @["src/jolt"] :compiled-cache @{} :type-registry @{} :data-readers (let [dr @{}] diff --git a/test/integration/deps-loader-test.janet b/test/integration/deps-loader-test.janet new file mode 100644 index 0000000..521a36a --- /dev/null +++ b/test/integration/deps-loader-test.janet @@ -0,0 +1,48 @@ +# The loader resolves namespaces against an ordered list of source roots (the +# stdlib first, then deps.edn-resolved dirs), trying .clj then .cljc. This is the +# foundation for loading Clojure libraries via deps.edn — here we point a root at +# a hand-written "library" and require it. + +(use ../../src/jolt/api) +(use ../../src/jolt/types) + +(def tmp (string (os/getenv "TMPDIR") "jolt-deps-test-" (os/time))) +(os/mkdir tmp) +(os/mkdir (string tmp "/mylib")) +(os/mkdir (string tmp "/other")) + +# a .cljc library with its own ns form and a require of a sibling ns +(spit (string tmp "/mylib/core.cljc") + "(ns mylib.core (:require [other.util :as u]))\n(defn double [x] (* 2 x))\n(defn doubled-inc [x] (u/inc1 (double x)))\n") +# a .clj sibling, with a dash in the name (-> underscore in the path) +(os/mkdir (string tmp "/other")) +(spit (string tmp "/other/util.clj") "(ns other.util)\n(defn inc1 [x] (+ x 1))\n") + +(def ctx (init {:paths [tmp]})) +(ctx-set-current-ns ctx "user") + +(var fails 0) +(defn check [label expr expected] + (let [r (protect (eval-string ctx expr)) + got (if (r 0) (normalize-pvecs (r 1)) (string "ERR " (r 1)))] + (if (= got expected) + (print " ok " label) + (do (++ fails) (printf " FAIL %s: want %q, got %q" label expected got))))) + +# require a .cljc lib from the added root and call it +(check "require .cljc lib" "(do (require (quote [mylib.core :as m])) (m/double 21))" 42) +# transitive require (mylib.core requires other.util) resolved from the root too +(check "transitive .clj require" "(mylib.core/doubled-inc 10)" 21) +# the stdlib still resolves from its default root +(check "stdlib still loads" "(do (require (quote [jolt.interop :as j])) (j/janet-type 1))" :number) + +# clean up +(defn rmrf [p] + (if (= :directory (os/stat p :mode)) + (do (each e (os/dir p) (rmrf (string p "/" e))) (os/rmdir p)) + (os/rm p))) +(rmrf tmp) + +(if (> fails 0) + (error (string "deps-loader-test: " fails " failing check(s)")) + (print "\nAll deps-loader tests passed!")) From 4d8494d6207623b87780ac636e4372111464ae22 Mon Sep 17 00:00:00 2001 From: Yogthos Date: Fri, 5 Jun 2026 23:05:47 -0400 Subject: [PATCH 4/6] deps phase 2: jolt-deps tool resolves deps.edn git/local deps MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A separate jolt-deps executable (mirroring jpm beside janet) resolves a deps.edn into source roots and runs jolt with them on JOLT_PATH; the runtime stays deps-agnostic. - src/jolt/deps.janet: read deps.edn (Janet parses it directly), resolve :git/* and :local/root via jpm/pm's download-bundle (git fetch + cache, no build), recurse for transitive deps, return de-duped roots. jpm loaded lazily so it's never embedded. Roots cached on a deps.edn hash. - src/jolt/deps_cli.janet + project.janet: the jolt-deps tool — path/run/repl/-e. - main: apply JOLT_PATH at runtime (the CLI ctx is frozen into the image at build, so init's env read wouldn't see it). Verified end to end against a local dep and a real git dep (medley). git only, pure clj/cljc. Test: deps-resolve-test (local deps, no network). --- doc/tools-deps.md | 46 +++++++++---- project.janet | 6 ++ src/jolt/deps.janet | 88 ++++++++++++++++++++++++ src/jolt/deps_cli.janet | 41 +++++++++++ src/jolt/main.janet | 6 ++ test/integration/deps-resolve-test.janet | 61 ++++++++++++++++ 6 files changed, 234 insertions(+), 14 deletions(-) create mode 100644 src/jolt/deps.janet create mode 100644 src/jolt/deps_cli.janet create mode 100644 test/integration/deps-resolve-test.janet diff --git a/doc/tools-deps.md b/doc/tools-deps.md index 8f41ea7..7e8f4f7 100644 --- a/doc/tools-deps.md +++ b/doc/tools-deps.md @@ -73,8 +73,8 @@ hit interpreter gaps, which is fine). so the stdlib always wins. (Two small changes: a root list in the ctx, and trying `.cljc`.) 4. **Dev vs build.** - - *Dev:* on REPL/CLI start, if `deps.edn` is present, resolve once (cache keyed - on a hash of deps.edn so it's a no-op when unchanged) and register the roots. + - *Dev:* `jolt-deps` resolves `deps.edn` (cached on a hash of it, so it's a + no-op when unchanged) and runs jolt with the roots on `JOLT_PATH`; `(require '[medley.core])` then just works. - *Build:* the dep namespaces a project actually uses get compiled into the image the same way the embedded `jolt.nrepl` source is today — load them at @@ -87,14 +87,29 @@ semantics, or version conflict resolution. Resolution is a tree walk over git deps; "the classpath" is just the list of `src` dirs of the clones. The loader already does path-based namespace lookup — we widen it from one root to a few. -## Integration with jpm +## A separate tool, like jpm beside janet -jpm stays the build tool for the Jolt binary; this lives beside it and *calls -into* `jpm/pm` for the git/cache work (import the module at dev/build time — jpm -is always present then; the shipped binary doesn't need it). **Decision: reuse -`jpm/pm`** (`resolve-bundle` + `download-bundle`, after `jpm/config/load-default`) -rather than shelling `git` ourselves — less code, and it shares jpm's cache. The -internal-API risk is acceptable since it's only used at dev/build time. +The jolt *runtime* knows nothing about deps.edn — exactly as the `janet` binary +knows nothing about jpm. Resolution lives in a separate `jolt-deps` tool (its own +`declare-executable`), and the runtime's only interface is the source roots in +`JOLT_PATH`: + +``` +jolt-deps path # print the resolved roots, ':'-joined +jolt-deps run FILE [args] # resolve, then run `jolt FILE …` with JOLT_PATH set +jolt-deps repl # resolve, then start a jolt REPL with JOLT_PATH set +jolt-deps -e EXPR [args] # resolve, then `jolt -e …` +``` + +It **reuses `jpm/pm`** (`resolve-bundle` + `download-bundle`, after +`jpm/config/load-default`) for the git fetch/cache rather than shelling `git` — +less code, shares jpm's cache. jpm is loaded lazily (`require`, not `import`), so +it's pulled in only when resolving (dev time), never embedded in a binary. The +internal-API risk is acceptable since it's dev-time only. + +One gotcha worth recording: `jolt`'s context is built into its image at build +time, so `JOLT_PATH` is read at runtime in `main` (not in `init`, whose env read +would be frozen at build). ## Limitations @@ -111,12 +126,15 @@ internal-API risk is acceptable since it's only used at dev/build time. `init` adds roots from the `:paths` opt and `JOLT_PATH` (colon-separated). Verified loading a real lib (medley) from an added root. See `test/integration/deps-loader-test.janet`. -2. **Resolve git deps via jpm.** Read `deps.edn`, resolve `:git/*` (+ - `:local/root`) through `jpm/pm` into `jpm_tree/.cache`, recurse for transitive - git deps, register the roots. `jolt deps` to resolve/print; auto on startup - when `deps.edn` exists. +2. **Resolve git deps via jpm.** *(done)* `src/jolt/deps.janet` reads `deps.edn`, + resolves `:git/*` + `:local/root` through `jpm/pm` into `jpm_tree/.cache`, + recurses for transitive deps, and returns the roots (cached on a deps.edn + hash). The separate `jolt-deps` tool (`src/jolt/deps_cli.janet`, + `declare-executable`) exposes `path`/`run`/`repl`/`-e`. See + `test/integration/deps-resolve-test.janet`. 3. **Build-time compile-in.** Fold the used dep namespaces into the image at - build (as with embedded `jolt.nrepl`). + build (as with embedded `jolt.nrepl`), so a built artifact needs neither the + deps nor jpm. *(not started)* 4. **Conformance.** Pull a few popular pure-`cljc` git libs, see what loads/runs, and drive interpreter gaps from the failures — same loop as the clojure-test-suite battery. diff --git a/project.janet b/project.janet index 2676169..d6d9ddd 100644 --- a/project.janet +++ b/project.janet @@ -8,3 +8,9 @@ (declare-executable :name "jolt" :entry "src/jolt/main.janet") + +# Separate tool (like jpm beside janet): resolves deps.edn into Jolt source +# roots. The jolt runtime stays deps-agnostic — it just reads JOLT_PATH. +(declare-executable + :name "jolt-deps" + :entry "src/jolt/deps_cli.janet") diff --git a/src/jolt/deps.janet b/src/jolt/deps.janet new file mode 100644 index 0000000..2e487bf --- /dev/null +++ b/src/jolt/deps.janet @@ -0,0 +1,88 @@ +# deps.edn resolution for Jolt. +# +# Resolve git and :local/root dependencies from a deps.edn into a list of source +# directories, which the loader then searches (see evaluator/find-ns-file). We +# reuse jpm's git fetch + cache (jpm/pm) rather than shipping a package manager. +# Maven (:mvn/version) deps are ignored — git only, pure clj/cljc only. +# +# jpm is loaded lazily (require, not import) so it's needed only at resolve time +# (dev/build), never embedded in the shipped binary. + +# A typical deps.edn is also valid Janet data, so we read it with Janet's parser. +# (EDN-only forms — #{} sets, tagged literals, namespaced maps — aren't handled; +# deps.edn rarely uses them in :deps/:paths.) +(defn read-edn [path] + (when (os/stat path) + (try (parse (slurp path)) ([_] nil)))) + +(defn- jpm-fn [mod sym] + (get-in (require mod) [sym :value])) + +(defn- ensure-jpm-config [tree] + ((jpm-fn "jpm/config" 'load-default)) + (setdyn :modpath tree) + (setdyn :gitpath (dyn :gitpath "git"))) + +(defn- clone-git [spec] + # spec is a deps.edn dep value: {:git/url ... :git/sha/:git/tag ...} + (def resolve-bundle (jpm-fn "jpm/pm" 'resolve-bundle)) + (def download-bundle (jpm-fn "jpm/pm" 'download-bundle)) + (def b (resolve-bundle {:url (get spec :git/url) + :sha (get spec :git/sha) + :tag (get spec :git/tag) + :shallow false})) + (download-bundle (b :url) (b :type) (b :tag) (b :shallow))) + +(defn- src-roots + "Source dirs of a project/dep at `dir`: its deps.edn :paths joined to dir + (default [\"src\"])." + [dir edn] + (map |(string dir "/" $) (or (and edn (get edn :paths)) ["src"]))) + +(defn resolve-deps + "Resolve the git/:local deps of `deps-edn-path` into an ordered, de-duplicated + array of source dirs (the project's own :paths first, then each dependency's, + transitively). `tree` is where jpm's clone cache lives (default ./jpm_tree)." + [deps-edn-path &opt tree] + (default tree (string (os/cwd) "/jpm_tree")) + (os/mkdir tree) + (ensure-jpm-config tree) + (def roots @[]) + (def seen @{}) + (defn add-root [r] (unless (index-of r roots) (array/push roots r))) + (defn process [edn base-dir own-paths?] + (when (dictionary? edn) + (when own-paths? (each r (src-roots base-dir edn) (add-root r))) + (eachp [lib spec] (or (get edn :deps) {}) + (def k (string lib)) + (unless (get seen k) + (put seen k true) + (def dir + (cond + (and (dictionary? spec) (get spec :git/url)) (clone-git spec) + (and (dictionary? spec) (get spec :local/root)) + (let [lr (get spec :local/root)] + (if (string/has-prefix? "/" lr) lr (string base-dir "/" lr))) + nil)) # :mvn/* and anything else: skip + (when dir + (def dep-edn (read-edn (string dir "/deps.edn"))) + (each r (src-roots dir dep-edn) (add-root r)) + (process dep-edn dir false)))))) + (process (read-edn deps-edn-path) (os/cwd) true) + roots) + +(defn resolve-deps-cached + "Like resolve-deps, but caches the resolved roots in the tree keyed on a hash + of the deps.edn, so an unchanged deps.edn resolves without re-fetching." + [deps-edn-path &opt tree] + (default tree (string (os/cwd) "/jpm_tree")) + (when (os/stat deps-edn-path) + (os/mkdir tree) + (def cache-file (string tree "/.jolt-deps-roots.jdn")) + (def h (hash (slurp deps-edn-path))) + (def cached (when (os/stat cache-file) (try (parse (slurp cache-file)) ([_] nil)))) + (if (and cached (= h (get cached :hash))) + (get cached :roots) + (let [roots (resolve-deps deps-edn-path tree)] + (spit cache-file (string/format "%j" {:hash h :roots roots})) + roots)))) diff --git a/src/jolt/deps_cli.janet b/src/jolt/deps_cli.janet new file mode 100644 index 0000000..ae0c990 --- /dev/null +++ b/src/jolt/deps_cli.janet @@ -0,0 +1,41 @@ +# jolt-deps — a separate tool that resolves a deps.edn into Jolt source roots. +# +# Mirrors how jpm is a tool beside the janet runtime: the jolt runtime knows +# nothing about deps.edn — it just searches the roots in JOLT_PATH (see +# api/init). This tool does the resolution (git + :local deps, via jpm's fetch +# cache) and either prints the roots or launches jolt with JOLT_PATH set. +# +# jolt-deps path print the resolved roots (':'-joined), e.g. for +# JOLT_PATH=$(jolt-deps path) jolt file.clj +# jolt-deps run FILE [args] resolve, then run `jolt FILE args` with JOLT_PATH set +# jolt-deps repl resolve, then start a jolt REPL with JOLT_PATH set +# jolt-deps -e EXPR [args] resolve, then `jolt -e EXPR ...` with JOLT_PATH set +# +# The jolt binary is found via $JOLT_BIN, else `jolt` on PATH. + +(import ./deps :as deps) + +(defn- roots [] + (if (os/stat "deps.edn") (deps/resolve-deps-cached "deps.edn") @[])) + +(defn- exec-jolt [extra-args] + # Set JOLT_PATH in our own env and let the child inherit it (os/execute's env + # arg isn't honored here; inheriting is reliable). + (def rs (string/join (roots) ":")) + (def existing (os/getenv "JOLT_PATH")) + (os/setenv "JOLT_PATH" (if (and existing (> (length existing) 0)) (string rs ":" existing) rs)) + (os/execute [(os/getenv "JOLT_BIN" "jolt") ;extra-args] :p)) + +(defn- usage [] + (print "usage: jolt-deps [path | run FILE [args] | repl | -e EXPR [args]]")) + +(defn main [&] + (def argv (tuple/slice (or (dyn :args) @[]) 1)) + (def cmd (get argv 0)) + (cond + (or (nil? cmd) (= cmd "help") (= cmd "-h") (= cmd "--help")) (usage) + (= cmd "path") (print (string/join (roots) ":")) + (= cmd "run") (os/exit (exec-jolt (tuple/slice argv 1))) + (= cmd "repl") (os/exit (exec-jolt [])) + (= cmd "-e") (os/exit (exec-jolt argv)) + (do (eprint "jolt-deps: unknown command " cmd) (usage) (os/exit 1)))) diff --git a/src/jolt/main.janet b/src/jolt/main.janet index 06d9842..dca9fb2 100644 --- a/src/jolt/main.janet +++ b/src/jolt/main.janet @@ -309,6 +309,12 @@ (def args (or (dyn :args) @[])) # @["jolt" arg1 arg2 ...] (def argv (if (> (length args) 1) (array/slice args 1) @[])) (ctx-set-current-ns ctx "user") + # JOLT_PATH must be applied at runtime: this `ctx` is built into the image at + # build time, so its source-paths can't capture the runtime environment. + # `jolt-deps` sets JOLT_PATH to the resolved deps.edn source roots. + (when-let [jp (os/getenv "JOLT_PATH")] + (each p (string/split ":" jp) + (when (> (length p) 0) (array/push (get (ctx :env) :source-paths) p)))) (cond (empty? argv) (run-repl) (or (= (argv 0) "-h") (= (argv 0) "--help")) (print-help) diff --git a/test/integration/deps-resolve-test.janet b/test/integration/deps-resolve-test.janet new file mode 100644 index 0000000..f71c6fb --- /dev/null +++ b/test/integration/deps-resolve-test.janet @@ -0,0 +1,61 @@ +# deps.edn resolution into source roots, then loading a library through them. +# Uses :local/root deps only so it needs no network (the git path is the same +# code, exercised manually against real repos). + +(use ../../src/jolt/api) +(use ../../src/jolt/types) +(import ../../src/jolt/deps :as deps) + +(def base (string (os/getenv "TMPDIR") "/jolt-deps-resolve-" (os/time))) +(defn rmrf [p] + (when (os/stat p) + (if (= :directory (os/stat p :mode)) + (do (each e (os/dir p) (rmrf (string p "/" e))) (os/rmdir p)) + (os/rm p)))) +(rmrf base) + +(defn mkdirs [p] + (def abs (string/has-prefix? "/" p)) + (var acc nil) + (each seg (filter |(not= "" $) (string/split "/" p)) + (set acc (cond (nil? acc) (if abs (string "/" seg) seg) (string acc "/" seg))) + (unless (os/stat acc) (os/mkdir acc)))) + +# project -> depends on lib-a (local), which depends on lib-b (local, transitive) +(each d ["proj/src" "a/src/liba" "b/src/libb"] (mkdirs (string base "/" d))) +(spit (string base "/proj/deps.edn") `{:paths ["src"] :deps {my/a {:local/root "../a"}}}`) +(spit (string base "/a/deps.edn") `{:paths ["src"] :deps {my/b {:local/root "../b"}}}`) +(spit (string base "/b/deps.edn") `{:paths ["src"]}`) +(spit (string base "/a/src/liba/core.clj") "(ns liba.core (:require [libb.core :as b]))\n(defn val [] (b/n))\n") +(spit (string base "/b/src/libb/core.clj") "(ns libb.core)\n(defn n [] 99)\n") + +(var fails 0) +(defn check [label got expected] + (if (= got expected) (print " ok " label) + (do (++ fails) (printf " FAIL %s: want %q, got %q" label expected got)))) + +(os/cd (string base "/proj")) +(def roots (deps/resolve-deps "deps.edn" (string base "/proj/jpm_tree"))) +# roots: proj/src, a/src (transitive: b/src) — at least the two dep srcs present +(check "resolves transitive local dep roots" + (truthy? (and (some |(string/has-suffix? "/a/src" $) roots) + (some |(string/has-suffix? "/b/src" $) roots))) + true) + +# load through the resolved roots: liba.core requires libb.core transitively +(def ctx (init {:paths roots})) +(ctx-set-current-ns ctx "user") +(let [r (protect (eval-string ctx "(do (require (quote [liba.core :as a])) (a/val))"))] + (check "require local lib + transitive dep" (if (r 0) (r 1) (string "ERR " (r 1))) 99)) + +# cached resolution returns the same roots without re-walking +(check "cached resolve matches" + (deep= roots (deps/resolve-deps-cached "deps.edn" (string base "/proj/jpm_tree"))) + true) + +(os/cd "/") +(rmrf base) + +(if (> fails 0) + (error (string "deps-resolve-test: " fails " failing check(s)")) + (print "\nAll deps-resolve tests passed!")) From 503b0af050b5ceef44d56d646a1520c0f06792a6 Mon Sep 17 00:00:00 2001 From: Yogthos Date: Fri, 5 Jun 2026 23:07:20 -0400 Subject: [PATCH 5/6] test: fall back to /tmp when TMPDIR is unset (Linux CI) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The deps tests built temp paths from $TMPDIR, which isn't set on the CI runners — deps-resolve-test then tried to mkdir at the filesystem root and failed. Default to /tmp. --- test/integration/deps-loader-test.janet | 2 +- test/integration/deps-resolve-test.janet | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/test/integration/deps-loader-test.janet b/test/integration/deps-loader-test.janet index 521a36a..96cca27 100644 --- a/test/integration/deps-loader-test.janet +++ b/test/integration/deps-loader-test.janet @@ -6,7 +6,7 @@ (use ../../src/jolt/api) (use ../../src/jolt/types) -(def tmp (string (os/getenv "TMPDIR") "jolt-deps-test-" (os/time))) +(def tmp (string (or (os/getenv "TMPDIR") "/tmp") "/jolt-deps-test-" (os/time))) (os/mkdir tmp) (os/mkdir (string tmp "/mylib")) (os/mkdir (string tmp "/other")) diff --git a/test/integration/deps-resolve-test.janet b/test/integration/deps-resolve-test.janet index f71c6fb..dadfbd5 100644 --- a/test/integration/deps-resolve-test.janet +++ b/test/integration/deps-resolve-test.janet @@ -6,7 +6,7 @@ (use ../../src/jolt/types) (import ../../src/jolt/deps :as deps) -(def base (string (os/getenv "TMPDIR") "/jolt-deps-resolve-" (os/time))) +(def base (string (or (os/getenv "TMPDIR") "/tmp") "/jolt-deps-resolve-" (os/time))) (defn rmrf [p] (when (os/stat p) (if (= :directory (os/stat p :mode)) From 91df7233b37fc709cbeae81bf3e8540fd9986b5f Mon Sep 17 00:00:00 2001 From: Yogthos Date: Fri, 5 Jun 2026 23:11:16 -0400 Subject: [PATCH 6/6] docs: add building-and-deps guide, link from README Move build details, JOLT_PATH/namespace resolution, and the jolt-deps/deps.edn usage into doc/building-and-deps.md instead of growing the README; README Build now links to it. --- README.md | 13 +++--- doc/building-and-deps.md | 97 ++++++++++++++++++++++++++++++++++++++++ 2 files changed, 102 insertions(+), 8 deletions(-) create mode 100644 doc/building-and-deps.md diff --git a/README.md b/README.md index 87374bc..28bb36e 100644 --- a/README.md +++ b/README.md @@ -10,16 +10,13 @@ A Clojure interpreter running on [Janet](https://janet-lang.org). Jolt reads Clo git clone https://github.com/jolt-lang/jolt.git cd jolt git submodule update --init # pulls vendor/sci -jpm build # compiles build/jolt +jpm build # builds build/jolt and build/jolt-deps ``` -Requires `jpm` and a recent Janet — developed and CI-tested against **1.41**. The -futures and core.async layers rely on Janet's threaded `ev/` channels (`ev/thread`, -`ev/thread-chan`), so an older Janet may not run the full suite. - -`jpm build` doesn't always detect source changes — run `jpm clean && jpm build` -after editing `src/` to be sure `build/jolt` is current. The test suite (`jpm -test`) runs against the source directly, so it never goes stale. +Requires `jpm` and a recent Janet (CI-tested against 1.41). See +[doc/building-and-deps.md](doc/building-and-deps.md) for build details, the +`jpm clean` caveat, how namespaces are resolved (`JOLT_PATH`), and pulling +Clojure libraries from a `deps.edn` with the `jolt-deps` tool. ## Run diff --git a/doc/building-and-deps.md b/doc/building-and-deps.md new file mode 100644 index 0000000..0c63634 --- /dev/null +++ b/doc/building-and-deps.md @@ -0,0 +1,97 @@ +# Building and dependencies + +How to build Jolt from source and how to pull Clojure libraries into a project. + +## Building + +```bash +git clone https://github.com/jolt-lang/jolt.git +cd jolt +git submodule update --init # vendor/sci (used by the SCI bootstrap tests) +jpm build +``` + +This produces two executables under `build/`: + +- **`jolt`** — the runtime: REPL, file/expr runner, nREPL server. +- **`jolt-deps`** — a separate tool that resolves a `deps.edn` (see below). It + sits beside the runtime the way `jpm` sits beside `janet`; the runtime itself + knows nothing about deps.edn. + +Needs `jpm` and a recent Janet — developed and CI-tested against **1.41**. The +futures and core.async layers use Janet's threaded `ev/` channels (`ev/thread`, +`ev/thread-chan`), so older Janets may not run the full suite. + +`jpm build` doesn't always notice source changes; run `jpm clean && jpm build` +after editing `src/` to be sure the binaries are current. `jpm test` runs against +the source directly, so it never goes stale. + +## How namespaces are found + +`(require ...)` resolves a namespace to a file by searching an ordered list of +source roots — the stdlib first, then any extra roots — trying `.clj` then +`.cljc` (dots become directories, dashes become underscores). Extra roots +come from: + +- `JOLT_PATH` — a colon-separated list of directories (like a classpath), applied + at runtime; +- the `:paths` option to `init` when embedding Jolt as a library. + +So you can point Jolt at a directory of Clojure source with no deps machinery at +all: + +```bash +JOLT_PATH=/path/to/lib/src build/jolt myfile.clj +``` + +## Dependencies via deps.edn + +`jolt-deps` reads a `deps.edn` in the current directory, fetches its +dependencies, and runs `jolt` with the resolved source directories on +`JOLT_PATH`. + +```bash +jolt-deps path # print the resolved roots (':'-joined) +jolt-deps run FILE [args] # resolve, then run `jolt FILE …` +jolt-deps repl # resolve, then start a REPL +jolt-deps -e EXPR [args] # resolve, then evaluate EXPR +``` + +`jolt-deps` launches the `jolt` binary it finds on `PATH` (override with +`$JOLT_BIN`). + +Example `deps.edn`: + +```clojure +{:paths ["src"] + :deps {weavejester/medley {:git/url "https://github.com/weavejester/medley" + :git/tag "1.0.0"} + my/helpers {:local/root "../helpers"}}} +``` + +```bash +jolt-deps run -m myapp.main +``` + +### What's supported + +- **git deps** — `{:git/url … :git/tag …}` or `{:git/url … :git/sha …}` (use a + full SHA; `git fetch` can't resolve a short one). Transitive deps from each + dependency's own `deps.edn` are resolved too. +- **local deps** — `{:local/root "../path"}`. +- The project's own `:paths` (default `["src"]`) are included. + +Resolution reuses jpm's git fetch and cache (a dependency is cloned once into +`jpm_tree/.cache` and reused). Resolved roots are cached on a hash of `deps.edn`, +so an unchanged `deps.edn` doesn't re-fetch. + +### What's not + +- **No Maven.** `:mvn/version` deps are ignored — git and local only. +- **Pure `clj`/`cljc` only.** A library that needs the JVM (Java interop, host + classes) or a `clojure.core` feature Jolt doesn't implement will fail to load + or fail at a call. Coverage is per-function: a namespace can load with most + functions working and a few not. + +See [`tools-deps.md`](tools-deps.md) for the design rationale and the road to +compiling used dependency namespaces into the build.