marketplace/skills/sync/scripts/pull.py

#!/usr/bin/env python3
"""
pull.py — Gitea issues -> the local store.

Writes flat markdown the domain layer owns and prints a compact index; the raw
API payload never reaches the conversation.

**This is how you get a pushed issue back.** `push.py` deletes the local file
once Gitea has confirmed it, so pulling is not a refresh of a copy you kept —
it is how the copy comes to exist. It lands under the SAME slug it had before,
even after a rename in the web UI and even on a machine that has never seen the
issue: the slug travels in the body as `<!-- tea:id … -->`, and
tmp/issues/.remote.json indexes it by number. See `id_for` for the order those
are consulted in. The marker itself is stripped out of what is written to disk.

Two ways to name what to pull:

    pull.py 42 [17 …]                  by key: 42 | #42 | owner/repo#42 | URL
    pull.py --milestone 6              by filter: whole milestone in ONE request
    pull.py --label type/bug --state all
    pull.py -q sqlc --limit 20

Filter mode costs one request per 50 issues — the list payload already carries
the bodies. Gitea silently ignores an unresolvable `milestones=` filter and
returns the whole backlog, so the milestone is resolved up front and every
issue is re-checked locally. Projects are NOT filterable: the projects API is
not exposed (404 on Gitea 1.26) — use milestones or labels, or the web UI.

A closed issue is not a unit of work, so filter mode enumerates it but leaves
it out of the store: `--state all` still shows the whole picture, and only
`--state closed` writes one. An issue already on disk is refreshed either way,
so the local copy learns it was closed instead of staying open forever, and the
count of the ones left out goes to stderr. Key mode is exempt: an address is not
a bulk read, and `pull.py 1` fetches a closed issue as it always did.

**`--limit` is on the write, not on the selection.** It counts the issues this
run puts in the store — written, or left in place by `--cached` — and never the
closed ones it enumerated and threw away. `--limit 20` over a milestone whose
first 30 issues are closed still writes 20, if 20 open ones are there to write:
pages keep coming until the budget is full. Two boundaries keep that honest:

- Pages stop the moment the budget is full. Never one page more.
- A filtered read may scan at most `_gitea.PAGE_SLACK` times the pages the limit
  would need if nothing were dropped. A filter that matches almost only closed
  issues therefore ends in a warning and a short answer, not in a walk of the
  whole tracker. Narrow the filter, or raise `--limit`, which raises the budget
  with it.
- Dependencies are outside the count: a blocker is followed because a stored
  issue named it, not because the filter selected it. `--limit 20` can
  therefore leave more than 20 files behind — the budget counts the selection's
  writes, and the graph is not part of the selection.

`remote.py` is the deliberate exception, and it is not the same flag twice: it
writes nothing at all, so there is no write to bound and its `--limit` means
what it says — how many lines to print.

Comments ride along by default, in both modes and for every issue written:
the thread lands in tmp/issues/<id>.comments.md, beside the issue. It costs
nothing when there is nothing to fetch — the payload already carries the
comment count, so an issue with none makes no request, and a file left over
from an earlier pull is deleted. An absent file therefore means "no comments",
never "not asked for". The thread is pull-only: editing it changes nothing in
Gitea (post with comment.py).

**Dependencies come with every pull.** A pull answers with the whole unit of
work — the issue and what blocks it — so `depends:` is filled from Gitea's
native dependency graph and every blocker is pulled too, recursively, down to
`--depth` (default 3). That graph is the only source there is: `map.from_api`
writes slugs into the `## Depends on` prose and never `#N`, so an edge cannot be
recovered from the body. `--no-deps` turns off both halves — no `depends:`, no
recursion, and no request spent on either. `--deps` is still accepted and now
does nothing; it names what already happens.

What it costs, stated rather than hidden:

- **One request per issue that lands in the store** — `GET …/issues/{n}/dependencies`,
  fetched once and used twice, since the same links both fill `depends:` and
  tell the walk where to go next. A closed issue that filter mode drops costs
  nothing: nothing was stored, so there is no unit of work to complete.
- **One request per blocker the selection did not already carry** — a `GET` for
  the issue itself, then its own links, and so on until `--depth`.
- So `--milestone X` over 50 open issues is one list request + 50 link requests
  + one pair for every blocker outside the milestone, where it used to be one
  request flat. `--no-deps` is the way back to one.

**In filter mode a blocker the filter did not select still lands in the store,
and that is deliberate.** `--milestone X` can leave an issue from milestone Y on
disk and `--label` an unlabelled one: a blocker is followed because a stored
issue names it, not because it matched. The one blocker that does not land is a
closed one — closed is not a unit of work, filter mode drops it the way it drops
any other closed issue, and the `depends:` edge to it goes with it, so nothing
points at a file that is not there. Key mode has no such rule and stores it.

Other flags:
    --no-deps            do not fill depends:, do not follow blockers
    --deps               accepted, does nothing: it is the default now
    --depth N            how deep to follow blockers (default 3)
    --cached             skip issues already on disk (body AND comments)
    --repo owner/repo    default: auto-detect from the CWD git remote

Pulling overwrites the local body: it is a fetch, not a merge. Local edits you
have not pushed are lost — with exactly one exception, checkbox state. A `[x]`
on either side wins for any item whose text matches, because a tick is monotone
and unioning the two sides is not conflict resolution (gmap.merge_checkbox_state
has the rule and its price). `--cached` skips an issue before any of that: it is
not read and not merged — it still costs its one link request, because a cached
issue's blockers can be missing from disk even when it is not (`--cached
--no-deps` is the free one). Draw the graph afterwards with the domain's own
issue_tree.py — it needs no network.

Login: the operator's pin from .claude/settings.local.json (see /tea:auth).
"""
import argparse
import os
import sys

_HERE = os.path.dirname(os.path.abspath(__file__))
sys.path[:0] = [_HERE, os.path.normpath(os.path.join(_HERE, "..", "..", "issue", "scripts"))]

import _gitea  # noqa: E402
import issue  # noqa: E402
import issue_index  # noqa: E402
import map as gmap  # noqa: E402


def id_for(payload, store_ids, remote_map, repo, root):
    """The slug this remote issue belongs under. Three sources, in order.

    1. **`.remote.json`, keyed by number.** The local ledger, and the only one
       that knows about a file sitting on disk right now, so it wins. A
       retitled issue keeps the slug it was first pulled under.
    2. **The `<!-- tea:id … -->` marker in the body** (`gmap.id_in_body`). What
       makes push -> delete -> pull a round trip rather than a rename: the
       ledger can be lost (a fresh clone, another machine, a deleted
       `.remote.json`) and the tracker still remembers what this issue is called
       here — even after the title was changed in the web UI.
    3. **The title, slugified.** Issues filed in the web UI have no marker and
       have never had a local name; this is where they get one.

    A marker is only taken at its word when the slug is free. If a file of that
    name is already in the store, or the ledger has it under another number, the
    marker is a collision and not an identity — the name is uniquified
    (`marked-2`) rather than allowed to overwrite somebody else's issue."""
    got = remote_map.get(gmap.remote_key(repo, payload["number"]))
    if got:
        return got
    marked = gmap.id_in_body(payload.get("body") or "")
    if marked and marked not in store_ids and marked not in set(remote_map.values()):
        return marked
    return issue.unique_id(root, marked or issue.slugify(payload.get("title", "")),
                           taken=store_ids)


def lands_in_store(payload, drop_closed, store_ids, remote_map, repo, root):
    """Would this payload leave a file in the store? The `--limit` predicate.

    It has to be the same test the walk below applies, or the budget is spent on
    issues that never land — which is the bug this exists to prevent. So: a
    closed issue counts only when the store already has it (it is refreshed, and
    that is a write); anything else counts, including one `--cached` will skip,
    because a skipped issue is still an issue the store holds when the run ends.

    Cheap in the common case: only a closed payload costs an `id_for`, and that
    is a lookup plus, at worst, a stat."""
    if not (drop_closed and payload.get("state") == "closed"):
        return True
    id = id_for(payload, store_ids, remote_map, repo, root)
    return os.path.isfile(issue.path_of(root, id))


def comments_path(root, id):
    """Where an issue's comment thread lives — beside it, under the same slug.
    Named in `_gitea` because push.py has to delete the same file."""
    return _gitea.comments_path(root, id)


def sync_comments(login, base, root, id, number, count):
    """Bring <id>.comments.md in line with the server; return it, or None when
    the issue has no thread.

    `count` is the payload's own comment count, so an issue with none costs no
    request. A file from an earlier pull is removed when the thread is empty:
    the absence of the file is the answer, not a gap in what was asked for."""
    path = comments_path(root, id)
    comments = _gitea.get_comments(login, base, number) if count else []
    if comments:
        with open(path, "w") as f:
            f.write(gmap.render_comments(comments))
        return path
    if os.path.isfile(path):
        os.remove(path)  # stale thread from an earlier pull
    return None


def main():
    ap = argparse.ArgumentParser(description="Pull Gitea issues into the local store")
    ap.add_argument("keys", nargs="*", help="issue keys: 42, #42, owner/repo#42, URL")
    ap.add_argument("--milestone", help="pull a whole milestone (id or title)")
    ap.add_argument("--label", action="append", default=[],
                    help="filter by label; repeat for AND")
    ap.add_argument("-q", "--query", help="search text in title/body")
    ap.add_argument("--state", default="open", choices=["open", "closed", "all"],
                    help="filter mode only (default: open)")
    ap.add_argument("--limit", type=int, default=100,
                    help="filter mode: how many issues to STORE, not to enumerate"
                         " (default: 100)")
    # Dependencies are the default: a pull answers with the unit of work, not
    # one row of it. `--deps` stays accepted so the calls and command tables
    # written against the old default keep working — it now sets what is
    # already set.
    ap.add_argument("--no-deps", dest="deps", action="store_false",
                    help="do not fill depends: and do not follow blockers")
    ap.add_argument("--deps", dest="deps", action="store_true",
                    help="accepted, does nothing: dependencies are followed by default")
    ap.set_defaults(deps=True)
    ap.add_argument("--depth", type=int, default=3, help="max dependency depth (default: 3)")
    ap.add_argument("--cached", action="store_true",
                    help="skip issues already on disk instead of refetching")
    ap.add_argument("--repo", help="owner/repo (default: auto-detect from CWD git remote)")
    ap.add_argument("--out", default=issue.ISSUE_ROOT,
                    help="store root (default: <repo>/tmp/issues)")
    args = ap.parse_args()

    filtered = bool(args.milestone or args.label or args.query)
    if args.keys and filtered:
        _gitea.die("pass issue keys OR filters, not both")
    if not args.keys and not filtered:
        _gitea.die("nothing to pull: pass issue keys, or --milestone / --label / -q")

    root = args.out
    # A first pull into a fresh checkout has to create the store; it says so,
    # and the path is absolute, so it cannot be a stray cwd.
    if issue.create_store(root):
        sys.stderr.write("created store %s\n" % os.path.abspath(root))

    login = _gitea.require_login()

    # ---- which repo ------------------------------------------------------
    repo_arg = args.repo
    if not repo_arg and args.keys:
        repos = {_gitea.parse_key(k)[1] for k in args.keys} - {None}
        if len(repos) > 1:
            _gitea.die("all keys must belong to one repo, got: %s" % ", ".join(sorted(repos)))
        repo_arg = repos.pop() if repos else None
    base = _gitea.repo_base(repo_arg)
    repo = _gitea.repo_slug(login, repo_arg)

    issues = issue.load_all(root)
    remote_map = _gitea.load_map(root) or _gitea.rebuild_map(root, issues)
    store_ids = set(issues)
    number_of_id = {gmap.parse_remote_key(k)[1]: v for k, v in remote_map.items()
                    if gmap.parse_remote_key(k)[0] == repo}

    # A closed issue is not a unit of work: filter mode enumerates it but keeps
    # it out of the store unless the operator named the state. A key is an
    # address, not a bulk read, so key mode is exempt.
    drop_closed = filtered and args.state != "closed"

    written, skipped, dropped, pending = [], [], [], []

    # ---- seeds -----------------------------------------------------------
    if filtered:
        # The limit bounds the write, so the transport is told what a write is
        # and counts those; the closed ones it enumerated on the way come back
        # in the list anyway, to be reported and dropped below.
        payloads, ms_title = _gitea.list_issues(
            login, base, state=args.state, labels=args.label, query=args.query,
            milestone=args.milestone, limit=args.limit,
            keep=lambda p: lands_in_store(p, drop_closed, store_ids, remote_map,
                                          repo, root))
        if not payloads:
            _gitea.die("no issues match that filter")
        what = []
        if args.milestone:
            what.append("milestone %s" % ms_title)
        what += ["label %s" % l for l in args.label]
        if args.query:
            what.append("q=%r" % args.query)
        sys.stderr.write("%d issue(s) match %s (%s)\n"
                         % (len(payloads), " + ".join(what), args.state))
        queue = [(p, 0) for p in payloads]
        seen_numbers = {p["number"] for p in payloads}
    else:
        numbers = [_gitea.parse_key(k)[0] for k in args.keys]
        queue = [(_gitea.get_issue(login, base, n), 0) for n in numbers]
        seen_numbers = set(numbers)

    # ---- walk ------------------------------------------------------------
    while queue:
        payload, depth = queue.pop(0)
        number = payload["number"]
        id = id_for(payload, store_ids, remote_map, repo, root)
        stored = os.path.isfile(issue.path_of(root, id))

        # Closed and not already ours: nothing is written and nothing is asked
        # of the server for it — not its comments, not its links, and its own
        # blockers are not followed. The slug stays unclaimed too, so no other
        # issue ends up pointing `depends:` at a missing file.
        if drop_closed and payload.get("state") == "closed" and not stored:
            dropped.append(number)
            continue  # not stored: no unit of work here, so no links are fetched

        store_ids.add(id)
        number_of_id[number] = id

        # The native links, fetched ONCE for the two things they are for:
        # filling this issue's `depends:` and telling the walk where to go next.
        # One request per issue that lands in the store, and only one — the cost
        # the docstring quotes is this line.
        deps = _gitea.native_deps(login, base, number) if args.deps else []

        if args.cached and stored:
            skipped.append(id)  # body and thread unread; only the links cost
        else:
            # The copy already on disk, as it was when this run started. It
            # contributes its ticked checkboxes and nothing else; None when
            # the store has never seen this issue.
            prev = issues.get(id)
            iss, unresolved = gmap.from_api(payload, id, repo,
                                            id_for_number=number_of_id,
                                            extra_numbers=deps,
                                            synced=_gitea.now_iso(),
                                            local_body=prev.body if prev else None)
            issue.save(root, iss)
            sync_comments(login, base, root, id, number, payload.get("comments") or 0)
            remote_map[gmap.remote_key(repo, number)] = id
            written.append(id)
            pending.append((id, unresolved))

        if args.deps and depth < args.depth:
            child_numbers = gmap.numbers_in_body(payload.get("body") or "") + deps
            for n in child_numbers:
                if n in seen_numbers:
                    continue
                seen_numbers.add(n)
                queue.append((_gitea.get_issue(login, base, n), depth + 1))

    # Nothing is dropped in silence — say how many closed ones stayed out.
    if dropped:
        sys.stderr.write("%d closed issue(s) enumerated, not stored"
                         " (--state closed to pull them)\n" % len(dropped))

    # ---- second pass: dependencies that were not yet known on first write --
    for id, unresolved in pending:
        newly = [number_of_id[n] for n in unresolved
                 if n in number_of_id and number_of_id[n] != id]
        if not newly:
            continue
        iss = issue.load(root, id)
        for slug in newly:
            if slug not in iss.depends:
                iss.depends.append(slug)
        issue.save(root, iss)

    _gitea.save_map(root, remote_map)
    index_path, _ = issue_index.build(root)

    # Compact output — the only thing that lands in the model's context. The
    # thread rides on the issue's own line; no file means no comments.
    graph = False
    for id in sorted(set(written) | set(skipped)):
        iss = issue.load(root, id)
        graph = graph or bool(iss.depends)
        note = "  (cached)" if id in skipped else ""
        cpath = comments_path(root, id)
        if os.path.isfile(cpath):
            note += "  +%s comments: %s" % (iss.extra.get("comments") or "?", cpath)
        print("%s [%s] %s — %s    %s%s" % (
            id, ", ".join(iss.labels) or "no labels", iss.title, iss.state,
            issue.path_of(root, id), note))
    print("index: %s" % index_path)
    # Now that dependencies are the default, the hint is worth printing when
    # there is something to draw, not on every run that could have drawn it.
    if graph:
        print("graph: run issue_tree.py (offline) to draw it")


if __name__ == "__main__":
    main()