ascii: make the fallback reachable, and the clone survive the gateway

TWO UNRELATED THINGS THE LAST RED PIPELINE TAUGHT, one of which was not a
bug in this repository at all.

THE PIPELINE DID NOT FAIL ON A TEST. Job 16530892473 never ran a line of
script: the runner could not reach GitLab's own CI gateway, spent 130 seconds
on a connect timeout, and git exited 128. The API reports that as
`script_failure`, which is how it read as a code regression at the exact commit
that reworked the banner -- it was a coincidence, and `ascii` was simply the
job that drew the short straw. GET_SOURCES_ATTEMPTS makes the runner retry the
fetch, which is the narrow fix; `retry:when: script_failure` would also re-run
genuinely failing tests until they passed once.

THE FALLBACK WAS NEVER EXERCISED. Chasing the above turned up the real defect.
_enable_utf8 calls stdout.reconfigure(encoding="utf-8"), which SUCCEEDS on a
pipe whatever PYTHONIOENCODING asked for -- so UNICODE was always True, the
`ascii` job spent its whole life on the UTF-8 branch it was written to avoid,
and it would have passed with BANNER_ASCII deleted. Confirmed by running it in
python:3.11-slim: the blocks printed as blocks under ascii:replace instead of
degrading to `?`. test_env.py already documented this as intended, so it is
a design decision reversed here rather than an oversight caught.

An explicit PYTHONIOENCODING is now read as a statement about what the far end
can take, not a default worth improving on. Only a DECLARED encoding gets that
treatment: a Windows console reporting cp1252 because that is the machine's
code page has declared nothing and is still reconfigured, so the rescue this
function was written for does not become dead code. The probe is GLYPHS, every
non-ASCII character the CLI can print, because a stdout that cannot hold the
tick cannot hold the fallback's promise either.

The job asserts the fallback was REACHED rather than merely survived, by
grepping for the bar only the ASCII spelling draws -- verified to fail when the
encoding is utf-8, so it has teeth. And it declares `ascii`, not
`ascii:replace`: under :replace a glyph escaping the UNICODE gate is rewritten
to `?` and the job stays green while the thing it tests is already broken.

Measured: all four jobs pass in python:3.11-slim, the suite on both matrix
legs, 82% coverage, and the ASCII wordmark now reaches a declared-ascii stdout.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

.gitlab-ci.yml

@@ -17,6 +17,18 @@
 
 stages: [corpus, test]
 
+# The clone, not the script. A pipeline was already lost to the runner failing
+# to reach GitLab's own CI gateway -- 130 seconds of connect timeout, git exiting
+# 128, and the job reported as a script_failure that never ran a line of script.
+# The runner retries fetching sources by itself if asked, and this is the narrow
+# way to ask: `retry:when: script_failure` would also silently re-run genuinely
+# failing tests until they passed once.
+#
+# Global rather than under `default:`, which takes no variables, and global
+# rather than per-job, because every job in the file pays the same clone.
+variables:
+  GET_SOURCES_ATTEMPTS: 3
+
 default:
   image: python:3.11-slim
   interruptible: true
@@ -92,11 +104,27 @@ ascii:
     TERM: "dumb"
     # Force the legacy-code-page path. This is what a Windows console stuck on
     # cp437 hands the banner, and the ASCII fallback exists for exactly it.
-    PYTHONIOENCODING: "ascii:replace"
+    #
+    # Strict, not `ascii:replace`. Under :replace a glyph that escaped the
+    # UNICODE gate is silently rewritten to `?` and the job stays green while
+    # the fallback it is testing is already broken. Strict turns that same
+    # glyph into a UnicodeEncodeError and a red job, which is the only version
+    # of this that is worth running.
+    PYTHONIOENCODING: "ascii"
   script:
     - python nexa.py
     - python nexa.py help ask
     - python nexa.py validate
+    # The fallback has to be REACHED, not merely survived. An explicit
+    # PYTHONIOENCODING used to be overridden by stdout.reconfigure, so this job
+    # spent its whole life on the UTF-8 branch it was written to avoid and
+    # would have passed with the ASCII art deleted. This is the bar the ASCII
+    # spelling draws and the Unicode one does not.
+    - |
+      if ! python nexa.py | grep -qF '########++++++++++............'; then
+        echo "the ASCII wordmark did not appear; the fallback was not reached"
+        exit 1
+      fi
     # Colour off means no escape sequences anywhere in the output. printf is
     # used to build the escape because the shell here is dash, which has no
     # $'...' syntax.

nexa.py

@@ -38,6 +38,24 @@ ADVISE_MODEL = "arch-advisor"
 
 # --------------------------------------------------------------------- colour
 
+# Every non-ASCII character the CLI can print: the wordmark's four blocks and
+# the seven glyphs defined against UNICODE further down. _enable_utf8 probes a
+# declared encoding with these rather than with the banner alone, because a
+# stdout that cannot hold the tick cannot hold the fallback's promise either.
+# A test asserts this string stays a superset of what the two actually use.
+GLYPHS = "█▓▒░─·▸◆◇✓✗"
+
+
+def _declared_encoding():
+    """The codec named in PYTHONIOENCODING, without its error handler.
+
+    The variable is `<encoding>[:<errors>]` and either half may be empty, so
+    a bare `:replace` declares a handler and no codec and must not be read as
+    one.
+    """
+    return os.environ.get("PYTHONIOENCODING", "").split(":")[0].strip()
+
+
 def _enable_utf8():
     """Windows consoles default to a legacy code page that cannot encode the
     block-drawing characters in the banner. Reconfigure stdout if possible;
@@ -46,6 +64,24 @@ def _enable_utf8():
     encoding = (getattr(sys.stdout, "encoding", "") or "").lower()
     if encoding.replace("-", "") in ("utf8", "utf8mb4"):
         return True
+    # An explicit PYTHONIOENCODING is a statement about what the far end can
+    # take, not a default worth improving on. reconfigure() re-encodes a pipe
+    # whatever was asked for, so without this check the declaration is silently
+    # overridden and the blocks reach a sink that already said it cannot hold
+    # them -- and the ASCII branch becomes unreachable from outside a
+    # monkeypatch. The CI job that declares ascii to exercise the fallback
+    # spent its whole life on the UTF-8 branch it was written to avoid, and
+    # would have passed with the ASCII art deleted.
+    #
+    # Only a DECLARED encoding gets this treatment. A Windows console reporting
+    # cp1252 because that is the machine's code page has declared nothing, and
+    # is still rescued below.
+    declared = _declared_encoding()
+    if declared:
+        try:
+            GLYPHS.encode(declared)
+        except (UnicodeEncodeError, LookupError):
+            return False
     try:
         sys.stdout.reconfigure(encoding="utf-8")
         sys.stderr.reconfigure(encoding="utf-8")

tests/test_cli.py

@@ -199,6 +199,33 @@ def test_the_ascii_fallback_is_actually_ascii():
     assert nexa.BANNER_ASCII.isascii()
 
 
+def test_glyphs_lists_every_non_ascii_character_the_module_can_print():
+    """GLYPHS is what _enable_utf8 probes a declared encoding with.
+
+    A glyph added to the UI and not to GLYPHS makes the probe pass on an
+    encoding that cannot hold it, and the character reaches the terminal
+    anyway -- the exact crash the fallback exists to prevent, reintroduced one
+    character at a time. Asserted against the source rather than against the
+    constants, because those are already degraded to ASCII when UNICODE is
+    False and would have nothing to say in the run that matters.
+
+    Comments are excluded. They are never printed, and the ones describing the
+    bar name the very characters this is looking for.
+    """
+    import io
+    import tokenize
+    with open(os.path.join(ROOT, "nexa.py"), encoding="utf-8") as handle:
+        source = handle.read()
+    printable = set()
+    for token in tokenize.generate_tokens(io.StringIO(source).readline):
+        if token.type == tokenize.COMMENT:
+            continue
+        printable.update(ch for ch in token.string if ord(ch) > 127)
+    assert printable, "the scan found nothing, so it is not reading the art"
+    missing = sorted(printable - set(nexa.GLYPHS))
+    assert not missing, "used but not in nexa.GLYPHS: " + " ".join(missing)
+
+
 def _cube(n):
     """An xterm-256 cube index as its (r, g, b) coordinates, each 0-5."""
     n -= 16

tests/test_env.py

@@ -258,18 +258,102 @@ def test_colour_is_off_when_stdout_is_not_a_terminal():
     assert "\033" not in out
 
 
+def _art_rows(art):
+    """The art's rows, indentation intact, without the blank lines the triple-
+    quoted literal starts and ends with. The same shape test_cli.py uses, kept
+    local because these tests read the art out of a subprocess's stdout rather
+    than out of the module."""
+    return [row for row in art.splitlines() if row.strip()]
+
+
 def test_a_legacy_code_page_does_not_crash_the_banner():
     """chcp 437, in the form that is portable to a test.
 
-    Not an assertion about which banner appears: with a cp437 stdout,
-    _enable_utf8 still succeeds, because stdout.reconfigure can re-encode a
-    pipe whatever PYTHONIOENCODING asked for. What matters is that the command
-    completes instead of dying on a UnicodeEncodeError halfway through a
-    wordmark, which is what the fallback exists to prevent.
+    This used to be unable to assert which banner appeared: reconfigure()
+    re-encodes a pipe whatever PYTHONIOENCODING asked for, so _enable_utf8
+    succeeded and the cp437 declaration was overridden. A declared encoding is
+    now read as a statement about the far end, so the fallback is reached from
+    outside a monkeypatch and the test can say so.
     """
     code, out = cli([], {"PYTHONIOENCODING": "cp437:replace"})
     assert code == 0
     assert "NEXA" in out
+    import nexa
+    for row in _art_rows(nexa.BANNER_ASCII):
+        assert row in out, "the ASCII wordmark did not reach a cp437 stdout"
+
+
+def test_a_declared_ascii_stdout_gets_the_ascii_wordmark():
+    """The CI job's case, and the one the fallback is named for.
+
+    Strict `ascii` rather than `ascii:replace`: under :replace a glyph that
+    escaped the UNICODE gate comes back as `?` and nothing fails, which is how
+    a fallback rots while its job stays green.
+    """
+    code, out = cli([], {"PYTHONIOENCODING": "ascii"})
+    assert code == 0
+    assert out.isascii(), "a non-ASCII character survived a declared ASCII stdout"
+    import nexa
+    for row in _art_rows(nexa.BANNER_ASCII):
+        assert row in out
+
+
+def test_a_declared_utf8_still_gets_the_unicode_wordmark():
+    """The common case, asserted so the narrowing above cannot swallow it."""
+    import nexa
+    code, out = cli([], {"PYTHONIOENCODING": "utf-8"})
+    assert code == 0
+    for row in _art_rows(nexa.BANNER_UNICODE):
+        assert row in out
+
+
+def test_an_undeclared_code_page_is_still_rescued():
+    """The Windows console this was written for declares nothing.
+
+    cp1252 as the machine's code page is not a statement about the far end, so
+    it is still reconfigured to UTF-8. The check keys on PYTHONIOENCODING, not
+    on the encoding stdout happens to report -- otherwise every Windows console
+    would drop to the fallback and the rescue would be dead code.
+
+    In-process because `cli` always names an encoding, and the situation here
+    is the one where nobody has.
+    """
+    import nexa
+
+    class MachineCodePage(object):
+        """Reports cp1252 because that is the machine's code page, and agrees
+        to be re-encoded, which is what a real console does."""
+        encoding = "cp1252"
+        reconfigured = False
+
+        def reconfigure(self, **kwargs):
+            MachineCodePage.reconfigured = True
+
+    monkey = MachineCodePage()
+    saved_out, saved_err = nexa.sys.stdout, nexa.sys.stderr
+    declared = os.environ.pop("PYTHONIOENCODING", None)
+    try:
+        nexa.sys.stdout = nexa.sys.stderr = monkey
+        assert nexa._enable_utf8() is True
+        assert MachineCodePage.reconfigured, "the console was never rescued"
+    finally:
+        nexa.sys.stdout, nexa.sys.stderr = saved_out, saved_err
+        if declared is not None:
+            os.environ["PYTHONIOENCODING"] = declared
+
+
+def test_a_bare_error_handler_declares_no_codec():
+    """`:replace` is a handler and no encoding. Read as a codec it raises
+    LookupError, which would send every such run to the fallback."""
+    import nexa
+    for value, expected in (
+            (":replace", ""), ("", ""), ("utf-8", "utf-8"),
+            ("utf-8:replace", "utf-8"), ("  ascii : strict ", "ascii")):
+        os.environ["PYTHONIOENCODING"] = value
+        try:
+            assert nexa._declared_encoding() == expected, value
+        finally:
+            del os.environ["PYTHONIOENCODING"]
 
 
 def test_ascii_banner_when_stdout_cannot_be_made_utf8(monkeypatch):