hermes.pds.lassul.us / superconfig
forked from lass.pds.lassul.us/superconfig
This repository has no description
0
Fork 0

Configure Feed

Select the types of activity you want to include in your feed.

tools.hermes-voice: keep following runs after the call, send unspoken requests on hang-up

Hermes does not stop a run when the /v1/runs/{id}/events subscriber goes
away (the SSE handler only detaches it), so runs already survived the
page. But nothing listened any more: an approval request then waited for
nobody and was denied by Hermes's timeout. hermes-voice now follows every
run it starts in its own thread, page or no page; the page replays the
run's items and gets blank keepalive lines so a vanished page is noticed.

On hang-up (button, idle, page close) the page posts /hangup with
keepalive: runs of the conversation count as unheard from then on, and
what the user said since the last delegation becomes a run of its own
with a note that nobody listens. Lines that the model's delegation
already took are not sent again, and a delegation still in its settle
delay leaves them to the hang-up. Only a correction within the same call
(replace) stops the previous run; a new call no longer stops work an
earlier call started.

An approval no call can answer is posted to the owner's Matrix DM (the
gateway's Matrix home room) through the gateway's webhook platform
(--matrix-webhook, signed with the API server key) with a short code;
/approve takes that code, from the page or from the Matrix side with the
API server key.

Approvals also failed outright before: under the gateway's session
authority the approval request is named prompt_id and the answer must be
exactly {request_id, execution_generation, choice}, while hermes-voice
sent only {choice} (invalid_params).

The persona and the server's texts name no particular user: they speak
of "the user" and of the owner's Matrix DM.

author
Hermes committer
lassulus date (Oct 17, 2026, 9:13 AM CEST) commit d90dc398 parent 105cac40
+340 -86
+303 -72
tools/hermes-voice/hermes_voice.py
··· 8 8 streams the reply back. The page then gives the answer to the voice model, 9 9 which says it aloud. 10 10 11 - A run lives independently of any HTTP connection, so a turn survives the page 12 - going away. A newer delegation for the same conversation stops the previous 13 - run (the user corrected or replaced the request); hanging up just ends the 14 - call, and the run finishes whatever the user asked for. 11 + A run lives independently of any HTTP connection, and this server follows 12 + every run it started to its end, whether or not a page is still listening. A 13 + newer delegation within the same call stops the previous run (the user 14 + corrected or replaced the request); hanging up just ends the call, and the run 15 + finishes whatever the user asked for. On hang-up the page sends what the user 16 + said since the last delegation, so a request spoken right before hanging up 17 + still reaches Hermes. When a run asks for approval while no page is listening, 18 + the question goes to the owner's Matrix DM (the gateway's Matrix home room, 19 + through its webhook platform), and the hermes-voice plugin's 20 + `/assist approve|deny <code>` command answers it. 15 21 16 22 This server exists so neither secret reaches the browser: it exchanges the 17 23 page's SDP offer for a Live session with the OpenAI key, and it calls Hermes ··· 24 30 """ 25 31 26 32 import argparse 33 + import hashlib 27 34 import hmac 28 35 import json 29 36 import logging 30 37 import os 31 38 import re 39 + import secrets 32 40 import socket 33 41 import threading 34 42 import time ··· 88 96 # The run event stream sends a keepalive every 10s, so a read that stalls this 89 97 # long means the API server is gone. 90 98 EVENTS_READ_TIMEOUT = 60 99 + # Same cadence towards the page: a write is the only way to notice that a page 100 + # went away while a run is quiet (e.g. waiting for an approval). 101 + PAGE_KEEPALIVE_SECONDS = 10 91 102 SESSION_ID_RE = re.compile(r"^[A-Za-z0-9_-]{1,64}$") 92 - # Hermes run ids are "run_<hex>", approval request ids hex. 93 - RUN_ID_RE = re.compile(r"^[A-Za-z0-9_.:-]{1,256}$") 94 103 95 104 # Frontend persona, adapted from Hermes's own (tools/voice_live.py): role, 96 105 # style and a labelled delegation policy. The backend carries the real 97 106 # instructions, tools and memory. 98 - PERSONA = """You are Hermes, lass's warm, upbeat and genuinely helpful voice assistant. Sound friendly and glad to help: positive and encouraging, never curt, annoyed, bored or sarcastic, even when delivering bad news or when the user is short with you. Talk fast: speak at a rapid, energetic pace, like someone who talks quickly by nature, with no slow drawn-out words and no long pauses between sentences. Keep replies short; a brief warm acknowledgment before the facts is good. 107 + PERSONA = """You are Hermes, the user's warm, upbeat and genuinely helpful voice assistant. Sound friendly and glad to help: positive and encouraging, never curt, annoyed, bored or sarcastic, even when delivering bad news or when the user is short with you. Talk fast: speak at a rapid, energetic pace, like someone who talks quickly by nature, with no slow drawn-out words and no long pauses between sentences. Keep replies short; a brief warm acknowledgment before the facts is good. 99 108 100 109 Backchannel policy: Use moderate backchannels. Acknowledge naturally without competing with the main response. 101 110 ··· 103 112 104 113 Delegation policy: 105 114 Backend tools: 106 - - Hermes agent: a full AI agent with tools. It can run commands on lass's machines, read and edit files, manage the calendar, browse the web, search, and remember things across sessions. It is the one who actually does work and knows facts. 115 + - Hermes agent: a full AI agent with tools. It can run commands on the user's machines, read and edit files, manage the calendar, write and read messages in the user's Matrix chat, browse the web, search, and remember things across sessions. It is the one who actually does work and knows facts. 107 116 108 117 Delegate to the backend when: 109 118 - The user asks a question that needs facts, current information, or careful reasoning. ··· 121 130 122 131 # Ephemeral system prompt for every delegated Hermes turn (Hermes layers system 123 132 # messages on top of its core prompt without storing them). 124 - BACKEND_NOTE = """This turn is a delegation from a live spoken conversation. The user message is a speech transcript: it may contain mis-hearings, phonetic spellings of technical terms, hesitations and later corrections; use the latest intent. Your reply will be spoken aloud by a friendly voice model that paraphrases it: answer in warm, plain conversational sentences, keep it short (a few sentences unless the user asked for detail), no markdown, no lists, no code blocks, no URLs read out character by character. Do the work with your tools as usual; only the final facts need to be spoken. Report an action as done only after it actually succeeded.""" 133 + BACKEND_NOTE = """This turn is a delegation from a live spoken conversation. The user message is a speech transcript: it may contain mis-hearings, phonetic spellings of technical terms, hesitations and later corrections; use the latest intent. Your reply will be spoken aloud by a friendly voice model that paraphrases it: answer in warm, plain conversational sentences, keep it short (a few sentences unless the user asked for detail), no markdown, no lists, no code blocks, no URLs read out character by character. Do the work with your tools as usual; only the final facts need to be spoken. Report an action as done only after it actually succeeded. The user may hang up before your reply is spoken: when they asked for a result or a question somewhere else, such as their Matrix chat (matrix_dm_send), send it there yourself.""" 134 + 135 + # Instead of BACKEND_NOTE for what the user said right before hanging up, when 136 + # the voice model had not delegated it yet. 137 + HUNG_UP_NOTE = """This turn comes from a live spoken conversation the user has already hung up. The user message is a speech transcript of what they said since the last request: it may contain mis-hearings, phonetic spellings of technical terms, hesitations and later corrections; use the latest intent. Nobody hears your reply. If the transcript asks for something, do it with your tools; when the user asked for a result somewhere, such as their Matrix chat (matrix_dm_send), send it there, and if you need something from them, ask in their Matrix chat. If it only says goodbye or thanks, or repeats a request already handled in this conversation, do nothing and reply with a short note saying so.""" 125 138 126 139 127 140 def glossary_text(glossary): ··· 138 151 139 152 140 153 class Config: 141 - def __init__(self, raw, openai_key, hermes_url, hermes_key): 154 + def __init__(self, raw, openai_key, hermes_url, hermes_key, matrix_webhook=None): 155 + self.matrix_webhook = matrix_webhook 142 156 self.model = raw.get("model", DEFAULT_MODEL) 143 157 self.voice = raw.get("voice", DEFAULT_VOICE) 144 158 self.idle_seconds = int(raw.get("idle_seconds", DEFAULT_IDLE_SECONDS)) ··· 159 173 parts.append(self.extra_instructions) 160 174 return "\n\n".join(parts) 161 175 162 - def backend_note(self): 176 + def backend_note(self, hung_up=False): 177 + note = HUNG_UP_NOTE if hung_up else BACKEND_NOTE 163 178 if not self.glossary: 164 - return BACKEND_NOTE 179 + return note 165 180 return ( 166 - BACKEND_NOTE 181 + note 167 182 + "\n\nTerms the transcript may contain as phonetic misspellings:\n" 168 183 + glossary_text(self.glossary) 169 184 ) ··· 285 300 return json.load(response) 286 301 287 302 288 - def hermes_start_run(config, hermes_session, lines): 289 - """Start one Hermes turn as a run; return its run id.""" 303 + def hermes_start_run(config, hermes_session, lines, hung_up=False): 304 + """Start one Hermes turn as a run; return its run id. Hermes queues it 305 + behind a run of the same session that is still going.""" 290 306 # A plain string input: Hermes then loads the history of session_id itself. 291 307 return hermes_call( 292 308 config, ··· 294 310 "/v1/runs", 295 311 { 296 312 "input": transcript_message(lines), 297 - "instructions": config.backend_note(), 313 + "instructions": config.backend_note(hung_up), 298 314 "session_id": hermes_session, 299 315 }, 300 316 )["run_id"] ··· 316 332 yield {"type": "tool", "status": status, "label": payload.get("tool")} 317 333 elif event == "approval.request": 318 334 # The run blocks until POST /v1/runs/{run_id}/approval answers. 335 + # Under the gateway's session authority the request is named 336 + # prompt_id and the answer must carry its execution_generation. 319 337 yield { 320 338 "type": "approval", 321 - "run_id": run_id, 322 - "request_id": payload.get("request_id"), 339 + "request_id": payload.get("request_id") or payload.get("prompt_id"), 340 + "execution_generation": payload.get("execution_generation"), 323 341 "command": payload.get("command"), 324 342 "description": payload.get("description"), 343 + "timeout_s": payload.get("timeout_s"), 325 344 } 326 345 elif event == "run.completed": 327 346 # Only the final answer: message.delta also streams commentary. ··· 336 355 yield {"type": "error", "message": "Hermes event stream ended early"} 337 356 338 357 339 - def hermes_approve(config, run_id, request_id, choice): 358 + def hermes_approve(config, run_id, approval, choice): 340 359 """Answer a pending approval of a Hermes run.""" 341 360 answer = {"choice": choice} 342 - if request_id: 343 - answer["request_id"] = request_id 361 + if approval.get("request_id"): 362 + answer["request_id"] = approval["request_id"] 363 + if approval.get("execution_generation") is not None: 364 + answer["execution_generation"] = approval["execution_generation"] 344 365 return hermes_call(config, "POST", f"/v1/runs/{run_id}/approval", answer) 345 366 346 367 347 - # The active run per Hermes session, so a newer delegation can stop it. 368 + def notify_matrix(config, text): 369 + """Post text to the owner's Matrix DM (the gateway's Matrix home room) 370 + through the gateway's webhook platform (a deliver_only route, signed with 371 + the API server key).""" 372 + body = json.dumps({"text": text}).encode() 373 + stamp = str(int(time.time())) 374 + signature = hmac.new( 375 + config.hermes_key.encode(), stamp.encode() + b"." + body, hashlib.sha256 376 + ).hexdigest() 377 + request = urllib.request.Request( 378 + config.matrix_webhook, 379 + data=body, 380 + method="POST", 381 + headers={ 382 + "Content-Type": "application/json", 383 + "X-Webhook-Timestamp": stamp, 384 + "X-Webhook-Signature-V2": signature, 385 + }, 386 + ) 387 + with urllib.request.urlopen(request, timeout=30): 388 + pass 389 + 390 + 391 + # Live runs by run id, and the latest run per Hermes session, which a 392 + # correction in the same call replaces. 348 393 RUNS = {} 394 + LATEST = {} 349 395 RUNS_LOCK = threading.Lock() 396 + # Pending approvals by short code: the page and the Matrix command answer 397 + # with the code. 398 + APPROVALS = {} 399 + APPROVALS_LOCK = threading.Lock() 350 400 351 401 352 - def take_run(hermes_session, run_id=None): 353 - """Remove and return the session's run (only if it is run_id, when given).""" 354 - with RUNS_LOCK: 355 - if run_id is None or RUNS.get(hermes_session) == run_id: 356 - return RUNS.pop(hermes_session, None) 357 - return None 402 + class Run: 403 + """A Hermes run this server follows to its end, page or no page. Pages 404 + replay its items; an approval nobody can hear goes to Matrix.""" 405 + 406 + def __init__(self, config, hermes_session, run_id, hung_up=False): 407 + self.config = config 408 + self.hermes_session = hermes_session 409 + self.run_id = run_id 410 + self.items = [] 411 + self.finished = False 412 + self.pages = 0 413 + self.hung_up = hung_up 414 + self.cond = threading.Condition() 415 + with RUNS_LOCK: 416 + RUNS[run_id] = self 417 + if not hung_up: 418 + LATEST[hermes_session] = self 419 + threading.Thread(target=self.follow, daemon=True).start() 420 + 421 + def follow(self): 422 + started = time.monotonic() 423 + tools = 0 424 + try: 425 + for item in hermes_follow_run(self.config, self.run_id): 426 + if item["type"] == "tool" and item["status"] == "running": 427 + tools += 1 428 + self.add(item) 429 + except urllib.error.HTTPError as error: 430 + detail = error.read().decode("utf-8", "replace")[:400] 431 + self.add({"type": "error", "message": f"Hermes {error.code}: {detail}"}) 432 + except (urllib.error.URLError, OSError, ValueError) as error: 433 + self.add({"type": "error", "message": f"Hermes unreachable: {error}"}) 434 + settle_approvals(self) 435 + with self.cond: 436 + self.finished = True 437 + self.cond.notify_all() 438 + with RUNS_LOCK: 439 + RUNS.pop(self.run_id, None) 440 + if LATEST.get(self.hermes_session) is self: 441 + del LATEST[self.hermes_session] 442 + last = self.items[-1] if self.items else {} 443 + log.info( 444 + "run %s ended (%s) after %.1fs, %d tool calls", 445 + self.run_id, 446 + last.get("finish_reason") or last.get("message"), 447 + time.monotonic() - started, 448 + tools, 449 + ) 450 + 451 + def add(self, item): 452 + # Anything after an approval request means Hermes moved on (answered, 453 + # or its own timeout). 454 + settle_approvals(self) 455 + if item["type"] == "approval": 456 + item["code"] = open_approval(self, item) 457 + with self.cond: 458 + self.items.append(item) 459 + self.cond.notify_all() 460 + self.ask_unheard() 461 + 462 + def attach(self): 463 + with self.cond: 464 + self.pages += 1 465 + 466 + def detach(self): 467 + with self.cond: 468 + self.pages -= 1 469 + self.ask_unheard() 470 + 471 + def hang_up(self): 472 + with self.cond: 473 + self.hung_up = True 474 + self.ask_unheard() 475 + 476 + def ask_unheard(self): 477 + """Send a pending approval to Matrix once no call can answer it.""" 478 + with self.cond: 479 + if self.pages and not self.hung_up: 480 + return 481 + with APPROVALS_LOCK: 482 + unasked = [ 483 + (code, item) 484 + for code, (run, item) in APPROVALS.items() 485 + if run is self and not item.get("asked") 486 + ] 487 + for _, item in unasked: 488 + item["asked"] = True 489 + for code, item in unasked: 490 + ask_matrix(self.config, code, item) 491 + 492 + 493 + def open_approval(run, item): 494 + with APPROVALS_LOCK: 495 + code = secrets.token_hex(2) 496 + while code in APPROVALS: 497 + code = secrets.token_hex(2) 498 + APPROVALS[code] = (run, item) 499 + return code 500 + 501 + 502 + def settle_approvals(run): 503 + with APPROVALS_LOCK: 504 + for code in [code for code, (owner, _) in APPROVALS.items() if owner is run]: 505 + del APPROVALS[code] 506 + 507 + 508 + def ask_matrix(config, code, item): 509 + if not config.matrix_webhook: 510 + log.info("approval %s has no listener and no Matrix webhook", code) 511 + return 512 + text = ( 513 + "Hermes needs your approval for work from a voice call: " 514 + + (item.get("description") or "run a command") 515 + ) 516 + if item.get("command"): 517 + text += f"\n```\n{item['command']}\n```" 518 + text += f"\nAnswer with `/assist approve {code}` or `/assist deny {code}`." 519 + if item.get("timeout_s"): 520 + text += f" Unanswered, it is denied after {round(item['timeout_s'] / 60)} minutes." 521 + try: 522 + notify_matrix(config, text) 523 + log.info("approval %s asked in Matrix", code) 524 + except (urllib.error.URLError, OSError) as error: 525 + log.warning("asking approval %s in Matrix failed: %s", code, error) 358 526 359 527 360 528 def stop_run(config, run_id): ··· 372 540 static: Path 373 541 access_token: "str | None" = None 374 542 375 - def authorized(self): 376 - """Check the page's bearer token; answer 401 when it is wrong.""" 543 + def authorized(self, also_hermes_key=False): 544 + """Check the page's bearer token (or, where allowed, the API server key 545 + the hermes-voice plugin in the gateway holds); answer 401 when it is 546 + wrong.""" 377 547 if self.access_token is None: 378 548 return True 379 549 header = self.headers.get("Authorization", "") 380 550 scheme, _, token = header.partition(" ") 381 - if scheme == "Bearer" and hmac.compare_digest( 382 - token.encode(), self.access_token.encode() 551 + accepted = [self.access_token] 552 + if also_hermes_key: 553 + accepted.append(self.config.hermes_key) 554 + if scheme == "Bearer" and any( 555 + hmac.compare_digest(token.encode(), key.encode()) for key in accepted 383 556 ): 384 557 return True 385 558 self.send_json(HTTPStatus.UNAUTHORIZED, {"error": "access token required"}) ··· 435 608 self.wfile.write(body) 436 609 437 610 def do_POST(self): 438 - if not self.authorized(): 611 + if not self.authorized(also_hermes_key=self.path == "/approve"): 439 612 return 440 613 try: 441 614 request = self.read_json() ··· 446 619 self.handle_session(request) 447 620 elif self.path == "/delegate": 448 621 self.handle_delegate(request) 622 + elif self.path == "/hangup": 623 + self.handle_hangup(request) 449 624 elif self.path == "/approve": 450 625 self.handle_approve(request) 451 626 else: ··· 488 663 if not isinstance(lines, list) or not lines: 489 664 self.send_json(HTTPStatus.BAD_REQUEST, {"error": "lines required"}) 490 665 return 491 - # One run per conversation: a newer delegation (usually a correction) 492 - # supersedes and stops the previous run. 493 - previous = take_run(hermes_session) 494 - if previous: 495 - stop_run(self.config, previous) 666 + # A correction within the same call (the page says so) supersedes and 667 + # stops the run it replaces; runs from earlier calls keep going. 668 + if request.get("replace") is True: 669 + with RUNS_LOCK: 670 + previous = LATEST.pop(hermes_session, None) 671 + if previous: 672 + stop_run(self.config, previous.run_id) 496 673 try: 497 - run_id = hermes_start_run(self.config, hermes_session, lines) 674 + run = Run( 675 + self.config, 676 + hermes_session, 677 + hermes_start_run(self.config, hermes_session, lines), 678 + ) 498 679 except urllib.error.HTTPError as error: 499 680 detail = error.read().decode("utf-8", "replace")[:400] 500 681 self.send_json(HTTPStatus.BAD_GATEWAY, {"error": f"Hermes {error.code}: {detail}"}) ··· 502 683 except (urllib.error.URLError, OSError, ValueError, KeyError) as error: 503 684 self.send_json(HTTPStatus.BAD_GATEWAY, {"error": f"Hermes unreachable: {error}"}) 504 685 return 505 - with RUNS_LOCK: 506 - RUNS[hermes_session] = run_id 507 686 self.send_response(HTTPStatus.OK) 508 687 self.send_header("Content-Type", "application/x-ndjson") 509 688 self.send_header("Cache-Control", "no-store") 510 689 self.send_header("X-Accel-Buffering", "no") 511 690 self.end_headers() 691 + run.attach() 512 692 try: 513 - try: 514 - started = time.monotonic() 515 - tools = 0 516 - for item in hermes_follow_run(self.config, run_id): 517 - if item["type"] == "tool" and item["status"] == "running": 518 - tools += 1 693 + sent = 0 694 + while True: 695 + with run.cond: 696 + if sent == len(run.items) and not run.finished: 697 + run.cond.wait(PAGE_KEEPALIVE_SECONDS) 698 + new = run.items[sent:] 699 + sent += len(new) 700 + done = run.finished and sent == len(run.items) 701 + for item in new: 519 702 self.write_line(item) 520 - log.info( 521 - "hermes turn took %.1fs (%d tool calls)", time.monotonic() - started, tools 522 - ) 523 - except urllib.error.HTTPError as error: 524 - detail = error.read().decode("utf-8", "replace")[:400] 525 - self.write_line({"type": "error", "message": f"Hermes {error.code}: {detail}"}) 526 - except (urllib.error.URLError, OSError, ValueError) as error: 527 - self.write_line({"type": "error", "message": f"Hermes unreachable: {error}"}) 528 - take_run(hermes_session, run_id) 703 + if done: 704 + break 705 + if not new: 706 + self.wfile.write(b"\n") 707 + self.wfile.flush() 529 708 except (BrokenPipeError, ConnectionResetError): 530 - # The page went away; the run keeps going and stays registered, so 531 - # a later delegation still supersedes it. 532 - log.info("delegation for %s abandoned by the page", hermes_session) 709 + # Only the page is gone: this server keeps following the run. 710 + log.info("page left run %s", run.run_id) 711 + finally: 712 + run.detach() 713 + 714 + def handle_hangup(self, request): 715 + """The call ended. No page hears this conversation's runs any more, and 716 + what the user said since the last delegation becomes its own run.""" 717 + hermes_session = request.get("hermes_session") 718 + lines = request.get("lines") or [] 719 + if ( 720 + not isinstance(hermes_session, str) 721 + or not SESSION_ID_RE.match(hermes_session) 722 + or not isinstance(lines, list) 723 + ): 724 + self.send_json(HTTPStatus.BAD_REQUEST, {"error": "invalid hang-up"}) 725 + return 726 + with RUNS_LOCK: 727 + LATEST.pop(hermes_session, None) 728 + runs = [run for run in RUNS.values() if run.hermes_session == hermes_session] 729 + for run in runs: 730 + run.hang_up() 731 + if not any(line.get("speaker") == "user" for line in lines): 732 + self.send_json(HTTPStatus.OK, {"run_id": None}) 733 + return 734 + try: 735 + run_id = hermes_start_run(self.config, hermes_session, lines, hung_up=True) 736 + except urllib.error.HTTPError as error: 737 + detail = error.read().decode("utf-8", "replace")[:400] 738 + self.send_json(HTTPStatus.BAD_GATEWAY, {"error": f"Hermes {error.code}: {detail}"}) 739 + return 740 + except (urllib.error.URLError, OSError, ValueError, KeyError) as error: 741 + self.send_json(HTTPStatus.BAD_GATEWAY, {"error": f"Hermes unreachable: {error}"}) 742 + return 743 + Run(self.config, hermes_session, run_id, hung_up=True) 744 + log.info("hang-up of %s left a request: run %s", hermes_session, run_id) 745 + self.send_json(HTTPStatus.OK, {"run_id": run_id}) 533 746 534 747 def handle_approve(self, request): 535 - run_id = request.get("run_id") 536 - request_id = request.get("request_id") 748 + code = request.get("code") 537 749 choice = request.get("choice") 538 750 # No "always": a permanent allowlist entry is not a voice decision. 539 - if ( 540 - choice not in ("once", "session", "deny") 541 - or not isinstance(run_id, str) 542 - or not RUN_ID_RE.match(run_id) 543 - or not ( 544 - request_id is None 545 - or (isinstance(request_id, str) and RUN_ID_RE.match(request_id)) 546 - ) 751 + if choice not in ("once", "session", "deny") or not ( 752 + code is None or isinstance(code, str) 547 753 ): 548 754 self.send_json(HTTPStatus.BAD_REQUEST, {"error": "invalid approval answer"}) 549 755 return 756 + with APPROVALS_LOCK: 757 + if code is None and len(APPROVALS) == 1: 758 + code = next(iter(APPROVALS)) 759 + pending = APPROVALS.get(code) 760 + if pending is None: 761 + self.send_json( 762 + HTTPStatus.CONFLICT, 763 + {"error": f"no pending approval {code}" if code else "no single pending approval"}, 764 + ) 765 + return 766 + run, item = pending 550 767 try: 551 - result = hermes_approve(self.config, run_id, request_id, choice) 768 + result = hermes_approve(self.config, run.run_id, item, choice) 552 769 except urllib.error.HTTPError as error: 553 770 detail = error.read().decode("utf-8", "replace")[:400] 554 771 self.send_json(HTTPStatus.BAD_GATEWAY, {"error": f"Hermes {error.code}: {detail}"}) ··· 556 773 except (urllib.error.URLError, OSError, ValueError) as error: 557 774 self.send_json(HTTPStatus.BAD_GATEWAY, {"error": f"Hermes unreachable: {error}"}) 558 775 return 559 - log.info("approval %s for run %s: %s", choice, run_id, result.get("resolved")) 560 - self.send_json(HTTPStatus.OK, result) 776 + with APPROVALS_LOCK: 777 + APPROVALS.pop(code, None) 778 + if result.get("status") == "already_resolved": 779 + self.send_json(HTTPStatus.CONFLICT, {"error": f"approval {code} was already answered"}) 780 + return 781 + log.info("approval %s (run %s): %s %s", code, run.run_id, choice, result.get("status")) 782 + self.send_json( 783 + HTTPStatus.OK, 784 + {"code": code, "choice": choice, "description": item.get("description")}, 785 + ) 561 786 562 787 def write_line(self, item): 563 788 self.wfile.write(json.dumps(item).encode() + b"\n") ··· 571 796 parser.add_argument("--hermes-key-file", required=True, help="file holding API_SERVER_KEY") 572 797 parser.add_argument("--openai-key-file", required=True, help="file holding the OpenAI API key") 573 798 parser.add_argument("--access-token-file", help="file holding the token the page must present") 799 + parser.add_argument( 800 + "--matrix-webhook", 801 + help="Hermes webhook route (deliver_only to Matrix, signed with API_SERVER_KEY) that " 802 + "receives approvals no call can answer", 803 + ) 574 804 parser.add_argument("--config", help="JSON file: voice, model, idle_seconds, glossary, instructions") 575 805 parser.add_argument( 576 806 "--static", ··· 591 821 openai_key=Path(args.openai_key_file).read_text().strip(), 592 822 hermes_url=args.hermes_url, 593 823 hermes_key=Path(args.hermes_key_file).read_text().strip(), 824 + matrix_webhook=args.matrix_webhook, 594 825 ) 595 826 Handler.config = config 596 827 Handler.static = Path(args.static)
+37 -14
tools/hermes-voice/static/app.js
··· 315 315 this.answerApproval(id); 316 316 return; 317 317 } 318 + const replace = Boolean(this.delegation); 318 319 if (this.delegation) this.delegation.controller.abort(); 319 320 const controller = new AbortController(); 320 321 const delegation = { id, controller }; ··· 322 323 setStatus("Hermes is working…"); 323 324 324 325 await new Promise((resolve) => setTimeout(resolve, DELEGATION_SETTLE_MS)); 325 - if (this.delegation !== delegation) return; 326 + // A hang-up meanwhile sent these lines along itself. 327 + if (this.delegation !== delegation || this.closed) return; 326 328 const fresh = state.lines 327 329 .slice(state.delegated) 328 330 .filter((line) => line.speaker !== "hermes" && line.speaker !== "error"); ··· 342 344 const response = await api("/delegate", { 343 345 method: "POST", 344 346 headers: { "Content-Type": "application/json" }, 345 - body: JSON.stringify({ hermes_session: state.hermesSession, lines }), 347 + body: JSON.stringify({ 348 + hermes_session: state.hermesSession, 349 + lines, 350 + replace, 351 + }), 346 352 signal: controller.signal, 347 353 }); 348 354 if (!response.ok) ··· 360 366 buffered += value; 361 367 let newline; 362 368 while ((newline = buffered.indexOf("\n")) >= 0) { 363 - const item = JSON.parse(buffered.slice(0, newline)); 369 + const line = buffered.slice(0, newline); 364 370 buffered = buffered.slice(newline + 1); 371 + // Blank lines are the server's keepalives. 372 + if (!line.trim()) continue; 373 + const item = JSON.parse(line); 365 374 this.lastActivity = Date.now(); 366 375 if (item.type === "approval") { 367 376 this.askApproval(id, item); ··· 422 431 } 423 432 424 433 askApproval(id, item) { 425 - this.approval = { 426 - delegationId: id, 427 - runId: item.run_id, 428 - requestId: item.request_id, 429 - }; 434 + this.approval = { delegationId: id, code: item.code }; 430 435 const what = item.description || "run a command"; 431 436 setStatus("Hermes needs your approval"); 432 437 addLine( ··· 484 489 const response = await api("/approve", { 485 490 method: "POST", 486 491 headers: { "Content-Type": "application/json" }, 487 - body: JSON.stringify({ 488 - run_id: approval.runId, 489 - request_id: approval.requestId, 490 - choice, 491 - }), 492 + body: JSON.stringify({ code: approval.code, choice }), 492 493 }); 493 494 const result = await response.json(); 494 495 ok = response.ok; ··· 530 531 this.closeTimer = setTimeout(() => this.finish("close timed out"), 15000); 531 532 } 532 533 534 + // Tell the server the call is over, with what the user said since the last 535 + // delegation: a request spoken right before hanging up still reaches 536 + // Hermes. keepalive lets the request outlive the page. 537 + hangUp() { 538 + const lines = state.lines 539 + .slice(state.delegated) 540 + .filter( 541 + (line) => line.speaker === "user" || line.speaker === "assistant", 542 + ); 543 + state.delegated = state.lines.length; 544 + saveState(); 545 + api("/hangup", { 546 + method: "POST", 547 + keepalive: true, 548 + headers: { "Content-Type": "application/json" }, 549 + body: JSON.stringify({ hermes_session: state.hermesSession, lines }), 550 + }).catch(() => {}); 551 + } 552 + 533 553 finish(reason) { 534 554 if (this.closed) return; 535 555 this.closed = true; 536 556 clearTimeout(this.closeTimer); 537 557 clearInterval(this.idleTimer); 538 558 clearInterval(this.probe); 539 - // Only the stream is dropped; the Hermes run keeps going on the server. 559 + this.hangUp(); 560 + // Only the stream is dropped; the server keeps following the Hermes run. 540 561 if (this.delegation) this.delegation.controller.abort(); 541 562 this.audioContext?.close().catch(() => {}); 542 563 this.mic?.getTracks().forEach((track) => track.stop()); ··· 577 598 call.finish("failed to start"); 578 599 } 579 600 }); 601 + 602 + window.addEventListener("pagehide", () => call?.finish("page closed")); 580 603 581 604 document.getElementById("new").addEventListener("click", () => { 582 605 if (call) call.close();