Self-repair
Every 30 minutes the box checks whether it is getting slow. If it is, it clears leftover browsers and memory buildup by itself. It never touches anything that is being used.
Where you see it
Almost never, which is the goal. The Health tab has a Self-repair check card. It shows what the numbers looked like on the last check. If the same repair is needed three days running, the daily checkup raises it.
What happens, step by step
Self-repair has four layers.
- Prevention. Every browser the agents use runs inside
browser-session.sh(a wrapper). The wrapper always closes the browser when the job ends. A separate watchdog kills the browser if the program driving it disappears. - Automatic repair. Every 30 minutes,
box-fresh --if-neededreads the box's pressure numbers. If none is over its line, it does nothing and says nothing. If one is, it cleans up. - Manual repair.
sudo box-freshruns the same cleanup plus a restart of the services. It is for when a person or the fixer decides it is needed. - Repair, then report. The daily checkup runs safe repairs itself, checks they worked, and emails only what is left.
What trips the automatic repair:
| Signal | Line |
|---|---|
| Some work waiting on memory (1-minute average) | over 10% |
| All work stalled on memory (1-minute average) | over 3% |
| Work waiting on the disk (1-minute average) | over 25% |
| Swap used (disk standing in for memory) | over 300 MB |
| Free memory | under 250 MB |
| Browser programs running | over 12 |
Only memory and disk waiting trip it. A busy processor does not. Several real jobs running at once is not "broken", and cleaning cannot fix it.
What powers it
| Part | What it does |
|---|---|
box-fresh.sh | The repair. Closes stray browsers, clears swap when safe. |
agent-fresh.timer | Runs the automatic repair every 30 minutes. |
browser-session.sh | The prevention wrapper every browser runs inside. |
selfrepair_card.py | Reads the same numbers and writes the Health tab card. Decides nothing, repairs nothing. |
fresh-watcher.timer | Lets the chat agent ask for memory relief without having admin rights. |
The card is a separate program on purpose. The repair must stay simple and silent. Teaching it to also write a card would give its "do nothing" path new work and a new way to fail.
Why it works this way
A crashed job once left 15 browsers running. The box ran out of memory and started swapping. That is why prevention is the layer that matters most. The rest is cleanup after a failure that should not happen.
An automatic action needs a way to know what is in use. If it cannot know, it must not act.
So the automatic repair refuses to:
- close a browser in the middle of a job,
- clear swap unless it fits in free memory,
- restart the chat while an answer is being written,
- restart anything at all. Restarts are the disruptive part, kept for manual repair.
How it knows a browser is in use. Two signals. First, a running browser-session process, matched on its exact program name. Second, a heartbeat file the job touches every 5 seconds. The repair waits for 120 seconds of silence before it treats a browser as abandoned. That is slower than the watchdog on purpose: this guard matters most when the box is starved and a busy job might stutter.
Connected to
- The daily checkup: the fourth layer, and where repeats are raised.
- The Health tab: where the Self-repair card shows.
- Watch the browser: the browsers this protects.
- How changes reach the box: the same request-file idea, used for applying changes.