Thanks Ben, this makes a lot of sense, and it fits a direction we already started on.
The first step will be in the next version: each agent's header now has a small plug button that tells you whether that session really loaded the AgentsRoom tools. Hover it to see every tool server with its tool count, the status hooks and the skills injected at launch (with a warning if one was edited since). Click it to get the full list of tools by name, grouped by server and by family (backlog, memory, mailbox, etc.). So you can see exactly what each agent can see, for every supported CLI.
What is still missing is the control part you describe: turning tool families on or off so they stop taking context (for people who use their own memory, backlog or delegation system), and possibly splitting them into separate servers you can switch off one by one. We are keeping this ticket open for that and will add it in a future version. Your three points are noted, and we will tell you here when it ships.