271 lines
9.7 KiB
Markdown
271 lines
9.7 KiB
Markdown
---
|
|
name: tv-app-a11y-fallback
|
|
description: Use when driving a TV app (webOS SmartTV, Kinopoisk, etc.) where the accessibility tree is broken or empty — no interactive elements, buttons, links, or inputs in snapshots. Covers synthetic key events with webOS key codes, React fiber introspection to call onPress/onClick directly, focus detection via computed styles, and bundle analysis to discover key handling. Also use when agent-browser press/click/fill fail on a TV app.
|
|
allowed-tools: Bash(agent-browser:*), Bash(npx agent-browser:*)
|
|
---
|
|
|
|
# TV app control when the a11y tree is bad
|
|
|
|
TV apps (Kinopoisk SmartTV, Yandex OTT, etc.) are custom SPAs rendered as divs with
|
|
their own focus system. The accessibility tree is often useless:
|
|
|
|
```bash
|
|
agent-browser snapshot -i # → "(no interactive elements)"
|
|
```
|
|
|
|
There are no `<button>`, `<a>`, `<input>` elements, no `role` attributes, no
|
|
`tabindex`. `click @eN`, `fill`, and `press` from the core skill will not work.
|
|
This skill covers the fallback techniques.
|
|
|
|
## 1. Confirm the situation
|
|
|
|
```bash
|
|
# a11y tree empty?
|
|
agent-browser snapshot -i
|
|
|
|
# no standard interactive elements in DOM?
|
|
cat <<'EOF' | agent-browser eval --stdin
|
|
(() => {
|
|
const inputs = document.querySelectorAll('input, button, a, [role="button"], [role="link"]');
|
|
return {count: inputs.length};
|
|
})()
|
|
EOF
|
|
```
|
|
|
|
If both are empty/zero, you are in a custom-focus TV app. Proceed below.
|
|
|
|
## 2. Connect to the TV
|
|
|
|
The TV exposes a CDP session (e.g. `127.0.0.1:9998`). Use a named session:
|
|
|
|
```bash
|
|
export AGENT_BROWSER_SESSION="$(agent-browser session id --scope worktree --prefix mytask)"
|
|
agent-browser --cdp 9998 snapshot
|
|
```
|
|
|
|
Check what page/app is attached:
|
|
|
|
```bash
|
|
curl -s http://127.0.0.1:9998/json/list | python3 -m json.tool
|
|
```
|
|
|
|
## 3. Discover how the app handles keys
|
|
|
|
TV apps listen for `keydown` on `window`/`document` with **capture** and map
|
|
**webOS key codes** to logical keys. Find the app's JS bundles:
|
|
|
|
```bash
|
|
cat <<'EOF' | agent-browser eval --stdin
|
|
(() => Array.from(document.querySelectorAll('script[src]')).map(s => s.src))()
|
|
EOF
|
|
```
|
|
|
|
Download the main/platform bundles and grep for key handling:
|
|
|
|
```bash
|
|
curl -s -o main.js "<bundle-url>"
|
|
grep -o 'addEventListener("keydown"[^)]*' main.js | head
|
|
grep -o 'keyCode[^,;)]*' main.js | sort -u | head
|
|
```
|
|
|
|
Typical webOS key map (Yandex OTT / Kinopoisk SmartTV):
|
|
|
|
| Key | keyCode |
|
|
|-----|---------|
|
|
| SELECT (OK/Enter) | 13 |
|
|
| UP | 38 |
|
|
| DOWN | 40 |
|
|
| LEFT | 37 |
|
|
| RIGHT | 39 |
|
|
| BACK | 461 |
|
|
| PLAY | 415 |
|
|
| PAUSE | 19 |
|
|
| SEEK_FORWARD | 417 |
|
|
| SEEK_BACKWARD | 412 |
|
|
| STOP | 413 |
|
|
| CHANNEL_UP / CHANNEL_DOWN | 33 / 34 |
|
|
|
|
The platform handler typically looks like:
|
|
|
|
```js
|
|
window.addEventListener("keydown", handler, true); // capture!
|
|
// handler: n.preventDefault(); n.stopPropagation();
|
|
// key = keyMap[n.keyCode] || "Unknown"; emitter.emit({key, ...})
|
|
```
|
|
|
|
## 4. Send synthetic key events
|
|
|
|
`agent-browser press` sends trusted CDP events that may not reach the app's
|
|
custom handler. Instead, dispatch synthetic events on `window` (the handler is
|
|
attached with capture, so `window.dispatchEvent` triggers it):
|
|
|
|
```bash
|
|
cat <<'EOF' | agent-browser eval --stdin
|
|
(() => {
|
|
const ev = new KeyboardEvent('keydown', {key: 'ArrowRight', code: 'ArrowRight', keyCode: 39, which: 39, bubbles: true, cancelable: true});
|
|
window.dispatchEvent(ev);
|
|
return 'sent RIGHT';
|
|
})()
|
|
EOF
|
|
```
|
|
|
|
Helper script for repeated use:
|
|
|
|
```bash
|
|
cat <<'EOF' > /tmp/tv_key.sh
|
|
#!/bin/bash
|
|
# Usage: tv_key.sh <keyCode> <keyName>
|
|
export AGENT_BROWSER_SESSION="<session>"
|
|
cat <<EOS | agent-browser eval --stdin
|
|
(() => {
|
|
const ev = new KeyboardEvent('keydown', {key: '$2', code: '$2', keyCode: $1, which: $1, bubbles: true, cancelable: true});
|
|
window.dispatchEvent(ev);
|
|
return 'sent $2';
|
|
})()
|
|
EOS
|
|
sleep 1
|
|
EOF
|
|
chmod +x /tmp/tv_key.sh
|
|
```
|
|
|
|
Notes:
|
|
- Navigation keys (UP/DOWN/LEFT/RIGHT) usually work this way — the focus system
|
|
moves focus between elements.
|
|
- **SELECT may not trigger actions** even though navigation works. The focus
|
|
system routes navigation keys itself, but SELECT must reach the focused
|
|
element's `onKeyDown` → `onPress`. If it doesn't fire, use fiber
|
|
introspection (section 6) to call `onPress`/`onClick` directly.
|
|
- Some apps need `keyup` too (long-press path): dispatch both keydown and keyup.
|
|
|
|
## 5. Detect the focused element
|
|
|
|
TV apps indicate focus visually. Common indicators:
|
|
|
|
- **CSS transform scale** (focused card is scaled up, e.g. `matrix(1.07, ...)`):
|
|
```bash
|
|
cat <<'EOF' | agent-browser eval --stdin
|
|
(() => {
|
|
const cards = Array.from(document.querySelectorAll('[class*="<card-class>"]'));
|
|
return cards.map(c => ({text: (c.textContent||'').trim().slice(0,40), transform: getComputedStyle(c).transform}));
|
|
})()
|
|
EOF
|
|
```
|
|
- **Opacity** (unfocused items dimmed to 0.4, focused = 1)
|
|
- **Focus classes** (`focus`, `active`, `selected`, `current` in className)
|
|
- **React fiber `focused` prop** (see section 6)
|
|
|
|
## 6. React fiber introspection (the key technique)
|
|
|
|
React attaches fiber metadata to DOM elements. Keys look like
|
|
`__reactFiber$<hash>` and `__reactProps$<hash>`. Walk the fiber tree to find
|
|
components with `onPress`/`onClick`/`onKeyDown` handlers and call them directly.
|
|
|
|
```bash
|
|
cat <<'EOF' | agent-browser eval --stdin
|
|
(() => {
|
|
// Find a DOM element by text, then walk up the fiber tree
|
|
const els = Array.from(document.querySelectorAll('*'))
|
|
.filter(e => e.textContent && e.textContent.trim() === '<TEXT>' && e.children.length === 0);
|
|
const el = els[0];
|
|
const fiberKey = Object.keys(el).find(k => k.startsWith('__reactFiber'));
|
|
let fiber = el[fiberKey];
|
|
const chain = [];
|
|
for (let i = 0; i < 20 && fiber; i++) {
|
|
const t = fiber.type;
|
|
const name = typeof t === 'string' ? t : (t && (t.displayName || t.name)) || 'anon';
|
|
const props = fiber.memoizedProps || {};
|
|
chain.push({i, name, hasOnPress: typeof props.onPress === 'function', hasOnClick: typeof props.onClick === 'function'});
|
|
if (typeof props.onPress === 'function' || typeof props.onClick === 'function') break;
|
|
fiber = fiber.return;
|
|
}
|
|
return chain;
|
|
})()
|
|
EOF
|
|
```
|
|
|
|
Call the handler directly:
|
|
|
|
```bash
|
|
cat <<'EOF' | agent-browser eval --stdin
|
|
(() => {
|
|
const els = Array.from(document.querySelectorAll('*'))
|
|
.filter(e => e.textContent && e.textContent.trim() === '<TEXT>' && e.children.length === 0);
|
|
const el = els[0];
|
|
const fiberKey = Object.keys(el).find(k => k.startsWith('__reactFiber'));
|
|
let fiber = el[fiberKey];
|
|
for (let i = 0; i < 20 && fiber; i++) {
|
|
const t = fiber.type;
|
|
const name = typeof t === 'string' ? t : (t && (t.displayName || t.name)) || 'anon';
|
|
const props = fiber.memoizedProps || {};
|
|
if (typeof props.onPress === 'function') {
|
|
try { props.onPress(<order>); return 'onPress called on ' + name; }
|
|
catch (e) { return 'error: ' + e.message; }
|
|
}
|
|
if (typeof props.onClick === 'function') {
|
|
try { props.onClick(); return 'onClick called on ' + name; }
|
|
catch (e) { return 'error: ' + e.message; }
|
|
}
|
|
fiber = fiber.return;
|
|
}
|
|
return 'no handler found';
|
|
})()
|
|
EOF
|
|
```
|
|
|
|
Tips:
|
|
- `onPress` often takes an index/order argument (e.g. card index in a row).
|
|
Read `props.order` from the same fiber and pass it.
|
|
- `onClick` usually takes no arguments.
|
|
- If the handler is a closure over props (`e.onPress && e.onPress(e.order, n)`),
|
|
the real callback may live in a parent fiber — walk further up.
|
|
- After calling, wait 2-4s for the SPA to navigate, then re-snapshot.
|
|
|
|
## 7. Verify state changes
|
|
|
|
The app is a SPA — the URL won't change. Verify by checking the DOM text:
|
|
|
|
```bash
|
|
cat <<'EOF' | agent-browser eval --stdin
|
|
(() => {
|
|
const body = (document.body.innerText||'');
|
|
const v = document.querySelector('video');
|
|
return {
|
|
bodyStart: body.slice(0, 300),
|
|
videoPaused: v ? v.paused : null,
|
|
videoTime: v ? Math.round(v.currentTime) : null,
|
|
videoDuration: v ? Math.round(v.duration) : null,
|
|
videoSrc: v ? (v.currentSrc||'').slice(0, 100) : null
|
|
};
|
|
})()
|
|
EOF
|
|
```
|
|
|
|
For a video player: check `video.currentTime` progresses, `video.duration`
|
|
matches the expected episode length, and the player controls show the right
|
|
title/episode when revealed (press any key to show controls).
|
|
|
|
## 8. Troubleshooting
|
|
|
|
| Problem | Fix |
|
|
|---------|-----|
|
|
| `snapshot -i` empty | Normal for TV apps — use `eval` + fiber introspection |
|
|
| `press` does nothing | App uses custom key handler — dispatch synthetic events on `window` |
|
|
| Synthetic keys move focus but SELECT does nothing | Call `onPress`/`onClick` directly via fiber (section 6) |
|
|
| Element not found by text | Text may be split across nodes — search with `includes()` instead of exact match |
|
|
| Handler is a closure, real callback elsewhere | Walk further up the fiber tree (up to 20+ levels) |
|
|
| Page changed, refs stale | Re-snapshot / re-run eval — refs are assigned fresh each snapshot |
|
|
| `eval` says "Identifier already declared" | Wrap code in an IIFE: `(() => { ... })()` — eval state persists between calls |
|
|
| Video not playing | Check `video.paused`, `video.readyState`; wait for `readyState === 4` |
|
|
| Need to see the screen | `agent-browser screenshot` — but if the model can't view images, rely on DOM text + fiber state |
|
|
|
|
## 9. Worked example (Kinopoisk SmartTV, Dorohedoro)
|
|
|
|
1. `agent-browser --cdp 9998 snapshot` — app already on home screen
|
|
2. Found "Продолжить просмотр" row with Дорохедоро card (2 сезон, 7 серия)
|
|
3. Focus detection: card with `transform: matrix(1.07,...)` = focused
|
|
4. Called `onPress(0)` on the card fiber → player opened with episode 7
|
|
5. Called `onClick` on the "Серии" button fiber → episode list opened
|
|
6. Found "8 серия" card, called `onPress(7)` (order 7 = 8th episode) → player switched
|
|
7. Verified: player controls showed "Дорохедоро, 2 сезон, 8 серия", video playing
|
|
with duration 1408s (23 min, matching the episode)
|