kinopoisk-test/.pi/skills/tv-app-a11y-fallback/SKILL.md

---
name: tv-app-a11y-fallback
description: Use when driving a TV app where the accessibility tree is broken or empty — no interactive elements, buttons, links, or inputs in snapshots. Covers synthetic key events with webOS key codes, React fiber introspection to call onPress/onClick directly, focus detection via computed styles, DOM MouseEvent bypass for non-focus UI, and bundle analysis to discover key handling. Also use when agent-browser press/click/fill fail on a TV app. End-to-end Kinopoisk recipe lives in the kinopoisk-flow skill.
allowed-tools: Bash(agent-browser:*), Bash(npx agent-browser:*)
---

# TV app control when the a11y tree is bad

TV apps (Kinopoisk SmartTV, Yandex OTT, etc.) are custom SPAs rendered as divs with
their own focus system. The accessibility tree is often useless:

```bash
agent-browser snapshot -i    # → "(no interactive elements)"
```

There are no `<button>`, `<a>`, `<input>` elements, no `role` attributes, no
`tabindex`. `click @eN`, `fill`, and `press` from the core skill will not work.
This skill covers the fallback techniques.

## 1. Confirm the situation

```bash
# a11y tree empty?
agent-browser snapshot -i

# no standard interactive elements in DOM?
cat <<'EOF' | agent-browser eval --stdin
(() => {
  const inputs = document.querySelectorAll('input, button, a, [role="button"], [role="link"]');
  return {count: inputs.length};
})()
EOF
```

If both are empty/zero, you are in a custom-focus TV app. Proceed below.

## 2. Connect to the TV

The TV exposes a CDP session (e.g. `127.0.0.1:9998`). Use a named session:

```bash
export AGENT_BROWSER_SESSION="$(agent-browser session id --scope worktree --prefix mytask)"
agent-browser --cdp 9998 snapshot
```

Check what page/app is attached:

```bash
curl -s http://127.0.0.1:9998/json/list | python3 -m json.tool
```

## 3. Discover how the app handles keys

TV apps listen for `keydown` on `window`/`document` with **capture** and map
**webOS key codes** to logical keys. Find the app's JS bundles:

```bash
cat <<'EOF' | agent-browser eval --stdin
(() => Array.from(document.querySelectorAll('script[src]')).map(s => s.src))()
EOF
```

Download the main/platform bundles and grep for key handling:

```bash
curl -s -o main.js "<bundle-url>"
grep -o 'addEventListener("keydown"[^)]*' main.js | head
grep -o 'keyCode[^,;)]*' main.js | sort -u | head
```

Typical webOS key map (Yandex OTT / Kinopoisk SmartTV):

| Key | keyCode |
|-----|---------|
| SELECT (OK/Enter) | 13 |
| UP | 38 |
| DOWN | 40 |
| LEFT | 37 |
| RIGHT | 39 |
| BACK | 461 |
| PLAY | 415 |
| PAUSE | 19 |
| SEEK_FORWARD | 417 |
| SEEK_BACKWARD | 412 |
| STOP | 413 |
| CHANNEL_UP / CHANNEL_DOWN | 33 / 34 |

The platform handler typically looks like:

```js
window.addEventListener("keydown", handler, true);  // capture!
// handler: n.preventDefault(); n.stopPropagation();
//          key = keyMap[n.keyCode] || "Unknown"; emitter.emit({key, ...})
```

## 4. Send synthetic key events

`agent-browser press` sends trusted CDP events that may not reach the app's
custom handler. Instead, dispatch synthetic events on `window` (the handler is
attached with capture, so `window.dispatchEvent` triggers it):

```bash
cat <<'EOF' | agent-browser eval --stdin
(() => {
  const ev = new KeyboardEvent('keydown', {key: 'ArrowRight', code: 'ArrowRight', keyCode: 39, which: 39, bubbles: true, cancelable: true});
  window.dispatchEvent(ev);
  return 'sent RIGHT';
})()
EOF
```

Helper script for repeated use:

```bash
cat <<'EOF' > /tmp/tv_key.sh
#!/bin/bash
# Usage: tv_key.sh <keyCode> <keyName>
export AGENT_BROWSER_SESSION="<session>"
cat <<EOS | agent-browser eval --stdin
(() => {
  const ev = new KeyboardEvent('keydown', {key: '$2', code: '$2', keyCode: $1, which: $1, bubbles: true, cancelable: true});
  window.dispatchEvent(ev);
  return 'sent $2';
})()
EOS
sleep 1
EOF
chmod +x /tmp/tv_key.sh
```

Notes:
- Navigation keys (UP/DOWN/LEFT/RIGHT) usually work this way — the focus system
  moves focus between elements.
- **SELECT may not trigger actions** even though navigation works. The focus
  system routes navigation keys itself, but SELECT must reach the focused
  element's `onKeyDown` → `onPress`. If it doesn't fire, use fiber
  introspection (section 6) to call `onPress`/`onClick` directly.
- Some apps need `keyup` too (long-press path): dispatch both keydown and keyup.
- **Don't rely on visual focus indicators** (scale transform) — many apps keep
  the old transform stale. Trust behavior, not styling.
- For UI elements that the focus system doesn't drive (nav sidebar, keyboard
  letter cells, modals), skip the key handler chain entirely and dispatch
  `MouseEvent`s (`mousedown`/`mouseup`/`click`) directly on the DOM element —
  this works because React listens for synthetic clicks on its root.

## 5. Detect the focused element

TV apps indicate focus visually. Common indicators:

- **CSS transform scale** (focused card is scaled up, e.g. `matrix(1.07, ...)`):
```bash
cat <<'EOF' | agent-browser eval --stdin
(() => {
  const cards = Array.from(document.querySelectorAll('[class*="<card-class>"]'));
  return cards.map(c => ({text: (c.textContent||'').trim().slice(0,40), transform: getComputedStyle(c).transform}));
})()
EOF
```
- **Opacity** (unfocused items dimmed to 0.4, focused = 1)
- **Focus classes** (`focus`, `active`, `selected`, `current` in className)
- **React fiber `focused` prop** (see section 6)

## 6. React fiber introspection (the key technique)

React attaches fiber metadata to DOM elements. Keys look like
`__reactFiber$<hash>` and `__reactProps$<hash>`. Walk the fiber tree to find
components with `onPress`/`onClick`/`onKeyDown` handlers and call them directly.

```bash
cat <<'EOF' | agent-browser eval --stdin
(() => {
  // Find a DOM element by text, then walk up the fiber tree
  const els = Array.from(document.querySelectorAll('*'))
    .filter(e => e.textContent && e.textContent.trim() === '<TEXT>' && e.children.length === 0);
  const el = els[0];
  const fiberKey = Object.keys(el).find(k => k.startsWith('__reactFiber'));
  let fiber = el[fiberKey];
  const chain = [];
  for (let i = 0; i < 20 && fiber; i++) {
    const t = fiber.type;
    const name = typeof t === 'string' ? t : (t && (t.displayName || t.name)) || 'anon';
    const props = fiber.memoizedProps || {};
    chain.push({i, name, hasOnPress: typeof props.onPress === 'function', hasOnClick: typeof props.onClick === 'function'});
    if (typeof props.onPress === 'function' || typeof props.onClick === 'function') break;
    fiber = fiber.return;
  }
  return chain;
})()
EOF
```

Call the handler directly:

```bash
cat <<'EOF' | agent-browser eval --stdin
(() => {
  const els = Array.from(document.querySelectorAll('*'))
    .filter(e => e.textContent && e.textContent.trim() === '<TEXT>' && e.children.length === 0);
  const el = els[0];
  const fiberKey = Object.keys(el).find(k => k.startsWith('__reactFiber'));
  let fiber = el[fiberKey];
  for (let i = 0; i < 20 && fiber; i++) {
    const t = fiber.type;
    const name = typeof t === 'string' ? t : (t && (t.displayName || t.name)) || 'anon';
    const props = fiber.memoizedProps || {};
    if (typeof props.onPress === 'function') {
      try { props.onPress(<order>); return 'onPress called on ' + name; }
      catch (e) { return 'error: ' + e.message; }
    }
    if (typeof props.onClick === 'function') {
      try { props.onClick(); return 'onClick called on ' + name; }
      catch (e) { return 'error: ' + e.message; }
    }
    fiber = fiber.return;
  }
  return 'no handler found';
})()
EOF
```

Tips:
- `onPress` often takes an index/order argument (e.g. card index in a row).
  Read `props.order` from the same fiber and pass it.
- `onClick` usually takes no arguments.
- If the handler is a closure over props (`e.onPress && e.onPress(e.order, n)`),
  the real callback may live in a parent fiber — walk further up.
- After calling, wait 2-4s for the SPA to navigate, then re-snapshot.

## 7. Verify state changes

The app is a SPA — the URL won't change. Verify by checking the DOM text:

```bash
cat <<'EOF' | agent-browser eval --stdin
(() => {
  const body = (document.body.innerText||'');
  const v = document.querySelector('video');
  return {
    bodyStart: body.slice(0, 300),
    videoPaused: v ? v.paused : null,
    videoTime: v ? Math.round(v.currentTime) : null,
    videoDuration: v ? Math.round(v.duration) : null,
    videoSrc: v ? (v.currentSrc||'').slice(0, 100) : null
  };
})()
EOF
```

For a video player: check `video.currentTime` progresses, `video.duration`
matches the expected episode length, and the player controls show the right
title/episode when revealed (press any key to show controls).

## 8. Troubleshooting

| Problem | Fix |
|---------|-----|
| `snapshot -i` empty | Normal for TV apps — use `eval` + fiber introspection |
| `press` does nothing | App uses custom key handler — dispatch synthetic events on `window` |
| Synthetic keys move focus but SELECT does nothing | Call `onPress`/`onClick` directly via fiber (section 6) |
| Element not found by text | Text may be split across nodes — search with `includes()` instead of exact match |
| Handler is a closure, real callback elsewhere | Walk further up the fiber tree (up to 20+ levels) |
| Page changed, refs stale | Re-snapshot / re-run eval — refs are assigned fresh each snapshot |
| `eval` says "Identifier already declared" | Wrap code in an IIFE: `(() => { ... })()` — eval state persists between calls |
| Video not playing | Check `video.paused`, `video.readyState`; wait for `readyState === 4`. Or call `video.play()` directly with `muted=true` to bypass the play gesture (section 10) |
| Sidebar item click does nothing via fiber | Use DOM `MouseEvent` (`mousedown`/`mouseup`/`click`) directly on the element instead — fiber `onKeyDown`/`onFocus` may not trigger navigation for nav items |
| Keyboard has no input element | On-screen keyboard is a grid of clickable `<div>` letters — type by dispatching mouse events on each letter div (section 10) |
| Focus indicator (scale transform) seems stuck | Stale transform stays on the old element — don't rely on it; trust which element accepts Enter |
| Text appears duplicated (sidebar + main) | Filter by `getBoundingClientRect().x` to disambiguate — sidebar icons at `x=60`, expanded text at `x=150`, keyboard letters at `x>500` |
| Need to see the screen | `agent-browser screenshot` — but if the model can't view images, rely on DOM text + fiber state |

## 9. Worked example (Kinopoisk SmartTV, Dorohedoro)

1. `agent-browser --cdp 9998 snapshot` — app already on home screen
2. Found "Продолжить просмотр" row with Дорохедоро card (2 сезон, 7 серия)
3. Focus detection: card with `transform: matrix(1.07,...)` = focused
4. Called `onPress(0)` on the card fiber → player opened with episode 7
5. Called `onClick` on the "Серии" button fiber → episode list opened
6. Found "8 серия" card, called `onPress(7)` (order 7 = 8th episode) → player switched
7. Verified: player controls showed "Дорохедоро, 2 сезон, 8 серия", video playing
   with duration 1408s (23 min, matching the episode)

## 10. Kinopoisk-specific findings (what worked)

Kinopoisk SmartTV (app id `com.webos.app.test.youtube`) — notes only,
end-to-end recipe lives in the `kinopoisk-flow` skill.

### What works

- **Navigation keys** (`ArrowUp/Down/Left/Right` on `window`): reliably move
  focus inside main content (rows, hero carousels). Don't use for sidebar
  nav items.
- **Profile picker**: `FocusView.onKeyDown({keyCode:13})` works at fiber
  depth ~4. No direct `onPress`/`onClick` on profile cards.
- **Sidebar nav items** ("Поиск", "Главное", etc.): **DOM `MouseEvent`
  (`mousedown`/`mouseup`/`click`)** at the element's center. Fiber
  `onKeyDown`/`onFocus` does not navigate. Sidebar has two views —
  collapsed icons at `x=60`, expanded text at `x=150` — filter by `x`.
- **On-screen keyboard** (no `<input>` exists): Russian ЙЦУКЕНГ layout is a
  grid of clickable `<div>` letters around `x>500, y 150–400`. Click each
  letter via `MouseEvent`. No space key — letters only, `123`, `⌫`,
  language switch. Search is fuzzy/substring, so partial queries
  (e.g. just "Поезд") already surface the right film.
- **Search-result cards**: `onPress(props.order)` on fiber at depth ~4
  (component name is a 1-char string like `_`). Pass `props.order` as the
  arg.
- **"Смотреть" button on details**: multiple wrappers have `onClick`. The
  outermost (i=3) is a no-op; the inner one (i=4, component named like
  `s`/`ct`) does navigation. Try i=3 first, fall back to i=4.
- **18+ age gate** ("Вам исполнилось 18 лет?"): standard `onClick` at
  fiber depth ~3 (component named `v`).
- **Player "Пропустить" (Skip) button**: **no React fiber handlers** —
  rendered in an isolated sub-tree or iframe overlay. The video plays
  underneath anyway; just hide the overlay via `display: none` on the
  ancestors of the leaf "Пропустить" text, or wait the countdown out.
  Do not waste time on fiber walks or synthetic Enter — none of it
  fires. The HLS source URL pattern is
  `https://strm.yandex.ru/vod/vh-ottenc-converted/vod-content/<id>...`.

### What doesn't work / traps

- **`agent-browser press`**: doesn't reach the app's custom key handler.
  Use synthetic `KeyboardEvent` on `window` (section 4) or DOM
  `MouseEvent`.
- **Focus indicator (CSS scale transform)**: stale — the previous focused
  element keeps its `matrix(1.05,...)` transform even after focus
  moves. Don't trust it; trust which element accepts Enter.
- **Sidebar `x` ambiguity**: text like "Поиск" appears both as collapsed
  icon (`x=60`) and expanded label (`x=150`). Always filter by `x` to
  pick the right one.
- **Fiber component names**: many components have obfuscated
  1-character names (`v`, `s`, `ct`, `_`, `J`). Don't rely on name for
  identification — match by `props.order` or by traversing parent
  fiber context.
- **Text-split nodes**: account names and button labels may appear in
  nested `<span>`s; filter by `e.children.length === 0` to find the leaf
  text node before walking fibers.