kinopoisk-test/.pi/skills/tv-app-a11y-fallback/SKILL.md

name, description, allowed-tools

name	description	allowed-tools
tv-app-a11y-fallback	Use when driving a TV app where the accessibility tree is broken or empty — no interactive elements, buttons, links, or inputs in snapshots. Covers synthetic key events with webOS key codes, React fiber introspection to call onPress/onClick directly, focus detection via computed styles, DOM MouseEvent bypass for non-focus UI, and bundle analysis to discover key handling. Also use when agent-browser press/click/fill fail on a TV app. End-to-end Kinopoisk recipe lives in the kinopoisk-flow skill.	Bash(agent-browser:*), Bash(npx agent-browser:*)

TV app control when the a11y tree is bad

TV apps (Kinopoisk SmartTV, Yandex OTT, etc.) are custom SPAs rendered as divs with their own focus system. The accessibility tree is often useless:

agent-browser snapshot -i    # → "(no interactive elements)"

There are no <button>, <a>, <input> elements, no role attributes, no tabindex. click @eN, fill, and press from the core skill will not work. This skill covers the fallback techniques.

1. Confirm the situation

# a11y tree empty?
agent-browser snapshot -i

# no standard interactive elements in DOM?
cat <<'EOF' | agent-browser eval --stdin
(() => {
  const inputs = document.querySelectorAll('input, button, a, [role="button"], [role="link"]');
  return {count: inputs.length};
})()
EOF

If both are empty/zero, you are in a custom-focus TV app. Proceed below.

2. Connect to the TV

The TV exposes a CDP session (e.g. 127.0.0.1:9998). Use a named session:

export AGENT_BROWSER_SESSION="$(agent-browser session id --scope worktree --prefix mytask)"
agent-browser --cdp 9998 snapshot

Check what page/app is attached:

curl -s http://127.0.0.1:9998/json/list | python3 -m json.tool

3. Discover how the app handles keys

TV apps listen for keydown on window/document with capture and map webOS key codes to logical keys. Find the app's JS bundles:

cat <<'EOF' | agent-browser eval --stdin
(() => Array.from(document.querySelectorAll('script[src]')).map(s => s.src))()
EOF

Download the main/platform bundles and grep for key handling:

curl -s -o main.js "<bundle-url>"
grep -o 'addEventListener("keydown"[^)]*' main.js | head
grep -o 'keyCode[^,;)]*' main.js | sort -u | head

Typical webOS key map (Yandex OTT / Kinopoisk SmartTV):

Key	keyCode
SELECT (OK/Enter)	13
UP	38
DOWN	40
LEFT	37
RIGHT	39
BACK	461
PLAY	415
PAUSE	19
SEEK_FORWARD	417
SEEK_BACKWARD	412
STOP	413
CHANNEL_UP / CHANNEL_DOWN	33 / 34

The platform handler typically looks like:

window.addEventListener("keydown", handler, true);  // capture!
// handler: n.preventDefault(); n.stopPropagation();
//          key = keyMap[n.keyCode] || "Unknown"; emitter.emit({key, ...})

4. Send synthetic key events

agent-browser press sends trusted CDP events that may not reach the app's custom handler. Instead, dispatch synthetic events on window (the handler is attached with capture, so window.dispatchEvent triggers it):

cat <<'EOF' | agent-browser eval --stdin
(() => {
  const ev = new KeyboardEvent('keydown', {key: 'ArrowRight', code: 'ArrowRight', keyCode: 39, which: 39, bubbles: true, cancelable: true});
  window.dispatchEvent(ev);
  return 'sent RIGHT';
})()
EOF

Helper script for repeated use:

cat <<'EOF' > /tmp/tv_key.sh
#!/bin/bash
# Usage: tv_key.sh <keyCode> <keyName>
export AGENT_BROWSER_SESSION="<session>"
cat <<EOS | agent-browser eval --stdin
(() => {
  const ev = new KeyboardEvent('keydown', {key: '$2', code: '$2', keyCode: $1, which: $1, bubbles: true, cancelable: true});
  window.dispatchEvent(ev);
  return 'sent $2';
})()
EOS
sleep 1
EOF
chmod +x /tmp/tv_key.sh

Notes:

• Navigation keys (UP/DOWN/LEFT/RIGHT) usually work this way — the focus system moves focus between elements.
• SELECT may not trigger actions even though navigation works. The focus system routes navigation keys itself, but SELECT must reach the focused element's onKeyDown → onPress. If it doesn't fire, use fiber introspection (section 6) to call onPress/onClick directly.
• Some apps need keyup too (long-press path): dispatch both keydown and keyup.
• Don't rely on visual focus indicators (scale transform) — many apps keep the old transform stale. Trust behavior, not styling.
• For UI elements that the focus system doesn't drive (nav sidebar, keyboard letter cells, modals), skip the key handler chain entirely and dispatch MouseEvents (mousedown/mouseup/click) directly on the DOM element — this works because React listens for synthetic clicks on its root.

5. Detect the focused element

TV apps indicate focus visually. Common indicators:

• CSS transform scale (focused card is scaled up, e.g. matrix(1.07, ...)):

cat <<'EOF' | agent-browser eval --stdin
(() => {
  const cards = Array.from(document.querySelectorAll('[class*="<card-class>"]'));
  return cards.map(c => ({text: (c.textContent||'').trim().slice(0,40), transform: getComputedStyle(c).transform}));
})()
EOF

• Opacity (unfocused items dimmed to 0.4, focused = 1)
• Focus classes (focus, active, selected, current in className)
• React fiber focused prop (see section 6)

6. React fiber introspection (the key technique)

React attaches fiber metadata to DOM elements. Keys look like __reactFiber$<hash> and __reactProps$<hash>. Walk the fiber tree to find components with onPress/onClick/onKeyDown handlers and call them directly.

cat <<'EOF' | agent-browser eval --stdin
(() => {
  // Find a DOM element by text, then walk up the fiber tree
  const els = Array.from(document.querySelectorAll('*'))
    .filter(e => e.textContent && e.textContent.trim() === '<TEXT>' && e.children.length === 0);
  const el = els[0];
  const fiberKey = Object.keys(el).find(k => k.startsWith('__reactFiber'));
  let fiber = el[fiberKey];
  const chain = [];
  for (let i = 0; i < 20 && fiber; i++) {
    const t = fiber.type;
    const name = typeof t === 'string' ? t : (t && (t.displayName || t.name)) || 'anon';
    const props = fiber.memoizedProps || {};
    chain.push({i, name, hasOnPress: typeof props.onPress === 'function', hasOnClick: typeof props.onClick === 'function'});
    if (typeof props.onPress === 'function' || typeof props.onClick === 'function') break;
    fiber = fiber.return;
  }
  return chain;
})()
EOF

Call the handler directly:

cat <<'EOF' | agent-browser eval --stdin
(() => {
  const els = Array.from(document.querySelectorAll('*'))
    .filter(e => e.textContent && e.textContent.trim() === '<TEXT>' && e.children.length === 0);
  const el = els[0];
  const fiberKey = Object.keys(el).find(k => k.startsWith('__reactFiber'));
  let fiber = el[fiberKey];
  for (let i = 0; i < 20 && fiber; i++) {
    const t = fiber.type;
    const name = typeof t === 'string' ? t : (t && (t.displayName || t.name)) || 'anon';
    const props = fiber.memoizedProps || {};
    if (typeof props.onPress === 'function') {
      try { props.onPress(<order>); return 'onPress called on ' + name; }
      catch (e) { return 'error: ' + e.message; }
    }
    if (typeof props.onClick === 'function') {
      try { props.onClick(); return 'onClick called on ' + name; }
      catch (e) { return 'error: ' + e.message; }
    }
    fiber = fiber.return;
  }
  return 'no handler found';
})()
EOF

Tips:

• onPress often takes an index/order argument (e.g. card index in a row). Read props.order from the same fiber and pass it.
• onClick usually takes no arguments.
• If the handler is a closure over props (e.onPress && e.onPress(e.order, n)), the real callback may live in a parent fiber — walk further up.
• After calling, wait 2-4s for the SPA to navigate, then re-snapshot.

7. Verify state changes

The app is a SPA — the URL won't change. Verify by checking the DOM text:

cat <<'EOF' | agent-browser eval --stdin
(() => {
  const body = (document.body.innerText||'');
  const v = document.querySelector('video');
  return {
    bodyStart: body.slice(0, 300),
    videoPaused: v ? v.paused : null,
    videoTime: v ? Math.round(v.currentTime) : null,
    videoDuration: v ? Math.round(v.duration) : null,
    videoSrc: v ? (v.currentSrc||'').slice(0, 100) : null
  };
})()
EOF

For a video player: check video.currentTime progresses, video.duration matches the expected episode length, and the player controls show the right title/episode when revealed (press any key to show controls).

8. Troubleshooting

Problem	Fix
snapshot -i empty	Normal for TV apps — use eval + fiber introspection
press does nothing	App uses custom key handler — dispatch synthetic events on window
Synthetic keys move focus but SELECT does nothing	Call onPress/onClick directly via fiber (section 6)
Element not found by text	Text may be split across nodes — search with includes() instead of exact match
Handler is a closure, real callback elsewhere	Walk further up the fiber tree (up to 20+ levels)
Page changed, refs stale	Re-snapshot / re-run eval — refs are assigned fresh each snapshot
eval says "Identifier already declared"	Wrap code in an IIFE: (() => { ... })() — eval state persists between calls
Video not playing	Check video.paused, video.readyState; wait for readyState === 4. Or call video.play() directly with muted=true to bypass the play gesture (section 10)
Sidebar item click does nothing via fiber	Use DOM MouseEvent (mousedown/mouseup/click) directly on the element instead — fiber onKeyDown/onFocus may not trigger navigation for nav items
Keyboard has no input element	On-screen keyboard is a grid of clickable <div> letters — type by dispatching mouse events on each letter div (section 10)
Focus indicator (scale transform) seems stuck	Stale transform stays on the old element — don't rely on it; trust which element accepts Enter
Text appears duplicated (sidebar + main)	Filter by getBoundingClientRect().x to disambiguate — sidebar icons at x=60, expanded text at x=150, keyboard letters at x>500
Need to see the screen	agent-browser screenshot — but if the model can't view images, rely on DOM text + fiber state

9. Worked example (Kinopoisk SmartTV, Dorohedoro)

1. agent-browser --cdp 9998 snapshot — app already on home screen
2. Found "Продолжить просмотр" row with Дорохедоро card (2 сезон, 7 серия)
3. Focus detection: card with transform: matrix(1.07,...) = focused
4. Called onPress(0) on the card fiber → player opened with episode 7
5. Called onClick on the "Серии" button fiber → episode list opened
6. Found "8 серия" card, called onPress(7) (order 7 = 8th episode) → player switched
7. Verified: player controls showed "Дорохедоро, 2 сезон, 8 серия", video playing with duration 1408s (23 min, matching the episode)

10. Kinopoisk-specific findings (what worked)

Kinopoisk SmartTV (app id com.webos.app.test.youtube) — notes only, end-to-end recipe lives in the kinopoisk-flow skill.

What works

• Navigation keys (ArrowUp/Down/Left/Right on window): reliably move focus inside main content (rows, hero carousels). Don't use for sidebar nav items.
• Profile picker: FocusView.onKeyDown({keyCode:13}) works at fiber depth ~4. No direct onPress/onClick on profile cards.
• Sidebar nav items ("Поиск", "Главное", etc.): DOM MouseEvent (mousedown/mouseup/click) at the element's center. Fiber onKeyDown/onFocus does not navigate. Sidebar has two views — collapsed icons at x=60, expanded text at x=150 — filter by x.
• On-screen keyboard (no <input> exists): Russian ЙЦУКЕНГ layout is a grid of clickable <div> letters around x>500, y 150–400. Click each letter via MouseEvent. No space key — letters only, 123, ⌫, language switch. Search is fuzzy/substring, so partial queries (e.g. just "Поезд") already surface the right film.
• Search-result cards: onPress(props.order) on fiber at depth ~4 (component name is a 1-char string like _). Pass props.order as the arg.
• "Смотреть" button on details: multiple wrappers have onClick. The outermost (i=3) is a no-op; the inner one (i=4, component named like s/ct) does navigation. Try i=3 first, fall back to i=4.
• 18+ age gate ("Вам исполнилось 18 лет?"): standard onClick at fiber depth ~3 (component named v).
• Player "Пропустить" (Skip) button: no React fiber handlers — rendered in an isolated sub-tree or iframe overlay. The video plays underneath anyway; just hide the overlay via display: none on the ancestors of the leaf "Пропустить" text, or wait the countdown out. Do not waste time on fiber walks or synthetic Enter — none of it fires. The HLS source URL pattern is https://strm.yandex.ru/vod/vh-ottenc-converted/vod-content/<id>....

What doesn't work / traps

• agent-browser press: doesn't reach the app's custom key handler. Use synthetic KeyboardEvent on window (section 4) or DOM MouseEvent.
• Focus indicator (CSS scale transform): stale — the previous focused element keeps its matrix(1.05,...) transform even after focus moves. Don't trust it; trust which element accepts Enter.
• Sidebar x ambiguity: text like "Поиск" appears both as collapsed icon (x=60) and expanded label (x=150). Always filter by x to pick the right one.
• Fiber component names: many components have obfuscated 1-character names (v, s, ct, _, J). Don't rely on name for identification — match by props.order or by traversing parent fiber context.
• Text-split nodes: account names and button labels may appear in nested <span>s; filter by e.children.length === 0 to find the leaf text node before walking fibers.