kinopoisk-test/.pi/skills/tv-app-a11y-fallback/SKILL.md

name, description, allowed-tools

name	description	allowed-tools
tv-app-a11y-fallback	Use when driving a TV app (webOS SmartTV, Kinopoisk, etc.) where the accessibility tree is broken or empty — no interactive elements, buttons, links, or inputs in snapshots. Covers synthetic key events with webOS key codes, React fiber introspection to call onPress/onClick directly, focus detection via computed styles, and bundle analysis to discover key handling. Also use when agent-browser press/click/fill fail on a TV app.	Bash(agent-browser:*), Bash(npx agent-browser:*)

TV app control when the a11y tree is bad

TV apps (Kinopoisk SmartTV, Yandex OTT, etc.) are custom SPAs rendered as divs with their own focus system. The accessibility tree is often useless:

agent-browser snapshot -i    # → "(no interactive elements)"

There are no <button>, <a>, <input> elements, no role attributes, no tabindex. click @eN, fill, and press from the core skill will not work. This skill covers the fallback techniques.

1. Confirm the situation

# a11y tree empty?
agent-browser snapshot -i

# no standard interactive elements in DOM?
cat <<'EOF' | agent-browser eval --stdin
(() => {
  const inputs = document.querySelectorAll('input, button, a, [role="button"], [role="link"]');
  return {count: inputs.length};
})()
EOF

If both are empty/zero, you are in a custom-focus TV app. Proceed below.

2. Connect to the TV

The TV exposes a CDP session (e.g. 127.0.0.1:9998). Use a named session:

export AGENT_BROWSER_SESSION="$(agent-browser session id --scope worktree --prefix mytask)"
agent-browser --cdp 9998 snapshot

Check what page/app is attached:

curl -s http://127.0.0.1:9998/json/list | python3 -m json.tool

3. Discover how the app handles keys

TV apps listen for keydown on window/document with capture and map webOS key codes to logical keys. Find the app's JS bundles:

cat <<'EOF' | agent-browser eval --stdin
(() => Array.from(document.querySelectorAll('script[src]')).map(s => s.src))()
EOF

Download the main/platform bundles and grep for key handling:

curl -s -o main.js "<bundle-url>"
grep -o 'addEventListener("keydown"[^)]*' main.js | head
grep -o 'keyCode[^,;)]*' main.js | sort -u | head

Typical webOS key map (Yandex OTT / Kinopoisk SmartTV):

Key	keyCode
SELECT (OK/Enter)	13
UP	38
DOWN	40
LEFT	37
RIGHT	39
BACK	461
PLAY	415
PAUSE	19
SEEK_FORWARD	417
SEEK_BACKWARD	412
STOP	413
CHANNEL_UP / CHANNEL_DOWN	33 / 34

The platform handler typically looks like:

window.addEventListener("keydown", handler, true);  // capture!
// handler: n.preventDefault(); n.stopPropagation();
//          key = keyMap[n.keyCode] || "Unknown"; emitter.emit({key, ...})

4. Send synthetic key events

agent-browser press sends trusted CDP events that may not reach the app's custom handler. Instead, dispatch synthetic events on window (the handler is attached with capture, so window.dispatchEvent triggers it):

cat <<'EOF' | agent-browser eval --stdin
(() => {
  const ev = new KeyboardEvent('keydown', {key: 'ArrowRight', code: 'ArrowRight', keyCode: 39, which: 39, bubbles: true, cancelable: true});
  window.dispatchEvent(ev);
  return 'sent RIGHT';
})()
EOF

Helper script for repeated use:

cat <<'EOF' > /tmp/tv_key.sh
#!/bin/bash
# Usage: tv_key.sh <keyCode> <keyName>
export AGENT_BROWSER_SESSION="<session>"
cat <<EOS | agent-browser eval --stdin
(() => {
  const ev = new KeyboardEvent('keydown', {key: '$2', code: '$2', keyCode: $1, which: $1, bubbles: true, cancelable: true});
  window.dispatchEvent(ev);
  return 'sent $2';
})()
EOS
sleep 1
EOF
chmod +x /tmp/tv_key.sh

Notes:

• Navigation keys (UP/DOWN/LEFT/RIGHT) usually work this way — the focus system moves focus between elements.
• SELECT may not trigger actions even though navigation works. The focus system routes navigation keys itself, but SELECT must reach the focused element's onKeyDown → onPress. If it doesn't fire, use fiber introspection (section 6) to call onPress/onClick directly.
• Some apps need keyup too (long-press path): dispatch both keydown and keyup.

5. Detect the focused element

TV apps indicate focus visually. Common indicators:

• CSS transform scale (focused card is scaled up, e.g. matrix(1.07, ...)):

cat <<'EOF' | agent-browser eval --stdin
(() => {
  const cards = Array.from(document.querySelectorAll('[class*="<card-class>"]'));
  return cards.map(c => ({text: (c.textContent||'').trim().slice(0,40), transform: getComputedStyle(c).transform}));
})()
EOF

• Opacity (unfocused items dimmed to 0.4, focused = 1)
• Focus classes (focus, active, selected, current in className)
• React fiber focused prop (see section 6)

6. React fiber introspection (the key technique)

React attaches fiber metadata to DOM elements. Keys look like __reactFiber$<hash> and __reactProps$<hash>. Walk the fiber tree to find components with onPress/onClick/onKeyDown handlers and call them directly.

cat <<'EOF' | agent-browser eval --stdin
(() => {
  // Find a DOM element by text, then walk up the fiber tree
  const els = Array.from(document.querySelectorAll('*'))
    .filter(e => e.textContent && e.textContent.trim() === '<TEXT>' && e.children.length === 0);
  const el = els[0];
  const fiberKey = Object.keys(el).find(k => k.startsWith('__reactFiber'));
  let fiber = el[fiberKey];
  const chain = [];
  for (let i = 0; i < 20 && fiber; i++) {
    const t = fiber.type;
    const name = typeof t === 'string' ? t : (t && (t.displayName || t.name)) || 'anon';
    const props = fiber.memoizedProps || {};
    chain.push({i, name, hasOnPress: typeof props.onPress === 'function', hasOnClick: typeof props.onClick === 'function'});
    if (typeof props.onPress === 'function' || typeof props.onClick === 'function') break;
    fiber = fiber.return;
  }
  return chain;
})()
EOF

Call the handler directly:

cat <<'EOF' | agent-browser eval --stdin
(() => {
  const els = Array.from(document.querySelectorAll('*'))
    .filter(e => e.textContent && e.textContent.trim() === '<TEXT>' && e.children.length === 0);
  const el = els[0];
  const fiberKey = Object.keys(el).find(k => k.startsWith('__reactFiber'));
  let fiber = el[fiberKey];
  for (let i = 0; i < 20 && fiber; i++) {
    const t = fiber.type;
    const name = typeof t === 'string' ? t : (t && (t.displayName || t.name)) || 'anon';
    const props = fiber.memoizedProps || {};
    if (typeof props.onPress === 'function') {
      try { props.onPress(<order>); return 'onPress called on ' + name; }
      catch (e) { return 'error: ' + e.message; }
    }
    if (typeof props.onClick === 'function') {
      try { props.onClick(); return 'onClick called on ' + name; }
      catch (e) { return 'error: ' + e.message; }
    }
    fiber = fiber.return;
  }
  return 'no handler found';
})()
EOF

Tips:

• onPress often takes an index/order argument (e.g. card index in a row). Read props.order from the same fiber and pass it.
• onClick usually takes no arguments.
• If the handler is a closure over props (e.onPress && e.onPress(e.order, n)), the real callback may live in a parent fiber — walk further up.
• After calling, wait 2-4s for the SPA to navigate, then re-snapshot.

7. Verify state changes

The app is a SPA — the URL won't change. Verify by checking the DOM text:

cat <<'EOF' | agent-browser eval --stdin
(() => {
  const body = (document.body.innerText||'');
  const v = document.querySelector('video');
  return {
    bodyStart: body.slice(0, 300),
    videoPaused: v ? v.paused : null,
    videoTime: v ? Math.round(v.currentTime) : null,
    videoDuration: v ? Math.round(v.duration) : null,
    videoSrc: v ? (v.currentSrc||'').slice(0, 100) : null
  };
})()
EOF

For a video player: check video.currentTime progresses, video.duration matches the expected episode length, and the player controls show the right title/episode when revealed (press any key to show controls).

8. Troubleshooting

Problem	Fix
snapshot -i empty	Normal for TV apps — use eval + fiber introspection
press does nothing	App uses custom key handler — dispatch synthetic events on window
Synthetic keys move focus but SELECT does nothing	Call onPress/onClick directly via fiber (section 6)
Element not found by text	Text may be split across nodes — search with includes() instead of exact match
Handler is a closure, real callback elsewhere	Walk further up the fiber tree (up to 20+ levels)
Page changed, refs stale	Re-snapshot / re-run eval — refs are assigned fresh each snapshot
eval says "Identifier already declared"	Wrap code in an IIFE: (() => { ... })() — eval state persists between calls
Video not playing	Check video.paused, video.readyState; wait for readyState === 4
Need to see the screen	agent-browser screenshot — but if the model can't view images, rely on DOM text + fiber state

9. Worked example (Kinopoisk SmartTV, Dorohedoro)

1. agent-browser --cdp 9998 snapshot — app already on home screen
2. Found "Продолжить просмотр" row with Дорохедоро card (2 сезон, 7 серия)
3. Focus detection: card with transform: matrix(1.07,...) = focused
4. Called onPress(0) on the card fiber → player opened with episode 7
5. Called onClick on the "Серии" button fiber → episode list opened
6. Found "8 серия" card, called onPress(7) (order 7 = 8th episode) → player switched
7. Verified: player controls showed "Дорохедоро, 2 сезон, 8 серия", video playing with duration 1408s (23 min, matching the episode)