Architecture & Specs

fjFlow Caret Tracking Technology Guide

This document explains the caret tracking technologies implemented in fjFlow. It details how the utility tracks user inp...

fjFlow Caret Tracking Technology Guide

This document explains the caret tracking technologies implemented in fjFlow. It details how the utility tracks user input cursor coordinates (caret positions) with extremely low latency and low CPU load, allowing indicators to render accurately.

On the Windows OS, legacy Win32 apps (e.g. Notepad) run alongside Chromium-based web browsers (Chrome, Edge), Electron-based environments (VS Code), and UWP (Universal Windows Platform) applications. To track carets across all these diverse GUI frameworks, fjFlow implements an optimized, hybrid coordinates retrieval pipeline.

1. System Architecture & Flow

fjFlow runs a polling timer (WatchCaret) to monitor caret positions on the active window. At the core of this system is the GetCaret function, which determines coordinates using the following pipeline and fallback strategies:

graph TD
    A[WatchCaret Timer - 30ms] --> B{Active Window Check}
    B -- Chromium Browser --> C[1. GetCaretViaUIA]
    B -- Other Applications --> D[2. GetCaretViaOleacc]
    C -- Success --> E[Save Cache & Render Overlays]
    C -- Failure --> D
    D -- Success --> E
    D -- Failure --> F[3. GetCaretViaUIA]
    F -- Success --> E
    F -- Failure --> G[4. CaretGetPos Native API]
    G -- Success --> E
    G -- Failure --> H[Tracking Failed / Retain Previous Coordinates]

1.1. Cache Control & Thread Load Optimization

Querying caret positions (especially COM operations in UI Automation or oleacc) can be CPU-intensive. Running these calls continuously every 30ms degrades performance. To resolve this, fjFlow incorporates the following cache-control optimizations:

  • Focus Change Detection: If the active window (currHwnd) or focused control (focusHwnd) changes, or if the window size/position changes, the utility clears the coordinates cache and performs a fresh query.
  • Input-Idle Suspension: If no keyboard inputs, mouse movements, mouse scrolls, or mouse clicks are detected for more than 300ms, the utility assumes the caret is idle. It bypasses new UIA/oleacc calls and returns the cached coordinates (lastX, lastY). This reduces idle CPU usage to almost 0%.
  • Browser Type Caching: Determinations of whether a target window is Google Chrome or Microsoft Edge (matching class Chrome_WidgetWin_1 and process names chrome.exe/msedge.exe) are cached in a hash map (isTargetBrowserCache), avoiding repeated string matching.

2. Caret Coordinate Extraction Methods

2.1. Active Accessibility (MSAA / oleacc)

For traditional Win32 desktop apps, the MSAA-based GetCaretViaOleacc is used:

GetCaretViaOleacc(&X, &Y, hwnd) {
    ...
    hrCaret := DllCall("oleacc\AccessibleObjectFromWindow"
        , "Ptr", hwnd
        , "UInt", 0xFFFFFFF8 ; OBJID_CARET
        , "Ptr", IID
        , "Ptr*", &pacc
        , "Int")
    ...
    oAcc := ComValue(9, pacc, 1)
    oAcc.accLocation(..., 0)
    ...
}
  • Mechanism: Queries the IAccessible COM interface bound to the caret object using the AccessibleObjectFromWindow API in oleacc.dll with Object ID OBJID_CARET (0xFFFFFFF8). It then calls the accLocation method to retrieve the screen coordinates (X, Y) and bounding box dimensions.
  • Advantages: Very fast and lightweight. It works with high accuracy in Notepad (notepad.exe) and traditional Win32 text fields.
  • Thread Block Prevention: MSAA calls can occasionally hang if an application is busy, blocking the utility thread. To prevent this, during initialization or on the first call to a specific window handle, the utility avoids waiting for COM responses. It returns false immediately and attempts asynchronous retrieval on the next polling frame.
  • Sanity Checks: Coordinate results are filtered. Only values that are non-zero and contain valid screen boundaries (e.g. Y coordinate > -5000 to filter out off-screen/minimized window states) are accepted.

2.2. UI Automation (UIA)

For modern frameworks that render custom graphics instead of Win32 GUI controls—such as Chromium browsers (Chrome/Edge), Electron apps (VS Code), and UWP windows—oleacc cannot retrieve caret data. The utility falls back to Windows UI Automation (UIA) via GetCaretViaUIA.

A. Text Pattern Traversal Order

To find caret positions within the UIA tree, the utility traverses text ranges in the following order:

  1. TextPattern2 (ID: 10024): Attempts to obtain the TextPattern2 interface from the focused element and calls GetCaretRange to directly fetch the text range where the caret resides.
  2. TextPattern (ID: 10014): For legacy platforms that do not support TextPattern2, the utility fetches the traditional TextPattern interface and calls GetSelection to obtain the active text selection range. Even if no characters are selected, a text range object with length 0 is returned, marking the caret position.

B. Memory Leak Prevention for SAFEARRAY

When calling UIA APIs to get bounding boxes, the API returns coordinates wrapped in a double-precision floating-point array (SAFEARRAY). Handling raw COM pointers directly in AutoHotkey can cause memory leaks if SAFEARRAY objects are not destroyed. fjFlow resolves this via the following code:

psa := 0
ComCall(10, caretRange, "Ptr*", &psa) ; GetBoundingRectangles
if psa {
    Rect := ComValue(0x2005, psa) ; Wrap pointer as VT_ARRAY | VT_R8 (0x2005)
    ComObjFlags(Rect, 1)          ; Set F_OWNVALUE (1) to enable automatic disposal
    if Rect.MaxIndex() >= 3 {
        X := Round(Rect[0])
        Y := Round(Rect[1])
        return true
    }
}
  • Wraps the raw SAFEARRAY pointer (psa) inside an AutoHotkey COM wrapper object using ComValue(0x2005, psa).
  • Configures ComObjFlags(Rect, 1) (F_OWNVALUE = 1) to instruct the AutoHotkey engine to call SafeArrayDestroy automatically once the wrapper object goes out of scope, preventing memory leaks.

C. DPI Scaling Alignment

When running on high-DPI monitors or multi-monitor setups with different DPI scales (e.g. 125% and 150%), UIA coordinates can deviate from actual pixel positions.

prevDpi := DllCall("SetThreadDpiAwarenessContext", "ptr", -3, "ptr")
...
; (Execute UIA caret query)
...
if (prevDpi)
    DllCall("SetThreadDpiAwarenessContext", "ptr", prevDpi, "ptr")

Before querying UIA, the utility thread calls SetThreadDpiAwarenessContext with -3 (DPI_AWARENESS_CONTEXT_PER_MONITOR_AWARE_V2) to match physical screen coordinates. The previous DPI context is restored inside a finally block once the query completes.

2.3. Native Fallback (CaretGetPos)

If both UIA and MSAA queries fail, the utility falls back to the native AutoHotkey command CaretGetPos:

CoordMode "Caret", "Screen"
if CaretGetPos(&aX, &aY) {
    X := aX, Y := aY
    return true
}

This queries standard Windows OS caret APIs. While it is stable and safe, it does not support modern UI engines. It serves as a final safety net for traditional controls.

3. Advanced Coordinated Mechanisms

3.1. Chromium Accessibility Wake-Up

Chromium-based engines (Chrome, Edge) often suspend accessibility services (UIA/MSAA responses) when idle to save resources. When a target window is focused, fjFlow sends a message to activate accessibility features:

EnsureChromiumAccessibility(topHwnd) {
    ...
    for cls in ["Chrome_RenderWidgetHostHWND1", "Chrome_RenderWidgetHostHWND"] {
        childHwnd := ControlGetHwnd(cls, "ahk_id " topHwnd)
        if (childHwnd) {
            SendMessage(WM_GETOBJECT, 0, 1, , "ahk_id " childHwnd)
            sentAny := true
        }
    }
    ...
}
  • WM_GETOBJECT Message: Sends WM_GETOBJECT (0x003D) with lParam = 1 (OBJID_CLIENT) to the Chromium child renderer window (Chrome_RenderWidgetHostHWND). This notifies Chromium that an assistive tool is connected, triggering the engine to build its internal accessibility tree.
  • Debounce Waiting: Building the UIA tree can take up to several hundred milliseconds. To prevent repeated queries during tree initialization, the utility pauses caret checks for 8 ticks (approx. 240ms) after sending the message.
  • Self-Healing Recovery: If caret coordinates cannot be obtained for 100 consecutive frames (approx. 3 seconds) in a Chromium window (which can occur after tab swaps or engine resets), the utility clears its accessibility flags and re-sends the WM_GETOBJECT message to rebuild the tree.

3.2. Scroll and Click Synchronization

In Web apps such as Google Docs, scrolling coordinates change the visual caret position without modifying text character indices. UIA text selection events may fail to report these scroll offsets.

To track these changes, the utility hooks mouse clicks and wheel scrolls globally:

~WheelUp::
~WheelDown:: {
    global LastInputTime := A_TickCount
    global LastScrollTime := A_TickCount
    global IsScrollHidden := true
}

~LButton::
~MButton::
~RButton:: {
    global LastInputTime := A_TickCount
}
  • Scroll Updates: When a mouse scroll occurs, LastScrollTime is updated. Inside GetCaret, if a scroll event is detected, the utility invalidates the cached coordinates immediately, triggering a fresh UIA query on the next frame to keep the indicator aligned.
  • Click Sync: Clicking updates LastInputTime, instantly waking the caret tracker from idle suspension to snap overlays to the new click coordinate.

4. Summary

The caret coordinate tracker in fjFlow adapts to different application environments through the following designs:

  1. Hybrid UIA & MSAA Extraction for broad application support.
  2. WM_GETOBJECT Signaling to activate Chromium accessibility trees with self-healing recovery.
  3. F_OWNVALUE Flag Management to automatically manage COM array memory.
  4. SetThreadDpiAwarenessContext to resolve multi-monitor DPI scaling alignment issues.
  5. Idle Suspension Modes to reduce CPU cycles combined with click/scroll hooks for positioning.