windows gui 自动化 Windows GUI自动化控制工具之python uiAutomation

对 Windows GUI进行自动化控制的工具有很多，比如pywinauto、pyautogui、pywin32、Autoit、airtest、UIAutomation等，UI Automation API是微软提供的自动化框架，可在支持 Windows Presentation Foundation (WPF) 的所有操作系统上使用，支持的应用类型更多。本文介绍封装了UI Automation API的Python uiautomation 模块的使用方法。

目录

环境准备uiautomation安装进程查看器inspect.exeAccessibility Insights

控件对象模型uiautomation库示例控制计算器

参考文档

Python uiautomation 模块由yinkaisheng 开发，封装了微软 UI Automation API，支持自动化Win32，MFC，WPF，Modern UI(Metro UI), Qt, IE, Firefox, Chrome和基于Electron开发的应用程序。

环境准备

uiautomation安装

最新版uiautomation2.0只支持Python 3版本，但不要使用3.7.6和3.8.1这两个版本，因为comtypes包在这两个版本中不能正常工作。

pip安装uiautomation：

$ pip install uiautomation

检查是否安装成功：

$ pip list | findstr uiautomation

uiautomation 2.0.18

安装完成后，在Python的Scripts(我的路径为C:\Program Files\Python37\Scripts)目录中会有一个文件automation.py，是用来枚举控件树结构的一个脚本。

可运行 automation.py -h查看命令帮助:

$ python automation.py -h

UIAutomation 2.0.18 (Python 3.7.2, 64 bit)

usage

-h show command help

-t delay time, default 3 seconds, begin to enumerate after Value seconds, this must be an integer

you can delay a few seconds and make a window active so automation can enumerate the active window

-d enumerate tree depth, this must be an integer, if it is null, enumerate the whole tree

-r enumerate from root:Desktop window, if it is null, enumerate from foreground window

-f enumerate from focused control, if it is null, enumerate from foreground window

-c enumerate the control under cursor, if depth is < 0, enumerate from its ancestor up to depth

-a show ancestors of the control under cursor

-n show control full name, if it is null, show first 30 characters of control's name in console,

always show full name in log file @AutomationLog.txt

-p show process id of controls

if UnicodeError or LookupError occurred when printing,

try to change the active code page of console window by using chcp or see the log file @AutomationLog.txt

chcp, get current active code page

chcp 936, set active code page to gbk

chcp 65001, set active code page to utf-8

examples:

automation.py -t3

automation.py -t3 -r -d1 -m -n

automation.py -c -t3

进程查看器

对 Windows GUI进行自动化控制需要使用进程查看器工具对GUI界面元素进行定位，定位工具有很多，这里推荐使用微软提供的inspect.exe 或者 Accessibility Insights 这两款工具。

inspect.exe

inspect.exe 是 Windows SDK 自带的一个进程查看器，可以用来查看系统正在运行的进程信息、模块、线程、堆栈跟踪等详细数据。

Windows SDK下载地址为：https://developer.microsoft.com/en-us/windows/downloads/sdk-archive/

建议直接到这里下载inspect.exe：https://github.com/yinkaisheng/Python-UIAutomation-for-Windows/tree/master/inspect

64位系统版本的inspect.exe也可以点击这里下载。

Accessibility Insights

Accessibility Insights 是微软开发的一款辅助功能测试工具。它可以帮助开发者测试 web 应用、Windows 桌面应用和 Android 应用的可访问性，确保这些应用程序符合无障碍标准。

Accessibility Insights获取的控件属性信息没有inspect.exe全面，使用起来更加流畅。下载为：https://accessibilityinsights.io/downloads/

控件对象模型

微软 UIAutomation API定义了支持的控件类型和对应的模型（Pattern），所有支持的控件类型可参考：https://learn.microsoft.com/en-us/windows/win32/winauto/uiauto-controlpatternmapping

控件类型必须支持的模型可选模型Does not supportButtonNoneExpandCollapse, Invoke, Toggle, ValueNoneCalendarGrid, TableScroll, SelectionValueCheckBoxToggleNoneNoneEditNoneRangeValue, Text, ValueNoneListNoneGrid, MultipleView, Scroll, SelectionTableListItemSelectionItemCustomNavigation, ExpandCollapse, GridItem, Invoke, ScrollItem, Toggle, ValueNoneMenuNoneNoneNoneMenuBarNoneDock, ExpandCollapse, TransformNoneMenuItemNoneExpandCollapse, Invoke, SelectionItem, ToggleNoneRadioButtonSelectionItemNoneToggleSplitButtonExpandCollapse, InvokeNoneNoneTabSelectionScrollNoneTabItemSelectionItemNoneInvokeTableGrid, GridItem, Table, TableItemNoneNoneTextNoneGridItem, TableItem, TextValueTitleBarNoneNoneNoneToolBarNoneDock, ExpandCollapse, TransformNone

python uiautomation库对UIAutomation API定义的各个Control和Pattern进行了封装。

下面来看使用python uiautomation操作Windows自带计算器的例子。

uiautomation库示例

控制计算器

可以使用inspect.exe来定位计算器元素：

示例脚本如下：

import os

import uiautomation as auto

import subprocess

class uiautoCalc(Loggers):

"""uiautomation控制计算器

"""

def __init__(self):

super().__init__()

self.logger = Loggers().myLogger()

auto.uiautomation.DEBUG_SEARCH_TIME =True

auto.uiautomation.SetGlobalSearchTimeout(2) # 设置全局搜索超时时间

self.calcWindow = auto.WindowControl(searchDepth=1, Name='计算器', desc='计算器窗口') # 计算器窗口

if not self.calcWindow.Exists(0,0):

subprocess.Popen('calc')# 设置窗口前置

self.calcWindow = auto.WindowControl(

searchDepth=1, Name='计算器', desc='计算器窗口')

self.calcWindow.SetActive() # 激活窗口

self.calcWindow.SetTopmost(True) # 设置为顶层

def gotoScientific(self):

self.calcWindow.ButtonControl(AutomationId='TogglePaneButton', desc='打开导航').Click(waitTime=0.01)

self.calcWindow.ListItemControl(AutomationId='Scientific', desc='选择科学计算器').Click(waitTime=0.01)

clearButton = self.calcWindow.ButtonControl(AutomationId='clearEntryButton', desc='点击CE清空输入')

if clearButton.Exists(0,0):

clearButton.Click(waitTime=0)

else:

self.calcWindow.ButtonControl(AutomationId='clearButton', desc='点击C清空输入').Click(waitTime=0.01)

def getKeyControl(self):

automationId2key ={'num0Button':'0','num1Button':'1','num2Button':'2','num3Button':'3','num4Button':'4','num5Button':'5','num6Button':'6','num7Button':'7','num8Button':'8','num9Button':'9','decimalSeparatorButton':'.','plusButton':'+','minusButton':'-','multiplyButton':'*','divideButton':'/','equalButton':'=','openParenthesisButton':'(','closeParenthesisButton':')'}

calckeys = self.calcWindow.GroupControl(ClassName='LandmarkTarget')

keyControl ={}

for control, depth in auto.WalkControl(calckeys, maxDepth=3):

if control.AutomationId in automationId2key:

self.logger.info(control.AutomationId)

keyControl[automationId2key[control.AutomationId]]= control

return keyControl

def calculate(self, expression, keyControl):

expression =''.join(expression.split())

if not expression.endswith('='):

expression +='='

for char in expression:

keyControl[char].Click(waitTime=0)

self.calcWindow.SendKeys('{Ctrl}c', waitTime=0.1)

return auto.GetClipboardText()

def calc_demo(self):

"""计算器示例

:return :

"""

self.gotoScientific() # 选择科学计算器

keyControl = self.getKeyControl() # 获取按键控件

result = self.calculate('(1 + 2 - 3) * 4 / 5.6 - 7', keyControl)

print('(1 + 2 - 3) * 4 / 5.6 - 7 =', result)

self.calcWindow.CaptureToImage('calc.png', x=7, y=0, width=-14, height=-7) # 截图

self.calcWindow.GetWindowPattern().Close() # 关闭计算机

if __name__ == "__main__":

ui = uiautoCalc()

ui.calc_demo()

脚本执行动图：

参考文档

https://github.com/pywinauto/pywinauto https://cloud.tencent.com/developer/article/2213048 https://github.com/yinkaisheng/Python-UIAutomation-for-Windows Python UIAutomation文档：https://github.com/yinkaisheng/Python-UIAutomation-for-Windows/blob/master/readme_cn.md https://www.cnblogs.com/Yinkaisheng/p/3444132.html GitHub - jacexh/pyautoit: Python binding for AutoItX3.dll GitHub - mhammond/pywin32: Python for Windows (pywin32) Extensions Accessibility tools - Inspect - Win32 apps | Microsoft Learn Accessibility Insights

--THE END--

好文推荐