Sorry about the formatting - first time posting :-)
This is the actual code - I have an extension on CGPoint that makes it possible to simply provide the numbers without the labels (I'm lazy that way ;-) ).
Not being a very experienced coder, I took the code from https://stackoverflow.com/questions/27484330/simulate-keypress-using-swift, with some changes. That unfortunately means I don't really know the answer as to "what I want to do with hid"- I just figured it was necessary to effect the leftclick.
As for the correct creation of the event, a "po eventDown!.location" nicely came back with (100.0, 75.0), as I would expect.
What I find strange is that it worked perfectly fine yesterday, when I uploaded the code to the AppStore ('UpointDriver' - the Mac counterpart of the Upoint iPhone app that turns the iPhone into an 'interactive laserpointer' that positions the cursor where you physically point the phone) - before upgrading to Big Sur 11.5. Even when I download that uploaded UpointDriver now (with Big Sur 11.5), the left-click works fine. Compiling and executing it with Xcode, however, gives problems...
Thanks for all the help.
erik_banning
Users receive reputation points by contributing quality content to the forums.
Posts include post and replies published on the forums.
Post authors can mark replies as Accepted to indicate successful solutions.
Apple Developer Forums admins can mark replies as Apple Recommended to indicate an approved solution
Post
Replies
Boosts
Views
Activity
I read somewhere that disabling Sandbox is not an option if I want to distribute via AppStore, unfortunately.
As for your code: am I correct in understanding that this would work when I know the windowNumber the Event is meant for, and that it only works in windows the App controls? If so: I'm looking for ways to simulate a mouseclick outside my App's window - the idea being that my counterpart iPhone app acts as a remote mouse that can click anywhere - inside or outside the driver-app's window.
For example, I would like to activate a window on the Mac when the user uses the iPhone App to position the cursor on that window, and "leftclicks" on it using the iPhone. The Mac code should then activate that window.
I got a similar problem since updating to BigSur 11.5 concerning keyboard events. Previously, emulating a keystroke worked fine (code swiped from https://stackoverflow.com/questions/27484330/simulate-keypress-using-swift) but, at least from within Xcode, it no longer does. The code I use there was
let source = CGEventSource(stateID: CGEventSourceStateID.hidSystemState)
let numberKey: UInt16 = keyCode(key) // e.g., 11 for 'b'
let keyDown = CGEvent(keyboardEventSource: source, virtualKey: numberKey, keyDown: true)
let keyUp = CGEvent(keyboardEventSource: source, virtualKey: numberKey, keyDown: false)
let loc = CGEventTapLocation.cgSessionEventTap
keyDown?.post(tap: loc)
keyUp?.post(tap: loc)
I got the windowNumber - thanks - but still can't get it to work. I wrote a test function that I would think should activate a window that's positioned on the left side of the screen (whereas the Xcode or App window are to the right, so "out of the way").
func tryToActivateWindow() {
print("begin -----------")
let mousePosition = NSPoint(x: 5,y: 200) // This corresponds to a visible window that's not part of this App
print("Mouse coordinates (\(mousePosition.x),\(mousePosition.y))")
let screenHeight = NSScreen.main!.frame.height
print("Screenheight: \(screenHeight)")
var screenPosition: NSPoint = mousePosition
screenPosition.y = screenHeight - mousePosition.y
print("Screen coordinates (\(screenPosition.x),\(screenPosition.y))")
let numberOfHitWindow = NSWindow.windowNumber(at: screenPosition, belowWindowWithWindowNumber: 0)
print("Window hit: \(numberOfHitWindow)")
CGDisplayMoveCursorToPoint(0, mousePosition) // Just for visual reference
print("Creating MouseDown event")
let newMouseDownEvent = NSEvent.mouseEvent(
with: .leftMouseDown,
location: NSPoint(x:0, y:0), // this should be in the hitWindow
modifierFlags: NSEvent.ModifierFlags.control,
timestamp: ProcessInfo.processInfo.systemUptime,
windowNumber: numberOfHitWindow,
context: nil,
eventNumber: 0, //don't know what else to use
clickCount: 1,
pressure: 1)
self.perform(#selector(NSResponder.self.mouseDown(with:)), with: newMouseDownEvent)
print("Sleeping for a bit")
Thread.sleep(forTimeInterval: 0.1)
print("Woke up")
print("Creating MouseUp event")
let newMouseUpEvent = NSEvent.mouseEvent(
with: .leftMouseUp,
location: NSPoint(x:0, y:0), // this should be in the hitWindow
modifierFlags: NSEvent.ModifierFlags.control,
timestamp: ProcessInfo.processInfo.systemUptime,
windowNumber: numberOfHitWindow,
context: nil,
eventNumber: 1, //don't know what else to use
clickCount: 1,
pressure: 1)
self.perform(#selector(NSResponder.self.mouseUp(with:)), with: newMouseUpEvent)
print("end -----------")
}
It produces the following output:
begin -----------
Mouse coordinates (5.0,200.0)
Screenheight: 1120.0
Screen coordinates (5.0,920.0)
Window hit: 125
Creating MouseDown event
Sleeping for a bit
Woke up
Creating MouseUp event
end -----------
The cursor gets put in the right window, but the window isn't activated as hoped...
That works, but you don't need the mouseEvents anymore. The switchToApp is all you need to activate the window the mouse is aimed at.
Thing is: the point was to feed the leftclick to that window, so that my iPhone user can activate buttons etc on any window he chooses...
That's indeed what I expected. Yet, if I execute the findWindow() of the following code
func switchToAppAndClick(withWindow windowNumber: Int32) {
let options = CGWindowListOption(arrayLiteral: CGWindowListOption.excludeDesktopElements, CGWindowListOption.optionOnScreenOnly)
let windowListInfo = CGWindowListCopyWindowInfo(options, CGWindowID(0))
guard let infoList = windowListInfo as NSArray? as? [[String: AnyObject]] else { return }
if let window = infoList.first(where: { ($0["kCGWindowNumber"] as? Int32) == windowNumber}), let pid = window["kCGWindowOwnerPID"] as? Int32 {
let app = NSRunningApplication(processIdentifier: pid)
app?.activate(options: .activateIgnoringOtherApps)
let inWindowClickpoint = CGPoint(x: 100 , y:200) // a clickable point in the targeted window
print("creating eventMouseDown and Up events")
let eventMouseDown = CGEvent(mouseEventSource: nil, mouseType: .leftMouseDown, mouseCursorPosition: inWindowClickpoint, mouseButton: .left)
let eventMouseUp = CGEvent(mouseEventSource: nil, mouseType: .leftMouseUp, mouseCursorPosition: inWindowClickpoint, mouseButton: .left)
print("posting \(eventMouseDown) to pid \(pid)")
eventMouseDown?.postToPid(pid)
Thread.sleep(forTimeInterval: 0.1)
print("posting \(eventMouseUp) to pid \(pid)")
eventMouseUp?.postToPid(pid)
}
}
func findWindow() {
let mousePosition = NSPoint(x: 100,y: 350) // This corresponds to a visible window that's not part of this App
print("Mouse coordinates (\(mousePosition.x),\(mousePosition.y))")
let screenHeight = NSScreen.main!.frame.height
var screenPosition: NSPoint = mousePosition
screenPosition.y = screenHeight - mousePosition.y
print("Screen coordinates (\(screenPosition.x),\(screenPosition.y))")
let numberOfHitWindow = NSWindow.windowNumber(at: screenPosition, belowWindowWithWindowNumber: 0)
CGDisplayMoveCursorToPoint(0, mousePosition) // Just for visual reference
print("Window hit: \(numberOfHitWindow); switching to switchToAppAndClick")
switchToAppAndClick(withWindow: Int32(numberOfHitWindow))
}
I get as output
Mouse coordinates (100.0,350.0)
Screen coordinates (100.0,770.0)
Window hit: 1245; switching to switchToAppAndClick
creating eventMouseDown and Up events
posting Optional(<CGEvent 0x6000024ce480>) to pid 2176
posting Optional(<CGEvent 0x6000024ce4e0>) to pid 2176
... and, although the targeted window is selected, I do not get the expected "click" action on the targeted window. Moreover, the value ofinWindowClickpoint doesn't seem to have any influence at all.
inWindowClickpoint is in window's coordinates, and not in absolute screen position. One way or another, it doesn't seem to have any effect anyway.
Also, using .post(tap: .cghidEventTap) instead of .postToPid(pid) doesn't solve the problem either, I'm afraid: the window gets activated, but the item at the cursor position isn't "clicked"...
No luck still, I'm afraid. The code I have now is:
func switchToAppAndClick(withWindow windowNumber: Int32) {
let options = CGWindowListOption(arrayLiteral: CGWindowListOption.excludeDesktopElements, CGWindowListOption.optionOnScreenOnly)
let windowListInfo = CGWindowListCopyWindowInfo(options, CGWindowID(0))
guard let infoList = windowListInfo as NSArray? as? [[String: AnyObject]] else { return }
if let window = infoList.first(where: { ($0["kCGWindowNumber"] as? Int32) == windowNumber}), let pid = window["kCGWindowOwnerPID"] as? Int32 {
let app = NSRunningApplication(processIdentifier: pid)
app?.activate(options: .activateIgnoringOtherApps)
let inWindowClickpoint = CGPoint(x: 100 , y:200) // a clickable point in the targeted window coords
print("creating eventMouseDown and Up events")
let source = CGEventSource.init(stateID: .hidSystemState)
let position = CGPoint(x: 75, y: 100)
let eventDown = CGEvent(mouseEventSource: source, mouseType: .leftMouseDown, mouseCursorPosition: position , mouseButton: .left)
let eventUp = CGEvent(mouseEventSource: source, mouseType: .leftMouseUp, mouseCursorPosition: position , mouseButton: .left)
print("posting \(eventUp)")
eventDown?.post(tap: .cghidEventTap)
Thread.sleep(forTimeInterval: 0.1)
print("posting \(eventDown)")
eventUp?.post(tap: .cghidEventTap)
}
}
Same thing: window gets activated, but the control the cursor is aimed at is not - when I 'physically execute' a leftclick, the window responds as expected. The events are printed out as non-nil, by the way.
First and foremost: thanks much for your help. It’s highly appreciated. Maybe I’ll wait a bit longer before burning a ticket. Someone else might have an idea?
I guess it’s possible I may be doing something wrong….Does it work for you? That is: activating a control ‘by mouse’ on a foreign window?
In any case, This functionality is critical for a ‘remote mouse’ like the Upoint interactive laser pointer, so if push comes to shove: I guess I need to burn that ticket…
That works!
That said, I experimented a bit more, and it turns out the problem may be elsewhere than where we were looking:
Every time I make changes to the code and recompile, I have to go to Preferences-Security and Privacy-Accessibility and de-check and re-check my App's permissions there. If I don't, the code only activates the window but doesn't activate any controls on there (note that I used a DispatchQueue call for safety, as these functions may be called by other threads than the main UI)
With that, I tried my original code for left- and right click, and it works fine once I de-check and re-check the App's permissions in Preferences after each compile:
func leftClick() {
DispatchQueue.main.async {
let source = CGEventSource.init(stateID: .hidSystemState)
let position = self.correctedCursorPosition // use your own CGPoint location here
let eventDown = CGEvent(mouseEventSource: source, mouseType: .leftMouseDown, mouseCursorPosition: position , mouseButton: .left)
let eventUp = CGEvent(mouseEventSource: source, mouseType: .leftMouseUp, mouseCursorPosition: position , mouseButton: .left)
eventDown?.post(tap: .cghidEventTap)
// usleep(50_000) // there's no need to sleep, it turns out
eventUp?.post(tap: .cghidEventTap)
}
}
func rightClick() {
DispatchQueue.main.async {
let source = CGEventSource.init(stateID: .hidSystemState)
let position = self.correctedCursorPosition // use your own CGPoint location here
let eventDown = CGEvent(mouseEventSource: source, mouseType: .rightMouseDown, mouseCursorPosition: position , mouseButton: .right)
let eventUp = CGEvent(mouseEventSource: source, mouseType: .rightMouseUp, mouseCursorPosition: position , mouseButton: .right)
eventDown?.post(tap: .cghidEventTap)
// usleep(50_000) // there's no need to sleep, it turns out
eventUp?.post(tap: .cghidEventTap)
}
}
By the way, using your code, one call to testClick works fine, too. In fact, two calls works too for activating controls on the targeted window (I guess one click or two doesn't matter for most controls), but using two testClick calls on e.g. the Apple icon in TopLeft produces two clicks, so that you're activating and immediately de-activating the dropdown. Fortunately, one call to testClick works fine (at least for me) both for controls in other windows as for the Apple dropdown etc.
That leaves me with the double-click as still being an issue, unfortunately. I tried using your original code, where I can change the clickCount (tried both 1 and 2), instead of let eventDown = CGEvent(mouseEventSource: source, mouseType: .leftMouseDown, mouseCursorPosition: position , mouseButton: .left) , etc. No luck, though...
let newMouseDownEvent = NSEvent.mouseEvent(
with: .leftMouseDown,
location: position, //NSPoint(x: 205,y: 200), // this should be in the hitWindow
modifierFlags: NSEvent.ModifierFlags.control,
timestamp: ProcessInfo.processInfo.systemUptime,
windowNumber: numberOfHitWindow,
context: nil,
eventNumber: 0, //don't know what else to use
clickCount: 1,
pressure: 1)
self.perform(#selector(NSResponder.self.mouseDown(with:)), with: newMouseDownEvent)
Thread.sleep(forTimeInterval: 0.1)
let newMouseUpEvent = NSEvent.mouseEvent(
with: .leftMouseDown,
location: position, //NSPoint(x: 205,y: 200), // this should be in the hitWindow
modifierFlags: NSEvent.ModifierFlags.control,
timestamp: ProcessInfo.processInfo.systemUptime,
windowNumber: numberOfHitWindow,
context: nil,
eventNumber: 1, //don't know what else to use
clickCount: 1,
pressure: 1)
self.perform(#selector(NSResponder.self.mouseUp(with:)), with: newMouseUpEvent)
Great work! Highly appreciated, thanks again! Learned a lot - and to think that one little checkbox in Preferences was the culprit!
yes, no luck there, I'm afraid... Also tried two Down events and one Up event. Didn't work. Also tried sleeping 0.1 secs in-between. Also there: no dice....
Sounds like a plan. Here's the code I used:
func leftClick() {
DispatchQueue.main.async {
print("-----------------------")
let sleepTime = 0.05
let inBetweenClickSleepTime = 0.1
let source = CGEventSource.init(stateID: .hidSystemState)
let position = self.correctedCursorPosition // holds the intended click position
let eventDown = CGEvent(mouseEventSource: source, mouseType: .leftMouseDown, mouseCursorPosition: position , mouseButton: .left)
let eventUp = CGEvent(mouseEventSource: source, mouseType: .leftMouseUp, mouseCursorPosition: position , mouseButton: .left)
print("before D & before U post, leftMouseDown count: \(CGEventSource.counterForEventType(.hidSystemState, eventType: .leftMouseDown)), leftMouseUp count: \(CGEventSource.counterForEventType(.hidSystemState, eventType: .leftMouseUp))")
eventDown?.post(tap: .cghidEventTap)
Thread.sleep(forTimeInterval: sleepTime)
print("after D post and \(sleepTime) sec sleep, leftMouseDown count: \(CGEventSource.counterForEventType(.hidSystemState, eventType: .leftMouseDown)), leftMouseUp count: \(CGEventSource.counterForEventType(.hidSystemState, eventType: .leftMouseUp))")
eventUp?.post(tap: .cghidEventTap)
Thread.sleep(forTimeInterval: sleepTime)
print("after D and U post and \(sleepTime) sec sleep, leftMouseDown count: \(CGEventSource.counterForEventType(.hidSystemState, eventType: .leftMouseDown)), leftMouseUp count: \(CGEventSource.counterForEventType(.hidSystemState, eventType: .leftMouseUp))")
print("\nSleeping for \(inBetweenClickSleepTime) secs, then trying another click\n")
Thread.sleep(forTimeInterval: inBetweenClickSleepTime)
print("before D & before U post, leftMouseDown count: \(CGEventSource.counterForEventType(.hidSystemState, eventType: .leftMouseDown)), leftMouseUp count: \(CGEventSource.counterForEventType(.hidSystemState, eventType: .leftMouseUp))")
eventDown?.post(tap: .cghidEventTap)
Thread.sleep(forTimeInterval: sleepTime)
print("after D post and \(sleepTime) sec sleep, leftMouseDown count: \(CGEventSource.counterForEventType(.hidSystemState, eventType: .leftMouseDown)), leftMouseUp count: \(CGEventSource.counterForEventType(.hidSystemState, eventType: .leftMouseUp))")
eventUp?.post(tap: .cghidEventTap)
Thread.sleep(forTimeInterval: sleepTime)
print("after D and U post and \(sleepTime) sec sleep, leftMouseDown count: \(CGEventSource.counterForEventType(.hidSystemState, eventType: .leftMouseDown)), leftMouseUp count: \(CGEventSource.counterForEventType(.hidSystemState, eventType: .leftMouseUp))")
}
}
Some interesting stuff happens: if I put sleepTime and inBetweenClickSleepTime to 0, counters are not increased. The output becomes:
before D & before U post, leftMouseDown count: 4190, leftMouseUp count: 4190
after D post and 0.0 sec sleep, leftMouseDown count: 4190, leftMouseUp count: 4190
after D and U post and 0.0 sec sleep, leftMouseDown count: 4190, leftMouseUp count: 4190
Sleeping for 0.0 secs, then trying another click
before D & before U post, leftMouseDown count: 4190, leftMouseUp count: 4190
after D post and 0.0 sec sleep, leftMouseDown count: 4190, leftMouseUp count: 4190
after D and U post and 0.0 sec sleep, leftMouseDown count: 4190, leftMouseUp count: 4190
Even so, a single click is registered in the target window (provided I de- and re-checked the app's privileges after compile). If I put sleepTime and inBetweenClickSleepTime to 0.05 and 0.1 but forget to do the privileges trick, counters are not increased and the target window doesn't receive the event. If I do reset privileges, the output becomes:
before D & before U post, leftMouseDown count: 4036, leftMouseUp count: 4036
after D post and 0.05 sec sleep, leftMouseDown count: 4037, leftMouseUp count: 4036
after D and U post and 0.05 sec sleep, leftMouseDown count: 4037, leftMouseUp count: 4037
Sleeping for 0.1 secs, then trying another click
before D & before U post, leftMouseDown count: 4037, leftMouseUp count: 4037
after D post and 0.05 sec sleep, leftMouseDown count: 4038, leftMouseUp count: 4037
after D and U post and 0.05 sec sleep, leftMouseDown count: 4038, leftMouseUp count: 4038
That is, both Down events and both Up events are registered by the counters. Still, the target window seems to receive only 1 click. For example, if the target window is Mail and the click is on a message, it doesn't open up - it just gets selected. Things don't change if I set inBetweenClickSleepTime to 0 but keep sleepTime at 0.05. Removing the Dispatch also doesn't help....
It's already connected to my iPhone, so every test I reported was by actually moving the cursor using my iPhone to the place I wanted to click. That means the clicks were never in the exact same place. Every time, though (provided I did the privileges trick, and had the sleepTimes at reasonable values), the item in the targeted Mail window got activated - just not opened up (as a double click does)
ah, sorry, my mistake.
I tried it, but no luck there either, I'm afraid: creating two Up and two Down events, and then positing them as Up1, Down1, Up2, Down2, with sleep in-between, still only produces a single click, even though the counterForEventType registers two Up and 2 Down events.