emulate mouse click

When you post code, please use code formatter tool (<>)

Could you detail what you want to do using hid ?

let position = NSPoint(100,75)
CGDisplayMoveCursorToPoint(0, position)
let source = CGEventSource.init(stateID: .hidSystemState)
let eventDown = CGEvent(mouseEventSource: source, mouseType: .leftMouseDown, mouseCursorPosition: position , mouseButton: .left)
let eventUp = CGEvent(mouseEventSource: source, mouseType: .leftMouseUp, mouseCursorPosition: position , mouseButton: .left)
eventDown?.post(tap: .cghidEventTap)
usleep(50_000)
eventUp?.post(tap: .cghidEventTap)

Did you check your events are created correctly ?

Note: Is it the real code ? The call

        let position = NSPoint(100,75)

appears incorrect. Should be

        let position = NSPoint(x: 100, y: 75)

Sorry about the formatting - first time posting :-)

This is the actual code - I have an extension on CGPoint that makes it possible to simply provide the numbers without the labels (I'm lazy that way ;-) ).

Not being a very experienced coder, I took the code from https://stackoverflow.com/questions/27484330/simulate-keypress-using-swift, with some changes. That unfortunately means I don't really know the answer as to "what I want to do with hid"- I just figured it was necessary to effect the leftclick.

As for the correct creation of the event, a "po eventDown!.location" nicely came back with (100.0, 75.0), as I would expect.

What I find strange is that it worked perfectly fine yesterday, when I uploaded the code to the AppStore ('UpointDriver' - the Mac counterpart of the Upoint iPhone app that turns the iPhone into an 'interactive laserpointer' that positions the cursor where you physically point the phone) - before upgrading to Big Sur 11.5. Even when I download that uploaded UpointDriver now (with Big Sur 11.5), the left-click works fine. Compiling and executing it with Xcode, however, gives problems...

Thanks for all the help.

It is always risky to reuse code we don't understand… I suppose you got it here: https://stackoverflow.com/questions/49082243/swift-4-simulating-left-click-programmatically-causes-right-click

Did you notice that require disabling sandboxing (may be the cause of the problem).

Here is what I do to modify an event for a mouse click (using perform).

    let newMouseDownEvent = NSEvent.mouseEvent(
        with: .leftMouseDown, 
        location: theEvent.locationInWindow,      // Use your own location
        modifierFlags: NSEvent.ModifierFlags.control,
        timestamp: theEvent.timestamp,
        windowNumber: theEvent.windowNumber,
        context: theEvent.context,
        eventNumber: theEvent.eventNumber+1,
        clickCount: theEvent.clickCount,
        pressure: theEvent.pressure)
    
    self.perform(#selector(NSResponder.self.mouseDown(with:)), with: newMouseDownEvent) 

In your case, you could try to use NSEvent:

class func mouseEvent(with type: NSEvent.EventType, 
             location: NSPoint, 
        modifierFlags flags: NSEvent.ModifierFlags, 
           timestamp time: TimeInterval, 
        windowNumber wNum: Int, 
              context unusedPassNil: NSGraphicsContext?, 
          eventNumber eNum: Int, 
           clickCount cNum: Int, 
            pressure: Float) -> NSEvent?

let eventDown = NSEvent(…)
self.perform(#selector(NSResponder.self.mouseDown(with:)), with: eventDown) 

Parameters

• type: One of the modifier key masks described in NSEvent.EventType, or an NSInternalInconsistencyException is raised.
• location: The cursor location in the base coordinate system of the window specified by windowNum.
• flags: An integer bit field containing any of the modifier key masks described in Getting Unicode Values, combined using the C bitwise OR operator.
• time: The time the event occurred in seconds since system startup.
• windowNum: An integer that identifies the window device associated with the event, which is associated with the NSWindow that will receive the event.
• context: The display graphics context of the event.
• eventNumber: An identifier for the new event. It’s normally taken from a counter for mouse events, which continually increases as the application runs.
• clickNumber: The number of mouse clicks associated with the mouse event.
• pressure: A value from 0.0 to 1.0 indicating the pressure applied to the input device on a mouse event, used for an appropriate device such as a graphics tablet. For devices that aren’t pressure-sensitive, the value should be either 0.0 or 1.0.

I read somewhere that disabling Sandbox is not an option if I want to distribute via AppStore, unfortunately.

As for your code: am I correct in understanding that this would work when I know the windowNumber the Event is meant for, and that it only works in windows the App controls? If so: I'm looking for ways to simulate a mouseclick outside my App's window - the idea being that my counterpart iPhone app acts as a remote mouse that can click anywhere - inside or outside the driver-app's window.

For example, I would like to activate a window on the Mac when the user uses the iPhone App to position the cursor on that window, and "leftclicks" on it using the iPhone. The Mac code should then activate that window.

I got a similar problem since updating to BigSur 11.5 concerning keyboard events. Previously, emulating a keystroke worked fine (code swiped from https://stackoverflow.com/questions/27484330/simulate-keypress-using-swift) but, at least from within Xcode, it no longer does. The code I use there was

let source = CGEventSource(stateID: CGEventSourceStateID.hidSystemState)
let numberKey: UInt16 = keyCode(key) // e.g., 11 for 'b'
let keyDown = CGEvent(keyboardEventSource: source, virtualKey: numberKey, keyDown: true)
let keyUp = CGEvent(keyboardEventSource: source, virtualKey: numberKey, keyDown: false)
let loc = CGEventTapLocation.cgSessionEventTap
keyDown?.post(tap: loc)
keyUp?.post(tap: loc)

disabling Sandbox is not an option

You're right.

The code I posted was to modify a received event. Your case is different, you want to create a brand new one.

Have a look at this func, it finds windNum outside of your app.:

windowNumber(at:belowWindowWithWindowNumber:) Returns the number of the frontmost window that would be hit by a mouse-down at the specified screen location.

https://developer.apple.com/documentation/appkit/nswindow/1419210-windownumber

I got the windowNumber - thanks - but still can't get it to work. I wrote a test function that I would think should activate a window that's positioned on the left side of the screen (whereas the Xcode or App window are to the right, so "out of the way").

func tryToActivateWindow() {
    print("begin -----------")

    let mousePosition = NSPoint(x: 5,y: 200) // This corresponds to a visible window that's not part of this App
    print("Mouse coordinates (\(mousePosition.x),\(mousePosition.y))")
    let screenHeight = NSScreen.main!.frame.height
    print("Screenheight: \(screenHeight)")
    var screenPosition: NSPoint = mousePosition
    screenPosition.y = screenHeight - mousePosition.y
    print("Screen coordinates (\(screenPosition.x),\(screenPosition.y))")
    let numberOfHitWindow = NSWindow.windowNumber(at: screenPosition, belowWindowWithWindowNumber: 0)
    print("Window hit: \(numberOfHitWindow)")

    CGDisplayMoveCursorToPoint(0, mousePosition) // Just for visual reference

    print("Creating MouseDown event")
    let newMouseDownEvent = NSEvent.mouseEvent(
        with: .leftMouseDown,
        location: NSPoint(x:0, y:0), // this should be in the hitWindow
        modifierFlags: NSEvent.ModifierFlags.control,
        timestamp: ProcessInfo.processInfo.systemUptime,
        windowNumber: numberOfHitWindow,
        context: nil,
        eventNumber: 0,  //don't know what else to use
        clickCount: 1,
        pressure: 1)

    self.perform(#selector(NSResponder.self.mouseDown(with:)), with: newMouseDownEvent)

    print("Sleeping for a bit")
    Thread.sleep(forTimeInterval: 0.1)
    print("Woke up")

    print("Creating MouseUp event")
    let newMouseUpEvent = NSEvent.mouseEvent(
        with: .leftMouseUp,
        location: NSPoint(x:0, y:0), // this should be in the hitWindow
        modifierFlags: NSEvent.ModifierFlags.control,
        timestamp: ProcessInfo.processInfo.systemUptime,
        windowNumber: numberOfHitWindow,
        context: nil,
        eventNumber: 1, //don't know what else to use
        clickCount: 1,
        pressure: 1)
    self.perform(#selector(NSResponder.self.mouseUp(with:)), with: newMouseUpEvent)

    print("end -----------")
}

It produces the following output:

• begin -----------
• Mouse coordinates (5.0,200.0)
• Screenheight: 1120.0
• Screen coordinates (5.0,920.0)
• Window hit: 125
• Creating MouseDown event
• Sleeping for a bit
• Woke up
• Creating MouseUp event
• end -----------

The cursor gets put in the right window, but the window isn't activated as hoped...

I got it work (with help of this): https://stackoverflow.com/questions/47152551/activate-a-window-using-its-window-id

You need to activate the window you select. Here the changes I made (I connected to a button for easier testing instead of tryToActivateWindow() ) ; I also changed the NSPoint, but that's a detail.

Here is the modified code:

    func switchToApp(withWindow windowNumber: Int32) {
        let options = CGWindowListOption(arrayLiteral: CGWindowListOption.excludeDesktopElements, CGWindowListOption.optionOnScreenOnly)
        let windowListInfo = CGWindowListCopyWindowInfo(options, CGWindowID(0))
        guard let infoList = windowListInfo as NSArray? as? [[String: AnyObject]] else { return }
        if let window = infoList.first(where: { ($0["kCGWindowNumber"] as? Int32) == windowNumber}), let pid = window["kCGWindowOwnerPID"] as? Int32 {
            let app = NSRunningApplication(processIdentifier: pid)
            app?.activate(options: .activateIgnoringOtherApps)
        }
    }

    //    }
    //
    //    func tryToActivateWindow() {

    @IBAction func tryToActivateWindow(_ sender: NSButton) {
        print("begin -----------")
        print(self.view.window?.windowNumber)

        let mousePosition = NSPoint(x: 205,y: 200) // This corresponds to a visible window that's not part of this App
        print("Mouse coordinates (\(mousePosition.x),\(mousePosition.y))")
        let screenHeight = NSScreen.main!.frame.height
        print("Screenheight: \(screenHeight)")
        var screenPosition: NSPoint = mousePosition
        screenPosition.y = screenHeight - mousePosition.y
        print("Screen coordinates (\(screenPosition.x),\(screenPosition.y))")
        let numberOfHitWindow = NSWindow.windowNumber(at: screenPosition, belowWindowWithWindowNumber: 0)
        print("Window hit: \(numberOfHitWindow)")

        CGDisplayMoveCursorToPoint(0, mousePosition) // Just for visual reference

        print("Creating MouseDown event")
        let newMouseDownEvent = NSEvent.mouseEvent(
            with: .leftMouseDown,
            location: mousePosition, //NSPoint(x: 205,y: 200), // this should be in the hitWindow
            modifierFlags: NSEvent.ModifierFlags.control,
            timestamp: ProcessInfo.processInfo.systemUptime,
            windowNumber: numberOfHitWindow,
            context: nil,
            eventNumber: 0,  //don't know what else to use
            clickCount: 1,
            pressure: 1)

        self.perform(#selector(NSResponder.self.mouseDown(with:)), with: newMouseDownEvent)

        print("Sleeping for a bit")
        Thread.sleep(forTimeInterval: 0.1)
        print("Woke up")

        print("Creating MouseUp event")
        let newMouseUpEvent = NSEvent.mouseEvent(
            with: .leftMouseUp,
            location: mousePosition, //NSPoint(x: 205,y: 200), // this should be in the hitWindow
            modifierFlags: NSEvent.ModifierFlags.control,
            timestamp: ProcessInfo.processInfo.systemUptime,
            windowNumber: numberOfHitWindow,
            context: nil,
            eventNumber: 1, //don't know what else to use
            clickCount: 1,
            pressure: 1)
        self.perform(#selector(NSResponder.self.mouseUp(with:)), with: newMouseUpEvent)

        print("end -----------")
        switchToApp(withWindow: Int32(numberOfHitWindow))       // <---- THAT'S THE KEY POINT
    }

As for counter, 0 and 1 work:

https://stackoverflow.com/questions/2662128/how-to-get-the-current-eventnumber-for-creating-an-event-with-nsevent

For your other question:

In a related question: how does one emulate a double-click?

Just pas 2 as a value for clickCount in newMouseDownEvent.

If that works, don't forget to close the thread on the correct answer. Good luck.

That works, but you don't need the mouseEvents anymore. The switchToApp is all you need to activate the window the mouse is aimed at.

Thing is: the point was to feed the leftclick to that window, so that my iPhone user can activate buttons etc on any window he chooses...

That's indeed what I expected. Yet, if I execute the findWindow() of the following code

func switchToAppAndClick(withWindow windowNumber: Int32) {
    let options = CGWindowListOption(arrayLiteral: CGWindowListOption.excludeDesktopElements, CGWindowListOption.optionOnScreenOnly)
    let windowListInfo = CGWindowListCopyWindowInfo(options, CGWindowID(0))
    guard let infoList = windowListInfo as NSArray? as? [[String: AnyObject]] else { return }
    if let window = infoList.first(where: { ($0["kCGWindowNumber"] as? Int32) == windowNumber}), let pid = window["kCGWindowOwnerPID"] as? Int32 {
        let app = NSRunningApplication(processIdentifier: pid)
        app?.activate(options: .activateIgnoringOtherApps)
        
        let inWindowClickpoint = CGPoint(x: 100  , y:200) // a clickable point in the targeted window
        print("creating eventMouseDown and Up events")
        let eventMouseDown = CGEvent(mouseEventSource: nil, mouseType: .leftMouseDown, mouseCursorPosition: inWindowClickpoint, mouseButton: .left)
        let eventMouseUp = CGEvent(mouseEventSource: nil, mouseType: .leftMouseUp, mouseCursorPosition: inWindowClickpoint, mouseButton: .left)
        print("posting \(eventMouseDown) to pid \(pid)")
        eventMouseDown?.postToPid(pid)
        Thread.sleep(forTimeInterval: 0.1)
        print("posting \(eventMouseUp) to pid \(pid)")
        eventMouseUp?.postToPid(pid)
    }
}

func findWindow() {
    let mousePosition = NSPoint(x: 100,y: 350) // This corresponds to a visible window that's not part of this App
    print("Mouse coordinates (\(mousePosition.x),\(mousePosition.y))")
    let screenHeight = NSScreen.main!.frame.height
    var screenPosition: NSPoint = mousePosition
    screenPosition.y = screenHeight - mousePosition.y
    print("Screen coordinates (\(screenPosition.x),\(screenPosition.y))")
    let numberOfHitWindow = NSWindow.windowNumber(at: screenPosition, belowWindowWithWindowNumber: 0)
    CGDisplayMoveCursorToPoint(0, mousePosition) // Just for visual reference
    print("Window hit: \(numberOfHitWindow); switching to switchToAppAndClick")
    switchToAppAndClick(withWindow: Int32(numberOfHitWindow))
}

I get as output

• Mouse coordinates (100.0,350.0)
• Screen coordinates (100.0,770.0)
• Window hit: 1245; switching to switchToAppAndClick
• creating eventMouseDown and Up events
• posting Optional(<CGEvent 0x6000024ce480>) to pid 2176
• posting Optional(<CGEvent 0x6000024ce4e0>) to pid 2176

... and, although the targeted window is selected, I do not get the expected "click" action on the targeted window. Moreover, the value ofinWindowClickpoint doesn't seem to have any influence at all.

Just some ideas:

• is inWindowClickpoint in the window's coordinate (not absolute screen position ?)

• If I read correctly, post() works only for keyboard.

For mouse, one should use eventDown?.post(tap: .cghidEventTap)

inWindowClickpoint is in window's coordinates, and not in absolute screen position. One way or another, it doesn't seem to have any effect anyway.

Also, using .post(tap: .cghidEventTap) instead of .postToPid(pid) doesn't solve the problem either, I'm afraid: the window gets activated, but the item at the cursor position isn't "clicked"...

Did you try set the source (and not leave nil)?

let source = CGEventSource.init(stateID: .hidSystemState)
let position = CGPoint(x: 75, y: 100)
let eventDown = CGEvent(mouseEventSource: source, mouseType: .leftMouseDown, mouseCursorPosition: position , mouseButton: .left)
let eventUp = CGEvent(mouseEventSource: source, mouseType: .leftMouseUp, mouseCursorPosition: position , mouseButton: .left)

No luck still, I'm afraid. The code I have now is:

func switchToAppAndClick(withWindow windowNumber: Int32) {
    let options = CGWindowListOption(arrayLiteral: CGWindowListOption.excludeDesktopElements, CGWindowListOption.optionOnScreenOnly)
 let windowListInfo = CGWindowListCopyWindowInfo(options, CGWindowID(0))
    guard let infoList = windowListInfo as NSArray? as? [[String: AnyObject]] else { return }
    if let window = infoList.first(where: { ($0["kCGWindowNumber"] as? Int32) == windowNumber}), let pid = window["kCGWindowOwnerPID"] as? Int32 {
        let app = NSRunningApplication(processIdentifier: pid)
        app?.activate(options: .activateIgnoringOtherApps)
        let inWindowClickpoint = CGPoint(x: 100  , y:200) // a clickable point in the targeted window coords
        print("creating eventMouseDown and Up events")
        let source = CGEventSource.init(stateID: .hidSystemState)
        let position = CGPoint(x: 75, y: 100)
        let eventDown = CGEvent(mouseEventSource: source, mouseType: .leftMouseDown, mouseCursorPosition: position , mouseButton: .left)
        let eventUp = CGEvent(mouseEventSource: source, mouseType: .leftMouseUp, mouseCursorPosition: position , mouseButton: .left)
        print("posting \(eventUp)")
        eventDown?.post(tap: .cghidEventTap)  
        Thread.sleep(forTimeInterval: 0.1)
        print("posting \(eventDown)")
        eventUp?.post(tap: .cghidEventTap)
    }
}

Same thing: window gets activated, but the control the cursor is aimed at is not - when I 'physically execute' a leftclick, the window responds as expected. The events are printed out as non-nil, by the way.

I fear I've reached my limits. May be (just to understand, because that's not possible for an app on appstore), desactivating sandboxing ? If that's important, you could burn a DTS ticket…

First and foremost: thanks much for your help. It’s highly appreciated. Maybe I’ll wait a bit longer before burning a ticket. Someone else might have an idea?

I guess it’s possible I may be doing something wrong….Does it work for you? That is: activating a control ‘by mouse’ on a foreign window?

In any case, This functionality is critical for a ‘remote mouse’ like the Upoint interactive laser pointer, so if push comes to shove: I guess I need to burn that ticket…