CLLocationManager: getting kCLErrorDenied in widget

I am having some problem with accessing the CLLLocationManager location from my widget. It works fine with Xcode 15 running on a iOS17 simulator. But running it on a iOS17 device gives me an error in the delegate:

To access the location manager, I have this class:

class WidgetLocationManager: NSObject, CLLocationManagerDelegate {
    var locationManager: CLLocationManager?
    private var handler: ((CLLocation?) -> Void)?

    override init() {
        super.init()
        DispatchQueue.main.async {
            print("WidgetLocationManager: init")
            
            self.locationManager = CLLocationManager()
            if self.locationManager!.authorizationStatus == .notDetermined {
                print("WidgetLocationManager: init - auth status is Undetermined")
            } else {
                print("WidgetLocationManager: init - auth status = \(self.locationManager!.authorizationStatus)")
            }
        }
    }
    
    func fetchLocation(handler: @escaping (CLLocation?) -> Void) {
        self.handler = handler
        self.locationManager = CLLocationManager()
        self.locationManager!.delegate = self
        self.locationManager!.requestLocation()
    }

    func locationManager(_ manager: CLLocationManager, didUpdateLocations locations: [CLLocation]) {

        if let lastLocation = locations.last {
            if (CLLocationCoordinate2DIsValid(lastLocation.coordinate) == true && abs(lastLocation.timestamp.timeIntervalSinceNow) < 60 && lastLocation.horizontalAccuracy < 200.0 && lastLocation.horizontalAccuracy > 0.0) {
                
                self.handler!(locations.last!)
            }
        }
    }
    
    func locationManager(_ manager: CLLocationManager, didFailWithError error: Error) {
        print("WidgetLocationManager: locationManager didFailWithError = \(error)")
        self.handler!(nil)
    }
}

When run on device, I get an error:

WidgetLocationManager: locationManager didFailWithError = Error Domain=kCLErrorDomain Code=1 "(null)"

Code = 1 in CLError is kCLErrorDenied ("Access to location or ranging has been denied by the user")

This is despite getting the following output in the init method:

WidgetLocationManager: init - auth status = CLAuthorizationStatus(rawValue: 4)

The weird thing is that it works fine in the simulator. On device, I've tried deleting and reinstalling the app a couple of times, restarting the device, making sure the privacy setting is correct etc.

Also, on the iOS17 device, when I am in the "Add Widget" page, the location manager runs fine in the preview screen, and shows the 'current location' to the user. But as soon as I add the widget to the home screen, it starts giving me this problem where the location can't be found, and I have to display an error message in the widget.

I have the following keys in the Info.plist for the widget:

• NSLocationAlwaysAndWhenInUseUsageDescription
• NSLocationWhenInUseUsageDescription
• NSWidgetWantsLocation

The app target also has the following keys in the Info.plist:

• NSLocationWhenInUseUsageDescription
• NSLocationAlwaysAndWhenInUseUsageDescription

Any idea for what I can try to fix this problem?

Thanks.