Confusing pointer conversion

I've tried to follow to shifting model of pointers in Swift (I understand it's for the best)… but now I'm completly lost.


How would I cast/convert a `sockaddr_in6` to an `UnsafePointer<sockaddr>!` type?


var s = sockaddr_in6()
// ... assign values members/properties of `s`

var fileDescriptor = socket(socket(AF_INET6, SOCK_STREAM, IPPROTO_TCP)
// ... handle errors

Darwin.bind(fileDescriptor, &s ,socklen_t(MemoryLayout<sockaddr_in6>.size))
//                          ^^ ###### can't convert sockaddr_in6 to sockaddr



I've tried many things without success (withUnsafePointer(…), OpaquePointers<T>, etc.).


My (current) understanding is that I would need to go from:

sockaddr_in6 ⇒ UnsafePointer<sockaddr_in6> ⇒ UnsafePointer<sockaddr>!


But there doesn't seem to be anyway to cast UnsafePointer between themselves. I'm sure there's a logical reason somehwere but it eludes me.


Thank you!

Accepted Reply

Re-read documentation I had initially misunderstood. Had misread UnsafePointer's `withMemoryRebound<U>()`.


var s = sockaddr_in6();
// ...

withUnsafePointer(to: &s) {
    (p1: UnsafePointer<sockaddr_in6>) in
    p1.withMemoryRebound(to: sockaddr.self, capacity: 1) {
        (p2: UnsafeMutablePointer<sockaddr>) in
        bind(fileDescriptor, p2, /*....*/ )
    }
}

Replies

Re-read documentation I had initially misunderstood. Had misread UnsafePointer's `withMemoryRebound<U>()`.


var s = sockaddr_in6();
// ...

withUnsafePointer(to: &s) {
    (p1: UnsafePointer<sockaddr_in6>) in
    p1.withMemoryRebound(to: sockaddr.self, capacity: 1) {
        (p2: UnsafeMutablePointer<sockaddr>) in
        bind(fileDescriptor, p2, /*....*/ )
    }
}

I've tried to follow to shifting model of pointers in Swift (I understand it's for the best)… but now I'm completly lost.

Yeah, I’ve been struggling with this myself, especially with regards BSD Sockets, since Xcode 8.0b6 landed. FWIW I think the approach in your second is valid but I’m still working to confirm that.

But there doesn't seem to be anyway to cast UnsafePointer between themselves. I'm sure there's a logical reason somehwere but it eludes me.

Right. This is not an accident. Swift is attempting to define aliasing rules and to do that it needs to separate

UnsafePointer<X>
from
UnsafeRawPointer
.
  • UnsafePointer<X>
    is unsafe from a memory management perspective (that is, it doesn’t participate in ARC) but not safe from an aliasing perspective. That is, if
    UnsafePointer<X>
    is valid it will always point to something of type
    X
    .
  • OTOH,

    UnsafeRawPointer
    has no such limitations.

The difference is important for compiler optimisation. If you search the ’net for terms like strict aliasing, pointer aliasing, and type punning you’ll find plenty of folks explaining this. Some of them even use BSD Sockets in their examples (-:

Strict aliasing is one of the reasons I’ve started parsing network packets byte-by-byte, as I’ve explained in this post. However, that won’t help you in this case, where the BSD Sockets API forces you to worry about it.

Share and Enjoy

Quinn “The Eskimo!”
Apple Developer Relations, Developer Technical Support, Core OS/Hardware

let myEmail = "eskimo" + "1" + "@apple.com"
  • UnsafePointer<X>
    is unsafe from a memory management perspective (that is, it doesn’t participate in ARC) but not safe from an aliasing perspective. That is, if
    UnsafePointer<X>
    is valid it will always point to something of type
    X
    .
  • OTOH,
    UnsafeRawPointer
    has no such limitations.


Um, is there a typo in there? If it's unsafe from one perspective, "but" not safe from the other, what are the limitations that UnsafeRawPointer doesn't have?


I've actually been struggling with this today, and I did find some now-out-of-date documentation on UnsafePointer at swift.org:


github.com/atrick/swift/blob/type-safe-mem-docs/docs/TypeSafeMemory.rst


as well as on UnsafeRawPointer:


github.com/apple/swift-evolution/blob/master/proposals/0107-unsaferawpointer.md


What I've been trying to figure out is how to overlay various structs on Data read from a file, where the data must be serially decoded to find the offsets of the structs. This is pretty easy in Obj-C with NSData and C structs, but I'm not sure it's doable with UnsafePointer because:


— it's awkward to make of round-trips through UnsafeRawPointer to get UnsafePointers at an offset from the start of data


— the underlying data pointer of a Data instance, and hence any pointer derived from it, is not valid outside a 'withUnsafeBytes' closure


— UnsafePointer has structure alignment requirements that I'm not sure the data meets


If you've any enlightenment to offer on such difficulties, I'd be very happy to know about it.

Um, is there a typo in there? If it's unsafe from one perspective, "but" not safe from the other, what are the limitations that UnsafeRawPointer doesn't have?

Yeah, I could have phrased that better. Sorry. Let’s try that again:

  • In general Swift is safe. If you break the rules of the language, you get a well-defined failure. For example, if you index out of bounds on an array, you trap.

  • UnsafePointer<X>
    is unsafe in that sense. If you don’t use it correctly, anything could happen.
  • One of those use-it-correctly rules relates to aliasing. It’s not legal to have pointers of two different types pointing to the same memory.

  • UnsafeRawPointer
    does not suffer from that restriction but, as a side effect, you don’t get direct access to the underlying value (
    pointee
    ).

IMPORTANT It’s critical to note that C-based languages have similar aliasing restrictions. For example, this code is not legal C:

uint64_t f(const uint16_t * u16s) {
  return * (const uint64_t * ) u16s;
}
uint16_t u16s[4] = {1, 2, 3, 4};
uint64_t u64 = f(&u16s[0]);

While the expected behaviour for this snippet is obvious, there are cases where this sort of aliasing can cause the compiler to generate code that does not work the way you’d expect it to. As I said before, there’s lots of articles on the ’net explaining this.

However, due to historical reasons folks have generally ignored this restriction and thus C compilers have to avoid some optimisations that they could otherwise do. Notably, in Xcode you have to opt in to those optimisations via a build setting (Enforce Strict Aliasing,

GCC_STRICT_ALIASING
).

Clearly it would be nice if Swift didn’t have to do that.

What I've been trying to figure out is how to overlay various structs on Data read from a file, where the data must be serially decoded to find the offsets of the structs.

If you don’t care about copies, you can use

load(fromByteOffset:as:)
.
func test(_ p: UnsafeRawPointer) {
    NSLog("%8x", p.load(as: UInt32.self))
}

If you want to avoid making the copy, things get trickier. For trivial types (defined in SE-0107) it’s reasonable to use

bindMemory(to:capacity:)
as long as:
  • the memory hasn’t been bound to any other type

  • you don’t trip over some other unsafeness related to

    UnsafePointer<X>
  • you get the alignment right

For example:

func test(_ p: UnsafeRawPointer) {
    let p32 = p.bindMemory(to: UInt32.self, capacity: 1)
    NSLog("%8x", p32.pointee)
}

With regards alignment, you can check it using the new

MemoryLayout
type (see SE-0101). If the alignment is wrong you won’t be able to get an
UnsafePointer<X>
to the data because one of the rules of
UnsafePointer<X>
is that the pointer is correctly aligned.

Share and Enjoy

Quinn “The Eskimo!”
Apple Developer Relations, Developer Technical Support, Core OS/Hardware

let myEmail = "eskimo" + "1" + "@apple.com"

I am a bit perplexed too. Perhaps it my age/IQ at issue.


Writing some game networking code, and my method is to allocate a block of mutable memory with NSMutableData.


I get the data's .bytes property to get a pointer, and then I can happily (albeit unsafely) read and write to the that block of bytes. This is working code. And I can send the data item off by shooting across the network.


But now Data.bytes gives me a UnsafeRawPointer, and I suddenly can't read and write to it because I need a UnsafeMutablePointer<UInt8>


I am happy to see the language evolving, and becoming safer. But perhaps we need some more concrete examples for common usage cases.

https://developer.apple.com/library/prerelease/content/documentation/Swift/Conceptual/BuildingCocoaApps/InteractingWithCAPIs.html

Does not seem to cover this comprehensively.

I think I'm going to bite the bullet and rewrite the code to use a byte-stream approach, which I think was more-or-less your recommendation in another forum thread. Structure overlays don't end up saving a lot of code because there's a lot of byte-swapping boilerplate.

I think I'm going to bite the bullet and rewrite the code to use a byte-stream approach, which I think was more-or-less your recommendation in another forum thread.

Right. Here’s how things used to work on Planet Quinn™:

  1. I need to deal with some external data representation (typically a network packet).

  2. This is full of struct-like things, lets defines some structs that map on to it. Cool!

  3. Ah, that’s tricky.

  4. Oh, and that bit’s tricky too.

  5. Hmmm, that bit’s impossible. Drat!

  6. OK, let’s parse this as a stream of bytes.

Nowadays I just skip from step 1 straight to step 6. This is helped by Swift’s ability to support low-cost abstractions.

Share and Enjoy

Quinn “The Eskimo!”
Apple Developer Relations, Developer Technical Support, Core OS/Hardware

let myEmail = "eskimo" + "1" + "@apple.com"

I am a bit perplexed too. Perhaps it my age/IQ at issue.

No, this stuff is hard to begin with, and Xcode 8.0b6 seems like Swift 3’s ‘point of maximal thrash’, so there’s a lot to deal with all at once and the docs (and community experience) haven’t yet caught up.

But now Data.bytes gives me a UnsafeRawPointer, and I suddenly can't read and write to it because I need a UnsafeMutablePointer

Indeed. I’d love to help out here but I need more details about your goals. it’s probably best not to continue this thread because we’ve strayed a long way from

sockaddr
already. Please start a new thread with a high-level outline of what you’re trying to do.

Share and Enjoy

Quinn “The Eskimo!”
Apple Developer Relations, Developer Technical Support, Core OS/Hardware

let myEmail = "eskimo" + "1" + "@apple.com"

I'm similarly baffled by some of the recent changes...


I've fixed some of the problems/errors I was having (with help from the Swift-evolution docs), but I'm stumped by the withMemoryRebound thing...

Basically, I have a function to get a buffer of MIDIMetaEvent data that I can read in various places:

func dataBufferOfMIDIMetaEvent(_ mmePtr: UnsafePointer<MIDIMetaEvent>) -> UnsafeBufferPointer<UInt8> {
     let offset = OffsetOfDataInMIDIMetaEvent
     let bytePtr = UnsafePointer<UInt8>(mmePtr)     // <-- error here
     return UnsafeBufferPointer<UInt8>(start: bytePtr.advanced(by: offset), count: Int(mmePtr.pointee.dataLength))
}

The error in UnsafePointer<UInt8>(mmePtr) is suggesting that I use withMemoryRebound, but I honestly have no idea how that's supposed to work. If anybody can make a quick example, it would be a huge help.

Basically, I have a function to get a buffer of MIDIMetaEvent data that I can read in various places:

func dataBufferOfMIDIMetaEvent(_ mmePtr: UnsafePointer<MIDIMetaEvent>) -> UnsafeBufferPointer<UInt8>

The problem here is that your function signature encourages exactly the sort of behaviour that SE-0107 is trying to prevent. This function takes an

UnsafePointer<MIDIMetaEvent>
and returns an
UnsafeBufferPointer<UInt8>
for the data within it. Imagine calling it like this:
var event = MIDIMetaEvent()
let bytesBuffer = dataBufferOfMIDIMetaEvent(&event)
let bytes = bytesBuffer.baseAddress!
… stuff that accesses `event` and `bytes` …

At this point

event
and
bytes
are two pointers that point to the same memory but with different types. This is pointer aliasing (a form of type punning) and its behaviour is unspecified. If you rely on it, you may find that future compiler optimisations will cause your program to fail

This risk is why the SE-0107 changes are making things hard for you; Swift is trying to help you avoid future problems.

To solve this issue you must make sure that your code only accesses your memory via one type at a time. A good way to do this is

withMemoryRebound(of:capacity:body:)
. That does the following:
  1. tells the compiler to treat the memory as holding the new type

  2. calls the body

  3. on return, tells the compiler to treat the memory as holding the original type

In many cases you’ll want to wrap this up in your own convenience function. For example, imagine a routine with this signature:

func withData(for: UnsafePointer<MIDIMetaEvent>, body: (_ data: UnsafeBufferPointer<UInt8>))

You can then call it like this:

var event: MIDIMetaEvent = …
withData(for: &event) { bytesBuffer in
    … access `bytesBuffer` but not `event` …
}

As long as:

  • the pointer to the bytes (

    bytesBuffer.baseAddress
    ) doesn’t escape the block, and
  • you don’t access

    event
    within that block

your code will avoid any pointer aliasing problems.

A simple implementation might look like this:

func withData(for eventPtr: UnsafePointer<MIDIMetaEvent>, body: (_ data: UnsafeBufferPointer<UInt8>) -> Void) {
    let dataLength = Int(eventPtr.pointee.dataLength)
    return eventPtr.withMemoryRebound(to: UInt8.self, capacity: 8 + dataLength) { eventAsBytes in
        let bytesBuffer = UnsafeBufferPointer(start: eventAsBytes + 8, count: dataLength)
        return body(bytesBuffer)
    }
}

There’s some things to note here:

  • The first “8” is the size of MIDIMetaEvent minus the trailing

    data
    field. The second “8” is the offset of
    data
    field in MIDIMetaEvent. Presumably, based on the
    OffsetOfDataInMIDIMetaEvent
    value in the code you posted, you already have a way of calculating these. I just hard coded them, which isn’t too bad given that this is an on-the-wire format.
  • I set up

    dataLength
    before entering the
    withMemoryRebound
    block. This avoids accessing
    eventPtr
    within that block, which would be incorrect because by the time that block is called the compiler has been told to treat that memory as a sequence of bytes.

Here’s an example of it in use:

var event = MIDIMetaEvent()
event.dataLength = 1
event.data = 42
withData(for: &event) { bytesBuffer in
    for b in bytesBuffer {
        print(String(b, radix: 16))
        // prints "2a", or 42 as hex
    }
}

A more complex implementation would look like this:

func withData<ReturnType>(for eventPtr: UnsafePointer<MIDIMetaEvent>, body: (_ data: UnsafeBufferPointer<UInt8>) throws -> ReturnType) rethrows -> ReturnType {
    let dataLength = Int(eventPtr.pointee.dataLength)
    return try eventPtr.withMemoryRebound(to: UInt8.self, capacity: 8 + dataLength) { eventAsBytes in
        let bytesBuffer = UnsafeBufferPointer(start: eventAsBytes + 8, count: dataLength)
        return try body(bytesBuffer)
    }
}

This has two advantages:

  • The block can return a value of whatever type you choose (always helpful).

  • The

    rethrows
    means that the block can throw if necessary. However, if it does not throw, then
    withData(for:body:)
    is treated as not throwing.

Share and Enjoy

Quinn “The Eskimo!”
Apple Developer Relations, Developer Technical Support, Core OS/Hardware

let myEmail = "eskimo" + "1" + "@apple.com"

Thanks so much, eskimo! Your explanation and examples are extremely helpful.

ugh... my apologies, but in the context of my MIDI file parser, there's an additional step that I'm struggling with... I feel like I understand your example, but MIDI parsing has the added step of getting data from a MusicEventIterator using MusicEventIteratorGetEventInfo, which has the signature:

func MusicEventIteratorGetEventInfo(_ inIterator: MusicEventIterator,
_ outTimeStamp: UnsafeMutablePointer<MusicTimeStamp>,
_ outEventType: UnsafeMutablePointer<MusicEventType>,
_ outEventData: UnsafeMutablePointer<UnsafePointer<Void>>, // ??
_ outEventDataSize: UnsafeMutablePointer<UInt32>) -> OSStatus

The thing I don't understand here, with regard to your example, is how I get from UnsafeMutablePointer<UnsafePointer<Void>> to MIDIMetaEvent (or additionally, in my code below, ExtendedTempoEvent). The relevant chunk of code, which was working (however dangerously) up to beta 6, looks like this (I've marked the warnings/errors that I'm stumbling on):

// ...
// get the data field out of the MIDIMetaEvent C struct
func dataBufferOfMIDIMetaEvent(_ mmePtr: UnsafePointer<MIDIMetaEvent>) -> UnsafeBufferPointer<UInt8> {
  let offset = OffsetOfDataInMIDIMetaEvent
  let bytePtr = UnsafePointer<UInt8>(mmePtr)  // <-- error
  return UnsafeBufferPointer<UInt8>(start: bytePtr.advanced(by: offset), count: Int(mmePtr.pointee.dataLength))
}

func getTempoEventsFromMIDIFileTrack(_ track: MusicTrack, inSequence sequence: MusicSequence) -> (tempo: [TempoEvent], timeSignature: [TimeSignature]) {
  var tempoEvents: [TempoEvent] = [TempoEvent]()
  var timeSignatures: [TimeSignature] = [TimeSignature]()
  self.determineTimeResolutionWithTempoTrack(track)
  // Set up the event iterator
  var iterator: MusicEventIterator? = nil
  NewMusicEventIterator(track, &iterator)
  var hasCurrentEvent: DarwinBoolean = false
  MusicEventIteratorHasCurrentEvent(iterator!, &hasCurrentEvent)

  // Somewhere to put the raw midi data
  var type: MusicEventType = 0
  var stamp: MusicTimeStamp = -1
  var data: UnsafePointer<()>? = nil    // <-- warning to use UnsafeRawPointer
  var size: UInt32 = 0

  var tsChangeStamp: MusicTimeStamp = 0

  while(hasCurrentEvent != false) {
      MusicEventIteratorGetEventInfo(iterator!, &stamp, &type, &data, &size) // <-- error: cannot pass as inout argument

      // Get the bar and beat info for this event.
      var barBeat: CABarBeatTime = CABarBeatTime(bar: 1, beat: 1, subbeat: 0, subbeatDivisor: 0, reserved: 0)
      MusicSequenceBeatsToBarBeatTime(sequence, stamp, self.timeResolution, &barBeat)
      switch type {

      case 3 :    // kMusicEventType_ExtendedTempo
      let exTempo = UnsafePointer<ExtendedTempoEvent>(data) // <-- error: init is unavailable
      let tempoEvent = TempoEvent(timestamp: stamp, bar: barBeat.bar, beat: barBeat.beat, bpm: (exTempo?.pointee.bpm)!)
      tempoEvents.append(tempoEvent)
      case 5:    // kMusicEventType_Meta
      let meta = UnsafeMutablePointer<MIDIMetaEvent>(data)  // <-- error: init is unavailable
      let evType = meta?.pointee.metaEventType
      switch evType! {
            case 0x58:
            let dataBuffer = dataBufferOfMIDIMetaEvent(meta!)
            var ts_num: Int = 4
            var ts_denom: Int = 4
            ts_num = Int(dataBuffer[0])
            let pow = Float(dataBuffer[1])
            ts_denom = Int(powf(2.0, pow))
            let tempRange = CountableRange(Int(tsChangeStamp)...Int(stamp))
            var denominatorDuration = 1.0
            if ts_denom == 8 {
                denominatorDuration = 0.5
            }
            var timeSignature = TimeSignature(numerator: ts_num, denominator: ts_denom, denominatorDuration: denominatorDuration, bar: barBeat.bar, timestamp: stamp)
            timeSignature.beatRange = tempRange
            timeSignatures.append(timeSignature)
            tsChangeStamp = stamp
            default: ()
        }
      default: ()
      }
      // Advance the enumerator
      MusicEventIteratorHasNextEvent(iterator!, &hasCurrentEvent)
      MusicEventIteratorNextEvent(iterator!)
  }
  return (tempo: tempoEvents, timeSignature: timeSignatures)
}
// ...

Okay, I managed to get these functions to build:

func withMIDIDataOfType<T>(for eventPtr: UnsafePointer<MIDIMetaEvent>, body: (_ data: UnsafeBufferPointer<UInt8>) throws -> T) rethrows -> T {
    let dataLength = Int(eventPtr.pointee.dataLength)
    return try eventPtr.withMemoryRebound(to: UInt8.self, capacity: OffsetOfDataInMIDIMetaEvent + dataLength) { eventAsBytes in
        let bytesBuffer = UnsafeBufferPointer(start: eventAsBytes + OffsetOfDataInMIDIMetaEvent, count: dataLength)
        return try body(bytesBuffer)
    }
}
func getTempoEventsFromMIDIFileTrack(_ track: MusicTrack, inSequence sequence: MusicSequence) -> (tempo: [TempoEvent], timeSignature: [TimeSignature]) {
    var tempoEvents: [TempoEvent] = [TempoEvent]()
    var timeSignatures: [TimeSignature] = [TimeSignature]()
    self.determineTimeResolutionWithTempoTrack(track)
    /
    var iterator: MusicEventIterator? = nil
    NewMusicEventIterator(track, &iterator)
    var hasCurrentEvent: DarwinBoolean = false
    MusicEventIteratorHasCurrentEvent(iterator!, &hasCurrentEvent)

    /
    var type: MusicEventType = 0
    var stamp: MusicTimeStamp = -1
    let data: UnsafeMutablePointer<UnsafeRawPointer?> = UnsafeMutablePointer<UnsafeRawPointer?>.allocate(capacity: MemoryLayout<MusicEventType>.size)
    var size: UInt32 = 0

    var tsChangeStamp: MusicTimeStamp = 0

    while(hasCurrentEvent != false) {
   
        MusicEventIteratorGetEventInfo(iterator!, &stamp, &type, data, &size)
   
        print("data? \(data)")
   
        /
        var barBeat: CABarBeatTime = CABarBeatTime(bar: 1, beat: 1, subbeat: 0, subbeatDivisor: 0, reserved: 0)
        MusicSequenceBeatsToBarBeatTime(sequence, stamp, self.timeResolution, &barBeat)
        switch type {
       
        case 3 :    /
       
            let exTempo = data.pointee?.bindMemory(to: ExtendedTempoEvent.self, capacity: OffsetOfDataInMIDIMetaEvent)
            let tempoEvent = TempoEvent(timestamp: stamp, bar: barBeat.bar, beat: barBeat.beat, bpm: (exTempo!.pointee.bpm))
            tempoEvents.append(tempoEvent)
        case 5:     /
            let meta = data.pointee?.bindMemory(to: MIDIMetaEvent.self, capacity: 8)
            if let evType = meta?.pointee.metaEventType {
                switch evType {
                case 0x58:
                    withMIDIDataOfType(for: meta!, body: { dataBuffer in
                        var ts_num: Int = 4
                        var ts_denom: Int = 4
                        ts_num = Int(dataBuffer[0])
                        let pow = Float(dataBuffer[1])
                        ts_denom = Int(powf(2.0, pow))
                        let tempRange = CountableRange(Int(tsChangeStamp)...Int(stamp))
                        var denominatorDuration = 1.0
                        if ts_denom == 8 {
                            denominatorDuration = 0.5
                        }
                        var timeSignature = TimeSignature(numerator: ts_num, denominator: ts_denom, denominatorDuration: denominatorDuration, bar: barBeat.bar, timestamp: stamp)
                        timeSignature.beatRange = tempRange
                        timeSignatures.append(timeSignature)
                        tsChangeStamp = stamp
                    })
                default: ()
                }
            }
        default: ()
        }
        /
        MusicEventIteratorHasNextEvent(iterator!, &hasCurrentEvent)
        MusicEventIteratorNextEvent(iterator!)
    }
    return (tempo: tempoEvents, timeSignature: timeSignatures)
}

I have no idea yet whether this will work, since I have a long list of similar errors to fix before I can try it, but it seems to be a start.


Just an update. The above does work! Only minor change is using MemoryLayout<ExtendedTempoEvent>.size to set "capacity". Hopefully that's correct, but I am getting reasonable looking data.

That code is safe. But the withMemoryRebound workaround is not great because "rebinding" all of the MIDIMetaEvent and its payload data to be UInt8 is not really what you're after. I think code that views raw bytes in a piece of memory that's managed by someone else should use UnsafeRawPointer. I'll follow up with some suggestions.

func withMIDIDataOfType<T>(for eventPtr: UnsafePointer<MIDIMetaEvent>, body: (_ data: UnsafeRawPointer) throws -> T) rethrows -> T {
    let bytesBuffer = UnsafeRawPointer(eventPtr) + OffsetOfDataInMIDIMetaEvent
    return try body(bytesBuffer)
}
...
   case 0x58:
       withMIDIDataOfType(for: meta!, body: { dataBuffer in
           var ts_num: Int = 4
           var ts_denom: Int = 4
           ts_num = Int(dataBuffer.load(as:UInt8))
           let pow = Float(dataBuffer.load(fromByteOffset: 1, as: UInt8.self))