Fragment large size data sent and received using NSKeyedArchiver.archivedData in GameCenter

Trying to send and receive data in the GameCenter environment using the following methods:

func sendData(dictionaryWithData dictionary: Dictionary<String, Any>,toPeer targetPeers: [GKPlayer]) {
        guard let match = self.match else { return }
        do {
            let dataToSend = try NSKeyedArchiver.archivedData(withRootObject: dictionary, requiringSecureCoding: false)
            try match.send(dataToSend, to: targetPeers, dataMode: .reliable)
        }
        catch {
            #if DEBUG
            print("CONNECTION MANAGER SEND DATA ERROR")
            #endif
        }
    }
public func match(_ theMatch: GKMatch,didReceive data: Data,forRecipient recipient: GKPlayer,fromRemotePlayer player: GKPlayer) { 
        if match != theMatch { return } 
        DispatchQueue.main.async { 
           do { 
               guard let message = NSDictionary.unsecureUnarchived(from: data) as? Dictionary<String, Any> else {return} 
...
<CODE> 
...
} 

///Source: https://stackoverflow.com/questions/51487622/unarchive-array-with-nskeyedunarchiver-unarchivedobjectofclassfrom 
static func unsecureUnarchived(from data: Data) -> Self? {
        do {
            let unarchiver = try NSKeyedUnarchiver(forReadingFrom: data)
            unarchiver.requiresSecureCoding = false
            let obj = unarchiver.decodeObject(of: self, forKey: NSKeyedArchiveRootObjectKey)
            if let error = unarchiver.error {
                print("Error:\(error)")
            }
            return obj
        } catch {
            print("Error:\(error)")
        }
        return nil
    }

Everything works great until the data exceeds 87K (which, I understand, is the limit for exchanging data in GameCenter).

The data is not sent and gives the following error: Async message[1FCA0D11-05DE-47D0-9714-983C8023F5C1] send error: FailedToSendData: , InternalError: reliable, maxPayloadSizeExceeded

Interesting enough, I do not have this problem when using MCSession, as follows, even if data exceeds 87K:

func sendData(dictionaryWithData dictionary: Dictionary<String, Any>, toPeer targetPeers: [MCPeerID]) {
        do {
            let dataToSend = try NSKeyedArchiver.archivedData(withRootObject: dictionary, requiringSecureCoding: false)
            try session.send(dataToSend, toPeers: targetPeers, with: MCSessionSendDataMode.reliable)
        }
        catch {
            #if DEBUG
            print("CONNECTION MANAGER SEND DATA ERROR")
            #endif
        }
    }

I have been doing research and found that I need to fragment data and send and receive it in packages. But I could not find a good explanation how to do it.

Any help would be appreciated!

Answered by DTS Engineer in 796112022

So, there are two parts to this:

  • GameKit and its limitations

  • Fragmenting messages

I can help with the latter, but I’m not an expert on the former. If, for example, you say:

which, I understand, is the limit for exchanging data in GameCenter

I’m going to take your word on it (-:

In terms of how your fragment large messages, that’s a pretty standard networking technique. There are lots of ways to approach this, but a common technique is:

  1. Split the data into chunks.

  2. For each chunk, form a fragment by prepending a header that positions the chunk bytes within the larger message.

  3. Send it fragment as a separate message.

This assumes that your underlying network is reliable, that is, it guarantees that each fragment will eventually either arrive intact or you’ll be notified of a failure. If the underlying network isn’t reliable, you have to take further steps, like adding a checksum or implementing a retry mechanism.

I’ve included a very cut down example of how you might do this at the end of this post.


Still, I’m concerned about you doing this at all. When a networking API imposes a limit like this, there’s usually a good reason. If you try to get around that limit, you could gain some hard experience as to why the limit is present in the first place.

It might be better to not do this, and instead try to optimise the amount of data that you send. Notably, NSKeyedArchiver is not a compact format. It contains a lot of redundancy. If you switch to a more compact format, like a binary property list, you may find that your data fits under the limit, and that’ll get you out of the fragmentation business entirely. Oh, and Swift has great support for binary property list via Foundation’s Codable mechanism.


Finally…

WARNING When using keyed archiving for network data, you must not disable secure coding. Secure coding was designed to protect you from vulnerabilities that are fundamental to the keyed archiving mechanism. If you disable it, you open yourself up to those vulnerabilities.

Notably, binary properties lists don’t have this problem.

Share and Enjoy

Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"


import Foundation

// This code requires the `Data` parsing extensions from:
//
// <https://developer.apple.com/forums/thread/652469?answerId=786462022#786462022>.

… elided …

// The follow is an extension of the ideas in the above post.

extension Data {
    
    mutating func appendBigEndian<T>(_ n: T) where T: FixedWidthInteger {
        let bytes = (0..<(T.bitWidth / 8)).reversed().map { byteNumber -> UInt8 in
            UInt8((n >> (byteNumber * 8)) & 0xff)
        }
        self.append(contentsOf: bytes)
    }
    
}

/// Splits a message into fragments.

func fragmentsForMessage(_ message: Data, messageID: UInt32, maxMessageBytesInFragment: Int) -> [Data] {
    precondition(!message.isEmpty)
    precondition(maxMessageBytesInFragment >= 1)
    precondition(UInt16(exactly: maxMessageBytesInFragment) != nil)
    let fragmentCount = (message.count + (maxMessageBytesInFragment - 1)) / maxMessageBytesInFragment
    guard let fragmentCount16 = UInt16(exactly: fragmentCount) else { fatalError() }

    var result: [Data] = []
    var residual = message
    while !residual.isEmpty {
        var fragment = Data()
        fragment.appendBigEndian(messageID)
        fragment.appendBigEndian(UInt16(result.count))
        fragment.appendBigEndian(fragmentCount16)

        let fragmentBytes = residual.prefix(maxMessageBytesInFragment)
        fragment.append(fragmentBytes)
        residual = residual.dropFirst(fragmentBytes.count)

        result.append(fragment)
    }
    return result
}

/// Parses a fragment to extract the header info and the payload.

func infoForFragment(_ fragment: Data) -> (messageID: UInt32, index: Int, count: Int, payload: Data)? {
    var residual = fragment
    guard
        let messageID = residual.parseBigEndian(UInt32.self),
        let fragmentIndex16 = residual.parseBigEndian(UInt16.self),
        let fragmentCount16 = residual.parseBigEndian(UInt16.self)
    else { return nil }
    return (messageID, Int(fragmentIndex16), Int(fragmentCount16), Data(residual))
}

func main() {
    let message = Data("Hello Cruel World".utf8)
    let fragments = fragmentsForMessage(message, messageID: 0xa1a2a3a4, maxMessageBytesInFragment: 6)
    for fragment in fragments {
        print((fragment as NSData).debugDescription)
    }
    // prints:
    // <a1a2a3a4 00000003 48656c6c 6f20>
    // <a1a2a3a4 00010003 43727565 6c20>
    // <a1a2a3a4 00020003 576f726c 64>
    for fragment in fragments {
        print(infoForFragment(fragment)!)
    }
    // prints:
    // (messageID: 2711790500, index: 0, count: 3, payload: 6 bytes)
    // (messageID: 2711790500, index: 1, count: 3, payload: 6 bytes)
    // (messageID: 2711790500, index: 2, count: 3, payload: 5 bytes)
}

main()
Accepted Answer

So, there are two parts to this:

  • GameKit and its limitations

  • Fragmenting messages

I can help with the latter, but I’m not an expert on the former. If, for example, you say:

which, I understand, is the limit for exchanging data in GameCenter

I’m going to take your word on it (-:

In terms of how your fragment large messages, that’s a pretty standard networking technique. There are lots of ways to approach this, but a common technique is:

  1. Split the data into chunks.

  2. For each chunk, form a fragment by prepending a header that positions the chunk bytes within the larger message.

  3. Send it fragment as a separate message.

This assumes that your underlying network is reliable, that is, it guarantees that each fragment will eventually either arrive intact or you’ll be notified of a failure. If the underlying network isn’t reliable, you have to take further steps, like adding a checksum or implementing a retry mechanism.

I’ve included a very cut down example of how you might do this at the end of this post.


Still, I’m concerned about you doing this at all. When a networking API imposes a limit like this, there’s usually a good reason. If you try to get around that limit, you could gain some hard experience as to why the limit is present in the first place.

It might be better to not do this, and instead try to optimise the amount of data that you send. Notably, NSKeyedArchiver is not a compact format. It contains a lot of redundancy. If you switch to a more compact format, like a binary property list, you may find that your data fits under the limit, and that’ll get you out of the fragmentation business entirely. Oh, and Swift has great support for binary property list via Foundation’s Codable mechanism.


Finally…

WARNING When using keyed archiving for network data, you must not disable secure coding. Secure coding was designed to protect you from vulnerabilities that are fundamental to the keyed archiving mechanism. If you disable it, you open yourself up to those vulnerabilities.

Notably, binary properties lists don’t have this problem.

Share and Enjoy

Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"


import Foundation

// This code requires the `Data` parsing extensions from:
//
// <https://developer.apple.com/forums/thread/652469?answerId=786462022#786462022>.

… elided …

// The follow is an extension of the ideas in the above post.

extension Data {
    
    mutating func appendBigEndian<T>(_ n: T) where T: FixedWidthInteger {
        let bytes = (0..<(T.bitWidth / 8)).reversed().map { byteNumber -> UInt8 in
            UInt8((n >> (byteNumber * 8)) & 0xff)
        }
        self.append(contentsOf: bytes)
    }
    
}

/// Splits a message into fragments.

func fragmentsForMessage(_ message: Data, messageID: UInt32, maxMessageBytesInFragment: Int) -> [Data] {
    precondition(!message.isEmpty)
    precondition(maxMessageBytesInFragment >= 1)
    precondition(UInt16(exactly: maxMessageBytesInFragment) != nil)
    let fragmentCount = (message.count + (maxMessageBytesInFragment - 1)) / maxMessageBytesInFragment
    guard let fragmentCount16 = UInt16(exactly: fragmentCount) else { fatalError() }

    var result: [Data] = []
    var residual = message
    while !residual.isEmpty {
        var fragment = Data()
        fragment.appendBigEndian(messageID)
        fragment.appendBigEndian(UInt16(result.count))
        fragment.appendBigEndian(fragmentCount16)

        let fragmentBytes = residual.prefix(maxMessageBytesInFragment)
        fragment.append(fragmentBytes)
        residual = residual.dropFirst(fragmentBytes.count)

        result.append(fragment)
    }
    return result
}

/// Parses a fragment to extract the header info and the payload.

func infoForFragment(_ fragment: Data) -> (messageID: UInt32, index: Int, count: Int, payload: Data)? {
    var residual = fragment
    guard
        let messageID = residual.parseBigEndian(UInt32.self),
        let fragmentIndex16 = residual.parseBigEndian(UInt16.self),
        let fragmentCount16 = residual.parseBigEndian(UInt16.self)
    else { return nil }
    return (messageID, Int(fragmentIndex16), Int(fragmentCount16), Data(residual))
}

func main() {
    let message = Data("Hello Cruel World".utf8)
    let fragments = fragmentsForMessage(message, messageID: 0xa1a2a3a4, maxMessageBytesInFragment: 6)
    for fragment in fragments {
        print((fragment as NSData).debugDescription)
    }
    // prints:
    // <a1a2a3a4 00000003 48656c6c 6f20>
    // <a1a2a3a4 00010003 43727565 6c20>
    // <a1a2a3a4 00020003 576f726c 64>
    for fragment in fragments {
        print(infoForFragment(fragment)!)
    }
    // prints:
    // (messageID: 2711790500, index: 0, count: 3, payload: 6 bytes)
    // (messageID: 2711790500, index: 1, count: 3, payload: 6 bytes)
    // (messageID: 2711790500, index: 2, count: 3, payload: 5 bytes)
}

main()

Thank you so much. I also did notice that the size is unreasonably large. For example, an array of 52 elements of a custom object Card with the following init methods takes about 70KB:

required init?(coder aDecoder: NSCoder) {
    self.suit = Suit(rawValue: aDecoder.decodeInteger(forKey: "suit"))! // (custom enum)
    self.rank = Rank(rawValue: aDecoder.decodeInteger(forKey: "rank"))! // (custom enum)
    self.positionX = CGFloat (aDecoder.decodeDouble(forKey: "positionX"))
    self.positionY = CGFloat (aDecoder.decodeDouble(forKey: "positionY"))
    self.frontTexture = aDecoder.decodeInteger(forKey: "frontTexture")
    self.backTexture = SKTexture(imageNamed: "card_back")
    self.faceUp = aDecoder.decodeBool(forKey: "faceUp")
    self.possession = aDecoder.decodeObject(forKey: "possession") as! String
    self.possessionPrevious =   aDecoder.decodeObject(forKey: "possessionPrevious") as! String
    self.notDumpable = aDecoder.decodeBool(forKey: "notDumpable")
    self.intersectsDeck = aDecoder.decodeInteger(forKey: "intersectsDeck")
    super.init(coder: aDecoder)
}

override func encode(with aCoder: NSCoder) {
    super.encode(with: aCoder)
    aCoder.encode(suit.rawValue, forKey: "suit")
    aCoder.encode(rank.rawValue, forKey: "rank")
    aCoder.encode(Double(positionX), forKey: "positionX")
    aCoder.encode(Double(positionY), forKey: "positionY")
    aCoder.encode(frontTexture, forKey: "frontTexture")
    aCoder.encode(faceUp, forKey: "faceUp")
    aCoder.encode(possession, forKey: "possession")
    aCoder.encode(possessionPrevious, forKey: "possessionPrevious")
    aCoder.encode(notDumpable, forKey: "notDumpable")
    aCoder.encode(intersectsDeck, forKey: "intersectsDeck")
}

So, I did consider sending data using property list encoding and decoding. However, I read that it is not possible to use it to encode/decode Dictionary [String:Any], which I need to send (e.g., https://stackoverflow.com/questions/53585848/swift-encode-and-decode-a-dictionary-stringany-into-plist). Is this true?

Another option may be to use JSONSerialization, but I am not sure whether it is significantly smaller than NSKeyedArchiver.

I read that it is not possible to use it to encode/decode Dictionary [String:Any], which I need to send

Honestly, I think the best path forward is to change this requirement. [String: Any] is almost always a mistake in networking code because it’s not possible to decode the Any securely. Your dictionary values could be literally any time, so an attacker can supply an arbitrary type of their own choosing. That’s hard to defend against.

Both secure coding and Foundation’s Codable are designed to systematically protect you from this attack, and that’s the path I recommend.

However, if you absolutely insist on going down this path, it is possible to do it with Foundation’s untyped property list and JSON serialisation types, namely PropertyListSerialization and JSONSerialization.

Getting an error: Invalid type in JSON write (MCPeerID)

Correct. JSON serialisation is limited to JSON types, and MCPeerID is not one of those. Standard practice here is to get the UUID from the peer ID and serialise that as a string.

Finally, how is MCPeerID coming into this? I thought you were using GameKit?

Share and Enjoy

Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

Finally, how is MCPeerID coming into this? I thought you were using GameKit?

Actually, I am using both GameKit and MCSession.

Trying to play with fragmenting the data, as you prompted. Although it is more or less clear how to send data:

func sendData(dictionaryWithData dictionary: Dictionary<String, Any>, toPeer targetPeers: [MCPeerID]) {
    do {
        let dataToSend = try NSKeyedArchiver.archivedData(withRootObject: dictionary, requiringSecureCoding: true)
        
        let fragments = fragmentsForMessage(dataToSend, messageID: 0xa1a2a3a4, maxMessageBytesInFragment: 50000)
            for fragment in fragments {
                try session.send(fragment, toPeers: targetPeers, with: MCSessionSendDataMode.reliable)
            }
    }
    catch {
    }
}

... I am frankly a bit stuck as to how to handle it in

func session(_ session: MCSession,didReceive data: Data,fromPeer peerID: MCPeerID)
{
    DispatchQueue.main.async {
        do {
               ....

Append data to a property and then at some point implement NSDictionary.unsecureUnarchived? How to know when to implement? When reading the data, I am also getting errors

Error:Error Domain=NSCocoaErrorDomain Code=4864 "*** -[NSKeyedUnarchiver _initForReadingFromData:error:throwLegacyExceptions:]: incomprehensible archive (0xffffffa1, 0xffffffa2, 0xffffffa3, 0xffffffa4, 0x0, 0x0, 0x0, 0x6)" UserInfo={NSDebugDescription=*** -[NSKeyedUnarchiver _initForReadingFromData:error:throwLegacyExceptions:]: incomprehensible archive (0xffffffa1, 0xffffffa2, 0xffffffa3, 0xffffffa4, 0x0, 0x0, 0x0, 0x6)}

Actually, I am using both GameKit and MCSession.

Ah, a glutton for punishment then? (-:

Seriously though, I generally recommend against using Multipeer Connectivity. IMO that API should be deprecated. Sadly, that hasn’t happened, so there are limits to what I can say officially. See TN3151 Choosing the right networking API for that. OTOH, here on DevForums, I can go into more details.

Append data to a property and then at some point implement NSDictionary.unsecureUnarchived?

With reference to the code I posted earlier, you’d need to:

  1. Call infoForFragment(_:) on each incoming fragment.

  2. From that you get a message ID, index, count, and payload. For each message ID you need a data structure, which could just be an array, to hold incoming fragments for that message. Put this fragment into that data structure.

  3. If you’ve received all the fragments for this message, sort the payloads by the index, then merge them into one Data value, then deserialise that.

Share and Enjoy

Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

Fragment large size data sent and received using NSKeyedArchiver.archivedData in GameCenter
 
 
Q