iOS TCP Client: downloading big text file

Hello. I use my iOS application to download big text files from the server over TCP. This is the code I use:


func openNetworkCommunication()
    {
        if(self.inputStream != nil && self.outputStream != nil)
        {
            closeNetworkCommunication()
        }
    
        let ipAddress = delegate?.ipAddressString as! CFStringRef    
        let portNumber = delegate?.ipPort
    
        var readStream:  Unmanaged<CFReadStream>?
        var writeStream: Unmanaged<CFWriteStream>?
        CFStreamCreatePairWithSocketToHost(nil, ipAddress, portNumber!, &readStream, &writeStream)
    
        self.inputStream = readStream!.takeRetainedValue()
        self.outputStream = writeStream!.takeRetainedValue()
    
        self.outputBuffer = delegate?.messageToServer.dataUsingEncoding(NSUTF8StringEncoding) as? NSMutableData

        self.inputStream?.delegate = self
        self.outputStream?.delegate = self
    
        self.inputStream?.scheduleInRunLoop(NSRunLoop.currentRunLoop(), forMode: NSDefaultRunLoopMode)
        self.outputStream?.scheduleInRunLoop(NSRunLoop.currentRunLoop(), forMode: NSDefaultRunLoopMode)
    
        self.inputStream?.open()
        self.outputStream?.open()
    }


    func closeNetworkCommunication()
    {
        self.inputStream?.close()
        self.outputStream?.close()
    
        self.inputStream?.removeFromRunLoop(NSRunLoop.currentRunLoop(), forMode: NSDefaultRunLoopMode)
        self.outputStream?.removeFromRunLoop(NSRunLoop.currentRunLoop(), forMode: NSDefaultRunLoopMode)
    
        self.inputStream?.delegate = nil
        self.outputStream?.delegate = nil
    
        self.inputStream = nil;
        self.outputStream = nil;
    
        self.outputBuffer = nil;
    }


    func stream(theStream : NSStream, handleEvent streamEvent : NSStreamEvent)
    {
        switch (streamEvent)
        {
        case NSStreamEvent.None:
            println("NSStreamEventNone")
        
        case NSStreamEvent.OpenCompleted:
            println("NSStreamEventOpenCompleted")
        
        case NSStreamEvent.HasBytesAvailable:
            println("NSStreamEvent.HasBytesAvailable")
        
            var buffer = [UInt8](count: 4096, repeatedValue: 0)
            if ( theStream == self.inputStream)
            {
            
                while (self.inputStream!.hasBytesAvailable)
                {
                    var len = inputStream!.read(&buffer, maxLength: buffer.count)

                    if len < 0
                    {
                        self.error = self.inputStream!.streamError
                        println("Input stream has less than 0 bytes")
                        closeNetworkCommunication()
                    }
                    
                    else if len == 0
                    {
                        println("Input stream has 0 bytes")
                        closeNetworkCommunication()
                    }
                
                    if(len > 0)
                    {
                        var messageFromServer = NSString(bytes: &buffer, length: buffer.count, encoding: NSUTF8StringEncoding)
                        delegate?.messageReceived(messageFromServer!)
                    }
                }
            }
        
        case NSStreamEvent.HasSpaceAvailable:
            println("NSStreamEventHasSpaceAvailable")
        
            if self.outputBuffer?.length != 0
            {
                var bytesWritten : Int = self.outputStream!.write(UnsafePointer<UInt8>(self.outputBuffer.bytes), maxLength: self.outputBuffer.length)
            
                if bytesWritten <= 0
                {
                    self.error = self.outputStream?.streamError
                }
                
                else
                {
                    self.outputBuffer?.replaceBytesInRange(NSMakeRange(0, bytesWritten), withBytes: nil, length: 0)
                }
            }
        
        case NSStreamEvent.ErrorOccurred:
            println("NSStreamEventErrorOccurred");           
        
        case NSStreamEvent.EndEncountered:
            println("NSStreamEventEndEncountered")
            theStream.close()
            theStream.removeFromRunLoop(NSRunLoop.currentRunLoop(), forMode: NSDefaultRunLoopMode)       
        
        default:
            println("Unknown event")
        }
    }


I have two big problems:


1) After the text has been transmitted to the iOS client, the connection remains open. If I try to do closeNetworkCommunication() in NSStreamEvent.HasBytesAvailable, then not the whole text is downloaded -- it is truncated in the middle. Is there a way to close the connection only after the full text has been passed?


2) Sometimes, the text passed to the iOS application includes a piece from the beginning of the text. For example:


Original:

AAAAAAAAAAAAAA

BBBBBBBBBBBBBB


Passed:

AAAAAAAAAAAAAA

BBBBBBBBBBBBBBAAAA


I am checking what is transmitted by the server and it is the original text... Does the code in the client fills the buffer with the remaining text to have the full size buffer? And how to fix it?


Many thanks!

Answered by DTS Engineer in 50882022

You should factor out you creation of

dataToFind
because it’s effectively a global constant. Check out Swift’s static property feature.

Why are you searching backwards (

NSDataSearchOptions.Backwards
)? You want to search from the beginning of the buffer so that you find the first block. If you search backwards, you’ll find the end of the last block, and the data before that may contain multiple blocks.

Also,

-subdataWithRange:
will give you just the terminator. What you want it to get the data up to the terminator. So, you should have:
let subData = self.inputBuffer!.subdataWithRange(NSMakeRange(0, indexOfData.location))

or:

let subData = self.inputBuffer!.subdataWithRange(NSMakeRange(0, indexOfData.location + indexOfData.length))

depending on whether you want

subData
to include the terminator or not.

Also, keep in mind that the newly arrived chunk of data might contain two blocks, so this code needs to go into a loop. You can either remove that data from the buffer each time around the loop, or adjust

searchRange
each time around the loop and then remove all the data in one hit at the end.

The following code shows an example of that first approach.

class BlankLineUnframer {

    private var inputBuffer: NSMutableData

    init() {
        self.inputBuffer = NSMutableData(capacity: 65536)!
    }

    private static let terminator = "\n\n".dataUsingEncoding(NSUTF8StringEncoding)!

    func blocksFromData(newData: NSData) -> [String] {
        var result: [String] = []

        self.inputBuffer.appendData(newData)
        repeat {
            // Search for the terminator.
            let terminatorRange: NSRange = self.inputBuffer.rangeOfData(
                BlankLineUnframer.terminator,
                options: [],
                range: NSMakeRange(0, self.inputBuffer.length)
            )
            if terminatorRange.location == NSNotFound {
                // +++ here you should check for self.inputBuffer being ridiculously large
                // If it's not present, we're done.
                break
            }
            // The block is everything up to the terminator.  Extract it as a string
            // and add it to our result array.
            let blockRange = NSMakeRange(0, terminatorRange.location)
            let blockData = self.inputBuffer.subdataWithRange(blockRange)
            let blockString = String(data: blockData, encoding: NSUTF8StringEncoding)
            // +++ here you should check for blockString being nil
            result.append(blockString!)
            // Remove the terminator and everything before it (the block) from the buffer.
            let rangeToEndOfTerminator = NSMakeRange(0, terminatorRange.location + terminatorRange.length)
            self.inputBuffer.replaceBytesInRange(rangeToEndOfTerminator, withBytes: nil, length: 0)
        } while (true)

        return result
    }
}

IMPORTANT See how I’ve put this code into a separate ‘unframer’ class? This is important because it allows you to test the code in isolation. Parsing data that comes off the network is tricky because the edge cases will only show up in the field. Worse yet, before this is network facting, any bugs in your parse may introduce security vulnerabilities. So, you should test your parser in a unit test rather than in your live code.

ps A lot of the declarations in your code are

var
rather than
let
for no apparent reason. In Swift you should default to
let
.

Share and Enjoy

Quinn "The Eskimo!"
Apple Developer Relations, Developer Technical Support, Core OS/Hardware

let myEmail = "eskimo" + "1" + "@apple.com"

For 2), you need to copy len bytes, not buffer.count bytes:

var messageFromServer = NSString(bytes: &buffer, length: len, encoding: NSUTF8StringEncoding)

I’ve moved your question over to Core OS > Networking because this is more about networking than about Swift.

There’s more than one issue here. guywithmazda spotted one, which is cool because I probably wouldn’t have spotted it myself. OTOH, I have a few more comments which I’ll outline below.

First, code like this:

while (self.inputStream!.hasBytesAvailable) {
    …
}

is incorrect. It’ll probably work but it’s not how you’re supposed to use NSStream. Rather, when you get a

.HasBytesAvailable
event, you should do one read, process those bytes, and then return. Continuing to work while the
-hasXxxAvailable
returns true can cause subtle problems. For example:
  • On a fast network you can end up ‘stuck’ in your stream event handler, which prevents anything else on that thread doing any work (which is bad if it’s the main thread). This can happen on both the send and receive sides.

  • On the send side, you end up being incompatible with

    TCP_NOTSENT_LOWAT
    . See WWDC 2015 Session 719 Your App and Next Generation Networks.

Your data-to-string conversion could run into problems because TCP does not preserve message boundaries. Imagine you send UTF-8 data for

"na\u{ef}ve"
. This translates to the string of bytes 6E 61 C3 AF 76 65. Now imagine that the TCP stack breaks this up into two chunks 6E 61 C3 and AF 76 65. Neither of these is valid UTF-8 and thus your NSString construction will fail.

You can get around this problem using a 'streaming' UTF-8 to string converter but that may not be sufficient. You can still run into other Unicode problems. For example, if naïve was encoded as

"nai\u{0308}ve"
(that is, with an i and a U+0308 COMBINING DIAERESIS), your approach could split the base character and the combining mark which will probably end badly.

If the data consists of UTF-8 text, and thus you can presume it is broken up into lines (or paragraphs, depending on how you think about ti), you can resolve both of these issues by buffering data until you get to the end of the line and then converting the entire line. This is what I do in the RemoteCurrency sample code.

Finally, your remaining unanswered question:

Is there a way to close the connection only after the full text has been passed?

The approach I use is to ignore non-positive results from

-read:maxLength:
. If you do that then you will get the
.ErrorOccurred
or
.EndEncountered
event depending on whether the connection closed cleanly or not. If it closed cleanly, you know you got all the data that was sent.

Share and Enjoy

Quinn "The Eskimo!"
Apple Developer Relations, Developer Technical Support, Core OS/Hardware

let myEmail = "eskimo" + "1" + "@apple.com"

Guys. Thank you so much for your help!


guywithmazda. Yes, this does address the problem. However, since eskimo says it is the wrong approach, I had to reconsider it.


eskimo. I did read your RemoteCurrency code many times, particularly the QCommandConnection class. I tried to replicate it in swift, but faced difficulties converting some code. It would really be great if the example was updated in Swift.


Meanwhile, I tried to incorporate both comments and came up with the following:


func stream(theStream : NSStream, handleEvent streamEvent : NSStreamEvent)
    {
        switch (streamEvent)
        {
               ...

               case NSStreamEvent.HasBytesAvailable:
                    processInput()
  
               case NSStreamEvent.HasSpaceAvailable:
                    self.hasSpaceAvailable = true
                    processOutput()      

               case NSStreamEvent.ErrorOccurred:
                    println("NSStreamEventErrorOccurred");
                    var connectionError : NSError = theStream.streamError!
                    let errorMsg = NSString(format: "Obtaining navdata information requires connection with X-Plane.\n Error %li: %@", connectionError.code, connectionError.localizedDescription)
                    delegate?.displayErrorWithTitle("No Connection with Server", errorMessage: errorMsg as String) 

               case NSStreamEvent.EndEncountered:
                     theStream.close()
                     theStream.removeFromRunLoop(NSRunLoop.currentRunLoop(), forMode: NSDefaultRunLoopMode)

               default:
                    println("Unknown event")
        }
    }


func processInput()
    {

        let bufferSize = 1024
        var buffer = Array<UInt8>(count: bufferSize, repeatedValue: 0)

        let bytesRead = self.inputStream!.read(&buffer, maxLength: bufferSize)

        if bytesRead < 0
        {
            self.error = self.inputStream!.streamError
            println("Input stream has less than 0 bytes")
            closeNetworkCommunication()
        }
   
        else if bytesRead == 0
        {
            println("Input stream has 0 bytes")
            closeNetworkCommunication()
        }

        if bytesRead >= 0
        {
   
            var messageFromServer = NSString(bytes: &buffer, length: bytesRead, encoding: NSUTF8StringEncoding)
   
            delegate?.messageReceived(messageFromServer!) 
        }
}


I tried it several times. Looks like it works, and the full text is passed to the Client.


However, I still have a problem with closing the connection. I believe that the connect still exists since I got the message "Input stream has 0 bytes" only after I close the server, but not as soon as the buffer is read.


eskimo. Did you mean this?


if bytesRead < 0
        {
            self.error = self.inputStream!.streamError
            println("Input stream has less than 0 bytes")
            closeNetworkCommunication()
        }
   
        else if bytesRead == 0
        {
            println("Input stream has 0 bytes")
            closeNetworkCommunication()
        }


when you said:


<q>The approach I use is to ignore non-positive results from -read:maxLength:. If you do that then you will get the .ErrorOccurred or .EndEncountered event depending on whether the connection closed cleanly or not.</q>


EDIT:


Another option that I came up with would be this:


else if bytesRead >= 0
        {
            self.inputBuffer.appendBytes(&buffer, length: bytesRead)
        }
      
        var messageFromServer = NSString(bytes: &buffer, length: bytesRead, encoding: NSUTF8StringEncoding)
      
        delegate?.messageReceived(messageFromServer!)


However, it still does not address the issue of closing the connection right after the full text has been downloaded.


Another issue is performing functions on the downloaded text. I would like to extract a part of the full downloaded text. What happens is the application downloads the first buffer, performs a function (extracts some phrase from the text); then downloads another buffer and performs the same functions; then downloads the third buffer and performs the function. I need to perform the function only once after the full text has been downloaded. Using background thread does not seem to help.

Another option that I came up with would be this:

Right. That’s more along the lines of the approach I now use. Consider the code in TLSToolCommon in the TLSTool sample code:

bytesRead = [self.inputStream read:buffer maxLength:sizeof(buffer)];
if (bytesRead > 0) {
    (void) fwrite(buffer, 1, (size_t) bytesRead, stdout);
    (void) fflush(stdout);
}

It simply reads the bytes and ignores any non-positive result. That’s because a result of 0 will trigger a subsequent

.EndEncountered
event and a negative result will trigger a subsequent
.ErrorOccurred
event.

However, it still does not address the issue of closing the connection right after the full text has been downloaded.

Can you be more specific about that problem? Once you get the

.EndEncountered
event, you’ve received all the bytes you’re going to receive, so I’m not sure as to what problem you’re still having here.

Another issue is performing functions on the downloaded text. I would like to extract a part of the full downloaded text. What happens is the application downloads the first buffer, performs a function (extracts some phrase from the text); then downloads another buffer and performs the same functions; then downloads the third buffer and performs the function. I need to perform the function only once after the full text has been downloaded.

Processing streaming text is harder than in seems for two reasons:

  • the Unicode issues I discussed earlier

  • the text of interest may span two buffers

In a lot of cases breaking the text up into lines is the right choice here because:

  • it avoids the Unicode entanglements

  • it’s often reasonable to set a maximum line length, which limits the amount of data you have to carry over from one buffer to the next

However, that really depends on what “extracts some phrase from the text” means. Specifically:

  • Do you expect those phrases to span lines?

  • How Unicode correct do you need this to be?

If you need to support multi-line phrases that might contain arbitrary Unicode, things do get quite complex here.

Using background thread does not seem to help.

To (slightly mis)quote Buffy: Those things? Never helpful.

Share and Enjoy

Quinn "The Eskimo!"
Apple Developer Relations, Developer Technical Support, Core OS/Hardware

let myEmail = "eskimo" + "1" + "@apple.com"

eskimo. First of all, thank you so much for bearing with me and providing me with your extremely useful support. Reading your explanation helps me a lot.


1) I do not get the .EndEncountered event after the text has been loaded. I only get it when the server gets closed. This is printed after I have closed the server:


NSStreamEvent.HasBytesAvailable

NSStreamEventEndEncountered

CLOSE NETWORK CONNECTION


Until then, the connection remains open (no NSStreamEventEndEncountered).


2) The text is simple. I will try to illustrate what exactly I meant. The text is a number of blocks of lines separated by a space. Each line represents a comma separated data. The first element of each line is a key that I use to pull the appropriate data. For example, 'A' stands for airport and I try to extract its id and name from the line that has this key. And this is what I get (some lines are removed to mind the space). Look at the highlighted text. Once the first buffer is read, it extracts the id and name just fine. Then the second buffer is transmitted, and, of course it does not find the key 'A' and the id and name are empty.


A,EHAM,SCHIPHOL,52.308056,4.764167,-11,3000,0,12400,0

R,04,41,6608,148,0,0.000,0,52.300372,4.783483,-13,3.00,50,1,0


SID,ANDI1N,09,5

CA,0,87.0,2,500,0,0,0,0,0,0

DF,EH052,52.324167,4.899722,0, ,0.0,0.0,0,0,0,0,0,0,0,0

TF,EH043,52.425278,5.131667,0, ,0.0,0.0,0.0,0.0,0,0,0,0,0,0,0,0

TF,ANDIK,52.739403,5

ICAO: EHAM

name: SCHIPHOL

NSStreamEvent.HasBytesAvailable

.270489,0, ,0.0,0.0,0.0,0.0,0,0,0,0,0,0,0,0


SID,ANDI1S,24,5

CF,EH005,52.273889,4.697500,0,SPY,199.8,17.0,238.0,3.0,0,0,0,0,0,0,0,1

CF,EH008,52.208333,4.766667,1,AMS,163.2,7.8,118.0,3.0,0,0,0,1,220,0,0,0

TF,EH026,52.171389,4.869722,0, ,0.0,0.0,0.0,0.0,0,0,0,0,0,0,0,0

TF,PAM,52.334761,5.092161,0, ,0.0,0.0,0.0,0.0,0,0,0,0,0,0,0,0

TF,ANDIK,52.739403,5.270489,0, ,0.0,0.0,0.0,0.0,0,0,0,0,0,0,0,0


SID,ANDI2F,04,5

CF,EH019,52.338056,4.837222,0,SPL,83.7,3.2,41.0,3.0,0,0,

ICAO:

name:

NSStreamEvent.HasBytesAvailable

0,0,0,0,0,0

TF,EH020,52.336945,4.928889,0, ,0.0,0.0,0.0,0.0,0,0,0,0,0,0,0,0

TF,EH043,52.425278,5.131667,0, ,0.0,0.0,0.0,0.0,0,0,0,0,0,0,0,0

TF,ANDIK,52.739403,5.270489,0, ,0.0,0.0,0.0,0.0,0,0,0,0,0,0,0,0


SID,ARNE1V,36L,5

CF,EH012,52.420278,4.717500,0,SPY,214.8,8.8,3.0,5.0,0,0,0,0,0,0,0,0

TF,EH087,52.480000,4.772222,0, ,0.0,0.0,0.0,0.0,0,0,0,0,0,0,0,0

TF,EH088,52.487500,4.915278,0, ,0.0,0.0,0.0,0.0,0,0,0,0,0,0,0,0

TF,PAM,52.334761,5.092161,0, ,0.0,0

ICAO:

name:

NSStreamEvent.HasBytesAvailable

.0,0.0,0.0,0,0,0,0,0,0,0,0

TF,ARNEM,52.096447,6.076603,0, ,0.0,0.0,0.0,0.0,0,0,0,0,0,0,0,0


This is the messageReceived function in the delegate:


func messageReceived(message : NSString)
    {
        println(message)
        self.navdataAirportDep = message as String
        var portDepart = IGAAirportNavigraph(portNavigationData: self.navdataAirportDep)
   
        var icaoDep = portDepart.getAirportIcao()
        var nameDep = portDepart.getAirportName()
   
        println("ICAO: \(icaoDep)")
        println("name: \(nameDep)")
    }


I have no idea how to delay implementation of the function in the following until all bytes are read:


if bytesRead > 0
        {
            self.inputBuffer!.appendBytes(&buffer, length: bytesRead)
            var messageFromServer = NSString(bytes: &buffer, length: bytesRead, encoding: NSUTF8StringEncoding)
            delegate?.messageReceived(messageFromServer!)
        }


EDIT: Potentially, I could put the function in the NSStreamEventEndEncountered. However, as I said above, this does not happen automatically after the text has been downloaded.


Thank you so much!

1) I do not get the .EndEncountered event after the text has been loaded. I only get it when the server gets closed. This is printed after I have closed the server:

Until then, the connection remains open (no NSStreamEventEndEncountered).

Hmmm, OK. I was under the impression that the server closes the connection after sending you all the data. Clearly that’s not the case.

That makes things tricky because you have to decide when to close the connection. Is there something in the data that indicates an EOF? If not, you’ll have to use some heuristic (like a timer).

Regardless, I don’t think this should impact on how you parse the data, as I’ll discuss below.

2) The text is simple.

Indeed, we’re talking ASCII level simple here. Which is good news because it eliminates (literally) a world of potential complexity.

The text is a number of blocks of lines separated by a space.

By a space? Or a blank line?

From the example you posted it looks like a blank line. Presumably it’s reasonable to put a bound on the size of these blocks as well. That is, you wouldn’t expect the block to take more than, say, 64 KiB.

What you have here is an unframing problem. That is, you have an incoming byte stream and you want to break it up into frames (blocks of text). This can be quite tricky to get right, and there’s definitely a trade-off between performance and reliability.

IMPORTANT One critical aspect of reliability is security. It’s easy to write fast and elegant unframing code that is vulnerable to buffer overrun attacks (-:

In your case I’d do something like this:

  • maintain an NSMutableData buffer

  • on

    .HasBytesAvailable
    , read a chunk of data and append it to that
  • then use

    -rangeOfData:options:range:
    to search the buffer for your block terminator (more on this below)
  • if you find it:

    1. extract a subdata using

      -subdataWithRange:
    2. build an NSString from that

    3. use

      -replaceBytesInRange:withBytes:length:
      to remove it from the front of the buffer
  • if you don’t find the terminator in the buffer, check the buffer size; if it’s too big, shut everything down

The last step is important because it prevents a broken (or malicious) server from using all of your app’s memory.

This algorithm is not particularly efficient but that shouldn’t be a problem unless your dealing with thousands of records.

On the subject of terminators, remember that there are lots of potential options:

  • If the server is using standard Internet conventions, a line terminator will be CR LF, so you can find a blank line by searching for [13, 10, 13, 10].

  • Alternatively, the server could just be using LF, in which case it’s [10, 10].

  • Similarly, just CR results in [13, 13].

There’s also the chance that the server is doing something off the wall, like using LF for a line terminator and CR LF for a block terminator. You’ll have to dump the incoming data as hex to know for sure.

And if the server is doing something truly weird, it’s possible that it’ll be too complex to find using

-replaceBytesInRange:withBytes:length:
.

Share and Enjoy

Quinn "The Eskimo!"
Apple Developer Relations, Developer Technical Support, Core OS/Hardware

let myEmail = "eskimo" + "1" + "@apple.com"

Thank you very much, eskimo. One step forward. Following your advice, I am using this code now:


func processInput()
    {
        let bufferSize = 2056
        var buffer = Array<UInt8>(count: bufferSize, repeatedValue: 0)
        var bytesRead = self.inputStream!.read(&buffer, maxLength: buffer.count)

        if bytesRead > 0
        {
            self.inputBuffer!.appendBytes(&buffer, length: bytesRead)
          
            var searchRange : NSRange = NSMakeRange(0, self.inputBuffer!.length)
            var searchString : String = "\n\n"
            var dataToFind : NSData = searchString.dataUsingEncoding(NSUTF8StringEncoding, allowLossyConversion: false)!
            var indexOfData : NSRange = self.inputBuffer!.rangeOfData(dataToFind, options: NSDataSearchOptions.Backwards, range: searchRange)
          
            if (indexOfData.length > 0)
            {
                println("Index Of Data")     
                var subData = self.inputBuffer!.subdataWithRange(indexOfData)
                println(subData)
                var StringData = NSString(data: subData, encoding: NSUTF8StringEncoding)
                println(StringData)
            }
            else
            {
                println("Data not found ...")
            }
        }
    }


And this is my output from lines 18, 20, and 22:


NSStreamEvent.HasBytesAvailable

Index Of Data

<0a0a>

Optional(

)

NSStreamEvent.HasBytesAvailable

Index Of Data

<0a0a>

Optional(

)

NSStreamEvent.HasBytesAvailable

Index Of Data

<0a0a>

Optional(

)

...



So, on the one have, I believe that I have the right terminator (prints "Index Of Data"). On the other hand, the original text file has several times more blank lines than what I see in the console. Plus, -subdataWithRange does not seem to be working in my case. Anything I am missing now?


Many thanks!

Accepted Answer

You should factor out you creation of

dataToFind
because it’s effectively a global constant. Check out Swift’s static property feature.

Why are you searching backwards (

NSDataSearchOptions.Backwards
)? You want to search from the beginning of the buffer so that you find the first block. If you search backwards, you’ll find the end of the last block, and the data before that may contain multiple blocks.

Also,

-subdataWithRange:
will give you just the terminator. What you want it to get the data up to the terminator. So, you should have:
let subData = self.inputBuffer!.subdataWithRange(NSMakeRange(0, indexOfData.location))

or:

let subData = self.inputBuffer!.subdataWithRange(NSMakeRange(0, indexOfData.location + indexOfData.length))

depending on whether you want

subData
to include the terminator or not.

Also, keep in mind that the newly arrived chunk of data might contain two blocks, so this code needs to go into a loop. You can either remove that data from the buffer each time around the loop, or adjust

searchRange
each time around the loop and then remove all the data in one hit at the end.

The following code shows an example of that first approach.

class BlankLineUnframer {

    private var inputBuffer: NSMutableData

    init() {
        self.inputBuffer = NSMutableData(capacity: 65536)!
    }

    private static let terminator = "\n\n".dataUsingEncoding(NSUTF8StringEncoding)!

    func blocksFromData(newData: NSData) -> [String] {
        var result: [String] = []

        self.inputBuffer.appendData(newData)
        repeat {
            // Search for the terminator.
            let terminatorRange: NSRange = self.inputBuffer.rangeOfData(
                BlankLineUnframer.terminator,
                options: [],
                range: NSMakeRange(0, self.inputBuffer.length)
            )
            if terminatorRange.location == NSNotFound {
                // +++ here you should check for self.inputBuffer being ridiculously large
                // If it's not present, we're done.
                break
            }
            // The block is everything up to the terminator.  Extract it as a string
            // and add it to our result array.
            let blockRange = NSMakeRange(0, terminatorRange.location)
            let blockData = self.inputBuffer.subdataWithRange(blockRange)
            let blockString = String(data: blockData, encoding: NSUTF8StringEncoding)
            // +++ here you should check for blockString being nil
            result.append(blockString!)
            // Remove the terminator and everything before it (the block) from the buffer.
            let rangeToEndOfTerminator = NSMakeRange(0, terminatorRange.location + terminatorRange.length)
            self.inputBuffer.replaceBytesInRange(rangeToEndOfTerminator, withBytes: nil, length: 0)
        } while (true)

        return result
    }
}

IMPORTANT See how I’ve put this code into a separate ‘unframer’ class? This is important because it allows you to test the code in isolation. Parsing data that comes off the network is tricky because the edge cases will only show up in the field. Worse yet, before this is network facting, any bugs in your parse may introduce security vulnerabilities. So, you should test your parser in a unit test rather than in your live code.

ps A lot of the declarations in your code are

var
rather than
let
for no apparent reason. In Swift you should default to
let
.

Share and Enjoy

Quinn "The Eskimo!"
Apple Developer Relations, Developer Technical Support, Core OS/Hardware

let myEmail = "eskimo" + "1" + "@apple.com"

eskimo.


Thank you very much for your detailed response and corrections. I still struggle sometimes with Swift (especially things like 'var' vs 'let' and unwrapping of optionals). Your code did work and allowed me to do what it was supposed to do. After some thinking, I decided that I needed to simplify the whole process a bit and decided that I'd rather make changes in the server (which I also developed) and insert a terminator in each message passed from it to the Client. So, now every message finishes with "END_OF_DATA". Then I looked at all your sugestions and came up with the following:



init()
    {
        self.inputBuffer = NSMutableData(capacity: 65536)!
    }


func stream(theStream : NSStream, handleEvent streamEvent : NSStreamEvent)
    {
        switch (streamEvent)
        {
        case NSStreamEvent.None:
            println("NSStreamEventNone")
        
        case NSStreamEvent.OpenCompleted:
            println("NSStreamEventOpenCompleted")
        
        case NSStreamEvent.HasBytesAvailable:
            processInput()


        case NSStreamEvent.HasSpaceAvailable:
            println("NSStreamEventHasSpaceAvailable")
            self.hasSpaceAvailable = true
            processOutput()
         
        case NSStreamEvent.ErrorOccurred:
            println("NSStreamEventErrorOccurred");
            closeNetworkCommunication()
            var connectionError : NSError = theStream.streamError!
            let errorMsg = NSString(format: "Obtaining navdata information requires connection with X-Plane.\n Error %li: %@", connectionError.code, connectionError.localizedDescription)
            delegate?.displayErrorWithTitle("No Connection with Server", errorMessage: errorMsg as String)  
         
        case NSStreamEvent.EndEncountered:
            println("NSStreamEventEndEncountered")
            closeNetworkCommunication()
         
        default:
            println("Unknown event")
        }
    }


func processInput()
    {
        var result : [String] = []
        let bufferSize = 4116
        var buffer = Array<UInt8>(count: bufferSize, repeatedValue: 0)
        var bytesRead = self.inputStream!.read(&buffer, maxLength: buffer.count)
      
        if bytesRead > 0
        {
            self.inputBuffer?.appendBytes(&buffer, length: bytesRead)
            let terminatorRange: NSRange = self.inputBuffer!.rangeOfData(terminator,options: nil,range: NSMakeRange(0, self.inputBuffer!.length))
            let blockString = NSString(data: self.inputBuffer!, encoding: NSUTF8StringEncoding)
            result.append(blockString! as String)
          
            // When the terminator is found, process the data
            if terminatorRange.location != NSNotFound
            {
                delegate?.messageReceived("".join(result))
                closeNetworkCommunication()
                  
                // Clean the buffer
                let rangeToEndOfTerminator = NSMakeRange(0, self.inputBuffer.length)
                self.inputBuffer!.replaceBytesInRange(rangeToEndOfTerminator, withBytes: nil, length: 0)
            }
          
        }
      
        else if bytesRead < 0
        {
            self.error = self.inputStream!.streamError
            println("Input stream has less than 0 bytes")
            closeNetworkCommunication()
        }

        else if bytesRead == 0
        {
            println("Input stream has 0 bytes")
            closeNetworkCommunication()
        }
      
    }


Checked it a few dozens of times. Looks like it is working. Would not have been the case without your help! Since this is the most vulnerable part of my application (because it deals with TCP connection), I had to make it reliable. I am still wondering whether I need to have a condition in the case when the terminator has not been found after loading the text. On the one hand, this is TCP, so there should not be any missing data. And the server does add the terminator every time it creates the message. If the connection stops in the middle of transmission, then the code should close the connection. If you have any advice, it would be very appreciated.


So, thank you again for all your support!

iOS TCP Client: downloading big text file
 
 
Q