NSURLSession, streamTaskWithHostName and data in chunks

I have some equipment with sensors that sends data in chunks. I am trying to use NSURLSession with streamTaskWithHostName as shown in WWDC 15...


            task = NSURLSession.sharedSession().streamTaskWithHostName("10.0.0.1", port: 32076)
            task.resume()
         
            task.readDataOfMinLength(1, maxLength: 65536, timeout: 30.0) {
                (data: NSData?, eof: Bool, error: NSError?) in
                if data != nil {
                    let xmlString = String(NSString(data: data!, encoding:NSUTF8StringEncoding))
                    print(xmlString)
                    self.xmlParser = NSXMLParser(data: data!)
                    if self.xmlParser != nil {
                        print("xmlParser created")
                        self.xmlParser.delegate = self
                        self.xmlParser.parse()
                        print(self.xmlParser.lineNumber)
                    }
                }
            }


readDataOfMinLength is called just once and returns one line... How can I read the rest of the data? Should I forget NSURLSession and use the legacy NSStream?


Thanks

Replies

You’re running up against a fundamental feature of TCP: it does not preserve record boundaries. In theory, TCP can deliver data to you one byte at a time (in practice this doesn’t happen but see silly window syndrome). You also have to deal with the case where TCP delivers the end of record N and the beginning of record N+1 in the same chunk of data.

Ultimately you are going to have to write code that breaks the incoming data stream into records. How you do this depends on the data format. For example:

  • Some protocols use a binary framing format, where each chunk of data is preceded by a record header that includes the length

  • Some protocols use a textual framing format, where each chunk of data is terminated by a line break. This can get complex if there are multiple levels of framing, for example, HTTP headers, where each header is on a separate line and a blank line denotes the end of the headers.

  • Some protocols use byte stuffing. There’s lots of different types of byte stuffing, for example COBS.

For a concrete example of this, look at the RemoteCurrency sample code, where the QCommandConnection and QCommandLineConnection classes combine to split records based on line breaks.

ps That code is rather old. These days I tend to split my framing and unframing code out into a separate class so that I can test it. Pasted in below you’ll find an example of a class that implements line-based unframing.

Share and Enjoy

Quinn “The Eskimo!”
Apple Developer Relations, Developer Technical Support, Core OS/Hardware

let myEmail = "eskimo" + "1" + "@apple.com"
@import Foundation;

NS_ASSUME_NONNULL_BEGIN

@interface QLineUnframer : NSObject

// maxLineLength constrains the total buffer space used by this object
// to less than maxLineLength + the size of data you pass in to -linesFromData:

@property (nonatomic, assign, readwrite) NSUInteger maxLineLength;      // defaults to 4 KiB

- (nullable NSArray *)linesFromData:(NSData *)data;

@property (nonatomic, copy, readonly, nullable) NSError * error;

@end

NS_ASSUME_NONNULL_END
#import "QLineUnframer.h"

typedef NS_ENUM(NSInteger, QLineUnframerState) {
    QLineUnframerStateWantCR,
    QLineUnframerStateWantLF,
    QLineUnframerStateFailed
};

@interface QLineUnframer ()

@property (nonatomic, copy, readwrite, nullable) NSError * error;

@end

@implementation QLineUnframer {
    QLineUnframerState  _state;
    NSMutableData *    _buffer;
}

- (instancetype)init {
    self = [super init];
    if (self != nil) {
        self->_maxLineLength = 64 * 1024;
        self->_buffer = [[NSMutableData alloc] init];
    }
    return self;
}

- (NSArray *)linesFromBufferStartingAtOffset:(NSUInteger)startingOffset {
    NSMutableArray *    lines;
    NSUInteger          offset;
    NSUInteger          lineStartOffset;
    const uint8_t *    bufferBase;
    NSUInteger          bufferLength;
    QLineUnframerState  state;

    lines = [[NSMutableArray alloc] init];

    offset = startingOffset;
    lineStartOffset = 0;
    bufferBase = self->_buffer.bytes;
    bufferLength = self->_buffer.length;
    state = self->_state;
    do {
        uint8_t    thisChar;

        thisChar = bufferBase[offset];
        if ( (thisChar == '\r') && (state == QLineUnframerStateWantCR) ) {
            state = QLineUnframerStateWantLF;
        } else if ( (thisChar == '\n') && (state == QLineUnframerStateWantLF) ) {
            NSString *  thisLine;

            thisLine = [[NSString alloc] initWithBytes:&bufferBase[lineStartOffset] length:offset - lineStartOffset - 1 encoding:NSUTF8StringEncoding];
            if (thisLine == nil) {
                self.error = [NSError errorWithDomain:NSCocoaErrorDomain code:NSFileReadCorruptFileError userInfo:nil];
                state = QLineUnframerStateFailed;
                break;
            } else {
                [lines addObject:thisLine];
                lineStartOffset = offset + 1;
                state = QLineUnframerStateWantCR;
            }
        } else {
            // Not a CR, not a LF, or a CR or LF but in the wrong place.  Such a
            // character belongs in the string itself and always leaves us in the
            // WantCR state.
            //
            // Note the >= in the following expression.  That's because we haven't yet
            // incremented offset to account for this character.
            if ( (offset - lineStartOffset) >= self->_maxLineLength) {
                self.error = [NSError errorWithDomain:NSCocoaErrorDomain code:NSFileReadTooLargeError userInfo:nil];
                state = QLineUnframerStateFailed;
                break;
            }
            state = QLineUnframerStateWantCR;
        }
        offset += 1;
        if (offset == bufferLength) {
            break;
        }
    } while (YES);
    self->_state = state;

    [self->_buffer replaceBytesInRange:NSMakeRange(0, lineStartOffset) withBytes:NULL length:0];

    return lines;
}

- (nullable NSArray *)linesFromData:(NSData *)data {
    NSArray *      result;

    NSParameterAssert(data.length != 0);

    if (self->_state == QLineUnframerStateFailed) {
        result = nil;
    } else {
        NSUInteger      startingOffset;

        // Append the new data to our buffer.

        startingOffset = self->_buffer.length;
        [self->_buffer appendData:data];

        // Parse lines from the buffer and return them in an array.

        result = [self linesFromBufferStartingAtOffset:startingOffset];
        if ( (self.error != nil) && (result.count == 0) ) {
            result = nil;
        }
    }
    return result;
}

@end

Thanks...


In fact the data is sent in chunks of xml data. Each xml data is related to a specific period in time. End occurs when no more data is received. I just need to buffer all the data into one string.


Which class should I use? NSURLSession? NSStream? Or should I go to a more low level code? Could you elaborate your answer a little bit? and if possible in swift? I apologize for my lack of knowledge.

Which class should I use? NSURLSession? NSStream?

I typically use NSStream. NSURLSession’s stream task support only just rolled out, so you can’t use it unless your deployment target is the current OS only.

Could you elaborate your answer a little bit?

If you pose specific questions I’d be happy to answer them.

Share and Enjoy

Quinn “The Eskimo!”
Apple Developer Relations, Developer Technical Support, Core OS/Hardware

let myEmail = "eskimo" + "1" + "@apple.com"

Thanks again... Target is the last iOS version. NSURLSession should be fine. Is not NSStream being deprecated? Could I use the following?


           
....
          let configuration = NSURLSessionConfiguration.defaultSessionConfiguration()
            configuration.timeoutIntervalForRequest = 15
          
            let urlSession = NSURLSession(configuration: configuration , delegate: self, delegateQueue: nil)
          
              let url = NSURL(string: "http://10.0.0.1:32076")
          
            let sessionDataTask = urlSession.dataTaskWithURL(url)
            sessionDataTask.resume()
          
        }
      
    }
  
    func URLSession(session: NSURLSession, dataTask: NSURLSessionDataTask, didReceiveData data: NSData) {
....
    }
  
    func URLSession(session: NSURLSession, task: NSURLSessionTask, didCompleteWithError error: NSError?) {
        if error != nil {
            session.finishTasksAndInvalidate()
        }
    }


Doing so I get errors when connecting to the URL (even after changing ATS)


Shoud then I go for a pure NSStream solution without NSURLSession? Or can I use NSURLSession with the stream options? If so, how can I create a lopp so all the data is read?

Is not NSStream being deprecated?

We’ve not made any formal announcement to that effect. Taking a step back, the future of Foundation-level APIs for low-level networking is a bit murky. NSURLSessionStreamTask is an obvious replacement for the CFSocketStream variant of NSStream, but there are other complicating factors.

Right now my recommendation is that you use the API that best matches your requirements and worry about the future in… well… the future.

Could I use the following?

That uses NSURLSessionDataTask, which is very different from NSURLSessionStreamTask. The former is for HTTP[S] requests and the latter is for lower-level TCP networking.

If so, how can I create a [loop] so all the data is read?

No. Both NSStream and NSURLSessionStreamTask should be used asynchronous (you can’t use NSStream synchronously but I don’t recommend it) and, as such, you can’t just loop waiting for data. You have two options here:

  • write code to find the record boundaries in your stream

  • switch to a chunk-based XML parser, one that allows you to present data to the parser as it arrives in chunks

libxml2 has such a parser but I’m not sure whether it’s up to the task of find XML record boundaries for you.

Share and Enjoy

Quinn “The Eskimo!”
Apple Developer Relations, Developer Technical Support, Core OS/Hardware

let myEmail = "eskimo" + "1" + "@apple.com"