nw_connection_t and STARTTLS for SMTP connections

Implementing STARTTLS (aka Opportunistic TLS) with Network framework is possible, but certainly not trivial. The trick is to use a protocol framer. I’ve written this for the Swift interface but not the C interface, so I’m going to respond in Swift. You’ll have to translate that back to the C interface. Sorry!

To start, you need a framer:

final class STARTTLSFramer: NWProtocolFramerImplementation {
    …
}

The NWProtocolFramerImplementation protocol has a number of requirements, so implement each in turn. Let’s start with the easy ones, where the implementation does nothing:

init(framer: NWProtocolFramer.Instance) { }

func handleOutput(framer instance: NWProtocolFramer.Instance, message: NWProtocolFramer.Message, messageLength: Int, isComplete: Bool) {
    fatalError()
}

func wakeup(framer instance: NWProtocolFramer.Instance) {
    fatalError()
}

func stop(framer instance: NWProtocolFramer.Instance) -> Bool { true }

func cleanup(framer instance: NWProtocolFramer.Instance) { }

My framer doesn’t handle output, so there’s no need to worry about that.

For wakeup(…), my framer never calls scheduleWakeup(wakeupTime:) so it should never get a wakeup.

The other methods would be necessary if the framer were more complex.

Next you need a label:

static let label: String = "STARTTLSFramer"

Finally, there’s the start(…) and handleInput(…) methods. Let’s start with the start (-:

func start(framer instance: NWProtocolFramer.Instance) -> NWProtocolFramer.StartResult {
    instance.writeOutput(data: Data(plaintextGreeting.utf8))
    return .willMarkReady
}

In this snippet plaintextGreeting is a Swift string that includes the EHLO and STARTTLS commands. In a real framework this would be more complex because… well… SMTP is complex.

Note how it returns .willMarkReady. This tells Network framework that protocol negotiation is ongoing, so the framer isn’t ready. This is why handleOutput(…) can trap, because the framer can’t receive output until it’s ready and, by the time that it marks itself as ready, it’s already marked itself as ‘pass through’. More on that below.

Finally, there’s the tricky part:

private var accumulated = Data()

func handleInput(framer instance: NWProtocolFramer.Instance) -> Int {

    // MARK: accumulate
    
    repeat {
        let success = instance.parseInput(minimumIncompleteLength:1, maximumLength: 2048) { buffer, _ in
            let count = buffer?.count ?? 0
            if let buffer {
                accumulated.append(contentsOf: buffer)
            }
            return count
        }
        if !success { break }
    } while true

    // MARK: parse

    let parseResult = … parse self.accumulated …
    switch parseResult {
    case .success: break
    case .failure:
        instance.markFailed(error: .posix(.ENOTTY))
        return 0
    case .needMoreData:
        // wait for more
        return 0
    }

    // MARK: go go gadget pass through
    
    self.accumulated.removeAll()

    let options = NWProtocolTLS.Options()
    try! instance.prependApplicationProtocol(options: options)
    instance.passThroughInput()
    instance.passThroughOutput()
    instance.markReady()

    return 0
}

There are three blocks of code with marker comments:

• accumulate — This reads input from the server and accumulates it into a buffer.

• parse — This parses the accumulated input. In my kludgy example I’m looking for a response to both the EHLO and STARTTLS commands. I’m not showing the code because it’s ugly.

  There are three possible results here:

  • .success, which goes to the next step
  • .failure, which marks the framer as having failed
  • .needMoreData, which returns 0 so that the system will call the framer again when more data arrives

• go go gadget pass through — This empties the accumulator (to save space), appends the TLS protocol, switches this framer to pass through mode, and marks the framer as ready. Any data that the app sends will be handled by TLS and then pass directly through this framer. On the return path, incoming data will pass through this framer and go straight up to TLS.

Finally, I enable the framer on the connection like so:

let startTLSOptions = STARTTLSFramer.options()
let params = NWParameters.tcp
params.defaultProtocolStack.applicationProtocols = [startTLSOptions]
let connection = NWConnection(host: "example.com", port: .smtp, using: params)

where the options() static method looks like this:

static func options() -> NWProtocolFramer.Options {
    let startTLSDef = NWProtocolFramer.Definition(implementation: STARTTLSFramer.self)
    let result = NWProtocolFramer.Options(definition: startTLSDef)
    return result
}

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

Hmm, can't begin to fathom why this was stumping me, so simple! OK, off to try and figure out how to make this work. Thanks for the pointers and the speedy response time, most appreciated.

I successfully translated your Swift code to C (I think):

//STARTTLSFramer.h

#import <Cocoa/Cocoa.h>
#import <Network/Network.h>

typedef NS_ENUM(NSInteger, ParseResult) {
	ParseResultSuccess,
	ParseResultFailure,
	ParseResultNeedMoreData
};

@protocol STARTTLSFramerDelegate;

@interface STARTTLSFramer : NSObject
@property (assign) id <STARTTLSFramerDelegate> delegate;
- (nw_connection_t)connectionWithSTARTTLSToEndpoint:(nw_endpoint_t)endpoint;
@end

@protocol STARTTLSFramerDelegate <NSObject>
@required
- (ParseResult)STARTTLSFramer:(STARTTLSFramer *)STARTTLSFramer didReceiveData:(NSData *)data;
@end

//STARTTLSFramer.m

#import "STARTTLSFramer.h"

@interface STARTTLSFramer ()
@property (nonatomic, strong) NSMutableData *accumulated;
@property (nonatomic, strong) nw_framer_start_handler_t start;
@end

@implementation STARTTLSFramer
- (instancetype)init {
	if (self = [super init]) {
		_accumulated = [NSMutableData new];
	}
	return self;
}
- (ParseResult)parseAccumulated { return [self.delegate STARTTLSFramer:self didReceiveData:_accumulated]; }
- (nw_framer_start_handler_t)startHandler {
	if (!_start) {
		nw_framer_start_result_t (^start)(nw_framer_t) = ^nw_framer_start_result_t(nw_framer_t framer) {
			nw_framer_set_output_handler(framer, ^(nw_framer_t framer, nw_framer_message_t message, size_t message_length, bool is_complete) {
				@throw [NSException exceptionWithName:NSInternalInconsistencyException reason:@"method not implemented" userInfo:nil];
			});
			nw_framer_set_wakeup_handler(framer, ^(nw_framer_t framer) {
				@throw [NSException exceptionWithName:NSInternalInconsistencyException reason:@"method not implemented" userInfo:nil];
			});
			nw_framer_set_stop_handler(framer, ^bool(nw_framer_t framer) {
				return true;
			});
			nw_framer_set_cleanup_handler(framer, ^(nw_framer_t framer) {
			});
			nw_framer_set_input_handler(framer, ^size_t(nw_framer_t framer) {
				// MARK: accumulate
				while (1) {
					BOOL complete = nw_framer_parse_input(framer, 1, 2048, NULL, ^size_t(uint8_t *buffer, size_t buffer_length, bool is_complete) {
						if (buffer) [self.accumulated appendBytes:(const void *)buffer length:(NSUInteger)buffer_length];
						return is_complete;
					});
					if (complete) break;
				}
				// MARK: parse
				switch(self.parseAccumulated) {
					case ParseResultSuccess: break;
					case ParseResultFailure: nw_framer_mark_failed_with_error(framer, ENOTTY); return 0;
					case ParseResultNeedMoreData: return 0;
				}
				// MARK: go go gadget pass through
				self.accumulated.length = 0;
				nw_protocol_options_t options = nw_tls_create_options();
				nw_framer_prepend_application_protocol(framer, options);
				nw_framer_pass_through_input(framer);
				nw_framer_pass_through_output(framer);
				nw_framer_mark_ready(framer);
				return 0;
			});
			/*
			NSString *plaintextGreeting = @"EHLO localhost\r\n"; //the delegate handles sending the greeting
			NSData *data = [plaintextGreeting dataUsingEncoding:NSUTF8StringEncoding];
			nw_framer_write_output(framer, (const uint8_t *)data.bytes, (size_t)data.length);
			*/
			return nw_framer_start_result_will_mark_ready;
		};
		self.start = start;
	}
	return _start;
}
- (nw_connection_t)connectionWithSTARTTLSToEndpoint:(nw_endpoint_t)endpoint {
	nw_parameters_t parameters = nw_parameters_create_secure_tcp(NW_PARAMETERS_DISABLE_PROTOCOL, NW_PARAMETERS_DEFAULT_CONFIGURATION);
	nw_protocol_stack_t protocol_stack = nw_parameters_copy_default_protocol_stack(parameters);
	nw_protocol_definition_t protocol_definition = nw_framer_create_definition("STARTTLSFramer", NW_FRAMER_CREATE_FLAGS_DEFAULT, self.startHandler);
	nw_protocol_options_t protocol_options = nw_framer_create_options(protocol_definition);
	nw_protocol_stack_prepend_application_protocol(protocol_stack, protocol_options);
	nw_connection_t connection = nw_connection_create(endpoint, parameters);
	nw_release(protocol_options);
	nw_release(protocol_definition);
	nw_release(protocol_stack);
	nw_release(parameters);
	return connection;
}
@end

Called from the delegate object like so:

_STARTTLSFramer = [STARTTLSFramer new];
_STARTTLSFramer.delegate = self;
_connection = [_STARTTLSFramer connectionWithSTARTTLSToEndpoint:endpoint];

But I'm running into the same issue I had when I was just using:

nw_endpoint_t endpoint = nw_endpoint_create_host("smtp.example.com", "587");
nw_parameters_t parameters = nw_parameters_create_secure_tcp(NW_PARAMETERS_DEFAULT_CONFIGURATION, NW_PARAMETERS_DEFAULT_CONFIGURATION);
_connection = nw_connection_create(endpoint, parameters);

When I open the connection, the SMTP server responds '220 Example SMTP...' and the nw_connection_set_state_changed_handler() is called with nw_connection_state_preparing but there is an error:

The operation couldn’t be completed. (OSStatus error -9836 - bad protocol version)

And in the log I see:

boringssl_context_handle_fatal_alert(2072) [[C2.1.1:1]:1][0x13610b420] write alert, level: fatal, description: protocol version
boringssl_context_error_print(2062) [[C2.1.1:1]:1][0x13610b420] Error: 5204110000:error:100000f7:SSL routines:OPENSSL_internal:WRONG_VERSION_NUMBER:/AppleInternal/Library/BuildRoots/a8fc4767-fd9e-11ee-8f2e-b26cde007628/Library/Caches/com.apple.xbs/Sources/boringssl/ssl/tls_record.cc:231:
boringssl_session_handshake_incomplete(210) [[C2.1.1:1]:1][0x13610b420] SSL library error
boringssl_session_handshake_error_print(44) [[C2.1.1:1]:1][0x13610b420] Error: 5204110000:error:100000f7:SSL routines:OPENSSL_internal:WRONG_VERSION_NUMBER:/AppleInternal/Library/BuildRoots/a8fc4767-fd9e-11ee-8f2e-b26cde007628/Library/Caches/com.apple.xbs/Sources/boringssl/ssl/tls_record.cc:231:
nw_protocol_boringssl_handshake_negotiate_proceed(779) [[C2.1.1:1]:1][0x13610b420] handshake failed at state 12288: not completed

If I don't use the connection configured by STARTTLSFramer or NW_PARAMETERS_DEFAULT_CONFIGURATION and use NW_PARAMETERS_DISABLE_PROTOCOL for configure_tls when creating the parameters, the connection connects and I when I send EHLO, the server responds with supported commands including STARTTLS. It's at this point that I believe I need to send STARTTLS, get confirmation from the server, then update the connection with the TLS parameters. If I start the connection with TLS already in place, I get the bad protocol error. So, back to the original question, how do I add TLS options to an already connected insecure connection? Or, am I just misunderstanding the STARTTLS procedure? Or, is my code just wrong?

Thanks!

I suspect I shouldn't enable the passthrough until I connect, send the greetings and STARTTLS, and finally get confirmation from the server. But when I use the connection created by STARTTLSFramer without enabling the passthrough, the server responds '220 Example SMTP...' and the nw_connection_set_state_changed_handler() is called with nw_connection_state_preparing but then doesn't get called again and never changes to nw_connection_state_ready. If I don't wait for nw_connection_state_ready and just send EHLO, I don't see any more incoming data on the connection.

Solved! I open the insecure connection and communicate with the SMTP server directly using the framer until it's ready to start TLS:

connected to smtp.example.com:587
<- 220 Example SMTP...
-> EHLO localhost
<- 250-example.com
<- 250-SIZE 28319744
<- 250-STARTTLS
<- 250-SMTPUTF8
<- 250 CHUNKING
-> STARTTLS
<- 220 2.0.0 Ready to start TLS
//[_framer enablePassthrough];
-> EHLO localhost
<- 250-example.com
<- 250-SIZE 28319744
<- 250-AUTH LOGIN PLAIN ATOKEN ATOKEN2 WSTOKEN WETOKEN
<- 250-SMTPUTF8
<- 250 CHUNKING
-> AUTH PLAIN AHVzZXJAZXhhbXBsZS5jb20AcGFzc3dvcmQ=
<- 235 2.7.0 Authentication successful
-> QUIT
<- 221 2.0.0 Bye
disconnected from smtp.example.com:587

This required implementing an actual nw_framer_set_output_handler() and allowing the delegate to send through the framer directly (framerSendData:). Once the "220 2.0.0" message is received, the delegate calls enablePassthrough and subsequent read/writes from the delegate use the connection object and not directly through the framer. Doing it this way traps all data sent through the framer until enablePassthrough is called so no need to use parseResult.

Thanks for your help. Revised code below.

//STARTTLSFramer.h
#import <Cocoa/Cocoa.h>
#import <Network/Network.h>

@protocol STARTTLSFramerDelegate;

@interface STARTTLSFramer : NSObject
@property (assign) id <STARTTLSFramerDelegate> delegate;
@property (readonly) BOOL isPassthroughEnabled;
- (nw_connection_t)connectionWithSTARTTLSToEndpoint:(nw_endpoint_t)endpoint;
- (void)framerSendData:(NSData *)data;
- (void)enablePassthrough;
@end

@protocol STARTTLSFramerDelegate <NSObject>
@required
- (void)STARTTLSFramer:(STARTTLSFramer *)framer didReceiveData:(NSData *)data;
@end

//STARTTLSFramer.m
#import "STARTTLSFramer.h"

@interface STARTTLSFramer ()
@property (nonatomic, strong) NSMutableData *accumulated;
@property (nonatomic, strong) nw_framer_start_handler_t start;
@property (nonatomic, assign) BOOL passthroughEnabled;
@end

@implementation STARTTLSFramer{
	nw_framer_t _framer;
}
- (instancetype)init {
	if (self = [super init]) {
		_accumulated = [NSMutableData new];
		_passthroughEnabled = NO;
	}
	return self;
}
- (BOOL)isPassthroughEnabled { return self.passthroughEnabled; }
- (void)enablePassthrough {
	// MARK: go go gadget pass through
	self.passthroughEnabled = YES;
	nw_protocol_options_t protocol_options = nw_tls_create_options();
	nw_framer_prepend_application_protocol(_framer, protocol_options);
	nw_release(protocol_options);
	nw_framer_pass_through_input(_framer);
	nw_framer_pass_through_output(_framer);
	nw_framer_mark_ready(_framer);
}
- (void)framerSendData:(NSData *)data {
	dispatch_data_t dispatch_data = dispatch_data_create(data.bytes, (size_t)data.length, dispatch_get_main_queue(), DISPATCH_DATA_DESTRUCTOR_DEFAULT);
	nw_framer_write_output_data(_framer, dispatch_data);
}
- (nw_framer_start_handler_t)startHandler {
	if (!_start) {
		nw_framer_start_result_t (^start)(nw_framer_t) = ^nw_framer_start_result_t(nw_framer_t framer) {
			_framer = framer;
			nw_framer_set_output_handler(framer, ^(nw_framer_t framer, nw_framer_message_t message, size_t message_length, bool is_complete) {
				if (message) {
					dispatch_data_t data = nw_framer_message_copy_object_value(message, "data");
					nw_framer_write_output_data(framer, data);
					nw_release(data);
				} else {
					nw_framer_write_output_no_copy(framer, message_length);
				}
			});
			nw_framer_set_wakeup_handler(framer, ^(nw_framer_t framer) {
				@throw [NSException exceptionWithName:NSInternalInconsistencyException reason:@"nw_framer_set_wakeup_handler not implemented" userInfo:nil];
			});
			nw_framer_set_stop_handler(framer, ^bool(nw_framer_t framer) {
				return true;
			});
			nw_framer_set_cleanup_handler(framer, ^(nw_framer_t framer) {
			});
			nw_framer_set_input_handler(framer, ^size_t(nw_framer_t framer) {
				// MARK: accumulate
				BOOL complete = NO;
				while (!complete) {
					complete = nw_framer_parse_input(framer, 1, 2048, NULL, ^size_t(uint8_t *buffer, size_t buffer_length, bool is_complete) {
						if (buffer) [self.accumulated appendBytes:(const void *)buffer length:(NSUInteger)buffer_length];
						return buffer_length;
					});
				}
				// MARK: parse
				[self.delegate STARTTLSFramer:self didReceiveData:_accumulated];
				self.accumulated.length = 0;
				return 0;
			});
			return nw_framer_start_result_will_mark_ready;
		};
		self.start = start;
	}
	return _start;
}
- (nw_connection_t)connectionWithSTARTTLSToEndpoint:(nw_endpoint_t)endpoint {
	nw_parameters_t parameters = nw_parameters_create_secure_tcp(NW_PARAMETERS_DISABLE_PROTOCOL, NW_PARAMETERS_DEFAULT_CONFIGURATION);
	nw_protocol_stack_t protocol_stack = nw_parameters_copy_default_protocol_stack(parameters);
	nw_protocol_definition_t protocol_definition = nw_framer_create_definition("STARTTLSFramer", NW_FRAMER_CREATE_FLAGS_DEFAULT, self.startHandler);
	nw_protocol_options_t protocol_options = nw_framer_create_options(protocol_definition);
	nw_protocol_stack_prepend_application_protocol(protocol_stack, protocol_options);
	nw_connection_t connection = nw_connection_create(endpoint, parameters);
	nw_release(protocol_options);
	nw_release(protocol_definition);
	nw_release(protocol_stack);
	nw_release(parameters);
	return connection;
}
@end

Calls from the delegate:

nw_endpoint_t endpoint = nw_endpoint_create_host("smtp.example.com", "587");
_framer = [STARTTLSFramer new];
_framer.delegate = self;
_connection = [_framer connectionWithSTARTTLSToEndpoint:endpoint];

- (void)STARTTLSFramer:(STARTTLSFramer *)framer didReceiveData:(NSData *)data {
	//parse the data pseudocode:
	if ([dataString hasPrefix:@"220 2.0.0"]) {
		[framer enablePassthrough];
		[self  sendData:@"EHLO localhost\r\n".data];
	} else if ([dataString hasPrefix:@"220 "]) {
		[self  sendData:@"EHLO localhost\r\n".data];
	} else if ([dataString containsString:@"250-STARTTLS"]) {
		[self  sendData:@"STARTTLS\r\n".data];
	}
}

- (void)sendData:(NSData *)data {
	if (!_framer.isPassthroughEnabled) {
		[_framer framerSendData:data];
		return;
	}
	//...
	nw_connection_send(_connection, dispatch_data, NW_CONNECTION_DEFAULT_MESSAGE_CONTEXT, is_complete, ^(nw_error_t error) { /*...*/ });
}

- (void)receive {
	nw_connection_receive(_connection, 1, UINT32_MAX, ^(dispatch_data_t content, nw_content_context_t context, bool is_complete, nw_error_t error) { /*...*/ });
}