Wondering if you have found any additional infromation about this?
I'm seeing the exact same thing when creating a NSSegmentedControl in interface builder.
Seems like a High Sierra bug!
( btw: I don't understand your comment "2 If I comment "setLabel" for both segments, its works." )
Try setting the segment style to a different explicit value.
I found that NSSegmentStyleRounded worked for me in this regard.
I'm having exactly the same issue, but my segmented controls are defined in NIB files and define small "palette" type windows that do not use NSDocument. At initial launch under 10.13, all of my segemented control labels are truncated with "...". HOWEVER - if my application opens an actual NSDocument-based window (our app scans and display TIFF and PDF files), the labels in the segmented control in the palette window immediately refresh and display normally and remain normal even after the document window is closed. I am using the "System Small" font in by segmented control labels, which should (i think) resolve to Luconda Grande 11. The control acts like it could not find the font for some reason so when it went to measure the length of the label strings (to see if they needed to be truncated), it was forced to use some default font metrics (perhaps assuming it was 12 point) and thus decided the text had to be truncated. That is a complete guess on my part.
Saw "mugginsoft"s post that suggested the style be explicitly set rather than leave at NSSegmentedStyleAutomatic, and when I looked at my NIB files, sure enough, the style was set to "Automatic", so I changed it to "Rounded", rebuilt and retested under 10.13. Sadly, there was no difference - still truncated until an NSDocument based window was opened.
I resolve this issue by setting Alignment property as left for programmatically created NSSegmentedControl as below.
Code : [segmentControl setAlignment:NSLeftTextAlignment];