1 Reply
      Latest reply on Apr 2, 2019 1:17 AM by midori
      philimanjaro Level 1 Level 1 (0 points)

        I've been struggling quite a bit with the BreakfastFinder demo app provided at WWDC 2018 for a month or so now and cannot figure out to get it to work properly in my situation when using a custom object detector model that I created using Turi Create.

         

        My primary issues in a nutshell when using the BreakfastFinder app:

        - Detected items regularly come back with a confidence of 0.999 or greater (why so high??)

        - Bounding Boxes are not drawn properly in the view and the offset varies based on the object's position in the view

         

        Background:

        • I trained an object detection model using Turi Create 5.1
        • I understand that by default in Turi 5.0+ that it exports CoreML models that return VNRecognizedObjectObservation, rather than VNCoreMLFeatureValueObservation
        • I exported two different .mlmodels from the Turi .model file, one with include_non_maximum_suppression set to True and the other set to False
        • My model predictions work great in Turi, with a mean_average_precision_50 of 88%
          • My real-world performance mirrors that also when using a separate demo app provided by a helpful internet tutorial (Using the non-default VNCoreMLFeatureValueObservation .mlmodel file that I created)
          • My model's performance in Apple's WWDC 2018 Breakfast app gives absurdly high confidence values and yields far too many false positives (using the VNRecognizedObjectObservationmlmodel file that I created)

         

        Dev System Specs:

        • Using MacOS 10.14 Stable
        • Using XCode 10 Stable, as well as the last three XCode 10.1 betas
        • Macbook Pro 2016
        • iPhone XS Max, iOS 12 (But also experienced the issue when testing using my iPhone X on iOS 12)

         

         

        Problem Scenario:

        When using the BreakfastFinder app from WWDC 2018 and the newer model type that returns VNRecognizedObjectObservations, I do receive bounding boxes and detected objects -- so I know my model is working and set up properly; however, the bounding boxes are very inaccurate and the confidence scores are wayyyy too high, which makes it difficult to filter out false positive identifications since they all clock in at 0.999+
        Link to BreakfastFinder app from WWDC 2018: https://developer.apple.com/documentation/vision/recognizing_objects_in_live_capture

        Works Fine Scenario:

        When using a random demo app that I found online from a helpful user with the .mlmodel type that returns VNCoreMLFeatureValueObservation, my objects are recognized in live capture accurately with a reasonable and expected level of accuracy (80-85% on average).
        Link to the project I'm mentioning here: https://blog.ichibod.com/posts/2017/12/27/coreml-machine-learning-part-1/

         

        Issue History:

        There is an existing Turi Create issue opened for this with many others experiencing these problems -- but it's unclear that it's related to Turi Create and perhaps an issue related to the BreakfastFinder Demo code: https://github.com/apple/turicreate/issues/1016

        Quick GIF of my bounding box issue using the BreakfastFinder App (also shared in the Turi Create issue linked above):

        https://user-images.githubusercontent.com/13277778/47176757-072fa080-d2dc-11e8-979c-ce2dfd4c1147.gif

         


        Final Thoughts:
        I'm happy to share my code -- but it's essentially the same as the two existing projects referenced here already. I can't share my model at this time -- but if necessary -- I can create another one using public data to help demonstrate my point and share that one out. The issue does not appear to be unique to my model, as others in that Turi Create issue are also experiencing similar issues with the bounding box placements and the confidence scores.

         

        I've looked all over the internet and have failed to find more documentation on how to fix these issues; hence why I've decided to post here asking for support and help. If anyone is aware of the issue and knows how to fix it, I'd really appreciate your help!

        Thanks everyone