Thanks for the answer. This website is for a client so that's not a great solution. One of the main reasons I wanted to use this for their site is that it's extremely easy for them to add images to the gallery on the page by simply uploading images to the page.
After testing a few things I found a 'solution' ..maybe this will help somebody else with the same question:
Instead of setting the featured image by clicking "set featured image > upload from computer" you should instead first upload the image to the media library. Then back on the page click "set featured image > media library". By doing that the featured image doesn't get added to the gallery.