I just did my upgrade to macOS High Sierra with a volume that had FileVault2 enabled and had this happen (prompt for disk password but would not accept any password). I ultimately called AppleCare... they had a note on the issue.
The solution was:
Boot into the recovery OS by
- Power off the Mac
- Press-and-hold CMD+R while powering on and wait for the machine to boot.
- Select "Disk Utility"
Within Disk Utility
- Select your internal volume (e.g. "Macintosh HD" unless you've renamed it)
- Select the "Mount" icon (along the top)
It will prompt for a disk password but this time YOUR normal login account password will work and the volume will mount.
Take note of the "Device" name (there's a table with information about the mounted volume... the device name will be in the lower right corner). The device name may be something like "disk2s1" - write it down... you need to know that device name for the next step.
Exit Disk Utility
Select the "Utilities" menu along the top and pick the "Terminal" app.
Within the terminal window you are now at a root prompt.
Type (at the root prompt #): diskutil apfs updatePreboot name-of-device
e.g. if your device was 'disk2s1' then the command would be: diskutil apfs updatePreboot disk2s1
That will take a moment (you'll see pages of messages fly by). Once you see the root prompt (#) return, it's ready.
Reboot your mac. You're done!
The mac should now boot normally (altough if you're just upgraded to macOS High Sierra then you'll get the normal set of screens that appear the first time you log in after the upgrade... e.g. it'll probably ask you to login to your iCloud account, etc. etc.)