This is a regression introduced by merging the EBS and S3 images. The
EBS images had a special marker /.ebs to prevent the initrd from using
ephemeral storage for the unionfs, but this marker was missing in the
consolidated image.
The fix is to check the file ami-manifest-path on the metadata server
to see if we're an S3-based instance. This does require networking in
the initrd.
Issue #12613.