This was an issue with the default surface created by sdl, which had the wrong color format. For the future it is advised to use the RGBA CFu8 format from babl, or things may break again.