I would not worry too much but I do suggest that more then just compost goes in.
You will want some top soil to bulk it up, and some manure for when the stuff in the compost leaches out - only a few weeks. Also dig over the top of what the raised bed is on, just integrates the whole lot.
My raised bed had had just about everything dug into it.