The bug is described here:
Can't auto-start xfce4 from slim when vbox services are enabled.
If this is a VirtualBox guest additions bug, there should be the prefix:
I'm using xfce in virtualbox but I use lightdm as a display manager with no issues.
I understand my opinion on this is not popular, but slim is actually unmaintained software and AFAIK does not implement all the latest freedesktop standards. I have friends who have various issues when using slim also without adding virtualbox to the mix.
Could you try another display manager?
Also, which virtualbox graphics contorller are you using in the VM? The virtualbox UI suggest using VMSVGA, but when the guest has additions installed I get much better results with VBoxSVGA. Just ignore the UI warning.
Had two issues with it: first, it autocompletes username with "User&", and I have to actively change to Other. With SLiM I already have the two fields in blank to enter and that's it. Also, on lightdm, the screen locked due to auto lock session enabled by default. When I entered again my username and password just didn't login, falls back to same login screen. With SLiM no issues.