The fast ramp on a the XE284H-10 cam makes it a bit noisey at idle.
oil and lifter bleed down trate make a big difference.
fast bleed "hi rev" lifters make more noise too. HT-817R
Why are you using the self aligning rocker with a old style head that has
guide plates or a push rod guide slot in the head.?
A thicker oil like rottella 15W 40 will probabily idle quieter with the xe HYD cam.
The solid lifter and solid roller XE street cams are actually quieter at idle.
You may have one pesky lifter with a bit of dirt inside it.
Make sure all the push rods spin when you rev up the motor.
Set the lifter preload as required with the engine hot and running.
If it needs 1 turn give it 1 turn preload.
When resetting the preload back off the rocker so it clicks a lot and let it normalize. Then set the 1/2 to 1 turn lifter preload.
Give it the preload it wants needs to idle quiet.