As @ott's comment mentions, you can't send unicast traffic to a machine that isn't active on the network at that time. 'Active on the network' means different things to different devices though... to a Layer 3 device like your PC or a router it means 'this device has talked to the machine at some time in the last 4 hours' (depends on arp timeout), to a Layer 2 device like a switch it means 'the switch has seen a packet with that source mac-address some time in the last 5 minutes' (depends on mac aging time).
You state that unicast WOL works when the device has been turned on and then back off, but you don't state if it stops working at some time after the machine is shut off. This works because the router (or the WOL-packet generating PC if in the same subnet) has an arp entry (4 hour timeout) and can create a packet destined for the correct machine, however the switch either knows the port the mac-address belongs to (WOL sent within mac aging time) or has already timed out that mac-address so it treats the unknown destination as a broadcast. Once the arp entry times out, the router (or PC) will fail to encapsulate the WOL packet since it has nothing to put in the dest-mac field.
To make this work you need to use broadcasts, there is no other reliable way. To make WOL work on a different network than where your server is connected, you need to use directed broadcast packets (unicast packets sent to broadcast address of remote network, properly configured remote router will allow the packet and send it as a broadcast on the intended network).
Why not Emby?