Ethically wrong? No way, but getting "caught" and in the dark no less then yes! LOL, shame on you using those trees***ds!
I've never had a confrontation and hope I never do but I also always try to remain invisible and absolutely silent when I'm "in country". There have been other hunters, nature lovers, dog walkers and hikers come into my area and all were oblivious to my presence. The hairs on my neck stand up along with a chill that runs down my spine every time I hear humans encroaching on the sanctity of my setup, even though I am legal to hunt there.
You did the right thing to deescalate the situation and move on, even though you were within your rights to hunt there, because at the end of the day your life or court proceedings isn't worth it. The authorities should be your first call after safely moving on, so said douche bag doesn't harrass or harm anyone else.
Another thing to consider if/when this type of situation arises and you decide to stay in the area or even return another day to hunt is do you want to sit there all day and wonder if douche bag is vandalizing your vehicle or if it will even be there? Sad to think this way but something similar happened to my buddy...
He was hunting this area, lets call it a gray area as it wasn't public and it wasn't part of the county park. A few guys hunted this area along with some local cops and some paramedics. My buddy got to know one of them because of where he'd park his truck and he would see him from time to time and chat for a few minutes then make his way. Well one day he comes out of the woods and his truck is gone. He walks to the other parking area and sees a state trooper so he walks up and explains what he was doing and that his truck is gone. The trooper states they got a report of a suspicious vehicle so they had it towed! Here is the best part, they offered to drive him to the tow yard to pick it up. When he got there the guy running the yard pointed to where his truck was and said have a nice day! No fine, no citation, and no fees to pay. It was a gesture for him to stay away from "their" hunting spot. It worked, neither of us hunted there again.