The trouble with that is that you're looking at it on a human time scale, not on a tree's time scale. Arborists routinely look at problems in trees that were the result of damage from years, decades, even many decades prior. Recently I was looking at some old Southern Live Oaks in bad shape and judging by the pruning practices that were used, the guys who trashed those trees are probably dead and gone by now. In their minds they probably thought they did a great job but there's a group of majestic trees with massive decay ready to fall over as a result of their handy work.
I'm not saying one hole by itself is going to kill a tree. Trees don't always die from one thing, they often go into a "spiral of decline". They get some damage from one thing, something else in it's environment changes, one stress factor leads to another and they start to coalesce together and cause the tree to go into decline. Again, that process can be decades in the making, hunting the same tree for a few years isn't a long enough time frame to access the long term damage from one's practices.
Just food for thought
