My point was more to the fact that whatever they're doing in the harbor, a location above land where multitudes of people pass through, would be more effectively done in the water, with little water traffic and at the peak of their power. No normal civilian would just happen to discover the Hōzuki clan living there, and no ninja passing by would be able to relay the information because they may not even escape the Hōzuki's grasp after discovering their clan.
Given that it's unlikely they'll even be discovered, we can go on the off chance that they do get discovered and discuss what would happen. They wouldn't be able to outright beat back the entirety of Konoha, but the entirety of Konoha wouldn't get mobilized to move them in the first place. Given the fact that they aren't going out of their way to mess with Konoha, the additional fact that Konoha has literally no reason to start a fight over the bottom of the ocean, and the fact that the war effort will cost more than its worth; I'm pretty sure the Hōzuki would be more or less fine because whatever Konoha would be willing to throw at them, they'd be able to handle, and what they couldn't handle, Konoha wouldn't be willing to throw at them.
Removing the Hōzuki clan would provide Konoha with 0 benefits whatsoever. So it simply wouldn't be worth the loss of war potential and collateral damage.
