I assume we’re talking about weapons, in which case, size and weight are both very important considerations, though possibly not for the reasons you’d think.
Size is critical for determining reach. This is how far you can reach out and impale someone. Generally speaking, longer weapons have a significant advantage over shorter ones. I say “generally” because there are a ton of specific exceptions, but if you can stab someone before they can reach you, that’s a combat advantage.
Weight is a major issue, but it’s never about being able to lift a weapon, (unless we’re talking about weapons designed to be used from an emplacement, like the M2 Browning) it’s about how agile the weapon is, and making sure that you can carry and use it all day.
This is why the heaviest swords intended for combat rarely exceed 8lbs. It needed to be light enough that its wielder could carry it and a couple other weapons and use them during constant physical exertion.
That “intended for combat” bit is a fairly important distinction, though. Parade swords were the historical equivalent of your friend’s gaudy katana display. They were there to look cool, not to be useful. Parade swords could get into the 20lb range. Some of those are amazing pieces of art in their own right, but they’re not practical weapons.
If we’re talking about your character? Then size and weight aren’t major considerations. Overall physical fitness is vitally important, but beyond that weight isn’t a huge issue. Depending on climate and diet, weight is semi-independent of physical fitness. I realize that may sound insane, but particularly in cold climates, it’s entirely possible for someone to bulk up while maintaining a layer of fat as insulation.
Size isn’t a huge issue unless your character is unusually large or unusually small. Characters that are less than a foot taller (or shorter) than their opponent should have roughly similar (unarmed) reach.
That said, shorter individuals do have lower centers of gravity, which makes it much easier for them to get into more stable stances.
It’s worth pointing out that: women have a lower center of gravity for their height than men.