Once that location becomes unavailable to the witcher, so does the potential opponent, so if you do not fight Fat Fred at the Inn in the Outskirts in Chapter I, you can never fight him, etc. Each opponent in the ascension rounds is only available in the one location. He also gets the boxer's tooth as a trophy. Geralt must beat him twice in a row before he can claim the title for himself. This final fight does not come easily, the champion will not relinquish his title after a single defeat. He must defeat four fighters before facing the Champion, in his cave in the swamp cemetery.
In each Chapter (except the epilogue) of the game, at the local tavern, Geralt is given the opportunity to challenge the local reigning fistfight champion for a chance at the 'Title', winning other trinkets and orens along the way.
In The Witcher, there are several quests that span the entire game, this being one the most noteworthy. 3.6.2 The Champion of Champions Defeated.