First it is important to consider all the facts involved in this story:
1. There was something that Ham did to his father that Noah was aware of when he awoke from his wine (vs.24).
2. Whatever Ham did and told his brothers about caused the brothers to abhor the sin so much that they covered their father by walking backwards into the tent so as not to see the nakedness of their father. (Obviously just the seeing of his nakedness was not the sin.)
3. There must be a connection between what Ham did and Canaan since there were four sons of Ham and neither Ham nor the other sons were cursed.
4. The curse, as well as the blessings on the other brothers, are prophecies dealing with the nations that would come from these people, and not statements about what would happen to the people themselves.
Fitting all this together, my conclusion is that Ham looked upon his father in lust and in some way committed a sin that would be both incestuous and homosexual and was later denounced in the Canaanites (See Leviticus 18:1-25).