I know how it relates to monoids, rather than to sets. For example, you cannot just multiply together any two matrices (like you can with monoids); they need to have the correct dimensions. So, in category theory, this corresponds to the composition of morphisms, so in this case, the objects are the number of rows/columns and the morphisms are the matrices.