Using the sumif function. So lets say your data range is from cells A1:B200 and the colors are in column A and the numbers are in column B.
Get the sum of the numbers for each color:
=SUMIF(A1:B200, "red",B1:B200)
Do that for each color.
Get the sum of all of the values:
=SUM(B1:B200)
Put them together to get the average, so to get the average of all colors marked "red", use:
=(SUMIF(A1:B200, "red",B1:B200))/(SUM(B1:B200))