Using the sumif function. So lets say your data range is from cells A1:B200 and the colors are in column A and the numbers are in column B.
Get the sum of the numbers for each color:
Do that for each color.
Get the sum of all of the values:
Put them together to get the average, so to get the average of all colors marked "red", use: