The historic sites built with slave money

Does it matter if historical monuments have links with slavery? Some claim that we are dwelling too much on the past, but others think that we should be ashamed of our colonial history.