``Experiments make unrealistic assumptions'', or ``The data was manipulated'', or ``It is impossible to quantify the variable of interest,'' are some of the criticisms. There are many more potential flaws: Experimenters may pick irrelevant questions, may neglect to provide enough detail for repeating experiments, may be nonchalant about control, may not validate observations, forget to bound errors, use inappropriate measurements, over-interpret their results, produce results that do not generalize, etc.
Good examples of solid experimentation in computer science are rare.
And there will always be questionable, even bad experiments. However,
the conclusion from this observation is not to discard the concept of
experimentation. We should keep in mind that other scientific fields
have been faced with bad experiments, even frauds. But the scientific
process on the whole has been self-correcting. Bad ideas, errors, and
downright hoaxes have been weeded out, sometimes promptly (see cold
fusion) sometimes belatedly (see the Piltdown man).
We can be sure of one thing, though: If scientists overlook experimentation or neglect reexamining others' claims, an important source of self-correction will be cut off and the field may drift into the wrong direction.