Scientific Hypotheses are Tested by means of Repeatable Observations.
Every scientific hypothesis ultimately stands or falls based on whether it fits with observed data. The best kind of data comes from experiment, in which the details of the situation are specified by the scientist and may be freely repeated or varied in order to see how the outcome changes. Some people might say that experiments are a necessary part of science, but I don't agree. That's because certain fields of Science don't seem to have them.
Unfortunately controlled experiments are not always feasible for historical sciences such as Geology or Evolutionary Biology, or for sciences which study objects that cannot be easily affected by humans such as Astronomy or Seismology. These sciences make up for the lack of experimentation by the fact that there is a large and ever increasing amount of observational data to test different hypotheses. So you can keep getting new data in a way that's reminiscent of the experimental sciences.
In the worst case all the observable effects of a hypothesis are mediated through a unique event. Such hypotheses are usually not subject to repeatable scrutiny, but there can be exceptions. A notable example relates to the highly regarded, but not completely established, theory of inflation in the early universe (a tiny fraction of a second after the so-called "Big Bang singularity", which as far as we know was the beginning of the universe.). In this theory there is a phenomenon known as Cosmic Variance, which implies that certain aspects of the theory can only be tested once, because the universe only happened once. Nothing stops you from looking again, but you'll see the same thing so it doesn't tell you anything new. (More precisely, there are two kinds of error, experimental error coming from randomness or bias in the experimental measurement, and cosmic variance coming from the way the early universe happened to evolve. You can beat down the former by doing more or better experiments, but you can't get rid of the the latter no matter how many experiments you do.)
Inflation says that, shortly after the Big Bang, the universe went through a period of extremely rapid exponential growth, caused by fields whose particle-excitations are too massive to be detectable in any forseeable particle accelerator. (This field is creatively named the "Inflaton". ) The inflationary hypothesis predicts that at the end of the period of inflation, the matter fields should be in a particular state, subject to statistical fluctuations originating from quantum mechanical uncertainty. As the universe expanded, these fluctuations were stretched out to enormous distance scales, whose size is now comparable to the entire observable universe. They were originally measured by the COBE (Cosmic Background Explorer) and WMAP (Wilkinson Microwave Anisotropy Probe) satellites, which measured the cosmic background radiation coming from different parts of the sky.
By looking in different directions in the sky, COBE or WMAP could detect tiny variations in the cosmic background radiation (one part in 
), which seems to mostly confirm the inflationary model. However, because these fluctuations come from a one-time-only event, if they should happen to have formed in a statistically unusual way, there is no way of compensating for this by repeating the experiment and averaging out all the results. Although repeating the experiment can reduce the uncertainty due to measurement error, there remains a residual uncertainty, the Cosmic Variance, which can never be reduced because it is always the same every time the experiment is performed.
Someone who was a stickler could say that the theory of inflation "doesn't count as Science" because you can't get an arbitrarily large amount of data on it. But that would be silly, since it's so similar to other kinds of Science, and you can still repeat in the sense of looking again.
So the requirement that Science be repeatable is flexible, depending on the nature of the particular thing being studied. Sometimes we have experiments; sometimes we have a lot of data; sometimes we have a finite amount of data that can never be increased, and we just do the best we can. But if it doesn't ultimately rely on observations, it isn't Science.
Aron, I'm intrigued, I don't understand Krauss' universe from nothing theory that well, but he uses repeatablity to argue for an eternal uni/multiverse, one that expands from quantum fields, back to quantum fields, and we can infer this happens because we know that particles & waves are interchangeable. Having come across from Unequally Yoked, I'm aware you are a Christian, so I would be interested on your take here, since Krauss & many others, claim this would put God out of a job as creator.
Smidoz,
I haven't read Krauss' book, but what you've summarized seems like nonsense. I'll have to read the book myself before saying anything more definite about it. But in general there are a fair number of ideas like this floating around, and since they are in my field (quantum gravity) I can promise I'll get around to saying something about them. But two really quick general points that apply to all proposals to explain physically the origin of the universe from nothing:
1) They are all extremely speculative. Separately, our most general well-tested physical theories (general relativity and quantum field theory) don't say anything of the kind. They are just guesses about what a hypothetical theory incorporating the principles of both theories might predict.
2) Even if true, such proposals would not explain where the laws of physics themselves come from, so it's not quite true that they would fully explain everything about the universe.
Thanks, looking forward to that, I haven't read Krauss' book, I've read interviews, & I wasn't impressed by his opinion of himself & his reference to all who disagree with him as morons. I have way too much reading to get through, & a real life to live to put his book at the front of the queue. I did get the impression from Hawking's A Brief History of Time that he believes in a universe from quantum fields, & that it would mean that a beginning wasn't required. Basically that our universe is simply part of a cycle, I inferred from Krauss' interviews that he was expanding on that kind of idea.