I would take it even farther and claim that very little in experimental psychology / sociology has any value whatsoever. Google the top papers of all time in psychology and its either been formally debunked (Stanford prison experiment) or it has no applicable impact on society.
Using computers to parse statistical data is interesting and can reveal truths about society. Trying to correlate the delay between eating a marshmellow as a child and income as an adult is ridiculous, yet collectively we have spent untold money and time on tenured professors, conferences, papers, and pop psychology books about such things. Another example is Power Posing. How Amy Cuddy still gets employment is a mystery.
Here is one of my favourite papers: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6880188/ - it is published in a psychiatry journal not a psychology journal, but it is absolutely an area which is studied in psychology-there is no clearcut boundary between psychology research and psychiatry research anyway
The paper adds to the growing body of evidence that DSM-5 diagnoses lack validity. The study specifically looks at ASD, ADHD and OCD in children/adolescents, but there are other studies similarly examining other diagnoses. I think this is very important research, because if we are going to use psychiatric diagnoses, we have to support research which questions the evidence behind them. If we don’t think that kind of research is worthwhile, then either we stop using them altogether, or else we are turning them into something faith-based rather than science-based.
I don't think anyone in the field seriously claims that DSM-5 diagnoses have some particular validity. Most of them are just clusters of highly correlated symptoms without a clear, objective causal link to any underlying pathology. But as a practical matter, in clinical practice it's useful to have labels for those clusters of symptoms. This helps to facilitate communications between providers, as well as in generating the documentation required by payers to justify insurance claims.
> I don't think anyone in the field seriously claims that DSM-5 diagnoses have some particular validity
In my personal experience, many clinicians – and even researchers – talk about diagnoses in a reified way which only makes sense if one presumes they do have some sort of validity. Many of them also appear to engage in the "motte-and-bailey fallacy" – talking about diagnoses as if they were valid, falling back on "we all know they aren't really" whenever that talk is challenged, and then going straight back to talking that way as soon as the challenge has departed.
If one takes the lack of validity of DSM-5 diagnoses seriously, that has certain consequences for research programming and study design – that one ought to prefer trans-diagnostic studies to those focusing on single diagnoses, that one ought to study cohorts which include subclinical cases along with those with a clinical diagnosis, etc – and yet, very many researchers continue to ignore all those recommendations, which suggests they don't actually take that lack of validity seriously at all.
In 2017, the journal Autism Research published a letter to the editors, by Lynn Waterhouse, Eric London and Christopher Gillberg, entitled "The ASD diagnosis has blocked the discovery of valid biological variation in neurodevelopmental social impairment" [0] – the authors argue that focusing research on a diagnosis which lacks validity is a scientific dead-end, that it is no wonder that it has failed to produce the results which its promoters had promised, and that the field is going to continue to be fruitless until research is re-oriented away from its current focus on an invalid diagnostic category. And I'm sure they'd say that, while their letter was about ASD specifically, the same is likely true for many other DSM-5 diagnoses as well
> Most of them are just clusters of highly correlated symptoms without a clear, objective causal link to any underlying pathology.
How strong is the evidence that these symptoms actually are "highly correlated"? To take ASD as an example, the diagnostic criteria refer to problems in two separate domains – social communication, and restricted/repetitive behaviours/interests (RRBIs) – with problems in both domains required for a positive diagnosis. This linkage is justified on the grounds that the two symptom domains are "highly correlated". However, the scientific evidence for the strength of that claimed correlation is actually rather weak and lacking – see [1]. And if that is true for ASD, it is probably true for many other diagnoses as well
> This helps to facilitate communications between providers
If the point was really to facilitate communication between providers, a better approach would be to just provide a list of symptoms for each patient/client. DSM-5 diagnoses are less useful than that, because many of them are defined in terms such as "Exhibits at least M of the following N:", so two people can both have the same diagnosis without actually having any symptoms in common.
> as well as in generating the documentation required by payers to justify insurance claims.
Well, here we get to the real point of diagnosis – a collection of cultural constructs in which policy-makers have faith, so they'll construct policies (insurance, disability service funding, research funding, drug approval, etc) oriented around those cultural constructs – and they display a studied ignorance of the lack of scientific evidence to support what they are doing, and also the very real possibility that doing things that way is at best useless and at worst positively harmful (see [2])
“Experimental psychology” might be a bigger field than you imagine.
It includes tons of work characterizing sensory perception. These studies tend to replicate pretty well and they have a pretty broad range of real-world applications: psychoacoustic models for audio compression, principles like pop-out and serial/parallel search that inform user interfaces, color reproduction. In fact, psychophysical experiments pinned down the properties of cone cells over a hundred years before they were directly measured.
Psychometrics, the study of test/assessment design, also shows up all over the place: education, obviously, but also marketing and other forms of evaluation.
Take what you said and apply the same logic to privacy-focused cryptocurrencies. You are armchair quarterbacking and speaking beyond your expertise, which is fine, but the claim you make is too strong for the supposed evidence you mention.
We'd probably all agree that's worth keeping an eye on your own credulity. That applies both to believing the scientists themselves and to believing critics that the science is overall bunk.
>Take what you said and apply the same logic to privacy-focused cryptocurrencies
Is this supposed to be a gotcha? Those have mostly been used as Ponzi schemes or hobbyist toys that extract wealth from those who don’t have it to lose
No, not a gotcha. Sorry if that wasn't clear. It was an invitation to a perspective shift.
My aging mother lost her retirement savings to crypto "investing" despite my attempts to dissuade her for a few years. I do think that privacy-focused cryptocurrency is a legitimately worthwhile area of study/development even though the current industry is beyond a dumpster fire. But the people who are serious about it as an area of inquiry are going to stick around and use the mistakes of the past to improve. The people in it only to make a quick buck or billions probably won't.
As with nearly everything, there's more nuance involved than armchair critics tend to appreciate.
'Other kinds of errors are more characteristic of poor science. When I was at Cornell. I often talked to the people in the psychology department. One of the students told me she wanted to do an experiment that went something like this—I don’t remember it in detail, but it had been found by others that under certain circumstances, X, rats did something, A. She was curious as to whether, if she changed the circumstances to Y, they would still do, A. So her proposal was to do the experiment under circumstances Y and see if they still did A.
I explained to her that it was necessary first to repeat in her laboratory the experiment of the other person—to do it under condition X to see if she could also get result A—and then change to Y and see if A changed. Then she would know that the real difference was the thing she thought she had under control.
She was very delighted with this new idea, and went to her professor. And his reply was, no, you cannot do that, because the experiment has already been done and you would be wasting time. This was in about 1935 or so, and it seems to have been the general policy then to not try to repeat psychological experiments, but only to change the conditions and see what happens.'
Judging by the replication crisis in psychology, not much has changed since Feynman's day.
I have nothing against you personally, but you have proved your own ignorance. I stand by my original comment.
It has never been acceptable in psychology or any other scientific discipline not to include a control condition in an experiment. You may not know that many statistical tests were developed by psychologists, and nearly all early development of statistical testing was done by by social scientists of all kinds. Same tests still used by data scientists today.
You seriously think that because Feynman happened to advise a psychology student with a poor adviser in 1935, that an indictment of the science 70-90 years later, that was then thoroughly embraced and prioritized by the entire discipline, even though the crisis was mostly applicable to a sub-field (social psychology), still discredits that entire area of science?
You're certain of that, even though you aren't in that discipline and have no professional responsibility to maintain that expertise?
Either way, rethink your standard for certainty. This whole jack-of-all-trades but also an expert-at-all-trades nonsense has to stop.
It has. You haven't read recent psychology textbooks. The biggest critics of Zimbardo (and Milgram) were and are psychologists. Those ethical issues (which are not the same as epistemologic issues that make things pseudoscience) helped psychology and social science develop immensely. Anyone can double-check the assertions we are both making. What is your deal exactly?
> which are not the same as epistemologic issues that make things pseudoscience
SPE was garbage science and all attempts to replicate the results failed. Your deflection of 'ethical issues' underscores the problem with psychology when you ignore the fact that a hallmark study in the field was rigged by the experimenter to get the results he wanted.
As for textbooks:
> Zimbardo’s 1971 Stanford Prison Experiment (SPE), one of the most famous studies in psychology, is discussed in most introductory textbooks. The present study is concerned with the nature of this coverage, given that there have been myriad criticisms, especially recently, of the SPE. These criticisms concern both Zimbardo’s situationist explanation of the outcome and the study’s methodology, such as the presence of strong demand characteristics. Thirteen contemporary introductory textbooks were analyzed for their coverage of the SPE and the ensuing criticisms of it. Eleven of these texts discussed the SPE, but only six even mentioned any of the criticisms. Possible explanations for such coverage and a plan to incorporate more accurate coverage within the discussion of research methods are offered.
> ...
> Of the 11 texts, 5 did not include any criticism of the SPE, and the other 6 provided very minimal discussions of such criticism. One text included a sentence questioning the ecological validity of the SPE results but did not provide a reference. Three texts briefly discussed ethical questions created by the SPE, such as the question of whether the ethical costs of the SPE outweighed its scientific gains but did not provide references. One of these texts also stated that many researchers have challenged the legitimacy of the SPE, which would seem to be referring to the recent criticisms of the SPE, but no references were provided.
So again, you are talking about social psychology, where much of the crisis occured and persists. The SPE, even done properly, is not foundational to any work that nearly any psychologist does.
Introductory survey textbooks are essentially enrollment propaganda. They are bad places to seek expertise. Some psychologists make it part of their life work to write better ones. But it's psychologists who are criticizing and working to improve the field.
Sorry to say, I have to disengage to drive my family home.
> I am not an expert in astrology either, but I am certain it is bunk.
Consider the nature of this style of thinking from a psychological and logical/epistemic perspective, the various different forms it can manifest in, etc.
Propaganda certainly seems to work quite well, perhaps it would work less well if (the good parts of) psychology was a mandatory part of basic curriculum.
The marshmallow effect replicated (IIRC), and is probably correct, unlike the power posing stuff. That behavior in a single trial in childhood moderately predicts success in adulthood seems like an interesting fact about humans.
What's the applicable result to this? Put children who eat marshmellows too quickly in special "delayed gratification" classes? Is the result such that all children who eat marshmellows too quickly should give up, as their success is predetermined, or is it the impact is barely statistically significant?
If I were the parent of a child who ate the marshmellow too quickly, I would do nothing different, as the whole thing is meaningless and doesn't change anything. It's more like, "Oh, OK, my son is 4% less likely to be in the top 20% of the income distribution (according to this barely coherent theory)." Why do we spend so much money, time, and effort discussing and researching this?
The experiment is part of a larger research program to study delayed gratification. It was never intended to help parents raise their children. Asking for immediate applications is an odd standard for basic science research. (Unless you reject the value of basic science research entirely.)
Further, the result in a recent replication [1] was a correlation of .28 between the time to ring the bell (to get the marshmallow) and academic achievement. That's not exactly the trivial "4% less likely" effect in your caricature.
(You may also wish to double check your spelling of "marshmallow.")
What is the skill they should teach
? To always eat lunch before doing a psych experiment? To like sugar, or marshmallows in particular, just slightly less? That high self-control in extremely low-reward situations is good?
Teaching a child to do basic budgeting with pocket money is likely to have a vastly higher impact than slight delayed gratification with candy.
I know it's super taboo to consider especially in the current climate but is it possible that people who have high self control are more likely to become rich because of their ability to delay gratification and that therefore those genes will be passed on to their children?
Biology will select against this. All primate females seem to be much more attracted by bold males taking risks, and more prone to use sex to appease dominant males showing violent episodes. The opposite argument could be equally valid.
The only thing that children of rich parents share is to have rich parents. One of the parents being over the mean in the "beautiful" trait, is also very common.
If I understand correctly, you're asserting that that's causal--that being born into affluence causes children to develop in a way that leads to them "passing" the marshmallow test.
Do you have evidence to exclude other possible interpretations, such as the possibility that children resisting the temptation of a marshmallow and their parents being rich are actually the effects of some other underlying single cause?
> If I understand correctly, you're asserting that that's causal--that being born into affluence causes children to develop in a way that leads to them "passing" the marshmallow test.
Stop and think.
If you are rich that marshmallow is probably very bland or you've eaten very well today. So maybe two makes more sense.
If you aren't, that marshmallow is super tasty and you probably didn't eat well today. Plus do you believe that grownup when your mom was lying. Someone might come and eat it.
Then some time passes. Rich kids grow up and are more successful. Delayed gratification is key to success!
The really big, splashy papers like the Stanford Prison Experiment and Milgram are partly well-known because of what bad science they are.
They're also 50 years old.
If that's your idea of what most of psychology is like, then I imagine you think 90% physics is all about things like phlogiston.
Just because there are some very well-known experiments that were very poorly conceived and conducted does not say anything about the field as a whole.
As someone who works in tech for a psychology department, I can tell you from a decade's experience that a) the people there care very deeply about performing good science, including replicating results, pre-registering papers, and doing good statistics (not p-hacking), b) there's a lot of psychological research that's decidedly unsexy, very much about things like proving what we already intuitively know to be true or sussing out small nuances between X and Y, and c) while they teach about Milgram and his ilk, it's almost entirely in a "What Not To Do" type of way.
SPE is still written about uncritically in psychology textbooks. If there are any criticisms, they're ethical criticisms rather than the fact that SPE was scientifically garbage. If physics textbooks still spoke uncritically about phlogiston theory, you'd have a point.
Thanks. Well, textbooks last copyrighted a decade ago, right as the reckoning in experimental psychology around the replication crisis was taking off. I suspect they may look different now.
When your life’s work and income rely on believing something, and you are surrounded by a tower of lies, it’s easy to avoid asking very hard questions. The replication crisis and recent situations of glowing NYT articles and mainstream acceptance of “scientific discoveries” like Power Posing never, ever would have happened in a hard science like math, physics, or chemistry. The “science” (I use the term as loosely as possible) is barely there and the entire facade is designed to extract ever more tax dollars in NIH and NSF grants. It is a mass delusion.
> never, ever would have happened in a hard science
If only!
Cancer biology has also struggled mightily with replicability, as has machine learning. Amgen could only replicate about half of the published findings in their reproducibility project; I’ve reviewed MLmreplications and the rate isn’t much better. Some of this is undoubtedly for petty reasons—-people run bad experiments, or put their thumb on the scale to get a particular result.
However, another major issue is that the “soft” sciences are actually quite hard. There are a million factors that might affect how an organism responds in a test. We often can’t control for all of them, and may not even know what they are. In a sense, electrons are easier to study. They’re all exactly identical, they only interact with each other in a few simple ways, and they don’t care at all about what you, the experimenter, might want or expect. Every patient (or rat) is unique and susceptible to all kinds of influences based on their interactions with each other, the experimenter, and the rest of the world. In that sense, the “replication crisis” just means we’re slowly and awkwardly discovering what matters.
It was on the softer side in Auguste Compte’s original 19th century definition, but things have moved on quite a bit since then. Platt’s famous article on “Strong Inference” explicitly calls out molecular biology as a success—-and does so ahead of physics! Storer calls biochemistry a hard science and botany a “medium hard” one.
The whole idea of a single, uni-dimensional spectrum of “hardness” is pretty silly, but even within it, the natural sciences (especially in the lab) are pretty unambiguously “hard”.
Milgram was unethical due to the severe emotional distress of many of the participants, and it was part of the impetus for the development of the modern Institutional Review Board structure and Informed Consent requirements for psychological experiments.
Its findings are somewhat less suspect than Zimbardo's in the Stanford Prison Experiment (which is now fairly widely panned as a godawful experimental design and implementation, even apart from its ethical problems), but it's still seen as a very big signpost saying What Not To Do.
> I would take it even farther and claim that very little in experimental psychology / sociology has any value whatsoever.
You are right, probably. I'm not exactly sure, because "top papers" like Standford Prison Experiment, are at the top for reasons that seems to me arbitrary and far away from a science. There are a lot of very good articles, some authors are really dig into their problems. Though these are not known for a public. They do not make good headlines. It is really annoying, because basically all popular psychology is a junk-science or have nothing to do with a science at all.
But in any case we can agree that there are a lot of papers with little to no value. And as I see it is because psychology and sociology are on the quest of creating the Method. The Method of research that works in their discipline. They do better now, then it was before. Now a paper like Zimbardo's wouldn't stand a chance. Papers from 30 years ago probably wouldn't seem rigorous enough now. The Method is becoming better, and probably it will become even better in a future and today's research will be deemed not good enough to be science, and will be read by undergraduates as a cautionary tale how one mustn't do Science.
It is a little like programming which changes with each passing decade, though a little slower, I'd guess. But socially it is different. I mean if we look not at results of research but on a social processes leading to this result, it is different from programming. Programmers are expected to create programs that work, they need to deliver. There is a final clear criterion of a success. Psychologists do not have such a criterion. They seek Truth, but what Truth is and what isn't is debatable. And they work for grants, not for revenue. Programmers are inventing new tools and techniques allowing to create better or more complex programs, but at the same time they also need to deliver something useful for non-programmers. Psychologists are inventing new tools and techniques of research (or sharpen old ones), but they do not feel the urge to deliver something to non-researchers. And to be honest oftentimes they are unable to deliver. They can find some grain of truth, but no one can find a practical application for this grain yet.
I think it will get better when neuroscience, AI, and psychology converge and some kind of a psychological theory will emerge. They are doing it now, and they are bound to collide at some point in a future. There are lots of psychology theories now, but no physicist will agree that any of them deserves to be called a theory. They are more like vague descriptions than theories. Or they are talk about some small part of mind.
I cannot say about prospects of sociology because I know little about it, but I'd say it would take more time to bring sociology to a high standards of an empirical science.
> Trying to correlate the delay between eating a marshmellow as a child and income as an adult is ridiculous, yet collectively we have spent untold money and time on tenured professors, conferences, papers, and pop psychology books about such things. Another example is Power Posing. How Amy Cuddy still gets employment is a mystery.
May be it is ridiculous. But if we want to know how human mind works, we better pay for it. Researchers played with electricity for centuries. Michael Faraday himself spent half a century designing experiments. All that didn't deliver nothing of a real practical value. But then James Clerk Maxwell published his equations. And suddenly electricity started to deliver. Electricity delivered probably more than any other technology.
Using computers to parse statistical data is interesting and can reveal truths about society. Trying to correlate the delay between eating a marshmellow as a child and income as an adult is ridiculous, yet collectively we have spent untold money and time on tenured professors, conferences, papers, and pop psychology books about such things. Another example is Power Posing. How Amy Cuddy still gets employment is a mystery.