A stats exercise with The Norwegian Frequency Project

My recent interview about training frequency and bro splits was hugely popular and caused quite a stir in the fitness industry. If you haven’t watched that yet, do so before reading this post, otherwise you won’t understand a word of it.

 

Lyle McDonald chimed in with a comment about the Norwegian Frequency Project.

 

Do pay close attention to the graphs for strength and size gains. There are a couple o big ouliers on both, guys who got amazing results. Everyone else was about the same and if you take out those outliers, the differences approach jack shit. It’s when you average them with that one or two big outlier data points

 

 

Norwegian Frequency Project results

 

Note: if you have no interest in educating yourself about statistical or scientific methodology or Lyle McDonald’s critique of the Norwegian Frequency Project, you can safely skip this post, i.e. this is for the hardcore science crowd.

 

Lyle’s comment is typical of that of a non-scientist who has no knowledge about statistical methodology. The claim is essentially that the data is confounded by a few extreme positive outliers that confound the results of the analysis, i.e. without these outliers, the significant between-group differences in favor of the high frequency group disappear.

 

Let’s deconstruct this argument like a real scientist, which I did together with my assistant researcher Richie Hedderman. First, we should ask if there really are outliers. Anyone can look at a graph and point towards the highest or lowest value, but a statistician understands that is not how you identify outliers. The position of the data points on a graph is greatly influenced by the choice of the axes. If you pick small increment with many lines, it will appear as if the data points are further away from each other. So visual inspection of any graph gives a statistician nothing more than a hunch.

 

The right way to identify outliers is to evaluate a data point’s relative distance from its expected distribution. So the first step is to evaluate what kind of distribution the data has. In this case, the theoretical assumption is that the data is normally distributed. I ran Kolmogorov-Smirnov and Shapiro-Wilk’s normality tests and indeed they confirm that the assumption of normality can be maintained and parametric analyses can be used.

 

Norwegian Frequency Project tests of normality

 

The next step is to identify the distance from the expected distribution for each data point. For a normal distribution, this distance should be measured in standard deviations from the mean. You should also take sample size and the type of statistical model into account, as the lower the sample size, the greater the expected deviations from theoretical distributions, so you should be more conservative with deleting data.

 

In general, statisticians are very wary of deleting data when there is no obvious cause for outliers, like gross measurement error or a typo made by the person that input the data, as data deletion is basically cherry picking.

 

Concretely, I used the cut-off criteria from Van Selst & Jolicoeur (1994), which in our case range from 4.475 to 6.20. As you can see, in neither group did any of the data points qualify as an outlier. In fact, the highest Z-score was only 2.13, which belonged to the person that gained the most strength in the 6-day group. Even the most liberal statisticians tend to use the rule of thumb that data is only considered suspect when it’s 3 or more standard deviations from the mean. 2 SDs from the mean is perfectly normal, as in fact 95% of data points are expected to fall within that range for any normal distribution.

 

Norwegian Frequency Project Z-scores

 

So we can conclude the authors from the Norwegian Frequency Project just maybe actually knew what they were doing when they didn’t delete the outliers. The lead author was Truls Raastad after all, who has a Researchgate score in the top 5% of all researchers on Researchgate, which is higher than that of Brad Schoenfeld, Alan Aragon, Bret Contreras and me.

 

But just to humor the idea that some genetic freaks completely confounded the analyses, we can look at some other statistics. One easy statistic is the median of both groups, which can be used instead of the average score when you want to ignore the impact of high and low values.

 

Median scores

 

As you can see, the results when using median instead of mean scores are still decidedly in favor of the 6-day group.

 

Alternatively, we can go on a data killing spree and just delete all data points that show up as potential outliers on boxplots and then rerun the analysis to see if indeed the pseudo-outliers change the results. This resulted in deleting the lowest strength gaining individual in the 3-day group, the lowest scoring individuals on Vastus Lateralis growth in both groups and the lowest scoring individual on total quad growth in the 6-day group. I’ve plotted all the values below. Note how clearly the 6-day group’s results tower above those of the 3-day group. (Also note that the closest values to becoming outliers here were in fact the low scores instead of the high scores, showing once again that visual inspection of absolute data points is misguided here and statistics are needed.)

 

Boxplot VL growth

Boxplot strength

Boxplot quad growth

 

Guess what happened? All values were still decidedly in favor of the 6-day group, as you can see in the analysis output below. The 6-day group had significantly greater gains for all 3 measures, both strength and size.

 

Results after outlier deletion Norwegian Frequency Project

 

Let this be a lesson for all lay individuals that are prone to just look at averages or graphs and jump to conclusions. Not only were there no true outliers in the Norwegian Frequency Project, deleting any pseudo-outliers actually strengthened the results

36 Comments

  1. Hi Menno,

    Liked the analysis. That’s we us epidemiologists call “sensitivity analysis.”

    Glad to see the studies’ results confirmed. Hadn’t heard of it: interesting results.

    But I think you’re being hard on Lyle.

    keep the good stuff coming.

    -Etienne

  2. Alex Honeycutt says:

    Here is a basic research term. Clinical significance. And this statistical significance has none of it. Therefore your splitting hairs here over nothing. doubling frequency with the same amount of volume for the sake of 2-5% of growth difference means nothing.

    • The measure you’re looking for to describe magnitude of effect here is called effect size. And effect sizes were actually not small at all as you can calculate based on the data I presented.

  3. lupulo says:

    hi,
    “genetic freaks”: we all know what this really means… :)

  4. Tomaz says:

    Hmm, I trained with bro split and had good results. Then I started to read websites like this where it is all about frequency, how you do not need to train arms directly etc. Guess what. I just stalled and I was getting smaller. I returned to 4 days a week bro split, training each muscle only once per week but with a lot more volume and guess what, I am growing again. No direct arm work? Yeah, if you want to be a power-lifter.. There was a study about how bodybuilders train and 95 % of them do tons of volume and only work out muscle once per week. Why? Argument goes oh they are jucing, but this is just stupid. If they juice, their recovery improve, so they would be able to train each muscle that much more, not with reduced frequency. I am very thin boned, a true ectomorph, but still more volume works much better for me than more frequency. More volume WITH more freqency would be best, but then CNS becomes a problem. You just cannot work all muscles effectively if you do upper / lower split for example. By the time you come to muscles worked at the end of the training, you are already fatigued. My current split which works for me is:

    mon: legs
    tue: off
    wed: shoulders – triceps
    thu: off
    fri: back – traps – abs
    sat: off
    sun: chest – biceps – abs

    I tryed upper – lower, but I had to remove a lot of exercises and if I started with chest I did it great, then back was already a bit less worked, then shoulders were already tired out and by the time I reached biceps and triceps I had no energy left, so I would just do a few sets of each, and this was not enough.. Now I do 3 exercises for shoulders and 3 for triceps and I am growing. I think best for me would be to work muscle every 5 days, but I could not recover so once every 7 days is a compromise. And every time I thought I could only do one exercise for shoulder and only one for triceps, or maybe two with less sets, it was just not enough. Maybe this is because I already train so long.. Beginners could probably fare better on 3 times a week full body, but for me it just does not work. Also check Mike Matthews. He has 6 days a week training and works each muscle only once with even more volume etc, works greats for him and his followers.

  5. Tomaz says:

    No need to get a reply from Meno, it was just my comment..

    Steve: like I said, it works for me and it works for a lot of people I know. And yeah, frequency also works for a lot of people. I know some people only need to do compound lifts and they grow like crazy, even their arms. For others, isolation exercises are a must. We are not built the same. We have different proportions, different mechanical advantages and disadvantages etc.. Some people squat a ton yet have chicken legs and vice versa.. For some people bench press builds great pecs, for others it works more shoulders and triceps etc.. You cannot explain everything with one principle, nor train all the people with the same routine, frequency and volume. Heck read the book The Sport gene. Genetics do matter and more than some people care to admit.

    • Chris says:

      Watch Mennos videos if you truly are interested and open about training knowledge.
      Some points:
      – Exercise selection and genetics is an own topic. Frequency considerations certainly doesnt mean to omit isolation exercises or to prefer one squat oder leg press over another – what makes u think that?
      – There is NO one frequency for everyone, true. BUT: The evidence is simply there that the more advanced you are, the more frequency you need.
      – What “works” can only be determined in an objective assessment. NOT by probably doing everything wrong you can do with high frequency and then claiming it wouldnt “work” for you, because you allegedly are super-unique. And of course it could very well be that you are not that advanced as you think you are (in fact thats very good for you) and bc of that – you wont need a high frequency.

      Again: Watch the 3 recent videos by Menno on the topic (links on the bayesianbb main site), learn about how implementing high frequency, learn about the evidence – then re-assess your opinion.

      • tomaz says:

        Hmm, I beg to differ with the statement that the more advanced you are, the more frequency you need. I think it is vice versa. Beginner can train each body part with more frequency, less exercises and less volume. You do not expect beginner to do like 4 exercises and 5 sets for each muscle part, do you? And I do not see how advanced bodybuilder would need more frequency.

        beginner: 3 times a week, full body, everything works at this stage, so why hammer your muscle with so many exercises and volume, when you progress linearly, adding weight from workout to workout etc..

        intermediate: more volume needed and more exercises. So here some upper / lower split is advisable, because you cannot train full body with so much volume.

        advanced: even more volume and more exercises should be added so body part split training is needed. Max two body parts. For more advanced even one body part per day.

        You have to add more and more volume the more advanced you become and at the same time REDUCE frequency. Either this, or all advanced bodybuilders have it all wrong and OMG, they should only start doing basic OLY lifting hitting each muscle 3 times a week..

        I am no way unique, even less advanced, far from it. But like I said I tryed full body 2-3 times a week and also upper – lower split 4 times a week and got nowhere. My motivation was also down, because I was not looking forward to doing all those muscle in the same day. Like I said when you hit your back hard, you do not have enough energy for another big muscle group like chest, let alone biceps, triceps and shoulders. And you then have to add traps, abs, even forearms if you wish.. And no, I did not get much bigger arms just doing tons of benching and rowing.. This is another stupidity flowing around.. If you want big arms, train them. If you just want to brag how much you lift, do only squats and bench.. Again, strength training is not bodybuilding.. I neglected my arms and shoulders because of this and now I have very overdeveloped chest and back, with lagging shoulders and lagging arms. I incorporated direct arms training and they are growing. I even had to add hammer curls to develop brachioradialis. So not only not training your biceps directly is stupid, you have to use different exercises for different parts to show. Even more with triceps. Close grip bench is not going to develop upper portion of triceps, pressdown will only develop part of triceps etc. So if you want good looking arms, you have to do like 3 exercises for triceps, 2 for biceps, at least 3 for back, 3 for chest, 3 for shoulders, 1 for traps, 2 for abs etc.. And you are going to do this all with enough volume in one session?
        By the time you come to muscles at the end of workout, you will not be able to hold an empty barbell, let alone work a muscle enough..

        And please do not quote studies showing MPS returning to normal after 24 hours in advanced lifters. This is not all there is to muscle growth. It has been proven that yes, MPS returns to normal even as low as 6 hours after the training, but that does not mean muscle if completely recovered. It can take 5-7 days for muscle to fully recover. Just looking at muscle protein synthesis is nonsense.

        Again, to each his own, but I know what works for me.. And I see too many people jumping on 5×5 or powerlifting routines, expecting that only squat, bench press and row will give them what they want. Leg extensions are for pussies right? But it just so happens they do wonders for my legs, because I have long femur bones and squats do not do much for my quads.. Also a lot of bodybuilders do not even deadlift and have very developed backs etc.. I could go on and on, when everyone want to have a bodybuilder look training like a powerlifter… Fu.. it, back to basics works just fine..

        • Chris says:

          Well, evidence is simply there that the more advanced you are, the more frequency (and of course, volume) you need. It´s logical foundations like MPS and repeated bout effect, it´s practial evidence by meta-analysiis, its studies like the NF Project – what else do you need?
          Its like that in every sports, thats just how the general adaptation model works.
          No professional athlete but (part, I might add) of a subculture of drug-using bodybuilders refuses to train a muscle or movement more often than once a week. Noone.
          Maybe bc its tradition (again: watch the video with Menno´s remark about the pre-Arnold-era), maybe because they do want to destroy every muscle once a week – after all, sport has an emotional component. And very probably its bc they dont know how to structure a high frequency program and have failed – like you – when trying one.
          Again, so many misunderstandings in your post like “high frequency means 5×5 powerlifter training” or “high frequency doesnt permit exercise variety”, “it looks like a beginner program” – all so far from true that I simply have to say: please do watch all the videos entirely and LEARN before posting. Instead of repeating false assumptions what high frequency means and programs look like. Thx!

          • Tomaz says:

            Well I will watch videos again. Maybe you are correct and I did it wrong when I tried high frequency. Can you recommend any good high frequency program then? Not just videos etc but actual program with detailed split, exercises, sets, reps. At least some template for starters?

            I just know that big 3 or big 5 or whatever is not enough for me. I have to do more exercises, include isolation exercises etc, and at the end of the day, this is hard to incorporate all in one single day or into AB split. I also do not want to train more than 3-4 days per week. I am 36 and my recovery was not that great even at 25 years, let alone now. So trying to do all this high frequency in 3-4 days means upper lower split and I am again at the beginning – too much exercises and volume cramped into one day.

            But I am open to some good program suggestion if you can list one or give me a link.

            thanks

            Tomaz

          • Eddie says:

            You really should digest every single article, video, and podcast here. Borge Fagerli has an excellent article on program design on his site.

          • Eddie says:

            That was meant for Tomaz, by the way.

          • François says:

            There are a couple of points:

            1) The debate is never about if bro splits “work” or not. It’s about their optimality. Higher frequency even with a given set volume would typically increase the real volume perceived by your body because you approach that muscle group with freshness, allowing you a bigger load.

            2) HFT is not required for beginners because they can inflict a greater damage on their muscle, therefore their “anabolic window” is longer. Though for rank beginners HFT is not a terrible idea from the standpoint of building the motor skills required to execute those lifts. Often, the real quick, linear progressive overload you notice with beginners mostly be increased neural efficiency.

            3) Literally, no one sensible will say shit like “oh leg exts are for pussies.” If you inferred that from Menno’s content, then I’m unsure what really were you reading/watching. Yes, there are the dumbass zealots on the powerlifting extreme (the IIYFM, Inzer Forever Level belt-carrying peeps who wear bright Romaleos), but no one is giving heed to those zealots here.

            4) Menno places a special emphasis repeatedly on the value of individualization of a workout and nutrition program. His (along with other academically inclined lifters) content is gold because they teach you principles. If you can conceptually grasp principles well, you can optimize what methods work for you… Optimization has to be done given a set of constraints, and those constraints vary individually. I’m an early intermediate lifter, and I’ve made mad gains since I’ve learned to take control of my own programming. I can only workout 4x a week max, and I typically hit each body part 2x a week. A rough example of my workout is as follows:

            Monday – Jump Training, Horizontal Press (Reverse Pyramid), Vertical Press (RP), Weighted Dips (Auto-regulated), Lateral Delt (AR), Rear Delts (AR), Leg Extensions (AR), Calves (AR), Overhead Tris (AR)

            Wednesday – Deadlift (RP), Weighted Chins (RP), BB Curls (RP), Hamstrings (AR), Lat isolation [cable rows/lat prayers] (AR), Face pulls (AR), Traps (AR)

            Friday – OHP (RP), Front Sq (RP), Lateral Delt (AR), Tric (AR), Rear Delts (AR), Chest [weighted pushups] (AR), Glutes (AR), OH Tric (AR)

            Sat – Jump Training, Weighted Chins (RP), Weighted Dips (RP), Lat iso (AR), Face pulls (AR), Traps (AR), Biceps (AR)

            Of course, volume varies on different factors… I might increase my frequency further in a couple of months. Although this kind of a template has given my absolutely astonishing results. I did a great recomp; my strength and mass have skyrocketed. Even my “big 3” (except I do Front Sqs instead of Back Sqs) have far exceeded what I could have imagined in my wildest dreams. My 1 year and 5 month progress has been quite sweet. Though when I started I wasn’t a complete beginner, but I had only dabbled in silly things like Occam’s protocol before this

            BW reduced from 110 kgs to 76kgs
            DL increased from 65×3 to 210×3
            FS: 40×3 to 140×3
            BP: 70×5 to 135×5
            OHP: 35×2 to 87.5×4
            WC: BWx4 to BW+37.5×5
            WD: BWx5 to BW+45×5

  6. jddavis says:

    http://www.mensfitness.com/mensfitness.com/strongin4weeks

    I’ve tried this simple one by Greg Nuckols. It says every other day, but he later commented that it should be 4 day/week, just no more than 2 days in a row . Also, I think there’s a typo on the bench numbers somewhere (can’t remember where), just make the bench percentages like the other lifts, follow the format. I’ve tried it and it seems to work, but I’d say there are tons like these. I also asked him if it’d be OK to throw in some OHP, his reply was, “If you like”.

  7. Darren says:

    Eddie, I don’t think Tomaz will be interested in Borge’s article as he says to just pick variations of ‘a push, pull, squat, and push around some heavy stuff’! That won’t satisfy most bodybuilders. I can sympathize with Tomaz here after following ‘basic’ routines, they caused a lot of imbalances for me.

    I don’t think high frequency will take off for bodybuilders because of that incessant need for training all the angles, total annihilation of a muscle group in one workout (or it won’t grow bro), going for the pump, taking each set to failure plus forced reps etc.

    Personally, I think I’m just going to run HST for a few cycles and see how it goes. At least it has a basic frame work of principles to follow and I still get to choose what exercises I run, how many sets, how many days per week etc.

  8. Tomaz says:

    Darren:

    imbalances, that is what I am talking about. Just do big 3, big 5, some accessory work etc.. Nope, not working. You develop big pecs, big quads, big back, but lack
    calves, arms and shoulders.

    I tried PPL routine, but the problem is that I do not like working chest, shoulders and triceps at the same time. If I start with chest then my shoulders are really tired already and I can only lift like 70 % of the weight. The same with triceps. And this lack of bigger weights leaves those muscles underdeveloped. So I rather pair chest – biceps to have both muscles fresh. Then when I start a few days later with shoulders, I can move a lot more weight doing military press and the difference in muscle mass shows the results. Besides doing split actually works almost all muscles twice weekly. When you do chest, shoulders and triceps are working, when you do back, biceps is also involved. When I do deadlift on back day, legs are working etc…

    • Eddie says:

      I think my comment could still apply. If you read Menno’s articles on Muscle-Specific Hypertrophy, and 7 Principles of Exercise Selection you would have a good idea of which exercies and rep ranges to use for all your body parts. Then you can implement Borge’s information to program your frequency and sets & reps. And of course, Menno’s latest videos explain this too.

      • Darren says:

        Eddie, 100% agree if one is willing to use a bit of brain power! A lot of guys just want a routine to follow that they don’t have to put much thought into :(

        Tomaz, have you seen the frequency study from Brad Schoenfeld done on experienced lifters?

        http://www.ncbi.nlm.nih.gov/pubmed/25932981

        http://www.lookgreatnaked.com/blog/bro-split-versus-total-body-training-which-builds-more-muscle/

        The results are that doing just 2 sets of curls 3 x per week after the compound work gave better gains than having an ‘arm day’ once per week.

        • François says:

          This is so true. Everyone wants results, but no one wants to put in the work to learn the principles which enable you to optimize results with your given constraints. I’m just so sick of lazy yahoos now, whenever someone comes to me for the second time at the gym to try to get me to tell him my secret, I just give them a generic 3/4/5/6 day bro split. People don’t like to educate themselves; when they try, they are either too lazy to keep digging or they are so aghast at the possibility to knowing that there might be better ways beyond what they are always convinced about that they just refuse to believe it outrightly.

  9. Jan Helge Uggedal says:

    Hi!
    whats your thoughts on TUT in hypertrophy benefits?
    for example the classic 4020 speed movement?

    and in the high frequency training program you use, are you using for example 4 x 10 on all exercise one day, and the next day 3 x 5? or is the reps pretty consistent in all 5 or 6 days a person trains.

    • TUT is the product of repetition tempo and volume. It’s not an independent variable to worry about.

      • Jan Helge says:

        thx for the answer :-)
        whats your thoughts on this: “and in the high frequency training program you use, are you using for example 4 x 10 on all exercise one day, and the next day 3 x 5? or is the reps pretty consistent in all 5 or 6 days a person trains.”

  10. Zamot says:

    If anyone is interested, check out this guy. Training once per 12-14 days, drugs free, incredible physique.. Making gainzzz not training each muscle 3 times per week but 3 times per month..

    https://www.youtube.com/user/bastionhead/videos

    As for once per week muscle training, I can give you tons of link at how good it works. From Muscle hack (5 days split) to Muscle for life (6 days split), and Tom Venuto (4 days split). How about Jeff Nippard and his youtube channel, where he explains why “BRO” splits are good for bodybuilding? People, open your eyes, there is MORE than one way to train. I have N = 1 experiment and for me, “BRO” split works best, blasting muscle once per week and leaving it to repair until the next session. Greg Nuckols have a whole article why this works, because if you train very often, muscle get used to it and stops growing. That is why so many people see their best growth after 2 weeks pause, or after one week deload… Try for youself and see how it works for you..

  11. YF says:

    Findings and interpretation of the Norwegian study have been critiqued here:

    http://muscleevo.net/double-the-muscle/

  12. Steve says:

    Menno, what about the comparison of Two-a-days with a squat/push/pull split for example where each movement is performed twice a week versus high frequency training?

    Thanks!

  13. Dave Winchester says:

    This is an older article but still very meh.

    For uninformed readers, don’t be fooled by the official looking tables; there’s no rigor here. From the start, KS test is likely not effective at these sample sizes. Why are you using Z scores at these quantities? T-test is far more appropriate as you’ll be more sensitive to weighting in the rails, as evidenced by your outliers.

    Paraphrasing you: most liberal statisticians use 3sigma to reject outliers.”No, this is just flat out not true. There’s no “universal” rule like this for outliers. Even if you consider only symmetric, continuous distributions–like you’ve done here–your comment still isn’t true.

    You’ve attacked Lyle as a non-scientist and then proceeded to perform probably the most rudimentary analysis using SPSS or something. I’m not disagreeing with the conclusions–I didn’t agree with Lyle’s comments–but billing this as the scientific rebuttal to Lyle’s claim is laughable.

    • 3 Sigma is a common rule found in many stats textbooks. I don’t particularly agree with its use myself, but it’s a good starting point to look at. How would your conclusions have differed here?

      • Dave WInchester says:

        I think you missed the part where I said I *DO* agree with your conclusions. My guess is Lyle considered them outliers because it supported his confirmation bias. Your comment of outliers and cherry picking was spot on in that case.

        I’ve certainly heard of such rules for outliers,but no they aren’t good. Sometimes its 2SD, sometimes it’s 3SD. Think about the former case though, if you’re tossing out any data with |d| > 2SD, you’re tossing 5%. 3SD, 0.3%… but why? Plus, outlier tests like Grubb’s use critical t-values, which as you’re aware change with DoF, and t-crit quickly drops below 3. I’m assuming you used 5% significance for everything based on your last table. The final method you actually settled on to make your point–box plots using the IQR–don’t even consider the distribution.

        I’ll repeat: my disagreement is that this is the “hardcore science” response. You have small samples, aren’t assessing your data graphically at all; does it even look symmetric? Did it *look* normal? You made no effort to really investigate this and stopped at a what amounted to a menu click -> perform KS test. Ultimately you’re relying on a calculation method that isn’t that powerful at sample sizes of 6 to 7. This whole thing showed some pretty-looking plots and tables, but in reality was a belabored explanation of a simple t-test of equal means and didn’t really go beyond it. Not hardcore science.

        • Of course I looked at all the data. I can’t very well dump all the output on my blog. In any case, the most valid way to assess the impact of outliers is arguably to run the analysis with and without them to see if they change anything, which I did, so I don’t see how anyone could come to any different conclusion here and I’m strongly getting the impression you’re more interested in appearing intellectual than in offering anything constructive.

Leave a Comment