Wikipedia talk:Avoiding MediaWiki expansion depth limit

Cases

The examples are all hypothetical. Are there any real-life cases where more than 40 levels are required (other than accidental recursions that is). --Redrose64 (talk) 20:27, 20 December 2010 (UTC)

• The problem has occurred numerous times where Template:Convert has been used inside multi-nested infoboxes displayed on a conditional /doc subpage. If fact, at one point, {Convert} could not even display its own examples, because sub-parts of the /doc page added too many levels of nesting. In that case, the doc page had been structured as several nested templates for each part of the page. The real-life examples are so utterly huge, most people would be unable to clearly see how the nesting was the cause of problems. Hence, the examples use hypothetical nesting to clearly show the effects of exceeding the limit.
  So, long-term, we just need to rewrite the templates with reduced-nesting of the logic. Fortunately, Template:Convert will bypass several levels of nested logic (which check precision) when a rounding parameter, such as "0" is appended to the conversion.
  However, knowing the 40-deep level exists, it would be simple to rewrite other templates such as {str_len} to use fewer levels of if-else-if-else nesting, and reduce combinations of templates to below 40 levels. All over Wikipedia, many people have assumed they could use 7 or 9 levels of if-else when not really needed. This is an easy crisis to avert, by avoiding all the extra if-else nesting. -Wikid77 (talk) 22:13, 20 December 2010 (UTC)

Levels of nested templates

Note that, confusingly, when transcluding a template with more than 20 levels of nested templates like in the examples of the section Checking the current expansion limit already an error occurs (try transcluding the subject page).--Patrick (talk) 00:11, 22 December 2010 (UTC)

During 2009-2016?

The article repeatedly says "during 2009-2016" or "during 2009-2017". To me as a reader new to the subject, this would mean that it has been changed now (it doesn't say "since 2009"). So what has it been changed to? Is the limit gone? Is there another way to program the templates to avoid the problem completely? Can Lua be used instead? --Jhertel (talk) 20:32, 14 October 2017 (UTC)

I think it means "between 2009 and up to when I [whoever was editing it] last checked".

The limit does not appear to have changed, since the errors still appear at Wikipedia:Avoiding_MediaWiki expansion depth limit#Checking the current expansion limit.

Coding in Lua is the way to avoid the problem, and templates that used to run into expansion problems have now all(?) been re-coded in Lua. Peter coxhead (talk) 16:45, 15 October 2017 (UTC)

Thanks, Peter. That all makes sense. --Jhertel (talk) 21:58, 15 October 2017 (UTC)

I'm not sure the advice on this page is correct

I've been carrying out some tests in an attempt to verify the content on this page experimentally:

• User:ais523/Template limit test 1
• User:ais523/Template limit test 2
• User:ais523/Template limit test 3
• User:ais523/Template limit test 4

The idea is to attempt to apply the template {{P1}} (which simply returns its first argument) ten times to a constant string, and observe the resulting "highest expansion depth". This is done either directly (simply nesting uses of {{P1}} by hand), or via a helper template, {{User:ais523/P1x5}}, which contains five nested instances of {{P1}}.

As far as I can tell, going indirectly via the helper template always seems to be the worst option, adding another 7 to template depth each time it's used (even though it saves five levels in the calling page, it costs 12 on its own). This is the opposite of the performance properties implied by this page (which suggests using subsidiary templates in order to reduce nesting). You'd expect the nesting depth to be reduced via the helper template if MediaWiki were using call by value for template parameters (i.e. evaluating them first, then passing them as arguments), as the "Optimising "recursive" depth" section suggests. Instead, the rule (as explained on, e.g., m:Help:Expansion depth#Transcluding a template) is that if a template calls another and uses one of its own parameters to produce the parameter given to the called template, the call adds two to the expansion depth, plus the depth of the parameter itself. This is a huge cost that makes it hard to see how any sort of indirection could give savings.

Perhaps the advice on the page needs updating to a) mention the actual formula for calculating expansion depth, and b) explain how to modify templates in a way that will give the best possible values under this formula. (I can't do the latter myself because I'm not sure how templates should be modified to be more expansion-depth friendly, given these rules.) --ais523 23:20, 30 May 2018 (UTC)

I can assure you that the advice given on this page is correct, although it's completely irrelevant now, since deeply nested templates should always be replaced by a Lua module, which is much more efficient and doesn't cause problems. I spent weeks trying to make maximize the depth the WP:Automated taxobox system could handle using the template language before I gave up and rewrote it in Lua.

Looking quickly, I don't think you are following the advice correctly.

You can only put 20 levels of nesting of P1 inside another template, "Proc1" say.

To get round it you replace the content of "Proc1" by, say, {{P1|{{P1|{{P1|{{P1|{{Proc2|{{P1|{{P1|{{P1|{{P1|{{P1|test}}}}}}}}}}}}}}}}}}}}. This processes 4 levels and then passes on the 5th level.

Then you write "Proc2" that contains, say, {{P1|{{P1|{{P1|{{P1|{{Proc3|{{P1|{{P1|{{P1|{{P1|{{P1|test}}}}}}}}}}}}}}}}}}}}. This again processes 4 levels and then passes on the 5th level to a different template.

If you continue in this way with the last "ProcN" template just processing 4 levels you can process more than 20 levels.

The real example of this working is in templates that have now been deleted because they have been replaced by Lua.

This whole page is now obsolete! Peter coxhead (talk) 06:55, 31 May 2018 (UTC)

No, it's really not. It is certainly less-often needed (which is not the same as being obsolete), if you believe Lua code is inherently preferable to wiki markup. --Pi zero (talk) 23:18, 8 June 2018 (UTC)

Lua code isn't inherently preferable generally to the template language (which isn't the same as wiki markup), but it is hugely preferable when deep recursion is necessary! Peter coxhead (talk) 20:26, 10 June 2018 (UTC)

Though we're likely getting rather off-topic, I don't agree. Deep recursion should, in most cases, be avoided entirely; for that small minority of us who can use Lua at all, it becomes too easy to rely too heavily on it in ways like this. Anything that isn't avoided should be in wiki markup where it's accessible to the community (rather than locked behind a language barrier where only a high priesthood can access it). --Pi zero (talk) 00:42, 12 June 2018 (UTC)

Last comment from me: the entire automated taxobox system, used on about 190,000 pages now, depends on recursing or iterating through taxonomy templates, sometimes to depths of 40–50, to recover the full classification of a taxon. Peter coxhead (talk) 06:02, 12 June 2018 (UTC)

Fwiw. My current solution to a similar problem (though less deep atm) at Wikibooks — searching through a subject/shelf hierarchy to list all ancestors of a particular element of the hierarchy, also a transitive closure though not of strictly a tree as a node may have multiple parents — is based on semi-automation and incremental computation. For each page in the hierarchy there is a secondary page with a list of its ancestors. Each interlocking page in the system does integrity checks on its neighbors, and if it sees a problem, it flags itself with a hidden category and provides a diagnostic message to help a human operator when considering how to remedy the situation. These checks include comparing a page's list of ancestors to its parents and their lists of ancestors. The ancestor lists are sorted alphabetically, to avoid propagating duplicates, which doesn't involve a full sort operation, only a merge of the presumably-already-sorted lists from the parents. For the marginal bits of computation I use {{evalx}} (thus minimizing syntactic overhead and keeping things embedded in wiki markup). I hope in future to improve the semi-automated assistance offered when a problem is detected by means of interactive wiki dialog. --Pi zero (talk) 12:04, 12 June 2018 (UTC)

The examples you've given there can't possibly be correct as written because your version of Proc1 is calling your version of Proc2 with a parameter, and yet your version of Proc2 doesn't take a parameter (it's just a hardcoded string). I can see how this would work without parameters involved. However, I can't see any way to make it work when the value we want to apply 20 copies of P1 to is a parameter taken from outside (i.e. a parameter to Proc1); as far as my experiments show, putting P1s either around the template as a whole, or around the parameter, both cost more expansion depth than they say. If you think there's some way to do it, could you set up a demonstration in userspace (either yours or mine)? Either that, or specify the exact code that's needed (including parameters) so that I can try it out myself. --ais523 16:12, 19 June 2018 (UTC)

Ok, yes, I was a bit sloppy in what I wrote, but the principle is correct. Put {{User:Peter coxhead/Proc1|Pteranodon}} on one of your user pages, by itself, and look at the expansion depth (e.g. via the table at the bottom of the page using Preview). You'll see that although it goes 20 deep in the taxonomy templates starting at Template:Taxonomy/Pteranodon, it only has an expansion depth of 13. If you care to, you can copy the "User:Peter coxhead/ProcN" templates and carry on (Proc5 is as far as I could be bothered to go). Peter coxhead (talk) 14:54, 21 June 2018 (UTC)

Aha, I see the difference. I'm putting the expansion in a parameter to the template, i.e. {{template|{{template|…}}}}. You're putting it in part of the template name, i.e. {{template/{{template/…}}}}. And the two have rather different performance properties (the names obviously have to be evaluated eagerly, whereas at least the template limits count parameters as though they're evaluated lazily – maybe they are!). In that case, the advice isn't wrong, but it's more specific than it looks. --ais523 19:57, 22 June 2018 (UTC)

If and when I have time I'll check it out. I worked intensively on this for several weeks, based on the template code that Wikid77 had written to increase the depth of taxonomic hierarchies that could be handled, and the advice that Wikid77 had written here. I did manage to increase the depth that could be handled by a small amount. Then I realized that it was much, much easier just to convert to Lua than to try to use special tricks to get round the limits placed on expansion depth. I've forgotten the details of these tricks now, and I'm not sure I want to recover them! I can only repeat that I think this page is obsolete: use Lua is the right advice if you run into expansion depth problems. Peter coxhead (talk) 11:41, 24 June 2018 (UTC)

Transduction depth

The way this behaves seems to have changed.

The example on this help page (previously) and the following example at wikimedia (currently) both allow transclusion of 40 nested template calls. However, this example now gives errors here after a transclusion depth of 20.

So

0 {{1x|1 {{1x|2 {{1x|3 {{1x|4 {{1x|5 {{1x|6 {{1x|7 {{1x|8 {{1x|9 {{1x|10 {{1x|11 {{1x|12 {{1x|13 {{1x|14 {{1x|15 {{1x|16 {{1x|17 {{1x|18 {{1x|19 {{1x|20 {{1x|21 {{1x|22 {{1x|23 {{1x|24 {{1x|25 {{1x|26 {{1x|27 {{1x|28 {{1x|29 {{1x|30 {{1x|31 {{1x|32 {{1x|33 {{1x|34 {{1x|35 {{1x|36 {{1x|37 {{1x|38 {{1x|39 {{1x|40 {{tc|1}} 40}} 39}} 38}} 37}} 36}} 35}} 34}} 33}} 32}} 31}} 30}} 29}} 28}} 27}} 26}} 25}} 24}} 23}} 22}} 21}} 20}} 19}} 18}} 17}} 16}} 15}} 14}} 13}} 12}} 11}} 10}} 9}} 8}} 7}} 6}} 5}} 4}} 3}} 2}} 1}} 0

gives

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 {{1}} 40 39 38 37 36 35 34 33 32 31 30 29 28 27 26 25 24 23 22 21 20 19 18 17 16 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1 0

with many errors instead of only one for the 41st call. While the Wikimedia example works on the page at Wikimedia, when it is previewed it behaves as above. Something has changed.   Jts1882 | talk  11:20, 19 September 2019 (UTC)