Wikipedia:Wikipedia Signpost/2012-01-30/Technology report

Technology report

Why "Lua" is on everybody's lips, and when to expect MediaWiki 1.19

Lua chosen as new template scripting language

Reducing the inefficient complexity of complicated templates has long been on the wishlists of those who design and maintain them. It is a complexity that can put a significant drain on server resources, not to mention one that makes accurate duplication of templates between wikis incredibly time consuming and error prone. This complexity is caused by a number of factors, but it has long been recognised that the introduction of certain constructions found in full programming languages (among them loops, arrays and a full range of string functions) would drastically simplify matters.

Of course, the introduction of a programming language would bring with it a number of other problems, particularly with regards to privacy, security and memory footprint. As a result, the long term consensus was that the introduction of a programming language would be a major undertaking, including time for discussing which languages would work best in a heavily sanitised environment. The project as a whole has been discussed at regular intervals, but it was only this week that a consensus seemed to emerge in favour of Lua, a lightweight but relatively unheard-of language generally regarded as being "easy to learn" (wikitech-l mailing list).

The preferred implementation at this time is via a separate namespace for scripts, perhaps augmented by the overloading of the double-curly-brace notation to incorporate both Script and Template namespaces. Work is expected to start after the release of 1.19 and could well centre around the Berlin Hackathon, held annually in May.

In brief

Not all fixes may have gone live to WMF sites at the time of writing; some may not be scheduled to go live for many weeks.

1.19 target missed: An end-of-the month target for 1.19 code review will be narrowly missed, but a short delay is unlikely to cause problems, says WMF Director of Platform Engineering Rob Lanphier (wikitech-l mailing list). Even a burst of code review activity on Friday proved insufficient to bring the number of revisions left to be checked (now around 70) within touching distance of zero; projects with many revisions left to review include recent work on the way files are stored. In a separate post on the same list, Volunteer Development Coordinator Sumana Harihareswara reiterated the Foundation's intention to deploy the latest version of the software between 13 February and 1 March this year. A post on the Wikimedia blog carried an appeal for more testers.
Wikimedia compares favourably on usability: Data extracted from a survey of Wikipedia readers show that they rate the user interface of Wikimedia sites higher than those of Facebook and YouTube (Wikimedia blog). The data, based on responses to two questions ("How appealing do you find the interface or look of the following sites?" and "How easy do you find it to navigate the following sites?") also showed lower comparative scores from countries where right-to-left languages predominate, apparently confirming earlier suspicions that this was an area that deserves developer attention. The potential for bias within the survey was not examined.
Looking back on the Google Android app release: WMF Director of Mobile and Special Projects Tomasz Finc used a post on the Wikimedia blog to explore the implications of the recent release of a Wikipedia app for devices running the Android operating system. Finc described how the new app was built using a cross-platform library known as PhoneGap and is therefore likely to form the basis for apps for other operating systems (including a future replacement to the existing Wikimedia iPhone app). The app is already available in 25 languages and has been downloaded 500,000 times.
FeaturedFeeds deployed: The FeaturedFeeds extension was deployed this week, exposing content such as "Today's Featured Article" and "In the news" via RSS and ATOM feeds (Wikimedia blog). "For now, links to the feeds only appear in page metadata; in the future, we will add them to the sidebar on main pages, if communities wish so", wrote the extension's developer Max Semenik.
Translation tutorial sessions: An online translation tools workshop was held via WebEx video conferencing on Saturday 28 January 2012, as announced by Localisation team member Gerard Meijssen on the Wikimedia blog. The workshop included "an introduction of use cases and features, as well as a Q&A". Another workshop will be held only if there is enough interest.
Wikimedia Foundation Director of Product Development: The Wikimedia Foundation announced that Howie Fung has been hired as Director of Product Development. He has previously been a consultant for the Wikimedia Foundation on usability projects and has been Senior Product Manager since May 2010. The Wikimedia Foundation is also internally restructuring its teams, formally creating a Product Development department, focusing on "Editor Engagement, Mobile, Analytics, and Internationalization/Localization". Erik Moeller said that the team is to include "Phil Chang, Brandon Harris, Fabrice Florin, Diederik van Liere, Siebrand Mazeland, Dario Taraborelli, Oliver Keyes, and the new Interaction Designer, when hired."

← Previous "Technology report"

Next "Technology report" →

In this issue

Discuss this story

These comments are automatically transcluded from this article's talk page. To follow comments, add the page to your watchlist. If your comment has not appeared here, you can try purging the cache.

So to write templates I need to learn a new language. This could be interesting. I hope that it has more documentation then the current language --Guerillero | My Talk 04:45, 31 January 2012 (UTC)[reply]

It strikes me that it's profoundly unlikely to be worse. As a programming language, ParserFunctions is horrible. At work this month I've been doing large and complicated things with ant ... I've nicknamed it mod_brainfuck. Turing complete is actually a sort of curse - if all you have is a hammer and it's Turing-complete, you will use it as a screwdriver, spanner, soda siphon, nail ... - David Gerard (talk) 08:28, 31 January 2012 (UTC)[reply]

I remember LUA from the old days scripting my World of Warcraft client (they it for the same reasons: security and light weight). It's not a bad language (even if I found it a bit confusing). -- Luk ^talk 09:27, 31 January 2012 (UTC)[reply]
"The potential for bias within the survey was not examined." what is the point in writing about useless surveys? Bulwersator (talk) 09:51, 31 January 2012 (UTC)[reply]
- It's not useless, you just have to take it for what it is: a survey of Wikipedia readers, not a survey of the population in general. - Jarry1250 ^{[Deliberation needed]} 13:42, 31 January 2012 (UTC)[reply]
Moving to Lua will reduce the programming complexity of templates, which is good, but it will increase the resource utilization of the scripts, so somehow limitations will need to be introduced to limit script cpu time. Does Lua make this easy to do?Jeff Kubina (talk) 23:08, 31 January 2012 (UTC)[reply]
- One of the positives of Lua discussed was the ease with which memory limits can be imposed to prevent accidental/deliberate massive memory allocations taking down whole servers. (Note that I would not be so sure average resource utilisation will go up rather than down: some templates are very resource-intensive at present.) - Jarry1250 ^{[Deliberation needed]} 23:19, 31 January 2012 (UTC)[reply]
  - Excellent! Thanks Jarry. So then in terms of current templates I agree, I suspect many of them would run much more efficiently in Lua. What I suspect will happen though is that templates will evolve to provide much more (better, useful) information using about the same amount of cpu time that they use now to be generated (and be easier to write); so it is a long term win situation. Jeff Kubina (talk) 04:12, 1 February 2012 (UTC)[reply]
    - I think that would be a very apt analysis. There will of course also be some widespread templates for which no new functionality is available, which would therefore see definite drops in resource intensity ({{convert}} perhaps?) - Jarry1250 ^{[Deliberation needed]} 11:22, 1 February 2012 (UTC)[reply]
      - A lot of the complexity of {{convert}} could be solved by exposing a printf type facility, to do all the fussy rounding of numbers. Many layers templates are needed to mimic this functionality.--Salix (talk): 13:41, 2 February 2012 (UTC)[reply]
Since when is Lua "relatively unheard-of"? --cmelbye (^t/_c) 00:21, 1 February 2012 (UTC)[reply]
- Of course "relatively unheard-of" a subjective term. Since it's an issue that interests me, I thought I might as well do some digging:
  - On Github it ranks at #18 for number of repos and #17 for number of users, way behind JavaScript, which was also considered for this role, and PHP, the main language of MediaWiki itself (source).
  - Probably due to the fact that it is mainly known in game programming circles, it did not rank in a 2007 survey of job adverts
  - It was not included in this top-ten of programming languages by book sales
  - In this comprehensive analysis it ranks at #15 by Yahoo! searches, #21 for job adverts, #26 for book sales, and #15 for projects.
- Of course, I have used ordinals, but I could have used cardinals. I agree that maybe people have heard of it (literally speaking) but the point I was trying to make was more figurative, and relates to the fact that Lua is an unfamiliar language to virtually all wiki admins and even a considerable percentage of the "hard core" of WMF techies. - Jarry1250 ^{[Deliberation needed]} 00:48, 1 February 2012 (UTC)[reply]
As a programmer, I look forward to this. I don't know any Lua, but I've heard good things about it, and I'm sure I can pick it up. I'm also confident it will be more satisfying to work on than ParserFunctions-based templates. Superm401 - Talk 22:56, 1 February 2012 (UTC)[reply]
I am very excited about Mediawiki supporting a real programming language for extensibility. This will dramatically reduce complexity and improve readability and maintainability, and remove a lot of very annoying limitations due to template nesting limits. It'll also enable new things that heretofore were considered too complex or inefficient to undertake. Dcoetzee 03:33, 2 February 2012 (UTC)[reply]
This sounds great! I can't wait to get my hands on Lua capabilities in my Semantic templates. Jamie (blog | hello) 03:44, 2 February 2012 (UTC)[reply]
Lua is a very nice programming language, that isn't that hard to learn. I'm looking forward to being able to use it in templates, ect. If you want documentation, you can look here. I presume for output, we'll use print("Hello World"), and stuff like that. It will be interesting to see what gets removed, and what stays when they update. (FYI, people get picky when you call it 'LUA' as it's not an Acronym, or when you call it 'lua', as 'lua' is portugues for moon. Make sure you call it Lua. :D) Idofen (talk | contribs | February 2) —Preceding undated comment added 07:39, 2 February 2012 (UTC).[reply]
Lua has no native Unicode string support. How will this be addressed? 67.6.156.62 (talk) 22:28, 2 February 2012 (UTC)[reply]
- At a guess, indicates that an extra library would needed such as Selene. With such a library, it's not likely to remain a big issue, though one that should be given considerable thought, so thanks for pointing it out. - Jarry1250 ^{[Deliberation needed]} 00:30, 3 February 2012 (UTC)[reply]
- Lua can store UTF-8, UTF-16 or whatever binary "strings". The APIs may be designed so that more advanced features (like comparison) can be performed on normalized unicode strings -- typically loading these strings from unicode files. More at Lua and UTF-8 - zertyz —Preceding undated comment added 22:01, 23 February 2012 (UTC).[reply]
Is there any example comparing the MediaWiki parser function and Lua? Thx. -- Sameboat - 同舟 (talk) 13:14, 5 February 2012 (UTC)[reply]
The ability to construct complex if statements will certainly be of great value. With ParserFunctions, the only real option at present is to resort to writing code such as this which looks more like obfuscated code than anything else. Having to write code this way also means it is extremely difficult for many people to understand and modify.
Probably the thing I find the most unusual about Lua is the use of and and or instead of && and || for logical operators. I'm sure there was a reason these keywords were chosen for logical operators, but these and the end keyword still seem odd.
If we are finally going to implement a real programming language, I also hope we end up with some support for true regular expressions (and not just pattern matching). Regular expressions would simplify far too many things to count. --Tothwolf (talk) 14:02, 5 February 2012 (UTC)[reply]
- You can also have complicated datastructures. For example, instead of having a 100 input thing for a input you can use tables. (I presume this stuff will be formatted as a function). It would look like this:

 function List(...)
 for _, Text in pairs({...}) do
 print([==[
 <div>
 ]==]..Text..[==[
 </div>
 ]==]
 end
 end

Which takes in every single input , and puts it in a <div </div> . Idofen (talk | contribs | April 26) —Preceding undated comment added 02:10, 26 April 2012 (UTC).[reply]

- Looking over some of the FAQs, it looks like Lrexlib might be an option for regular expression support. --Tothwolf (talk) 14:09, 5 February 2012 (UTC)[reply]
I was wondering, will this Lua implementation include the math library? I.e. can we implement inline calls like log₁₀ or power functions? Regards, RJH (talk) 23:26, 6 February 2012 (UTC)[reply]
At last, only... four years to reach a decision? Rich Farmbrough, 03:17, 7 February 2012 (UTC).[reply]
- Four years isn't that bad...in fact, for Wikipedia that is probably a new record. At least we won't have to wait 15 years like we did for Duke Nukem Forever. --Tothwolf (talk) 22:39, 13 February 2012 (UTC)[reply]
Lua isn't "relatively unheard of". It's viral, at least in the programming and games industry. Czech is Cyrillized (talk) 14:24, 4 March 2013 (UTC)[reply]

Keep up with The Signpost on Twitter, Facebook or Mastodon.

Home

About