Augoust

Outline of computer programming

You have done a very nice job here. The Transhumanist 12:17, 17 September 2015 (UTC)

Thanks! It looked like it needed attention. (It still does, of course.) And, I've noticed all the work you've put in over the years.

I have a question about your reworking of the lead. You turned it into a term–definition format, but most featured lists I've seen don't do that. For example: List of Sites of Special Scientific Interest in Greater London, List of freshwater islands in Scotland, List of current sovereign monarchs, and List of computer criminals. The examples from WP:LEADFORALIST fit the same pattern. It would be nice to have the list promoted to featured status at some point. Is there a practical reason you avoid a prosaic style? Augoust (talk) 14:15, 17 September 2015 (UTC)

Outlines are strange birds, and make up a department of their own. Standard lists include items (like dog breeds, shark species, scientists, movies, bike models, etc.), while outlines include all the topics that belong to a subject (and may include item lists in them). Outlines are one of the two types of general topics lists on Wikipedia (the other type being indexes).

While their topics are content, outlines are also part of Wikipedia's navigation system, like categories are. Categories are classified indexes, while outlines serve as subject-based tables of contents (each topic listed is also an article, and links thereto). So they aren't generally considered featured status worthy. Have you ever seen a featured category?

The Featured List department wasn't designed to handle outlines. Outlines have their own style guidelines. They are a type of tree structure, which is a nodal format, as opposed to a prose (sentence-paragraph) format (like articles, or the leads in standard lists). If outlines were to acquire featured status, they would need a featured department of their own, similar to the way portals have featured portals.

Also, the word "outline" has a very specific meaning. It is short for "hierarchical outline", and is the name of a textual tree structure format. Documents that are in prose format are by definition not hierarchical outlines.

Which brings us to the real power behind outlines. They are a taxonomical data format (computer memories). If they retain their nodal nature (tree trunk, branches, leaves), then they can be easily parsed by computer algorithm. One problem with Wikipedia's navigation systems is that they are woefully incomplete and they fall out of date very quickly. There just isn't enough manpower available to build and maintain them. There is great motivation to automate the construction and maintenance of these structures.

The mathematics department keeps long alphabetical indexes maintained using bots. I am looking into doing something similar for outlines using automatic taxonomy construction. It is a subfield of natural language processing, a branch of artificial intelligence.

I hope my explanation helps. The Transhumanist 05:52, 18 September 2015 (UTC)

Standard lists include items … while outlines include all the topics that belong to a subject … They are a type of tree structure, … They are a taxonomical data format

Right. This much I do know. It was the guiding principle in my edits.

The Featured List department wasn't designed to handle outlines. Outlines have their own style guidelines.

I don't see what about them intrinsically requires different lead styles nor such radically criteria for featured status (or any other type of formally reviewed status), but I will defer to our experience and judgement.

There just isn't enough manpower available to build and maintain them. There is great motivation to automate the construction and maintenance of these structures.

It's very unfortunate, but not surprising. Wikipedia is very large, covers nearly every topic possible, and is primarily a volunteer project. Even a fully manned team of tech experts with nothing else to do would have a hard time. I agree it would be great if a way to automate outline maintenance was devised.

Thank you very much for your answer. Augoust (talk) 14:07, 18 September 2015 (UTC)

Let me see if I can clarify the situation...

The scope of Wikipedia is knowledge (the most general topic there is). The scope of the outline department is knowledge, and Wikipedia's coverage of it (dual purpose). Because they both cover the same subjects, their scope matches exactly. Therein lies the main problem...

The outline department was at war for years defending against the WP:CFORK and redundancy arguments. I've lost track of the number of AFD noms, merge proposals, and threats of those that outlines have received. Let's not go there again.

Regular articles and outines are both root articles. They share exactly the same subjects in their titles (though outlines do tend to be more comprehensive than their prose counterparts, because it is more useful not to split them as much, and because list builders can be insanely good at gathering topics for their structured lists -- see Outline of Buddhism and Outline of chess for examples). If outlines and regular articles share the same format, there is nothing to differentiate them. Many articles include prose and embedded lists. If you do the same with outlines, then how do you tell them apart? Beware of WP:CFORK.

The saving grace of outlines is that they are navigation aids, like categories, which are allowed to overlap in scope with regular articles. But they can't be regular articles.

Content drift is a problem if the leads are not actually part of the outline. Outline is the name of a format. Prose format is not an outline format. A prose article or even partially prose article is not an outline. Having improperly structured prose in there renders the title a misnomer. If you want outlines to be taken seriously by the (WP:CFORK) rule hardliners (and in the outliner and NLP community by the state-of-the-art experts), then what you claim is an outline needs to actually be one.

However, the main danger for confusion is at the top of the page. When we first started adding leads for topic identification convenience, we used prose format. The lead often filled the screen. So upon first inspection, the reader could see no difference between the article on a subject and the outline on that subject, except for the article title. We had to explain many times that the lead was there merely for topic identification, and asked them to scroll further down the screen to see how the articles differed, but got tired of that and reformatted the leads for visual differentiation. That's the initial reason why the lead sentence explicitly states that the article is an outline, and the article's subject is presented as the top list entry (though without the bullet).

Featured list criteria is rather stringent, and based on different assumptions.

Outlines do not follow general MOS guidelines for lists and headings, among other things. They'll tear these things apart at FLC. For example, outlines are trees, and have branches which themselves are trees. Because the subtrees are often linked to by sectional redirects, to avoid confusion when users land there, the headings include the entire topic name in case they leave their computer momentarily or forget what they are looking at (see Branches of natural science, for example).

Outlines are not generally stable, because they need continuous updating, and because they fall out of date so quickly. This is especially so with technological subjects like computer programming. Almost all outlines have a history section, for example, which may fall out of date within weeks.

"Does not violate the content-forking guideline" is often interpreted as "does not share the same scope as a regular article" (but they do).

Hunting down references to prove topics meet inclusion criteria (belong to a general subject) is not a practical use of an editor's time.

Somewhere along the way, the realization sunk in that these things (outlines) were almost ontological, and that they had huge potential for use in the NLP field (which makes extensive use of Wikipedia as a text corpus by the way).

The term-definition format is explicit. Prose lead format is vague/ambiguous. Auto-maintenance, and even semi-automated maintenance is much much easier on explicit. Outlining knowledge by hand would take as many editors as Wikipedia has now, but would proceed at the same slow pace that Wikipedia currently grows, and thus would never catch up to regular article development.

An actual outline format can be parsed. A parsable outline can be loaded into an outliner (a specialized editor designed to edit outlines), without garbling the tree or losing data. A parsable outline can also be converted into other outline or outliner formats via computer program.

Basically, outlines have evolved and will continue to evolve as we try to automate them. They are a different species than other lists.

I hope this explanation has helped. The Transhumanist 17:56, 18 September 2015 (UTC)

By the way...

Please tell me about your programming skills. The Transhumanist 06:13, 18 September 2015 (UTC)

I have around a decade's experience in programming, no specific focus. I guess "jack of all trades, maser of none" would be applicable here. I primarily use C and Python. I get a lot of use out of things like sed and AWK. I also know Lua, C++, and Haskell. I have experience (but am very rusty) with Fortran, Basic, Java, Ada, Perl, Ruby, IA-32 Assebly, and a few others. Augoust (talk) 14:29, 18 September 2015 (UTC).

Nice. Python is used extensively in NLP. Maybe you can point me in the right direction there. And you are obviously proficient with regex. I'm currently using Perl 5, building regex-heavy scripts for automatic annotation and converting incidental outline formats (things that people don't generally identify as outlines). I haven't started applying POS taggers, semantic analysis, and the like yet -- probably a few months away before I dive into this corpus linguistics stuff. The Transhumanist 18:25, 18 September 2015 (UTC)

My go-to answer for most "where do I start with [programming topic x]?" questions is the relevant O'Reilly book. In this case, that's Bird, Klein, and Loper's Natural Language Processing with Python. You can find it at Amazon and nltk.org. Once you're beyond the level of a book like that, you'll learn more from language agnostic NLP books and the documentation of whatever libraries you find yourself using.

Yea, regex proficiency used to be a requirement. I'm also fond of it. It's very expressively efficient, as is the case with may domain specific languages.

Perhaps, I can help with those scripts. I'm pretty busy, so no promises, but I like seeing repetitive and otherwise predictable tasks automated. (They should.) Augoust (talk) 01:38, 19 September 2015 (UTC)

Understood: as time and circumstances allow. I appreciate your offer, and may take you up on it soon. Thank you. The Transhumanist 14:27, 21 September 2015 (UTC)

Programming assistance request: redlink checking

Dear Augoust,

Here's a problem I definitely need help on...

One thing I keep running into is a need for a method using perl to check for the red status of a link.

For example, "given a list x of links, insert those that are not red on page y". Or, "given list z of links, strip out the redlinks. To execute the desired actions, the script first must determine whether each link is red or not.

I look forward to your reply. The Transhumanist 17:56, 6 November 2015 (UTC)