Talk:Pipeline (Unix)

Pipemill?

I have never, in 35 years of working with virtually every major variant of UNIX from 5th Edition onwards, encountered the term "pipemill" before. This section should be removed or reworded, or the etymology of the term should be provided. -- Resuna (talk) 19:23, 1 April 2014 (UTC)

removed suggestion that pipes were borrowed

I removed the suggestion that pipes in Unix were borred from MS-DOS - that history was backwards.

limit to pipelines?

Is there a limit to the number of commands you can pipe together depending on the shell? -- Joseph Lorenzo Hall 23:42, 22 February 2006 (UTC)

Joseph, wikipedia is not a tech support forum. Anyway the article is about the conceptual use of pipes in UNIX ... covering both the use of pipes in shells and their use inside lower level programming languages (like C). Any limits would be implementation details of the OS and/or the specific shell. For example the OS (version of UNIX) defines a maximum command line length ... which is the longest "vector" that can be passed to the execve() system call can accept. Any particular shell may implement a different maximum command line length which may be (and generally is) larger than the kernel's maximum execve() vector limit. That's because a command line entered to the shell may map into a number of separate execve() system call vectors --- and many commands within any command line may not result in any system calls (built-ins within the same shell process) or may involve system calls other than execve() (for example `echo $foo` ... often also supported as $(echo $foo) would involve a fork() system call --- there is a subshell --- but might not entail any execve() ... when echo is a built-in). 158.140.1.25 23:44, 9 May 2007 (UTC)

There tend to be lots of limits, and they do indeed vary dramatically in type and quantity from one environment to another. Another common limit is the number of simultaneous processes a user can own (a pipeline stage is usually at least one process). As noted, the specific limits aren't germane to this article, but some discussion of the common types of limits might be. RossPatterson 01:33, 10 May 2007 (UTC)

removed Mac os X in history

Mac os X is an unix so there is no point of telling it has borrowed the "pipe" concept, it was part of the package. Actually Linux is not Unix, so you may add Linux in this field

It would be interesting to add something about C/PM and to see if someone can talk about graphical pipe à la smalltalk

Everything is a File Link

There does not seem to be a page for the link "everything is a file" under the Network Pipes section.

Afaik. the pipe symbol used to be a caret ^ on some older systems instead of the vertical line. Maybe someone has more informations about that fact? The description of the "tee" command (for fitting together two pipes) sounds mighty wrong to me, it's more exactly the opposite (or my knowledge of the English language fails there) --Volker

I noted the bad tee(1) description (where did the author get that from?) and fixed it. JöG (talk) 10:32, 21 July 2009 (UTC)

There were several notations for pipes, or "hoses". It was around 4th Edition when '|' was settled on, so I don't think the earlier syntaxes were seen much outside the labs. [1] 70.225.161.132 (talk) 11:09, 14 April 2011 (UTC)

WWIW, I distinctly remember that on the early Sun workstations the University of Nijmegen were using in the mid-80s, which were running some BSD, both | and ^ were supported pipe symbols, ^ being the older, deprecated one. Not all terminals had a | symbol back then. Rp (talk) 22:41, 28 September 2011 (UTC)

Fair use rationale for Image:Automator Icon.png

Image:Automator Icon.png is being used on this article. I notice the image page specifies that the image is being used under fair use but there is no explanation or rationale as to why its use in Wikipedia articles constitutes fair use. In addition to the boilerplate fair use template, you must also write out on the image description page a specific explanation or rationale for why using this image in each article is consistent with fair use.

Please go to the image description page and edit it to include a fair use rationale.

If there is other other fair use media, consider checking that you have specified the fair use rationale on the other images used on this page. Note that any fair use images uploaded after 4 May, 2006, and lacking such an explanation will be deleted one week after they have been uploaded, as described on criteria for speedy deletion. If you have any questions please ask them at the Media copyright questions page. Thank you.BetacommandBot 21:56, 31 May 2007 (UTC)

pipes = awesome —Preceding unsigned comment added by ReluctantPhilosopher (talk • contribs) 09:35, 21 October 2007 (UTC)

A less(1) example

I added a simple ls | less example before the complicated spellcheck-an-URL example. Honestly, I found all of the existing examples fairly daunting – and I use complex pipelines daily. If anyone feels like adding examples showing off other aspects of pipelines, I suggest (a) pipelines involving infinite data streams like yes(1) or tail -f; (b) pipelines which parallelizes work (e.g. a CPU-bound data producer feeding into gzip); (c) pipelines across networks (e.g. foo | ssh somewhere bar -o baz). JöG (talk) 22:18, 18 March 2008 (UTC)

I agree a simpler example was desperately needed; it could be even simpler if the more common (excuse the pun) more was used instead of less—I agree less is more useful, but more is more widespread! Particularly for those used to a standard Unix environment rather than a Linux or GNU one. MeekMark (talk) 03:42, 19 March 2008 (UTC)

I chose less because I feel it's widely spread these days (people either work on free Unixes, or on systems where much of the free goodies are installed) and because it is more useful in pipelines (last time I used more, you could not scroll up when reading from stdin). And by the way, the article seems to have improved in general since late March. Thanks! JöG (talk) 19:42, 16 April 2008 (UTC)

Newlines in filenames

I would like to point out that the first example under Example of Pipemill will not work properly for any filename containing a newline ('\n'). I didn't want to alter the article due to the simplicity of the example, but this is worth mentioning for correctness. Use of find ... -print0 | xargs -0 ... handles any filename. DRA (talk) 18:44, 5 November 2008 (UTC)

Complex example fails

Resolved

 – Revision 413409654

The complex example, which is very nice, does not work, because the file /usr/share/dict/words is not sorted properly (check "sort -c /usr/share/dict/words"). I don't know how to fix the example. —Preceding unsigned comment added by 94.208.225.204 (talk) 13:21, 16 January 2009 (UTC)

This was fixed in Revision 413409654 by 174.113.180.35, using <(sort /usr/share/dict/words). Note that the example was removed in Revision 471825278 and I re-added it in 1016504355 with section heading "Example" instead of "Complex example", because the simple example was removed in Revision 474808569 by Ormaaj. (Aside, regarding the simple example: It was originally added in Revision 199194488 by Jgrahn. Ormaaj claimed it was wrong in multiple ways but didn't elaborate. Since then a similar example was added to the introduction.) Hddqsb (talk) 16:51, 7 April 2021 (UTC)

Add info about bash pipefail

Please add information about the bash pipefail, because it is important for this article, but my knowledge of English do not allow this. --Важнов Алексей Геннадьевич (talk) 04:31, 28 April 2011 (UTC)

"Pipeline"?

I have been using Unix for 25 years, and I have only seen the name "pipe", never "pipeline". What is the justification for this article being called "Pipeline"? Rp (talk) 07:41, 28 April 2011 (UTC)

Technically, A pipeline is part of the shell grammar, and the pipe itself is an (optional) operator that's part of it's syntax. This usage of the term is consistent amongst the documentation of most bourne-derived shells that I'm aware of, as well as POSIX. One error here is that the pipeline and syntax as described in the article is a feature of the command shell, not the operating system. Though most shells use pipe(2) to set up communication between the subshells, that isn't technically a requirement AFAICT. Honestly, this article is extremely bad and needs a complete rewrite. Unfortunately, I don't have time. See the spec: http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_09_02 Ormaaj (talk) 12:18, 24 October 2011 (UTC)

I notice that the Bash shell's manpage confirms this usage. Rp (talk) 15:39, 31 October 2011 (UTC)

parallel

Hi. What about GNU Parallel ? Should be described her ? --Adam majewski (talk) 06:06, 25 February 2012 (UTC)

History

This contradicts what is currently in the History section of the article, but take a look at Brian Kernighan and Ken Thompson. A nerdy delight for any Vintage Computer Fest 2019 attendee: Kernighan interviewing Thompson about Unix. YouTube. Event occurs at 30m50s to 35m15s. Retrieved September 3, 2019. It seems as if this should be covered in the article. Wtmitchell (talk) (earlier Boracay Bill) 15:51, 3 September 2019 (UTC)

At 32:42, Thompson claims he invented them himself. The referenced document says "The basic redirectability of input-output made it easy to put pipes in when Doug McIlroy finally persuaded Ken Thompson to do it.". I'm not sure it's a direct contradiction - often, ideas float around for a while before they crystallize, and Thompson does mention that interconnecting programs in a "mesh" had been a goal for quite a while, but he didn't know how to do it. So perhaps McIlroy stated the goal, and Thompson suddenly saw how to implement it. Or perhaps McIlroy saw the implementation, but it didn't click with Thompson out of reluctance (he mentions concern about queues blowing up), until he actually did it and realized the benefits. Rp (talk) 09:29, 8 September 2019 (UTC)

I don't claim to be an expert about this, you likely have a better handle on it than I. I just happened to run across that video and thought it interesting and perhaps worthy of WP:DUE coverage in the article. I don't know enough about the details to take a stance about that and just thought to mention it here. Wtmitchell (talk) (earlier Boracay Bill) 10:38, 8 September 2019 (UTC)

I believe Rp's last guess is essentially right. Mdmi (talk) 23:16, 18 February 2022 (UTC)

Dead links to cm.bell-labs.com

There are two dead links with reference number 2 and 10! I suggest to replace them with: https://www.bell-labs.com/usr/dmr/www/mdmpipe.html https://www.bell-labs.com/usr/dmr/www/hist.html