How can I tail a log file in Python?
If you are on linux (as windows does not support calling select on files) you can use the subprocess module along with the select module.
import time import subprocess import select f = subprocess.Popen([tail,-F,filename], stdout=subprocess.PIPE,stderr=subprocess.PIPE) p = select.poll() p.register(f.stdout) while True: if p.poll(1): print f.stdout.readline() time.sleep(1)
This polls the output pipe for new data and prints it when it is available. Normally the
print f.stdout.readline() would be replaced with useful code.
You can use the subprocess module without the extra select module calls.
import subprocess f = subprocess.Popen([tail,-F,filename], stdout=subprocess.PIPE,stderr=subprocess.PIPE) while True: line = f.stdout.readline() print line
This will also print new lines as they are added, but it will block until the tail program is closed, probably with
Using the sh module (pip install sh):
from sh import tail # runs forever for line in tail(-f, /var/log/some_log_file.log, _iter=True): print(line)
Since sh.tail with
_iter=True is a generator, you can:
import sh tail = sh.tail(-f, /var/log/some_log_file.log, _iter=True)
Then you can getNewData with:
new_data = tail.next()
Note that if the tail buffer is empty, it will block until there is more data (from your question it is not clear what you want to do in this case).
This works if you replace -f with -F, but in Python it would be locking. Id be more interested in having a function I could call to get new data when I want it, if thats possible. – Eli
A container generator placing the tail call inside a while True loop and catching eventual I/O exceptions will have almost the same effect of -F.
def tail_F(some_file): while True: try: for line in sh.tail(-f, some_file, _iter=True): yield line except sh.ErrorReturnCode_1: yield None
If the file becomes inaccessible, the generator will return None. However it still blocks until there is new data if the file is accessible. It remains unclear for me what you want to do in this case.
Raymond Hettinger approach seems pretty good:
def tail_F(some_file): first_call = True while True: try: with open(some_file) as input: if first_call: input.seek(0, 2) first_call = False latest_data = input.read() while True: if n not in latest_data: latest_data += input.read() if n not in latest_data: yield if not os.path.isfile(some_file): break continue latest_lines = latest_data.split(n) if latest_data[-1] != n: latest_data = latest_lines[-1] else: latest_data = input.read() for line in latest_lines[:-1]: yield line + n except IOError: yield
This generator will return if the file becomes inaccessible or if there is no new data.
The second to last answer circles around to the top of the file it seems whenever it runs out of data. – Eli
I think the second will output the last ten lines whenever the tail process ends, which with
-f is whenever there is an I/O error. The
tail --follow --retry behavior is not far from this for most cases I can think of in unix-like environments.
Perhaps if you update your question to explain what is your real goal (the reason why you want to mimic tail –retry), you will get a better answer.
The last answer does not actually follow the tail and merely reads whats available at run time. – Eli
Of course, tail will display the last 10 lines by default… You can position the file pointer at the end of the file using file.seek, I will left a proper implementation as an exercise to the reader.
IMHO the file.read() approach is far more elegant than a subprocess based solution.
How can I tail a log file in Python?
The only portable way to
tail -f a file appears to be, in fact, to read from it and retry (after a
sleep) if the
read returns 0. The
tail utilities on various platforms use platform-specific tricks (e.g.
kqueue on BSD) to efficiently tail a file forever without needing
Therefore, implementing a good
tail -f purely in Python is probably not a good idea, since you would have to use the least-common-denominator implementation (without resorting to platform-specific hacks). Using a simple
subprocess to open
tail -f and iterating through the lines in a separate thread, you can easily implement a non-blocking
tail operation in Python.
import threading, Queue, subprocess tailq = Queue.Queue(maxsize=10) # buffer at most 100 lines def tail_forever(fn): p = subprocess.Popen([tail, -f, fn], stdout=subprocess.PIPE) while 1: line = p.stdout.readline() tailq.put(line) if not line: break threading.Thread(target=tail_forever, args=(fn,)).start() print tailq.get() # blocks print tailq.get_nowait() # throws Queue.Empty if there are no lines to read