TAGS :Viewed: 7 - Published at: a few seconds ago

[ How to implement a persistent counter in Python ]

Is there a way to have a "last known" counter in Python that will survive after server reboot?

I have a python script that is launched on schedule (by cron). It read files with document IDs in predefined format. I need to remember last processed document ID in order to ignore all the previous ones if there are any.

Answer 1

Any value you want to survive a reboot needs to be put into persistent storage - i.e. the disk. This means a file of some sort, whether that's a simple plaintext file or a database file is up to you. You've indicated in the comments you don't feel this is "cross-platform", but it would be a strange platform indeed that didn't have some sort of filesystem support.

If you need structured storage then SQLite support is built-in with Python's sqlite3 module. However, it sounds like you just need to store a single ID so a simple file will suffice. I suggest something like this:

import os

DATA_FILENAME = os.path.expanduser("~/document-counter.txt")

def update_document_id(new_id):
    with open(DATA_FILENAME, "w") as fd:
        fd.write(new_id + "\n")

def retrieve_document_id():
    with open(DATA_FILENAME, "r") as fd:
        return fd.readline().strip()

You should probably do better error checking (e.g. catch the exception that's thrown if the file doesn't exist, etc.) but this gives you the idea of how simple the solution can be. You're better off catching exceptions (EAFP is often considered more Pythonic than LBYL) but if you want to explicitly check for, say, file existence then that's easy in a portable manner too:

if not os.path.exists(DATA_FILENAME):
    print "No file found. Deal with it."

If you need to add more data fields later, I suggest SQLite - it's convenient, robust and allows you to interoperate with applications in other languages should that be required in the future. Plus you can use the standalone SQLite command-line client to manipulate your data if you need to. You just provide a filename to the sqlite3.connect() method so it's essentially as easy as opening a file, it's just you can then throw SQL at it.

However, for a simple single ID I'd just stick to a plaintext file - you really don't get much more compatible than that.

Answer 2

Store it in a database or a file. You might choose to write it to a file; you might choose to use ConfigParser or csv; or you might choose to use shelve (it's nice and simple to use; if this is all you want persistent storage for, it might be the most straightforward). There are other options too; take a look at what the standard library offers to get a feel for it. You will still need to specify the file you wish to store the results in, and permissions will need to be appropriate, but that shouldn't be difficult.