PYRO - Features and Guidelines

## 7. Features and Guidelines

This chapter discusses some of the more technical features of Pyro, that don't fit in elsewhere. Also it provides some guidelines you must follow when developing with Pyro.

- Nested attribute access
- Automatic rebinding
- Mobile code
- Multithreading in Pyro
- Thread Local Storage in Pyro objects
- DNS, IP addresses and firewalls
- Callbacks and Oneway calls
- Dealing with exceptions
- Remotely accessing the Daemon
- Timeouts and cleaning up unused objects
- Usage rules and guidelines

### Nested attribute access

As we've already seen in the Pyro Concepts chapter, there is a special Dynamic Proxy that allows direct remote attribute access on Pyro objects. You can use normal Python syntax for this (for instance, `print RemoteObj.name` prints the attribute 'name' of the remote object 'RemoteObj'). There are a few important points to keep in mind when you're using this:

- If attributes are objects of a certain class, that class (i.e. the module it's in) must be available on the client also. If it's not, you won't be able to access this attribute directly. You'll need to write classic getters and setters in this case.
- The above is also valid when you are using 'nested' attributes (such as `RemoteObj.person.address.street`) or 'nested' method calls (such as `RemoteObj.person.address.getStreet()`): the rule counts for each attribute.

For a better understanding why these rules have to be followed, I'll try to explain exactly what happens when accessing attributes on a Pyro proxy:

When you call `object.sub1.sub2` where `object` is a Pyro proxy to your Pyro server object, Python processes the statement left-to-right. First Python wants to get the `sub1` attribute from `object`. So Pyro intercepts a getattr call from Python and the remote Pyro object returns the `sub1` attribute. Unfortunately, this value is an object (i.e. a class instance)! The client receives a class instance of a class it doesn't know about! (that's the reason it crashes with AttributeError: 'module' object has no attribute 'SomeClass', when the class of the attribute object is not available on the client).

The easiest way to solve this is to add a delegation method to your Pyro object (something like `getSub2()` that returns `self.sub1.sub2`. Then, don't access the `object.sub1.sub2` attribute directly, but get it with `object.getSub2()`. The other solution is to make the class available to the client process: put it in a separate module and place the module somewhere the client code has access to (i.e. it must be able to import it). An explicit import statement is not necessary.

But there is a serious, confusing issue with nested attributes: you will have a local object instead of the remote object! What's going on: Say you took care of supplying all needed modules and classes to both server and client. Then you want to access `object.sub1.sub2`. What you get is the `sub2` attribute of a LOCAL sub1 object instance! This is no problem when you just want to read the value, but it is a huge problem when you want to perform some actions on this attribute, or change the value! This will be performed in your client, not in the server object!

Explanation: as pointed out above, Python searches for `sub1` in `object`. Because Pyro intercepts the getattr call, you get the remote `sub1` object. Its state is pickled on the server (by Pyro) and unpickled on the client (by Pyro), thereby recreating the remote `sub1` object on the client. Only after that, Python looks up `sub2` in `sub1`. What happens is that Python itself returns you the `sub2` attribute of the local `sub1` object! If you call methods on that object, or change it's value, that will only happen on the (temporary) local `sub2`! All changes are lost because they're not propagated to the server!

Currently the only working solution for this problem is to use extra methods on your original Pyro server object, that manipulate the `sub2` object on the server (for instance, you should call `object.setSub2(newvalue)` to update `sub2`'s value on the server, instead of `object.sub1.sub2=newvalue`).

Note that this problem only occurs when you're using nested attribute access. Manipulating the attributes of `object` itself (the Pyro proxy) is no problem, because all attribute accesses (read and write) are intercepted by Pyro and are remoted to the actual server object.

### Automatic rebinding

Pyro has an "auto rebind" feature. This means that - with a little help of yourself - your clients can recover from network errors that kill the connection with the server. More specifically, when your client detects a problem with the network (`ConnectionClosedError` or even `ProtocolError`) it can use a special 'hidden' method of the internal Pyro protocol adapter object to request it to reconnect to the server:

`yourpyroobject.adapter.rebindURI()`
  You can supply two arguments if desired, `tries` and `wait`. The first is the number of retries that is performed (default: sys.maxint), the second is the delay time between each retry in seconds (default: 1).

NOTES:

1. If your server crashes, and you want to restart it and let the clients reconnect to it, the server has to be prepared for this feature. It must not rely on any transient internal state to function correctly, because that state is lost when your server is restarted.
2. Your server must register the objects with the `connectPersistent` method of the PyroDaemon. Otherwise the reconnect feature does not work because the client tries to access an invalid URI. The `connectPersistent` method reuses the object's URI that is still known in the Name Server. It's no problem to always use `connectPersistent` instead of the regular `connect`, but there is a naming lookup overhead per call.
3. The NS does not have to be the persistent version on disk, as long as it keeps running. All that is needed is that the URI of the objects concerned stay available in the NS, so the server can reuse those URIs when it comes up again. When the NS dies, you're in deep trouble, unless it was the persistent NS, that reloads its naming database when it comes up again.
4. The client is responsible for detecting a network problem itself. It must call the 'hidden' `rebindURI` method of the object's protocol adapter itself if this is the case.
5. The method call that triggered the auto rebind is likely lost, i.e. it almost certainly has not been executed. You have to call it again explicitly if you want to be sure that it has been executed. However, it might be the case that the server received the call and that the connection was lost after the server executed the call but before returning the result. Calling the method again will result in two executions on the server. This is all usual transaction semantics stuff, and Pyro effectively gives you at-most-once semantics (methods are called zero or one times in case of failure).
6. If a daemon goes down, automatic rebinding only works if the daemon is started again on the same port number. This may be an issue if you start multiple daemons while relying on Pyro's automatic port selection feature. Start the daemons with an explicit `port` parameter to make sure they reuse the same port every time. The reason this is needed is that the rebinding of the proxy will use the PYRO:// uri that is stored in the proxy. That URI only contains the exact location of the object in a specific daemon. No new naming lookup via the name server is done.

You're probably wondering "Why isn't this transparent? Why no PYRO_AUTOREBIND config item?" The answer is: because you have to have control about it when a network problem occurs. Furthermore, only you can decide if your system needs this feature, and if your system can support this feature (see points 1 and 2 above).

About the difference between the exceptions that you can catch: You get a `ConnectionClosedError` when Pyro was able to establish a connection initially, but the connection gets interrupted later (because of network problems or something like that). You get a `ProtocolError` ('connection failed') when Pyro was unable to establish a connection at all, for instance, when the Pyro daemon on the server isn't running or reachable at the time Pyro tries to establish a connection. It is debatable if we should talk about reconnection in the second case, because there has never been a connection to begin with. But, if you want to handle this case too, just catch a `ProtocolError` instead of `ConnectionClosedError`.

Examine the example code provided in the "autoreconnect" example, and Pyro's internal adapter code, to see how the rebind feature works.

### Mobile code

Pyro supports the concept of mobile code (albeit with a few limitations). What does this mean? Imagine a Pyro object that accepts other objects as arguments in its methods. It will invoke various methods on these objects as they were passed in from the client. The client can pass any object as an argument to a remote method call. What happens on the server is that as soon as the method call arrives, Pyro needs the Python module that contains the code for the objects that were passed in. If this module is not available, you'd normally get an `ImportError`. However, Pyro intercepts this and returns a `NoModuleError`.

Unless you enable `PYRO_MOBILE_CODE`. Now, Pyro internally returns a special code, which makes the client submit the missing Python code to the server. Pyro then proceeds as if nothing was wrong, i.e. it can now load the previously missing module and call those objects! This happens automatically, you only have to enable mobile code on the server by setting the `PYRO_MOBILE_CODE` config item to 1!

There is one more thing: loading and running arbitrary code is dangerous. Pyro has codeValidators to help you protect your program; see the Security chapter for more info.

Pyro supports 2-way mobile code: your client can also receive code from the server that wasn't available before! This means the server can return objects that are unknown in the client. The module(s) on the server that define those objects will be sent to the client to make them available there, too. Just like enabling mobile code support on the server, you have to enable `PYRO_MOBILE_CODE` on the client to allow mobile code downloading from the server to the client. The codeValidator on the server checks both directions.

There are a few important limitations with the current mobile code implementation:

- Once the code is downloaded on the server, it stays there. It is not downloaded again. So there is a possible version control issue here. If the server keeps running and the client has updated its mobile object code in the meantime, the server will not notice this, and it will continue to use the previous (old) code.
- You have to define any mobile objects in a separate Python module. Any mobile objects defined in the module that runs as `__main__` will not work because of a Python peculiarity. There are no plans for including a workaround in Pyro.
- When you import other modules from your mobile object, you have to use the fully qualified module name. A package local name will not work. See the "agent3" example. You cannot import other such packages at the module level or in `__init__`, you have to import them from within the mobile object class methods.
- Mobile code does not work over a oneway method call! Any method calls that use mobile code must be normal calls. If you want to use oneway calls, first use a normal call (to transfer the agent) and then a second call (that is oneway) to start it.
- You have to use the same major Python version on all your systems where you use mobile code. For instance, mixing Python 2.4 with 2.3.x doesn't work and will cause your program to fail. This is because the bytecode format changes between major Python versions.

It is perfectly ok to put your mobile object module in a Python package. If your codeValidator allows it, your mobile object module can also import other modules that are unknown on the server (but only from within the mobile object class members, not at the module level or from `__init__`). They will be transported too (if you import them by their fully qualified name). It's easy enough to write an appropriate codevalidator. If you don't want cascaded loading, check for specific module names. If you want to allow cascaded loading, check for module name patterns, for instance allow everything that starts with "agent.". Have a look at the "agent2" example to see how this all works.

Note: if a compiled module (*.pyc or *.pyo) is available, Pyro will use that. If you have old *.pyc files around made with a different Python version, Pyro will crash (with a syntax error) because it can't recognise these compiled files and tries to compile them! Be sure to delete all *.pyc and *.pyo files if you switch Python versions.

### Multithreading in Pyro

Pyro has multithreading support in Pyro servers. It is turned on by default. This means that you can have a Pyro server that processes multiple remote object invocations in parallel. This is useful in cases where a single invocation takes a long time to complete. Without multithreading, the next invocation has to wait before your server is finished with the current one. With multithreading, each invocation runs in its own thread, and new invocations can be started while the others are still in progress.

Another case where multithreading is necessary, is when you are using callbacks in your Pyro object (or when you are calling other Pyro objects). Because this call may arrive at the same daemon, it must create a new thread to handle the new incoming call. When running in single threaded mode, Pyro will freeze because the calls are waiting on each other to be processed.

Pyro's use of threads on the server is as follows: the main daemon loop only waits for new connections. If a new connection arrives, a new thread is created that will serve this connection. The thread will process incoming calls sequentially, but that is good ofcourse since they can only arrive sequentially over the socket. The thread keeps running while the connection is still there. Be aware of this: if you ^C a program, you abort the main loop but many other threads might still be running. Pyro makes them 'daemon threads' so they will be terminated when your program exits, but it is preferred to clean up the nice way: call `daemon.shutdown()` when your program exits.

Of course, a little overhead is introduced. You can see this quite clearly when you are running the "benchmark" example in single- and multithreaded mode. Pyro will default to the multithreaded mode if your system supports it, because usually you'll need Pyro to be able to accept new invocations while others are still in progress. If you want, use the `PYRO_MULTITHREADED` config item to switch to singlethreaded mode (set it to zero). Pyro will default to single threaded mode if Python threads are not available. Switching to multithreaded mode if Python threads are not available, by setting `PYRO_MULTITHREADED` to 1, will crash Pyro. Please use `Pyro.util.supports_multithreading()` to test whether or not your system is able to use multithreading.

#### Concurrent method invocations, or perhaps not: `Pyro.core.SynchronizedObjBase`

Be aware that your Pyro objects can be accessed concurrently, i.e. method calls may occur simultaneously from different threads. It can be hard in some cases to create a program that behaves correctly with this. For instance, often a shared data structure may only be accessed by a single thread at the same time. To make it easier, there is the special `Pyro.core.SynchronizedObjBase` base class that you can use instead of the regular `Pyro.core.ObjBase`. When you use it, all (remote) method calls are automatically synchronized for you, because a thread lock object is used for your Pyro object. This has a huge negative impact on heavily multithreaded applications, but it saves you from much threading headaches. Note that other Pyro objects may still be accessed concurrently. If you share data over different Pyro objects, you still have to make sure that everything behaves in a thread-safe manner. Note that when you use this object base class, you don't have most of the threading issues mentioned above regarding further calls on proxies stored in your Pyro object, because you are running in a single thread all the time. However, the thread is a different one every call so you still have to transfer thread ownership on the proxy object!

#### Transferring proxies over remote calls

It's possible to pass proxies as parameters to remote calls. They will be pickled and unpickled in the Pyro object on the other side. Notice that the thread in which the call is executing on the remote side, is automatically taking ownership of the unpickled proxy object. That means that you don't have to worry about any threading issues (mentioned above) when you are using a proxy that you obtained via method parameters! It could be a good idea to just pass callback objects via method parameters on each call, instead of having the server store a callback object reference and the need to deal with all the gory threading details described above.

#### So when to use multithreading or not?

There are four situations where you don't want multithreading:

- Your platform/Python version doesn't support multithreading. Pyro will default to single threading mode if that is the case, so you don't have to worry.
- Your server is very much CPU bound, instead of I/O bound. Only I/O bound servers (this means: your system does a lot of I/O and not many CPU-intensive calculations) will benefit from parallel processing, unless your hardware is a multiprocessor system.
- You are sure that there is only a very small number of clients (preferably only one). If there is only one remote invocation in progress at a time, no multithreading is needed.
- You don't like the performance overhead. Think twice here, because usually the overall performance of your system will increase as the different client processes don't have to wait on each other.

All other cases will likely benefit from a multithreaded server, or even require one (callbacks/calling other Pyro objects!). Do some tests to find out what suits your needs better in your specific situation!

The "multithread" example shows what's going on, and how multithreading can help to improve performance and response times.

### Thread Local Storage in Pyro objects

Sometimes it is required to keep track of some data that should only be used by a single thread. An example could be an open database connection object, because the connection isn't thread safe, or it needs this because of transactions, or some other reason. If you don't want to re-establish the connection for every call you can store it on what is called the Thread Local Storage.

TLS is a container that is local to the current thread. Other threads have no access to that data, and you don't overwrite global data by accident when you use this kind of storage. Pyro's TLS can be accessed from within the remote methods of your Pyro objects ( it exists only when your object is invoked by Pyro, so you cannot access it from your `__init__` method for instance).

How do you gain access to the TLS? The `ObjBase` base class has a method `getLocalStorage()` that returns an instance of a storage class to put your data in. So you write something like this, to store something on it:

      	 self.getLocalStorage().something=...

The TLS object already has one predefined attribute: `caller`. This attribute contains the `Pyro.protocol.TCPConnection` object of the client proxy that is performing the current method call. You could use it to identifiy the unique client that is making the method call, but remember that it is just a single proxy that you are pinpointing. Your client application could have many proxies, even many different ones that connect to the same server. It is advised to leave the `caller` object alone, but you could do nasty things with this if you want to (such as dropping the connection by calling `close()` on it). You might also use the `caller` object to access any specific data associated with the connection, that you have stored there yourself (for instance, the authentication id placed as an attribute on it from within a custom connection validator). See the "user_passwd_auth" example. It puts it there to be able to uniquely identify the client, rather than just the calling proxy.

How do you initialize the TLS? Pyro allocates the TLS as soon as it's needed. You cannot initialize it when your objects are initialized because it doesn't exist yet at that time. Instead, you have to set a custom init function in the Daemon using the `setInitTLS(initfunc)` method of the Daemon. `initfunc` must be a callable object that takes a single argument: the TLS object to initialize.

For example, this function can make sure that several attributes are created on the TLS object that your methods assume to be there. If you would not do this, you need a check in all your methods for the existence of those attributes, creating them when not yet available, and so on.

Note: When running Pyro in singlethreaded mode the TLS is shared across every Pyro object and every method invocation! This is different than normal, when running in multithreaded mode!

If you use delegation instead of inheriting from `ObjBase`, you cannot use TLS because you don't have a means of getting to it (the getter is in `ObjBase`). This isn't bad because objects that use delegation know nothing about Pyro in the first place. (Well, actually, there is a sneaky way of getting to the TLS: use `threading.currentThread().localStorage` (it's an attribute of the current thread object). But if you do use this, be aware that your delegate object becomes dependent on the Pyro environment, and you probably chose to use delegation approach to avoid this!) Also, in a single-threaded environment, this is not possible because the `threading.currentThread()` call might not be available. In any case, when Pyro is running in single threaded mode, the current thread (or main thread) does not contain a `localStorage` attribute. To avoid any problems, it is probably best not to use it at all. Remember that you can use `self.getLocalStorage()` fine from a Pyro object that is inherited from `Pyro.core.ObjBase`, even in single threaded mode!

Examples Have a look at the 'user_passwd_auth' and 'sessions' examples to see possible usages of the TLS.

### DNS, IP addresses and firewalls

There are some issues to be aware of, depending on your network configuration.

Usually Pyro will locate its servers and objects using fixed IP numbers encoded in the Pyro URIs. This may or may not be appropriate. For instance, when a machine has an IP number that is only temporary, such as DHCP-leased IP numbers. The machine can get a new -different- IP number while Pyro is still running. URIs with the old IP number are now invalid! Therefore it is possible to tell Pyro to use symbolic DNS hostnames in URIs instead of raw IP numbers. Pyro will then use DNS to look up the actual IP number of the specified host (by name). You can enable this by setting the `PYRO_DNS_URI` config item to `1`. However note that each time Pyro has to look up a host, there is a DNS lookup delay.

If your machine has multiple IP addresses (for instance, when it has multiple network cards), you have to decide on what IP address your Pyro servers reside. When you create a Pyro Daemon, use the `host` argument to specify the hostname/ip address to bind the server on (defaults to '' - the default host). The Name Server can be started with a -n hostname argument, to specify the hostname/ip address to bind on. You can also use the `PYRO_HOST` config item.

Another issue is when you're using Pyro behind a firewall. There is one specific form of firewalling that is addressed in Pyro: simple Network Address Translating firewalls using port forwarding. Let's say you're at 192.168.0.1 (a private address) behind a NAT gateway that's 192.1.1.1. You have port forwarding on the NAT, so that Pyro requests go to the private box. However, with the way that Pyro is set up by default, the daemon will publish URIs as though they come from 192.168.0.1 -- an address unavailable to the rest of the Net. There is no way to have 192.168.0.1 publish URIs as though it were actually 192.1.1.1. But there is the extra `publishhost` parameter for the constructor of `Pyro.core.Daemon`. When constructing a Pyro Daemon, you can give it a special hostname or IP address that it should use when publishing URIs, via the `publishhost` parameter. The `host` parameter still is the "real" hostname of the machine the daemon is running on. When `publishhost` is not specified (it isn't by default) the value for `host` is taken. If that isn't specified either (it isn't by default) the hostname of the machine is queried and that is used. In our little example, `host` should be `192.168.0.1` (or just empty/omitted) and `publishhost` should be `192.1.1.1`, the address of the firewall/gateway. By the way, you can also use the `PYRO_PUBLISHHOST` config item to specify a publish hostname.

### Callbacks and Oneway calls

Callbacks are method invocations that are 'reversed'-- the server calls your client. This is useful when the server is something that publishes information, for instance, where the client cannot or doesn't want to poll for new info. Instead, the server calls a method on a callback object on the client to let it know that something happened.

Your client must publish a callback object that is a true Pyro object. In fact, for the callback part, your client must act as a true Pyro server. So you have to code your client as both a client and a server. This may require that your client must run a separate thread to handle Pyro messages (the Pyro daemon loop must be running to accept incoming calls). If your client only sits idle, and only waits for incoming callbacks, you can just run the daemon's `requestLoop` in the main thread. Have a look at the "callback" example for more details.

Be very aware of threading issues. Callbacks occur in their own thread. A callback may even occur while your client is still busy processing the previous callback. Your server should be even more prepared for callbacks. Usually you have a method that "registers" a client as a callback object. The client will call this method and has to pass a proxy to the callback object, not the object itself! (see usage rules, below).

Possible deadlock: if your objects enter a conversation, deadlock may occur easily. For instance, A calls B, B calls back to A, A calls B again... deadlock! B was still waiting for A to answer the callback, but A invokes a new method instead. Pyro cannot handle this yet. This issue might be addressed in a future Pyro version. In the meantime, please read on about possible alternatives.

Your server usually has a list of callback objects that it should call. Be very careful here: a client can unexpectedly disappear. You have to handle `ConnectionClosedError` exceptions and you must then remove the defunct callback object from the list. But you have to do this in a thread-safe way (the list must be under a thread-lock), because the server may be multithreaded! The server also has to handle any exceptions that occur in the callback object on the client. Don't trust it. Catch any exception that occurs, otherwise your server dies.
Be aware of a complex issue when your callback object raises an exception: if a callback occured in a remote method, that was called by the client, the exception might travel right back to the client if the server doesn't take precautions!

Not strictly callbacks, but when your Pyro object is calling other Pyro objects, you have to run in multithreaded mode. Because the new call may arrive at the same daemon, it must create a new thread to handle the new incoming call. When running in single threaded mode, Pyro will freeze because the calls are waiting on each other to be processed.

Please consider using the Pyro Event Service, instead of custom callback objects. It will handle all those nasty things for you, at the cost of some control. But it's very easy to use.

Special callback object `Pyro.core.CallbackObjBase`: Usually any exception that occurs in the callback object is silently transported back to the server, where it is raised again. For many callback objects, this is not exactly what you want, because usually the client is interested in an error within a callback object, and the server often doesn't care. If you use the special `Pyro.core.CallbackObjBase` as a base class for your callback objects (instead of the regular `ObjBase`), any exception that occurs in the callback object is not only sent to the server, but also raised again on the cient. You can see this work in one of the "callback" examples.

Oneway calls: These are remote method calls that do not expect any answer, so they return immediately after sending the remote call to the remote object. The call does not wait for the remote method to finish. At a lower level, it doesn't even wait for a protocol reply, so performance is much better too.

Normally, on the server side (the object that executes the oneway call), the call is executed in its own thread so that other calls can be processed in the meantime, if the oneway method takes some time to complete. This only works if multithreading is enabled. This property of oneway calls allows you, for instance, to start a computation and continue your own business while the computation runs. Later on you use another remote call to retrieve the results of the computation (or use a callback). This way, you don't have to worry about programming a multithreaded client, just use a oneway call. You can disable this by setting the PYRO_ONEWAY_THREADED config item to false (0). Oneway calls will then execute in the server's main thread. New method calls on the object will have to wait until the oneway method call has finished processing, as usual.

Note: There is no way to find out what the result of your request was - whether it succeeded or failed. No result nor any exception is ever returned. You're still quite sure that the call is performed though, because the request is sent using the regular PYRO protocol, but there is no guarantee (nor is there with regular calls by the way).

Oneway calls are very nice to have in a callback scenario. The Event Service also makes heavy use of them. Why? Your server is freed from the burden of handling exceptions that may occur in the remote method, and it doesn't block on slow or buggy clients. It just sends out the method invocations and continues on happily while the callback clients process the incoming method call.

You have to specify at runtime in your program which methods of what objects have this Oneway semantics. You do this by calling a special method on the Pyro proxy object:

    obj._setOneway(methods)

where obj is your proxy and methods is a list or tuple of method names that have to be called Oneway. It may also be a single method name. Currently there is no way to specify from within the remote object itself, or the creating process, that a method has to be called oneway. The calling party has to set this property. Ofcourse you could build some sort of inquiry method that has to be called first and that tells the caller what methods can have this property, and maybe this will become automatic in a future version, but it's not yet there.

### Dealing with exceptions

Pyro objects act just like regular Python objects. Your method calls are performed as if you're calling a regular local object. If the object raises an exception, the calling code will receive that exception and will act just as if the exception occurred in a local object.

Assume the remote Pyro object raises a `ValueError` exception. Your calling code will receive this and crash wit the same `ValueError` exception. However, the stacktrace in the traceback info is from your local code, not from the remote code. If you're just catching and processing exceptions, and don't want to deal with stacktrace/traceback info, there is no problem with this. But if you want to print the stacktrace, it is meaningless! It is a stacktrace from within the bowels of the local Pyro code! It provides no clue what piece of remote code caused the problem.

To help you with this, Pyro puts the remote stacktrace inside the exception that travels to your calling code. It is a list of text lines and can be obtained from the special attribute that is defined in `Pyro.constants.TRACEBACK_ATTRIBUTE`, but it's probably more convenient to use the utility function `Pyro.util.getPyroTraceback`. You pass the exception object and the function returns a list of lines that contain the remote traceback (if available) and the local traceback. Note that if you set `PYRO_DETAILED_TRACEBACK` to 1 on the server side, it will not be a normal traceback but a much more elaborated one including local variable values etc. An example:

    try:
            print thing.method()            # thing is a Pyro proxy
    except Exception,x:
            print ''.join(Pyro.util.getPyroTraceback(x))

This function is safe to call on all exceptions, also normal (local) exceptions. See the "simple" and "exceptions" examples. It is also possible to set a custom excepthook to automatically take care of the remote traceback printing:

        import sys
        import Pyro.util

        sys.excepthook = Pyro.util.excepthook

        #... call any method ...

### Remotely accessing the Daemon

Usually you won't care a bit, but the Pyro Daemon that is running in each Pyro server program, is exposed by a Pyro object itself. That means that you can access a limited set of functions of remote Daemons! Pyro itself uses this for the PYROLOC: direct lookup protocol that bypasses the Name Server; it queries the remote Daemon directly. Because the Daemon is always there, it has a special, fixed GUID: `Pyro.constants.INTERNAL_DAEMON_GUID`. (this is not a GUID in the true sense, because it is not unique). Here's an example to query all registered objects on a remote Daemon at 192.168.1.50, running on the default port:

    import Pyro.core
    Pyro.core.initClient()
    d=Pyro.core.PyroURI(host='192.168.1.50',objectID=Pyro.constants.INTERNAL_DAEMON_GUID).getProxy()
    print d.getRegistered()

The result is a dictionary that maps GUIDs to object names, like this: `{'c0a8013208585602469aec911dc92a20': ':Pyro.NameServer', 'c0000000011000001000000010000001': '__PYRO_Internal_Daemon'}`.

Currently there is only one other daemon method that is remotely accessible, `ResolvePYROLOC`. You will probably never call this yourself, it is used for the PYROLOC: protocol. Note: If the daemon is running in SSL mode you have to add an additional `prtcol="PYROSSL"` argument to the PyroURI constructor call above!

### Timeouts and cleaning up unused objects

If network errors occur, you often want to find out quickly. By default, Pyro may take a very long time to notice that a host is unreachable (it relies on the operating system to notice this). That's why it is possible to specify a custom timeout period. It is disabled by default because a default timeout period is very hard to decide on and the timeout logic also slightly decreases performance. But if you need, you can specify a timeout period on data transmission (sends and receives).

Because Pyro's connection protocol requires a handshake at connection time, the timeout also works for connecting to a Pyro daemon. This is nice, because evil clients now cannot eat connections indefinately by just connecting and never finishing the request. Once a connection has been established, it stays there.

How do you set the timeout?

            proxy._setTimeout(20)           # set 20-sec. timeout on proxy object (client)
            daemon.setTimeout(20)           # set 20-sec. timeout on daemon (server)

Clear it again by passing `None`. If a timeout occurs, Pyro raises a `TimeoutError` exception, which is subclassed from `ConnectionClosedException`. The connection has already been dropped by Pyro. (why is it `_setTimeout` -- with an underscore -- for the proxy? To avoid possible name clash with your own method names)

Note about NS and ES: the Name Server and Event Server have a built-in fixed timeout of 20 seconds. The connection is killed if data transmission takes longer than that, to prevent evil clients of clogging up the servers.

#### Cleaning up unused objects: reaping transients and passivate proxies

To save resources, Pyro facilitates in cleaning up unused connections or even objects. Your client may decide to release the connection if it doesn't need a proxy for some time. (remember that each proxy opens a network connection). Call `proxy._release()` to release the connection. Pyro will automatically create the connection again as soon as you start using the proxy again. An additional benefit is that the socket object associated with the proxy is destroyed, so you are able to pickle the proxy object again.

Things are a bit more complex on the server. Usually the Pyro objects you create there must stay active because it is not known when a client comes by to use them. This is especially true for objects that are registered in the Name Server, because that sort of tells the clients "this object is available there-and-there". But for transients, things are different. Remember that transients are objects created on the server, without a name, and usually a proxy is returned to the client to access these new objects. There's no way to tell when the client no longer needs the object! If it 'forgets' to call some sort of "release" method that destroys the object in the server, your server could overflow with all left-over unused transient objects. This is why you can specify an inactivity timeout for transients, by calling `daemon.setTransientsCleanupAge(timeout)` on your daemon. The argument is in seconds.
Pyro tracks the time an object has not been used, and destroys it when the timeout has expired. You may want to override the `_gotReaped()` method in the `ObjBase` to act on the destroy event (for instance, remove the object from a list).
Please be aware that if Pyro runs in single-threaded mode, the reaping of expired objects is only done when a method is invoked!
Also, cleaning up objects on the server doesn't automatically release any network connections and threads that might have been created for these objects. This is because these resources are not necessarily associated with a unique object, so they have to remain active. See also the "Bank2" example, it uses timeouts.

### Usage rules and guidelines

#### You should follow the guidelines below when using Pyro.

1. The remote class can't have a remote `__init__` method. You should use a regular initialization method that you must call explicitly after binding to the remote object. The `__init__` method will only be called on the server side when the object is created.
2. All objects that pass over the wire have to be pickleable. You cannot create a remote method like `openFile(filename)` that opens a file on the server and returns the file object, because file objects cannot be pickled. This also holds for sockets, and various other object types. As long as you don't access these objects in your client, you're OK though (there's nothing wrong with a Pyro object that has various open files or sockets as attributes -- if you don't access them from the client).
3. The remote class cannot support 'rich comparison', i.e. `object1==object2` for instance. This is because the proxy class needs to hijack the rich comparison mechanism to be able to compare two proxy classes with each other.
4. You have to choose explicitly for the special proxy that will allow direct attribute access, because it is slower than the regular proxy. Direct attribute access will not work with the regular proxy.
5. You have to use the built-in proxy for the Name Server, provided in `Pyro.naming.NameServerProxy`. When you use the Locator, you're safe.
6. All Python `.py` source files that contain the code of the objects that are used as parameters in remote method calls, must be available on the client and on the server. Otherwise the server cannot load the implementation code of an object that arrives in a remote method call. This is no longer necessary if you enable the mobile code feature. See the "agent2" example.
7. All Python `.py` source files that contain the code of the objects that are used as attributes of Pyro objects on the server, must be available on the client also. Otherwise the client cannot recreate the server object's attributes and will crash if you access these attributes. This cannot be fixed with enabling mobile code yet.
8. The class that is actually instantiated in the server should inherit from `Pyro.core.ObjBase`. You could define a new (probably empty) class in the server that inherits both from `Pyro.core.ObjBase` and the class that is made remotely available.
9. You could also use the Delegation pattern instead of subclassing from `Pyro.core.ObjBase`. This works as follows: create an object of your remote class, create a `Pyro.core.ObjBase` object, and tell the latter to use the former as delegate:

       ...
       impl = MyRemoteClass()
       obj = Pyro.core.ObjBase()
       obj.delegateTo(impl)
       ...

   and you then connect `obj` to the Daemon.
10. It is preferred to call `daemon.shutdown()` when your program exits. This will cleanly stop all threads that might still be running.
11. Note that, because Python doesn't do this automatically, you have to call the `__init__` from the base class in the `__init__` method -if you have one- of your remote class. You cannot omit this, your Pyro object will not work without the initialization:

        def __init__(self):
            Pyro.core.ObjBase.__init__(self)
            ...

12. If you create new Pyro objects on the server, and you want them to be accessed from the client, or vice versa (such as callback objects), you have to consider some things carefully. Make sure you:
    - derive the class from `Pyro.core.ObjBase`, or use the delegation approach, and call `Pyro.core.ObjBase.__init__(self)` from your own `__init__`
    - connect the new object to the Pyro Daemon (with or without a name). (This is needed to register the object as a Pyro object, and to set the daemon in the object for the next step.) It can be easily done from within the Pyro object that created the new object:

             URI = self.getDaemon().connect(object)        # no name, not in NS
             URI = self.getDaemon().connect(object,'name') # registered in NS with 'name'

    - return a proxy for the object, not the object itself. Instead of the normal `return object` that you would otherwise use, you do:

             return object.getProxy()      # regular dynamic proxy
             return object.getAttrProxy()  # dynamic proxy with attribute support

    - disconnect the object form the Pyro Daemon if you're done with it (don't just del it)

             self.getDaemon().disconnect(object)     # can also pass object's UID
             del object

13. Funny and unexpected things happen if you try to use a Pyro proxy object in various statements such as `while obj:`. Like the limitation with direct member access, this is also caused by the way the proxy object is currently implemented. It intercepts each and every method call. And testing for zero-ness or nonzero-ness or coercion are also method calls! (`__nonzero__, __coerce__` etc.)
14. Consider subclassing callback objects from `Pyro.core.CallbackObjBase` instead of the regular `ObjBase`. (see above at "callbacks")
15. If you want to use augmented assigment operators (such as +=, *=) and other special things on your Pyro object, please read the chapter of the Pytho/Language Reference, "Emulating numeric types". Also when you're using other special class methods such as `__nonzero__`. For example, to make `obj+=50` work, you need to implement the `__iadd__` method in your Pyro object and let it return the object itself (that is, a proxy for the object!) because the method returns the in-place modified object!

        def __iadd__(self,value):
            self.value+=value
            return self.getAttrProxy()

16. You have to make sure that you keep a reference to your Daemon at all time, because if the daemon is no longer referenced, it might be garbage collected (destroyed) by Python. Even if you connected Pyro objects to the daemon. This is recommended anyway because you can then cleanly terminate your Pyro application by calling `daemon.shutdown()` when it exits. Usually this is not a problem because your program creates a deamon and calls its `requestLoop`. But a situation might arise where you don't keep a reference to the daemon object, and then things break.
17. Pyro proxy objects (including the `NameServerProxy`) can be pickled, and therefore you can pass them around within your Pyro system. Thread ownership is taken by the thread that unpickled the proxy.
18. If your Pyro object calls other Pyro objects, it is strongly recommended to run the Daemon in multithreaded mode. Failing to do so will probably lock up Pyro, for instance when a "looping" request arrives.
19. Consider disconnecting objects from the daemon if you don't need them anymore; `daemon.disconnect(object)` (you can disconnect the object directly or provide its UID)
20. Be careful when using large data structures. For instance, when you are getting an element from a dictionary using attribute access on the remote object (like this: `value=remoteObject.somedict['somekey']`) Pyro will transfer the full dictionary to the caller and do the key lookup only after that. If it's a big data structure, you have a big performance hit. Consider writing getter and setter methods on the remote object instead.