Internal tests
==============

This document contains a number of internal tests of the
synchronization facility.

To start
--------

Let's first grok this package::

  >>> import grokcore.component.testing
  >>> grokcore.component.testing.grok('z3c.vcsync')

Serialization
-------------

In order to export content to a version control system, it first needs
to be possible to serialize a content object to a text representation.

For the purposes of this document, we have defined a simple item that
just carries an integer payload attribute::

  >>> class Item(object):
  ...   def __init__(self, payload):
  ...     self.payload = payload
  >>> item = Item(payload=1)
  >>> item.payload
  1

We will use an ISerializer adapter to serialize it to a file. Let's
define the adapter::

  >>> import grokcore.component as grok
  >>> from z3c.vcsync.interfaces import ISerializer
  >>> class ItemSerializer(grok.Adapter):
  ...     grok.provides(ISerializer)
  ...     grok.context(Item)
  ...     def serialize(self, f):
  ...         f.write(str(self.context.payload))
  ...         f.write('\n')
  ...     def name(self):
  ...         return self.context.__name__ + '.test'

Let's test our adapter::

  >>> from StringIO import StringIO
  >>> f= StringIO()
  >>> ItemSerializer(item).serialize(f)
  >>> f.getvalue()
  '1\n'

Let's register the adapter::

  >>> grok.testing.grok_component('ItemSerializer', ItemSerializer)
  True

We can now use the adapter::

  >>> f = StringIO()
  >>> ISerializer(item).serialize(f)
  >>> f.getvalue()
  '1\n'

Export persistent state to version control system checkout
----------------------------------------------------------

We have a object structure consisting of a container with some
items and sub-containers in it::

  >>> from z3c.vcsync.tests import Container
  >>> root = Container()
  >>> root.__name__ = 'root'
  >>> root['data'] = data = Container()
  >>> data['foo'] = Item(payload=1)
  >>> data['bar'] = Item(payload=2)
  >>> data['sub'] = Container()
  >>> data['sub']['qux'] = Item(payload=3)

This object structure has some test payload data::

  >>> data['foo'].payload
  1
  >>> data['sub']['qux'].payload
  3

We have a checkout in testpath on the filesystem::

  >>> from z3c.vcsync.tests import create_test_dir, TestCheckout
  >>> testpath = create_test_dir()
  >>> checkout = TestCheckout(testpath)

We also have a test state representing the object data::

  >>> from z3c.vcsync.tests import TestAllState
  >>> state = TestAllState(root)

The test state will always return a list of all objects. We pass in
``None`` for the revision_nr here, as the ``TestAllState`` ignores this
information anyway::

  >>> sorted([obj.__name__ for obj in state.objects(None)])
  ['bar', 'data', 'foo', 'qux', 'root', 'sub']

Now let's synchronize. For this, we need a synchronizer initialized
with the checkout and the state::
  
  >>> from z3c.vcsync import Synchronizer
  >>> s = Synchronizer(checkout, state)

We now save the state into that checkout. We are passing ``None`` for
the revision_nr for the time being::

  >>> s.save(None)

The filesystem should now contain the right objects. Everything is
saved in a directory called ``data``:
 
  >>> root_dir = testpath
  >>> data_dir = root_dir.join('data')
  >>> root_dir.check(dir=True)
  True

This root directory should contain the right objects::

  >>> sorted([entry.basename for entry in root_dir.listdir()])
  ['data']

The ``data`` subdirectory should contain the right sub-objects::

  >>> sorted([entry.basename for entry in data_dir.listdir()])
  ['bar.test', 'foo.test', 'sub']

We expect the right contents in ``bar.test`` and ``foo.test``::

  >>> data_dir.join('bar.test').read()
  '2\n'
  >>> data_dir.join('foo.test').read()
  '1\n'

``sub`` is a container so should be represented as a directory::

  >>> sub_dir = data_dir.join('sub')
  >>> sub_dir.check(dir=True)
  True

  >>> sorted([entry.basename for entry in sub_dir.listdir()])
  ['qux.test']

  >>> sub_dir.join('qux.test').read()
  '3\n'

Modifying an existing checkout
------------------------------

We will now change some data in the ZODB again.

Let's add ``hoi``::
  
  >>> data['hoi'] = Item(payload=4)

And let's delete ``bar``::

  >>> del data['bar']

Since we are removing something, we need inform the state about it. We
do this manually here, though in a real application typically you
would subscribe to the ``IObjectRemovedEvent``.

  >>> removed_paths = ['/data/bar']
  >>> state.removed_paths = removed_paths

The added object always will return with ``objects``, but in your
application you may also need to let the state know.
 
Let's save the object structure again to the same checkout::
 
  >>> s.save(None)

We expect the ``hoi.test`` file to be added::

  >>> data_dir.join('hoi.test').read()
  '4\n'

We also expect the ``bar.test`` file to be removed::

  >>> data_dir.join('bar.test').check()
  False

Modifying an existing checkout, some edge cases
-----------------------------------------------

The ZODB has changed again.  Item 'hoi' has changed from an item into
a container::

  >>> del data['hoi']
  >>> data['hoi'] = Container()

Let's create a new removed list. The item 'hoi' was removed before it
was removed with a new container with the same name, so we have to
remember this::

  >>> removed_paths = ['/data/hoi']
  >>> state.removed_paths = removed_paths

We put some things into the new container::

  >>> data['hoi']['something'] = Item(payload=15)

We export again into the existing checkout (which still has 'hoi' as a
file)::

  >>> s.save(None)

Let's check the filesystem state::

  >>> sorted([entry.basename for entry in data_dir.listdir()])
  ['foo.test', 'hoi', 'sub']

We expect ``hoi`` to contain ``something.test``::

  >>> hoi_path = data_dir.join('hoi')
  >>> something_path = hoi_path.join('something.test')
  >>> something_path.read()
  '15\n'

Let's now change the ZODB again and change the ``hoi`` container back
into a file::

  >>> del data['hoi']
  >>> data['hoi'] = Item(payload=16)
  >>> s.save(None)

This means we need to mark the path to the container to be removed::

  >>> removed_paths = ['/data/hoi']
  >>> state.removed_paths = removed_paths

We expect to see a ``hoi.test`` but no ``hoi`` directory anymore::

  >>> sorted([entry.basename for entry in data_dir.listdir()])
  ['foo.test', 'hoi.test', 'sub']

Note: creating a container with the name ``hoi.test`` (using the
``.test`` postfix) will lead to trouble now, as we already have a file
``hoi.test``. ``svn`` doesn't allow a single-step replace of a file
with a directory - as expressed earlier, an ``svn up`` would need to
be issued first, but this would be too early in the process. Solving
this problem is quite involved. Instead, we require the application to
avoid creating any directories with a postfix in use by items. The
following should be forbidden::

  data['hoi.test'] = Container() # XXX forbidden

multiple object types
---------------------

We will now introduce a second object type::

  >>> class OtherItem(object):
  ...   def __init__(self, payload):
  ...     self.payload = payload

We will need an ``ISerializer`` adapter for ``OtherItem`` too::

  >>> class OtherItemSerializer(grok.Adapter):
  ...     grok.provides(ISerializer)
  ...     grok.context(OtherItem)
  ...     def serialize(self, f):
  ...         f.write(str(self.context.payload))
  ...         f.write('\n')
  ...     def name(self):
  ...         return self.context.__name__ + '.other'
  >>> grok.testing.grok_component('OtherItemSerializer', OtherItemSerializer)
  True

Note that the extension we serialize to is ``.other``.

Let's now change the ``hoi`` object into an ``OtherItem``. First we remove
the original ``hoi``::

  >>> del data['hoi']

We need to mark this removal in our ``removed_paths`` list::

  >>> state.removed_paths = ['/data/hoi']

We then introduce the new ``hoi``::

  >>> data['hoi'] = OtherItem(23)

Let's serialize::

  >>> s.save(None)

We expect to see a ``hoi.other`` item now::

  >>> sorted([entry.basename for entry in data_dir.listdir()])
  ['foo.test', 'hoi.other', 'sub']

Let's change the object back again::

  >>> del data['hoi']
  >>> state.removed_paths = ['/data/hoi']
  >>> data['hoi'] = Item(payload=16)
  >>> s.save(None)

We expect to see a ``hoi.test`` item again::

  >>> sorted([entry.basename for entry in data_dir.listdir()])
  ['foo.test', 'hoi.test', 'sub']

loading a checkout state into python objects
--------------------------------------------

Let's load the current filesystem layout into python
objects. Factories are registered as utilities for the different
things we can encounter on the filesystem. Let's look at items
first. A ``IParser`` utility is registered for the ``.test``
extension::

  >>> from z3c.vcsync.interfaces import IParser
  >>> class ItemParser(grok.GlobalUtility):
  ...   grok.provides(IParser)
  ...   grok.name('.test')
  ...   def __call__(self, object, path):
  ...      object.payload = int(path.read())
  >>> grok.testing.grok_component('ItemParser', ItemParser)
  True
 
To have the ability to create new objects, a factory is registered for
the ``.test`` extension as well, implemented in terms of ``ItemParser``::

  >>> from z3c.vcsync.interfaces import IFactory
  >>> from zope import component
  >>> class ItemFactory(grok.GlobalUtility):
  ...   grok.provides(IFactory)
  ...   grok.name('.test')
  ...   def __call__(self, path):
  ...       parser = component.getUtility(IParser, '.test')
  ...       item = Item(None) # dummy payload
  ...       parser(item, path)
  ...       return item
  >>> grok.testing.grok_component('ItemFactory', ItemFactory)
  True

Now for containers. They are registered for an empty extension::

  >>> class ContainerParser(grok.GlobalUtility):
  ...   grok.provides(IParser)
  ...   def __call__(self, object, path):
  ...       pass # do nothing with existing containers
  >>> grok.testing.grok_component('ContainerParser', ContainerParser)
  True

We implement ``ContainerFactory`` in terms of
``ContainerParser``. Note that because ``ContainerParser`` doesn't
actually do something in this example, we could optimize it and remove
the use of ``IParser`` here, but we won't do this for consistency::

  >>> class ContainerFactory(grok.GlobalUtility):
  ...   grok.provides(IFactory)
  ...   def __call__(self, path):
  ...       parser = component.getUtility(IParser, '')
  ...       container = Container()
  ...       parser(container, path)
  ...       return container
  >>> grok.testing.grok_component('ContainerFactory', ContainerFactory)
  True

We need to maintain a list of everything modified or added, and a list
of everything deleted by the update operation. Normally this
information is extracted from the version control system, but for the
purposes of this test we maintain it manually. In this case,
everything is added so appears in the files list::

  >>> checkout._files = [data_dir, data_dir.join('foo.test'), 
  ...   data_dir.join('hoi.test'), data_dir.join('sub'), 
  ...   data_dir.join('sub', 'qux.test')]

Nothing was removed::

  >>> checkout._removed = []

Let's load up the contents from the filesystem now, into a new container::

  >>> container2 = Container()
  >>> container2.__name__ = 'root'

In order to load into a different container, we need to set up a new
synchronizer with a new state::

  >>> s = Synchronizer(checkout, TestAllState(container2))

We can now do the loading::

  >>> dummy = s.load(None)

We expect the proper objects to be in the new container::

  >>> sorted(container2.keys())
  ['data']
  >>> data2 = container2['data']
  >>> sorted(data2.keys())
  ['foo', 'hoi', 'sub']

We check whether the items contains the right information::

  >>> isinstance(data2['foo'], Item)
  True
  >>> data2['foo'].payload
  1
  >>> isinstance(data2['hoi'], Item)
  True
  >>> data2['hoi'].payload
  16
  >>> isinstance(data2['sub'], Container)
  True
  >>> sorted(data2['sub'].keys())
  ['qux']
  >>> data2['sub']['qux'].payload
  3

version control changes a file
------------------------------

Now we synchronize our checkout by synchronizing the checkout with the
central coordinating server (or shared branch in case of a distributed
version control system). We do a ``checkout.up()`` that causes the
text in a file to be modified.

The special checkout class we use for example purposes will call
``update_function`` during an update. This function should then
simulate what might happen during a version control system ``update``
operation. Let's define one here that modifies text in a file::

  >>> hoi_path = data_dir.join('hoi.test')
  >>> def update_function():
  ...    hoi_path.write('200\n')
  >>> checkout.update_function = update_function

Now let's do an update::

  >>> checkout.up()

We maintain the lists of things changed::

  >>> checkout._files = [hoi_path]
  >>> checkout._removed = []

We will reload the checkout into Python objects::

  >>> dummy = s.load(None)
 
We expect the ``hoi`` object to be modified::

  >>> data2['hoi'].payload
  200

version control adds a file
---------------------------

We update our checkout again and cause a file to be added::

  >>> hallo = data_dir.join('hallo.test').ensure()
  >>> def update_function():
  ...   hallo.write('300\n')
  >>> checkout.update_function = update_function

  >>> checkout.up()

We maintain the lists of things changed::

  >>> checkout._files = [hallo]
  >>> checkout._removed = []

We will reload the checkout into Python objects again::

  >>> dummy = s.load(None)
 
We expect there to be a new object ``hallo``::

  >>> 'hallo' in data2.keys()
  True

version control removes a file
------------------------------

We update our checkout and cause a file to be removed::

  >>> def update_function():
  ...   data_dir.join('hallo.test').remove()
  >>> checkout.update_function = update_function

  >>> checkout.up()

We maintain the lists of things changed::

  >>> checkout._files = []
  >>> checkout._removed = [hallo]

We will reload the checkout into Python objects::
  
  >>> dummy = s.load(None)

We expect the object ``hallo`` to be gone again::

  >>> 'hallo' in data2.keys()
  False

version control adds a directory
--------------------------------

We update our checkout and cause a directory (with a file inside) to be
added::

  >>> newdir_path = data_dir.join('newdir')
  >>> def update_function():
  ...   newdir_path.ensure(dir=True)
  ...   newfile_path = newdir_path.join('newfile.test').ensure()
  ...   newfile_path.write('400\n')
  >>> checkout.update_function = update_function
  
  >>> checkout.up()

We maintain the lists of things changed::

  >>> checkout._files = [newdir_path, newdir_path.join('newfile.test')]
  >>> checkout._removed = []

Reloading this will cause a new container to exist::

  >>> dummy = s.load(None)
  >>> 'newdir' in data2.keys()
  True
  >>> isinstance(data2['newdir'], Container)
  True
  >>> data2['newdir']['newfile'].payload
  400

version control removes a directory
-----------------------------------

We update our checkout once again and cause a directory to be removed::

  >>> def update_function():
  ...   newdir_path.remove()
  >>> checkout.update_function = update_function

  >>> checkout.up()

We maintain the lists of things changed::

  >>> checkout._files = []
  >>> checkout._removed = [newdir_path, newdir_path.join('newfile.test')]

And reload the data::

  >>> dummy = s.load(None)

Reloading this will cause the new container to be gone again::

  >>> 'newdir' in data2.keys()
  False

version control changes a file into a directory
-----------------------------------------------

Some sequence of actions by other users has caused a name that previously
referred to a file to now refer to a directory::

  >>> hoi_path2 = data_dir.join('hoi')
  >>> def update_function():
  ...   hoi_path.remove()
  ...   hoi_path2.ensure(dir=True)
  ...   some_path = hoi_path2.join('some.test').ensure(file=True)
  ...   some_path.write('1000\n')
  >>> checkout.update_function = update_function

We maintain the lists of things changed::

  >>> checkout._files = [hoi_path2, hoi_path2.join('some.test')]
  >>> checkout._removed = [hoi_path]

  >>> checkout.up()

Reloading this will cause a new container to be there instead of the file::

  >>> dummy = s.load(None)
  >>> isinstance(data2['hoi'], Container)
  True
  >>> data2['hoi']['some'].payload
  1000

version control changes a directory into a file
-----------------------------------------------

Some sequence of actions by other users has caused a name that
previously referred to a directory to now refer to a file::

  >>> def update_function():
  ...   hoi_path2.remove()
  ...   hoi_path = data_dir.join('hoi.test').ensure()
  ...   hoi_path.write('2000\n')
  >>> checkout.update_function = update_function

  >>> checkout.up()

We maintain the lists of things changed::

  >>> checkout._files = [hoi_path]
  >>> checkout._removed = [hoi_path2.join('some.test'), hoi_path2]

Reloading this will cause a new item to be there instead of the
container::

  >>> dummy = s.load(None)
  >>> isinstance(data2['hoi'], Item)
  True
  >>> data2['hoi'].payload
  2000

version control changes a file into one with a different file type
------------------------------------------------------------------

Some sequence of actions by other users has caused a name that
previously referred to one type of object to now refer to another kind.
Let's define an ``Item2``::

  >>> class Item2(object):
  ...   def __init__(self, payload):
  ...     self.payload = payload

And a parser and factory for it::
  
  >>> class Item2Parser(grok.GlobalUtility):
  ...   grok.provides(IParser)
  ...   grok.name('.test2')
  ...   def __call__(self, object, path):
  ...      object.payload = int(path.read()) ** 2
  >>> grok.testing.grok_component('Item2Parser', Item2Parser)
  True 
  >>> class Item2Factory(grok.GlobalUtility):
  ...   grok.provides(IFactory)
  ...   grok.name('.test2')
  ...   def __call__(self, path):
  ...       parser = component.getUtility(IParser, '.test2')
  ...       item = Item2(None) # dummy payload
  ...       parser(item, path)
  ...       return item
  >>> grok.testing.grok_component('Item2Factory', Item2Factory)
  True

Now we define an update function that replaces ``hoi.test`` with
``hoi.test2``::

  >>> hoi_path3 = data_dir.join('hoi.test2')
  >>> def update_function():
  ...    hoi_path.remove()
  ...    hoi_path3.ensure()
  ...    hoi_path3.write('44\n')
  >>> checkout.update_function = update_function
  >>> checkout.up()

We maintain the list of things changed::

  >>> checkout._files = [hoi_path3]
  >>> checkout._removed = [hoi_path]

Reloading this will cause a new type of item to be there instead of the old
type::

  >>> dummy = s.load(None)
  >>> isinstance(data2['hoi'], Item2)
  True
  >>> data2['hoi'].payload
  1936

Let's restore the original ``hoi.test`` object::
 
  >>> hoi_path3.remove()
  >>> hoi_path.write('2000\n')
  >>> del data2['hoi']
  >>> data2['hoi'] = Item(2000)

Complete synchronization
------------------------

Let's now exercise the ``sync`` method directly. First we'll modify
the payload of the ``hoi`` item::

  >>> data2['hoi'].payload = 3000
 
Next, we will add a new ``alpha`` file to the checkout when we do an
``up()``, so again we simulate the actions of our version control system::

  >>> alpha_path = data_dir.join('alpha.test').ensure()
  >>> def update_function():
  ...   alpha_path.write('4000\n')
  >>> checkout.update_function = update_function

We maintain the lists of things changed::

  >>> checkout._files = [alpha_path]
  >>> checkout._removed = []

The revision number before full synchronization::

  >>> checkout.revision_nr()
  8

Now we'll synchronize with the memory structure. We'll pass a special
function along that prints out all objects that have been created or
modified::

  >>> def f(obj):
  ...   print "modified:", obj.__name__
  >>> info = s.sync(message='', modified_function=f)
  modified: alpha
  >>> info.revision_nr
  9

We can get a report of what happened. No files were removed::

  >>> info.files_removed()
  []

One file, alpha, was added to the checkout during our update (by
someone else)::

  >>> info.files_changed()
  [local('.../data/alpha.test')]

We removed no objects from our database since the last update::

  >>> info.objects_removed()
  []

We did change one object, 'hoi', but the test infrastructure always returns
all objects here (returning more objects is allowed)::

  >>> info.objects_changed()
  ['/', '/data/foo', '/data/hoi', '/data', '/data/sub/qux', '/data/sub']

We expect the checkout to reflect the changed state of the ``hoi`` object::

  >>> data_dir.join('hoi.test').read()
  '3000\n'

We also expect the database to reflect the creation of the new
``alpha`` object::

  >>> data2['alpha'].payload
  4000

