#######################
PyFITS Developers Guide
#######################

This "developers guide" will be brief, as PyFITS will, in the near future,
be deprecated in favor of Astropy (which includes a port of PyFITS now dubbed
``astropy.io.fits``.  As such, it should be sufficient for any developers
wishing to contribute to PyFITS to look at the developer documentation for
Astropy, as much of it applies equally well.  In particular, please look at
the Astropy `Coding Guidelines`_ and the `Documentation Guidelines`_ before
getting started with any major contributions to PyFITS (don't worry if you
don't immediately absorb *everything* in those guidelines--it's just good to
be aware that they exist and have a rough understanding of how to approach the
source code).

Getting the source code
=======================

PyFITS was originally developed in SVN, but now most development has moved to
Git, primarily for ease of syncing changes to Astropy.  That said, the SVN
repository is still maintained for legacy purposes.  PyFITS' lead maintainer
at STScI will handle synchronizing the Git and SVN repositories, but the steps
for configuring git-svn are documented below for posterity.  Outside users
wishing to contribute to the source code should start with Astropy's guide to
`Contributing to Astropy`_.

The official PyFITS GitHub page is at: https://github.com/spacetelescope/pyfits

The best way to contribute to PyFITS is to create an account on GitHub, fork
your own copy of the PyFITS repository, and then make your changes in your
personal fork and make a pull request when they are ready to share.  The entire
process is described in Astropy's `Workflow for Developers`_ document.  That
documention was written for Astropy, but applies all the same to PyFITS.
Just replace any instance of ``astropy/astropy.git`` with
``spacetelescope/PyFITS.git`` and so on.  Use of virtualenv and
``./setup.py develop`` are strongly encouraged for developing on PyFITS--use of
this tools is also described in the aforementioned Workflow for Developers
document.

Synchronizing with SVN
----------------------

This section is primarily intended for developers at STScI who have commit
access to the PyFITS SVN repository (http://svn6.assembla.com/svn/pyfits).
The PyFITS Git and SVN repositories are synced using the git-svn command.
git-svn can be tricky to install as it requires the Perl bindings for SVN, as
well as SVN itself and of course Git.  The easiest way to get git-svn is to
ask a system administrator to install it from the OS packaging system.

Most guides for setting up git-svn start out with either the ``git svn init``
command or ``git svn clone``.  But because the work of synchronizing the Git
and SVN repositories up until this point has already been done, a faster,
though seemingly less straightforward approach, is to just clone the GitHub
reposiotry and add the git-svn metadata manually:

1. Clone the main spacetelescope GitHub repository::

       $ git clone git@github.com:spacetelescope/PyFITS.git

2. cd into the repository and open ``.git/config`` in an editor and add the
   following::

       [svn]
           authorsfile = .authors
       [svn-remote "svn"]
           url = http://svn6.assembla.com/svn/pyfits
           fetch = trunk:refs/remotes/trunk
           branches = branches/{3.1-stable,3.0-stable}:refs/remotes/branches/*
       [branch "3.0-stable"]
           remote = .
           merge = refs/remotes/branches/3.0-stable
       [branch "3.1-stable"]
           remote = .
           merge = refs/remotes/branches/3.1-stable

   Repeat the ``[branch "X.Y-stable"]`` section following the above pattern
   for any actively maintained release branches (see the "Maintenance" section
   below for more details on release branches). Likewise add each branch to
   the ``branches =`` option under the ``[svn-remote "svn"]`` section (you
   *may* put a ``*`` here to get all branches. It is also possible to sync
   all SVN tags.  But as most of those branches are defunct it is probably
   not desirable to sync them all (it is a very time consuming operation).

   .. warning::

       Do not forget to set the `[svn]/authorsfile = .authors` option, or
       the repository will get severely confused when trying to sync SVN
       changes with the git repository.  The .authors file maps SVN usernames
       to developers' name/e-mail address to use in git commits.  If you intend
       to synchronize changes you make with SVN, make sure to add yourself to
       the .authors file.  The format should be self-explanatory.

3. Put the hash of the latest revision of the upstream master branch in refs
   file for trunk, so git-svn knows where to start synchronizing with SVN's
   trunk::

       $ git rev-parse origin/master > .git/refs/remotes/trunk

   Likewise for each stable branch do, for example::

       $ mkdir .git/refs/remotes/branches/
       $ git rev-parse origin/3.1-stable > .git/refs/remotes/branches/3.1-stable

4. Finally, do::

       $ git svn fetch

   to synchronize any new revisions in the SVN repository.

   .. warning::

       This fetch operation can still take a pretty long time, especially on
       the branches.  It may appear to hang at some points--just be patient
       and leave it running.  I've yet to find a way to speed it up any
       further.

Syncing new changes to SVN
^^^^^^^^^^^^^^^^^^^^^^^^^^

The command for committing new changes in git to the SVN repository is
`git svn dcommit`.  This command goes through all commits on the current
branch that have *not* yet been committed to SVN and does so.

Whenever you are about to push new changes on the master branch to the remote
remote repository on GitHub it is best to first cross-commit those changes to
SVN.  This is because git-svn rewrites the commit messages on all your commits
to include a reference to the SVN revision that was created from that commit.
So if you push first, and then run `git svn dcommit` you will now have
different commits (as far as their SHA has is concerned) on your local
repository from what you just pushed to the remote repository.  The simplest
way to resolve this, when it happens, is to `git push --force`.  This will
override the old history with the new history that includes the SVN revisions
in the commit messages.

It's easier, however, to remember to always run `git svn dcommit` before doing
a `git push`.


Maintenance
===========

At any given time there are two to three lines of development on PyFITS
(possibly more if some critical bug is discovered that needs to be backported
to older release lines, though such situations are rare).  Typically there is
the mainline development in the 'master' branch, and at least one branch named
after the last minor release.  For example, if the version being developed in
the mainline is '3.2.0' there will be, at a minimum, a '3.1-stable' branch into
which bug fixes can be ported.  There may also be a '3.0-stable' branch and so
on so long as new bugfix releases are being made with '3.0.z' versions.

Bug fix releases should never add new public APIs or change existing ones--they
should only correct bugs or major oversights.  "Minor" releases, where the
second number in the version is increased, may introduce new APIs and may
*deprecate* old interfaces (see the ``@deprecated`` decorated in
``pyfits.util``, but may not otherwise remove or change (non-buggy) behavior of
old interfaces without backwards compatibility with the previous versions in
the same major version line.  Major releases may break backwards compatibility
so long as warning has been given through ``@deprecated`` markers and
documentation that those interfaces will break in future versions.

In general all development should be done in the 'master' branch, including
development of new features and bug fixes (though temporary branches should
certainly be used aggressively for any individual feature or fix being
developed, they should be merged back into 'master' when ready).

The only exception to this rule is when developing a bug fix that *only*
applies to an older release line.  For example it's possible for a bug to exist
in version '3.1.1' that no longer exists in the 'master' branch (perhaps
because it pertains to an older API), but that still exists in the '3.1-stable'
branch.  Then that bug should be fixed in the '3.1-stable' branch to be
included in the version '3.1.2' bugfix release (assuming a bugfix release is
planned).  If that bug pertains to any older release branches (such as
'3.0-stable') it should also be backported to those branches by way of
``git cherry-pick``.


Releasing
=========

Creating a PyFITS release consists 3 main stages each with several sub-steps
according to this rough outline:

1. Pre-release

   a. Set the version string for the release in the setup.cfg file

   b. Set the release date in the changelog (CHANGES.txt)

   c. Test that README.txt and CHANGES.txt can be correctly parsed as
      RestructuredText.

   d. Commit these preparations to the repository, creating a specific commit
      to tag as the "release"

2. Release

   a. Create a tag from the commit created in the pre-release stage

   b. Register the new release on PyPI

   c. Build a source distribution of the release and test that it is
      installable (specifically, installable with pip) and that all the tests
      pass from an installed version

3. Post-release

   a. Upload the source distribution to PyPI

   b. Set the version string for the "next" release in the setup.cfg file (the
      choice of the next version is based on inference, and does not mean the
      "next" version can't be changed later if desired)

   c. Create a new section in CHANGES.txt for the next release (using the same
      "next" version as in part b)

   d. Commit these "post-release" changes to the repository

   e. Push the release commits and the new tag to the remote repository
      (GitHub)

   f. Update the PyFITS website to reflect the new version

   g. Build Windows installers for all supported Python versions and upload
      them to PyPI

Most of these steps are automated by using `zest.releaser`_ along with some
hooks designed specifically for PyFITS that automate actions such as updating
the PyFITS website.

Prerequisites for performing a release
--------------------------------------

1. Because PyFITS is released (registered and uploaded to) on PyPI it is
   necessary to create an account on PyPI and get assigned a "Maintainer"
   role for the PyFITS package.  Currently the package owners--the only two
   people who can add additional Maintainers are Erik Bray <embray@stsci.edu>
   and Nicolas Barbey <nicolas.a.barbey@gmail.com>.  (It remains a "todo" item
   to add a shared "space telescope" account.  In the meantime, should both of
   those people be hit by a bus simultaneously the PyPI administrators will be
   reasonable if the situation is explained to them with proper documentation).

   Once your PyPI account is set up, it is necessary to add your PyPI
   credentials (username and password) to the ``.pypirc`` file in your home
   directory with the following format::

       [server-login]
       username: <your PyPI username>
       password: <your PyPI password>

   Unfortunately some the ``setup.py`` commands for interacting with PyPI
   are broken in that they don't allow interactive password entry.  Creating
   the ``.pypirc`` file is *currently* the most reliable way to make
   authentication with PyPI "just work".  Be sure to ``chmod 600`` this file.

2. Generate a signing key--all PyFITS tags are now cryptographically signed
   when creating the tag (using ``git tag -s``).  The `Astropy release
   process`_ page documents how to set this up.

3. Also make sure to have an account on readthedocs.org with administrative
   access to the PyFITS project on Read the Docs:
   https://readthedocs.org/projects/pyfits/
   This hosts documentation for all (recent) versions of PyFITS.  (TODO: Here
   also we need a "space telescope" account with administrative rights to all
   STScI projects that use RtD.)

4. It's best to do the release in a relatively "clean" Python environment, so
   make sure you have `virtualenv`_ installed and that you've had some practice
   in using it.

5. Make sure you have Numpy and nose installed and are able to run the PyFITS
   tests successfully without any errors.  Even better if you can do this with
   tox.

6. Make sure that at least someone can make the Windows builds.  This requires
   a Windows machine with at least Windows XP, Mingw32 with msys, and all of
   the Python development packages.  Python versions 2.5, 2.6, 2.7, 3.1, and
   3.2 should be installed with the installers from python.org, as well as a
   recent version of Numpy for each of those Python versions (currently Numpy
   1.6.x), as well as Git.  (TODO: More detailed instructions for setting up
   a Windows development environment.)

7. PyFITS also has a page on STScI's website:
   http://www.stsci.edu/institute/software_hardware/pyfits.  This is normally
   the first hit when Googling 'pyfits' so it's important to keep up to date.
   At a minimum each release should update the front page to mention the most
   recent release, the Release Notes page with an HTML rendering of the most
   recent changelog, and the download page with links to all the current
   versions.  See the exisint site for examples.  The STScI website has both
   a test server and a production server.  It's difficult for content creators
   to get direct access to the production server, but at least make sure you
   have access to the test server on port 8072, and that IT has given you
   permission to write to the PyFITS section of the site.

   Part of the PyFITS automated release script attempts to update the PyFITS
   website (on the test server) as part of the standard release process.  So
   it's important to test your access to the site and ability to make edits.
   If for any reason the automatic update fails (e.g. your authentication
   fails) it is still possible to update the site manually.

   Once the updates are made it's necessary to have IT push the updates to the
   production server.  As of writing the best person to ask is George Smyth--
   asking him directly is the fastest way to get it done, though if you send a
   ticket to IT it will be handled eventually.

8. Triage issues is milestones in the PyFITS bug tracker(s).  Currently this
   includes the Trac site: https://trac.assembla.com/pyfits/roadmap and the
   GitHub site: https://github.com/spacetelescope/PyFITS/issues/milestones

   No new tickets are being added in Trac, so after all open tickets in the
   Trac site have been addressed, milestones will only need to be managed in
   GitHub.

   First create a new milestone for the version after the version to be
   released.  If a major/minor release is being made, make the milestone for
   the next bugfix release in that series as well.  For example if releasing a
   bugfix release like 3.0.1, create a milestone for 3.0.2.  If releasing
   3.1.0, create milestones for 3.2.0 *and* 3.2.1.

   If the milestone for the to be released version still has any issues
   remaining in it, such as bugs that were not fixed, move them to the next
   appriopriate milestone if they will not be addressed before the release
   (or close issues that are no longer applicable).  Ensure that the milestone
   for the to be released version has no open issues remaining in it.



Release procedure
-----------------

(These instructions are adapted from the `Astropy release process`_
which itself was adapted from PyFITS' release process--the former just got
written down first.)

1. In a directory outside the pyfits repository, create an activate a
   virtualenv in which to do the release (it's okay to use
   ``--system-site-packages`` for dependencies like Numpy)::

       $ virtualenv --system-site-packages --distribute pyfits-release
       $ source pyfits-release/bin/activate

2. Obtain a *clean* version of the PyFITS repository. That is, one where you
   don’t have any intermediate build files. It is best to use a fresh
   ``git clone`` from the main repository on GitHub without any of the git-svn
   configuration. This is because the git-svn support in zest.releaser does not
   handle tagging in branches very well yet.

3. Use ``git checkout`` to switch to the appropriate branch from which to do
   the release.  For a new major or minor release (such as 3.0.0 or 3.1.0)
   this should be the 'master' branch.  When making a bugfix release it is
   necessary to switch to the appropriate bugfix branch (e.g.
   ``git checkout 3.1-stable`` to release 3.1.2 up from 3.1.1).

4. Install ``zest.releaser`` into the virtualenv; use ``--upgrade --force`` to
   ensure that the latest version is installed in the virtualenv (if you’re
   running a csh variant make sure to run rehash afterwards too)::

       $ pip install zest.releaser --upgrade --force

5. Install ``stsci.distutils`` which includes some additional releaser hooks
   that are useful::

       $ pip install stsci.distutils --upgrade --force

6. Ensure that any lingering changes to the code have been committed, then
   start the release by running::

       $ fullrelease

7. You will be asked to enter the version to be released.  Press enter to
   accept the default (which will normally be correct) or enter a specific
   version string.  A diff will then be shown of CHANGES.txt and setup.cfg
   showing that a release date has been added to the changelog, and that the
   version has been updated in setup.cfg.  Enter 'Y' when asked to commit these
   changes.

8. You will then be shown the command that will be run to tag the release.
   Enter 'Y' to confirm and run the command.

9. When asked "Check out the tag (for tweaks or pypi/distutils server upload)"
   enter 'Y': This feature is used when uploading the source distribution to
   our local package index.  When asked to 'Register and upload' to PyPI enter
   'N'.  We will do this manually later in the process once we've tested the
   release out first.  If asked to add the package to the "STScI package
   index" enter 'N'--this package index is no longer being maintained.

10. You will be asked to enter a new development version.  Normally the next
    logical version will be selected--press enter to accept the default, or
    enter a specific version string.  Do not add ".dev" to the version, as this
    will be appended automatically (ignore the message that says ".dev0 will be
    appended"--it will actually be ".dev" without the 0).  For example, if the
    just-released version was "3.1.0" the default next version will be "3.1.1".
    If we want the next version to be, say "3.2.0" then that must be entered
    manually.

11. You will be shown a diff of CHANGES.txt showing that a new section has been
    added for the new development version, and showing that the version has
    been updated in setup.py.  Enter 'Y' to commit these changes.

12. When asked to push the changes to a remote repository, enter 'N'.  We want
    to test the release out before pushing changes to the remote repository or
    registering in PyPI.

13. When asked to update the PyFITS homepage enter 'Y'.  The enter the name of
    the previous version (in the same MAJOR.MINOR.x branch) and then the name
    of the just released version.  The defaults will usually be correct.  When
    asked, enter the username and password for your Zope login.  As of writing
    this is not necessarily the same as your Exchange password.  If the update
    succeeeds make sure to e-mail IT and ask them to push the updated pages
    from the test site to the production site.

    This should complete the portion of the process that's automated at this point
    (though future versions will automate these steps as well, after a few needed
    features are added to zest.releaser).

14. Check out the tag of the released version.  For example::

        $ git checkout v3.1.0

15. Create the source distribution by doing::

        $ python setup.py sdist

16. Now, outside the repository create and activate another new virtualenv
    for testing the release::

        $ virtualenv --system-site-packages --distribute pyfits-release-test
        $ source pyfits-release-test/bin/activate

17. Use ``pip`` to install the source distribution built in step 13 into the
    new test virtualenv.  This will look something like::

        $ pip install PyFITS/dist/pyfits-3.2.0.tar.gz

    where the path should be to the sole ``.tar.gz`` file in the ``dist/``
    directory under your repository clone.

18. Try running the tests in the installed PyFITS::

        $ pip install nose --force --upgrade
        $ nosetests pyfits

    If any of the tests fail abort the process and start over.  Undo the
    previous two git commits (the one tagged as the release, and the one
    where you bumped to the next dev version)::

        $ git reset --hard HEAD^^

    Also delete the newly created tag::

        $ git tag -d v3.2.0

    Resolve the test failure, commit any new fixes, and start the release
    procedure over again (it's rare for this to be an issue if the tests
    passed *before* starting the release, but it is possible--the most likely
    case being if some file that *should* be installed is either not getting
    installed or is not included in the source distribution in the first
    place).

19. Assuming the test installation worked, change directories back into the
    repository and push the new tag/release to the main repository on GitHub::

        $ git push --tags

    This initial step is necessary since the tag was made off of a pure git
    commit.  But when we synchronize with SVN the commit history will change
    so we need to force an additional push to the GitHub repository::

        $ git svn dcommit
        $ git push --force

    Then register the release on PyPI with::

        $ python setup.py register

    Upload the source distribution to PyPI; this is preceded by re-running the
    sdist command, which is necessary for the upload command to know which
    distribution to upload::

        $ python setup.py sdist upload

    After registering on PyPI go to the URL:

    https://pypi.python.org/pypi?%3Aaction=pkg_edit&name=pyfits

    and mark any previous releases superceded by this release as hidden via the
    web UI.  Don't check "Auto-hide old releases" as we want to support
    discovery of bugfix releases of older versions.

20. When releasing a new major or minor version, create a bugfix branch for
    that version.  Starting from the tagged changset, just checkout a new
    branch and push it to the remote server.  For example, after releasing
    version 3.2.0, do::

        $ git checkout -b 3.2-stable

    Then edit the setup.cfg so that the version is ``'3.2.1.dev'``, and commit
    that change. Then, do::

        $ git push origin +3.2-stable

    .. note::
        You may need to replace ``origin`` here with ``upstream`` or whatever
        remote name you use for the main PyFITS repository on GitHub.

    The purpose of this branch is for creating bugfix releases like "3.2.1" and
    "3.2.2", while allowing development of new features to continue in the
    master branch.  Only changesets that fix bugs without making significant
    API changes should be merged to the bugfix branches.

21. On the other hand, if a bugfix release was made, the ``CHANGES.txt`` file
    will only be updated in the stable branch; the master branch also needs to
    be updated so that the release is reflected in its copy of ``CHANGES.txt``.
    Just run::

        $ git checkout master

    Say 3.2.1 was just released.  Use ``git log -p`` to find the commit
    that updated the changelog with the release date in the stable branch,
    like::

        $ git log -p 3.2-stable

    Copy the commit hash, and then cherry-pick it into master::

        $ git cherry-pick <sha1 hash>

    You will likely have to resolve a merge conflict, but just make sure that
    the section heading for the just released version is updated so that
    "(unreleased)" is replaced with today's date.  Also ensure that a new
    section is added for the next bugfix release in that release series.


22. Log into the Read the Docs control panel for PyFITS at
    https://readthedocs.org/projects/pyfits/.  Click on "Admin" and then
    "Versions".  Find the just-released version (it might not appear for a few
    minutes) and click the check mark next to "Active" under that version.
    Leave the dropdown list on "Public", then scroll to the bottom of the page
    and click "Submit".  If this is the release with the highest version
    number, make sure to set it as the "default" version as soon as the build
    finishes.

    Note: When you first activate the new version in Read the Docs, it
    immediately displays a "Build Failed" message for the build of the new
    docs.  This is a bug--all it really means is that those docs have never
    been built yet.  Give it a few minutes before checking that the build
    succeeded.  Then you can set that version as the default if needed.

23. We also mirror the most recent documentation at pythonhosted.org/pyfits (
    formerly packages.python.org).

    First it is necessary to build the docs manually.  Make sure all the
    dependencies are satisfied by running::

        $ pip install sphinx

    Then change directories into the docs/ directory and install the additional
    requirements for the docs::

        $ cd docs
        $ pip install -r requirements.txt

    Then make the HTML docs::

        make html

    Now change directories back to the source root and upload::

        $ cd ..
        $ python setup.py upload_docs

24. Mark the milestone of the released version as closed/completed in the
    PyFITS bug tracker(s).  If asked for a timestamp (as Trac does) use the
    timestamp of the git tag made for the release.

25. Build and upload the Windows installers:

    a. Launch a MinGW shell.

    b. Just as before make sure you have a ``pypirc`` file in your home
       directory with your authentication info for PyPI.  On Windows the file
       should be called just ``pypirc`` without the leading ``.`` because
       having some consistency would make this too easy :)

    c. Do a ``git clone`` of the repository or, if you already have a clone
       of the repository do ``git fetch --tags`` to get the new tags.

    d. Check out the tag for the just released version.  For example::

           $ git checkout v3.2.0

       (ignore the message about being in "detached HEAD" state).

    e. For each Python version installed, build with the mingw32 compiler,
       create the binary installer, and upload it.  It's best to use the full
       path to each Python version to avoid ambiguity.  It is also best to
       clean the repository between builds for each version.  For example::

           $ /C/Python25/python setup.py build -c mingw32 bdist_wininst upload
           < ... builds and uploads successfully ... >
           $ git clean -dfx
           $ /C/Python26/python setup.py build -c mingw32 bdist_wininst upload
           < ... builds and puloads successfully ... >
           $ git clean -dfx
           $ < ... and so on, for all currently supported Python versions ... >


.. _Coding Guidelines: http://astropy.readthedocs.org/en/v0.3/development/codeguide.html
.. _Documentation Guidelines: http://astropy.readthedocs.org/en/v0.3/development/docguide.html
.. _Contributing to Astropy: http://astropy.readthedocs.org/en/v0.3/development/workflow/index.html
.. _Workflow for Developers: http://astropy.readthedocs.org/en/v0.3/development/workflow/development_workflow.html
.. _Astropy release process: http://astropy.readthedocs.org/en/v0.3/development/releasing.html
.. _zest.releaser: https://pypi.python.org/pypi/zest.releaser
.. _virtualenv: https://pypi.python.org/pypi/virtualenv
