PyTables User's Guide
   Next

PyTables User's Guide

Hierarchical datasets in Python - Release 2.2.1

Francesc Alted

Ivan Vilata

Scott Prater

Vicent Mas

Tom Hedley

Antonio Valentino

Jeffrey Whitaker

Copyright © 2002, 2003, 2004 Francesc Alted

Copyright © 2005, 2006, 2007 Cárabos Coop. V.

Copyright © 2008, 2009, 2010 Francesc Alted

Copyright Notice and Statement for PyTables User's Guide.

Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:

a. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.

b. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.

c. Neither the name of Francesc Alted nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

$LastChangedDate: 2010-11-05 09:48:18 +0100 (dv, 05 nov 2010) $

Table of Contents

I. The PyTables Core Library
1. Introduction
1.1. Main Features
1.2. The Object Tree
2. Installation
2.1. Installation from source
2.1.1. Prerequisites
2.1.2. PyTables package installation
2.2. Binary installation (Windows)
2.2.1. Windows prerequisites
2.2.2. PyTables package installation
3. Tutorials
3.1. Getting started
3.1.1. Importing tables objects
3.1.2. Declaring a Column Descriptor
3.1.3. Creating a PyTables file from scratch
3.1.4. Creating a new group
3.1.5. Creating a new table
3.1.6. Reading (and selecting) data in a table
3.1.7. Creating new array objects
3.1.8. Closing the file and looking at its content
3.2. Browsing the object tree
3.2.1. Traversing the object tree
3.2.2. Setting and getting user attributes
3.2.3. Getting object metadata
3.2.4. Reading data from Array objects
3.3. Commiting data to tables and arrays
3.3.1. Appending data to an existing table
3.3.2. Modifying data in tables
3.3.3. Modifying data in arrays
3.3.4. And finally... how to delete rows from a table
3.4. Multidimensional table cells and automatic sanity checks
3.4.1. Shape checking
3.4.2. Field name checking
3.4.3. Data type checking
3.5. Using links for more convenient access to nodes
3.5.1. Interactive example
3.6. Exercising the Undo/Redo feature
3.6.1. A basic example
3.6.2. A more complete example
3.7. Using enumerated types
3.7.1. Enumerated columns
3.7.2. Enumerated arrays
3.8. Dealing with nested structures in tables
3.8.1. Nested table creation
3.8.2. Reading nested tables
3.8.3. Using Cols accessor
3.8.4. Accessing meta-information of nested tables
3.9. Other examples in PyTables distribution
4. Library Reference
4.1. tables variables and functions
4.1.1. Global variables
4.1.2. Global functions
4.2. The File class
4.2.1. File instance variables
4.2.2. File methods — file handling
4.2.3. File methods — hierarchy manipulation
4.2.4. File methods — tree traversal
4.2.5. File methods — Undo/Redo support
4.2.6. File methods — attribute handling
4.3. The Node class
4.3.1. Node instance variables — location dependent
4.3.2. Node instance variables — location independent
4.3.3. Node instance variables — attribute shorthands
4.3.4. Node methods — hierarchy manipulation
4.3.5. Node methods — attribute handling
4.4. The Group class
4.4.1. Group instance variables
4.4.2. Group methods
4.4.3. Group special methods
4.5. The Leaf class
4.5.1. Leaf instance variables
4.5.2. Leaf instance variables — aliases
4.5.3. Leaf methods
4.6. The Table class
4.6.1. Table instance variables
4.6.2. Table methods — reading
4.6.3. Table methods — writing
4.6.4. Table methods — querying
4.6.5. Table methods — other
4.6.6. The Description class
4.6.7. The Row class
4.6.8. The Cols class
4.6.9. The Column class
4.7. The Array class
4.7.1. Array instance variables
4.7.2. Array methods
4.7.3. Array special methods
4.8. The CArray class
4.8.1. Example of use
4.9. The EArray class
4.9.1. EArray methods
4.9.2. Example of use
4.10. The VLArray class
4.10.1. VLArray instance variables
4.10.2. VLArray methods
4.10.3. VLArray special methods
4.10.4. Example of use
4.11. The Link class
4.11.1. Link instance variables
4.11.2. Link methods
4.12. The SoftLink class
4.12.1. SoftLink special methods
4.13. The ExternalLink class
4.13.1. ExternalLink instance variables
4.13.2. ExternalLink methods
4.13.3. ExternalLink special methods
4.14. The UnImplemented class
4.15. The Unknown class
4.16. The AttributeSet class
4.16.1. Notes on native and pickled attributes
4.16.2. AttributeSet instance variables
4.16.3. AttributeSet methods
4.17. Declarative classes
4.17.1. The Atom class and its descendants.
4.17.2. The Col class and its descendants
4.17.3. The IsDescription class
4.18. Helper classes
4.18.1. The Filters class
4.18.2. The Index class
4.18.3. The Enum class
4.19. The Expr class - a general-purpose expression evaluator
4.19.1. Expr instance variables
4.19.2. Expr special tuning variables for input/output
4.19.3. Expr methods
4.19.4. Expr special methods
5. Optimization tips
5.1. Understanding chunking
5.1.1. Informing PyTables about expected number of rows in tables or arrays
5.1.2. Fine-tuning the chunksize
5.2. Accelerating your searches
5.2.1. In-kernel searches
5.2.2. Indexed searches
5.2.3. Indexing and Solid State Disks (SSD)
5.2.4. Achieving ultimate speed: sorted tables and beyond
5.3. Compression issues
5.3.1. A study on supported compression libraries
5.3.2. Shuffling (or how to make the compression process more effective)
5.4. Using Psyco
5.5. Getting the most from the node LRU cache
5.6. Compacting your PyTables files
II. Complementary modules
6. filenode - simulating a filesystem with PyTables
6.1. What is filenode?
6.2. Finding a filenode node
6.3. filenode - simulating files inside PyTables
6.3.1. Creating a new file node
6.3.2. Using a file node
6.3.3. Opening an existing file node
6.3.4. Adding metadata to a file node
6.4. Complementary notes
6.5. Current limitations
6.6. filenode module reference
6.6.1. Global constants
6.6.2. Global functions
6.6.3. The FileNode abstract class
6.6.4. The ROFileNode class
6.6.5. The RAFileNode class
7. netcdf3 - a PyTables NetCDF3 emulation API
7.1. What is netcdf3?
7.2. Using the tables.netcdf3 package
7.2.1. Creating/Opening/Closing a tables.netcdf3 file
7.2.2. Dimensions in a tables.netcdf3 file
7.2.3. Variables in a tables.netcdf3 file
7.2.4. Attributes in a tables.netcdf3 file
7.2.5. Writing data to and retrieving data from a tables.netcdf3 variable
7.2.6. Efficient compression of tables.netcdf3 variables
7.3. tables.netcdf3 package reference
7.3.1. Global constants
7.3.2. The NetCDFFile class
7.3.3. The NetCDFVariable class
7.4. Converting between true netCDF files and tables.netcdf3 files
7.5. tables.netcdf3 file structure
7.6. Sharing data in tables.netcdf3 files over the Internet with OPeNDAP
7.7. Differences between the Scientific.IO.NetCDF API and the tables.netcdf3 API
III. Appendixes
A. Supported data types in PyTables
B. Condition syntax
C. PyTables' parameter files
C.1. Tunable parameters in parameters.py.
C.1.1. Recommended maximum values
C.1.2. Cache limits
C.1.3. Parameters for the I/O buffer in Leaf objects
C.1.4. Miscellaneous
C.2. Tunable parameters in _parameters_pro.py
C.2.1. Parameters for the different internal caches
C.2.2. Parameters for general cache behaviour
D. Using nested record arrays
D.1. Introduction
D.2. NestedRecArray methods
D.3. NestedRecord objects
E. Utilities
E.1. ptdump
E.1.1. Usage
E.1.2. A small tutorial on ptdump
E.2. ptrepack
E.2.1. Usage
E.2.2. A small tutorial on ptrepack
E.3. nctoh5
E.3.1. Usage
F. PyTables File Format
F.1. Mandatory attributes for a File
F.2. Mandatory attributes for a Group
F.3. Optional attributes for a Group
F.4. Mandatory attributes, storage layout and supported data types for Leaves
F.4.1. Table format
F.4.2. Array format
F.4.3. CArray format
F.4.4. EArray format
F.4.5. VLArray format
F.5. Optional attributes for Leaves
Bibliography

List of Figures

1.1. An HDF5 example with 2 subgroups, 2 tables and 1 array.
1.2. A PyTables object tree example.
3.1. The initial version of the data file for tutorial 1, with a view of the data objects.
3.2. The final version of the data file for tutorial 1.
3.3. General properties of the /detector/readout table.
3.4. Table hierarchy for tutorial 2.
5.1. Creation time per element for a 15 GB EArray and different chunksizes.
5.2. File sizes for a 15 GB EArray and different chunksizes.
5.3. Sequential access time per element for a 15 GB EArray and different chunksizes.
5.4. Random access time per element for a 15 GB EArray and different chunksizes.
5.5. Times for non-indexed complex queries in a small table with 10 millions of rows: the data fits in memory.
5.6. Times for non-indexed complex queries in a large table with 1 billion of rows: the data does not fit in memory.
5.7. Times for indexing an Int32 and Float64 column.
5.8. Sizes for an index of a Float64 column with 1 billion of rows.
5.9. Times for complex queries with a cold cache (mean of 5 first random queries) for different optimization levels. Benchmark made on a machine with Intel Core2 (64-bit) @ 3 GHz processor with RAID-0 disk storage.
5.10. Times for complex queries with a cold cache (mean of 5 first random queries) for different compressors.
5.11. Times for complex queries with a cold cache (mean of 5 first random queries) for different disk storage (SSD vs spinning disks).
5.12. Times for complex queries with a cold cache (mean of 5 first random queries) for unsorted and sorted tables.
5.13. Comparison between different compression libraries.
5.14. Comparison between different compression levels of Zlib.
5.15. Writing tables with several compressors.
5.16. Selecting values in tables with several compressors. The file is not in the OS cache.
5.17. Selecting values in tables with several compressors. The file is in the OS cache.
5.18. Writing in tables with different levels of compression.
5.19. Selecting values in tables with different levels of compression. The file is in the OS cache.
5.20. Comparison between different compression libraries with and without the shuffle filter.
5.21. Writing with different compression libraries with and without the shuffle filter.
5.22. Reading with different compression libraries with the shuffle filter. The file is not in OS cache.
5.23. Reading with different compression libraries with and without the shuffle filter. The file is in OS cache.
5.24. Writing tables with/without Psyco.
5.25. Reading tables with/without Psyco.

List of Tables

5.1. Retrieval speed and memory consumption depending on the number of nodes in LRU cache.
A.1. Data types supported for array elements and tables columns in PyTables.
   Next
   Part I. The PyTables Core Library