****************************************************************

  lasprecision:

  reads LIDAR data in the LAS format and computes statistics that
  tell us whether the precision "advertised" in the header is
  really in the data. Often I find that the scaling factors of
  a LAS/LAZ file is miss-leading because they make it appear as
  if there was much more precision than there really is.

  To judge that we compute coordinate difference histograms that
  allow you to easily decide whether there is artificially high
  precision in the LAS/LAZ/BIN file. This may have been introduced
  during LAS conversion by "upsampling" to a higher precision
  "to play it safe".

  Once you figured out the "correct" precision you can also use
  this tool to resample the LAS/LAZ file to an appropriate level
  of precision. A big motivation to remove "fake" precision is
  that LAS files compress much better with laszip without this
  "fluff" in the low-order bits.

  For example I find "fluff" in those examples:
   - Grass Lake Small.las
   - MARS_Sample_Filtered_LiDAR.las
   - Mount St Helens Oct 4 2004.las
   - IowaDNR-CloudPeakSoft-1.0-UTM15N.las
   - LAS12_Sample_withRGB_QT_Modeler.las
   - Lincoln.las
   - Palm Beach Pre Hurricane.las

  There are also options to look at the '-gps' time and the '-rgb'
  colors in the same manner. You can change the amount of lines
  that are output per statistic with '-lines 30'.

  For updates check the website or join the LAStools mailing list.

  http://lastools.org/
  http://groups.google.com/group/lastools/
  http://twitter.com/lastools/
  http://facebook.com/lastools/
  http://linkedin.com/groups?gid=4408378

  Martin @lastools

****************************************************************

example usage:

What happens in the example below? We load -all the raw integer
x, y, and z coordinates of -i "MARS_Sample_Filtered_LiDAR.las"
into three separate arrays, sort each array in ascending sorted
orders, compute the difference between all neighboring values
and output a histogram in textural form.

What do these histograms tell us? We see that the original scale
factors are inflated/missleading. There is not actual millimeter
precision (=> 0.001) in the data. All x, and y, or, z values are
either 10, 20, 30, 40, or other multiple of 10 units spaced apart.
Hence the true precision is only centimeters (=> 0.01). This is
not nice and makes compression less efficient.

>> lasprecision -i MARS_Sample_Filtered_LiDAR.las -all
original scale factors: 0.001 0.001 0.001
loading first 8146178 of 8146178 points
X differences
          0 :    7822346   0
         10 :     323326   0.01
         20 :        500   0.02
         30 :          5   0.03
Y differences
          0 :    7533456   0
         10 :     609382   0.01
         20 :       3303   0.02
         30 :         36   0.03
Z differences
          0 :    8124512   0
         10 :      21175   0.01
         20 :        346   0.02
         30 :         75   0.03
         40 :         23   0.04
         50 :          7   0.05
         60 :          4   0.06
...

Let can take a closer look at the *raw* XYZ integers in this LAS
file to understand what is happening:

>>las2txt -parse XYZ -i "MARS_Sample_Filtered_LiDAR.las" -otxt | more
2136823840 1699144370 5192090
2136824490 1699143020 5192110
2136825130 1699141660 5192130
2136825780 1699140310 5192150
2136826420 1699138950 5192170
2136827070 1699137600 5192190
2136827710 1699136250 5192210
2136828360 1699134890 5192220
2136829000 1699133540 5192240
2136829650 1699132180 5192260
2136830290 1699130830 5192280
2136830940 1699129470 5192300
...

This is obviously not the right way to use the LAS format. Anyone peeking
in the header with lasinfo.exe would be lead to mistakenly assume that
the file has more precision that it actually has. We can fix that by
changing the scale factor to reflect the actual precision in this file:

>> lasprecision -i MARS_Sample_Filtered_LiDAR.las -rescale 0.01 0.01 0.01 -o MARS_Sample_Filtered_LiDAR_rescaled.las
new scale factors: 0.01 0.01 0.01

So lets look at the rescaled file and we see that now we get the type of
histogram one would expect.

>> lasprecision -i MARS_Sample_Filtered_LiDAR_rescaled.las -all
original scale factors: 0.01 0.01 0.01
loading first 8146178 of 8146178 points
X differences
          0 :    7822346   0
          1 :     323326   0.01
          2 :        500   0.02
          3 :          5   0.03
Y differences
          0 :    7533456   0
          1 :     609382   0.01
          2 :       3303   0.02
          3 :         36   0.03
Z differences
          0 :    8124512   0
          1 :      21175   0.01
          2 :        346   0.02
          3 :         75   0.03
          4 :         23   0.04
          5 :          7   0.05
          6 :          4   0.06

And in this particular example we did not even loose precision at all. The
double-precision coordinates after scaling are *identical* as lasdiff.exe
tell us (see below). That might not always be the case because usually tiny
rounding-error are introduced when excessive "fake" precision is in the file. 

>>lasdiff -i MARS_Sample_Filtered_LiDAR.las -i MARS_Sample_Filtered_LiDAR_rescaled.las
header is different
  WARNING: different x_scale_factor: 0.001 0.01
  WARNING: different y_scale_factor: 0.001 0.01
  WARNING: different z_scale_factor: 0.001 0.01
headers have 3 differences.
scaled points are identical. files are identical. both have 8146178 points.

Note that LAZ compression works *much* better on a properly scaled file:

01/10/2010  01:46 AM       163,225,753 MARS_Sample_Filtered_LiDAR.las
12/02/2010  01:02 PM        19,637,873 MARS_Sample_Filtered_LiDAR.laz
12/02/2010  11:04 AM       163,225,753 MARS_Sample_Filtered_LiDAR_rescale.las
12/02/2010  01:01 PM        17,932,612 MARS_Sample_Filtered_LiDAR_rescale.laz

****************************************************************

another (very bad) example (note the scale factors!!!):

C:\released_code\lastools\bin>lasprecision -i "Mount St Helens Oct 4 2004.las" -all -lines 105
original scale factors: 1.22154e-005 1.67604e-005 2.39289e-006
loading first 6743176 of 6743176 points
X differences
          3 :    2698410   3.66462e-005
          4 :    1591251   4.88616e-005
        804 :      85952   0.00982118
        805 :      34047   0.0098334
        807 :     523989   0.00985783
        808 :     993344   0.00987004
        811 :     355349   0.00990669
        812 :       9337   0.0099189
        814 :     220057   0.00994333
        815 :     144628   0.00995555
        817 :       4811   0.00997998
        818 :      15857   0.0099922
       1632 :      17269   0.0199355
       1633 :       3395   0.0199477
       2446 :       9077   0.0298789
       2447 :      11588   0.0298911
       3260 :        880   0.0398222
       3261 :      19783   0.0398344
...
Y differences
          2 :    1797569   3.35208e-005
          3 :    1511859   5.02812e-005
        591 :    2301976   0.0099054
        592 :     243796   0.00992216
        593 :     358431   0.00993892
        594 :     442732   0.00995568
        596 :      20476   0.0099892
        597 :        190   0.010006
       1189 :       9028   0.0199281
       1190 :      11637   0.0199449
       1783 :      18302   0.0298838
       1784 :       2363   0.0299006
       2376 :       6904   0.0398227
       2377 :      13763   0.0398395
       2970 :        652   0.0497784
       2971 :        183   0.0497951
...
Z differences
          0 :    1223726   0
          1 :         53   2.39289e-006
          2 :         46   4.78578e-006
...
         24 :         64   5.74294e-005
         25 :      13395   5.98223e-005
         26 :      13788   6.22152e-005
         27 :         66   6.46081e-005
...
         50 :         48   0.000119645
         51 :     141693   0.000122037
         52 :       1999   0.00012443
         53 :         43   0.000126823
...
         74 :         99   0.000177074
         75 :         74   0.000179467
         76 :      11837   0.00018186
         77 :      12752   0.000184253
         78 :         56   0.000186646
         79 :        112   0.000189038
         80 :        159   0.000191431
...
        101 :         41   0.000241682
        102 :    1527728   0.000244075
        103 :      43356   0.000246468
        104 :         57   0.000248861
...
        203 :         34   0.000485757
        204 :     983926   0.00048815
        205 :      57265   0.000490543
        206 :         42   0.000492936
...
        305 :         30   0.000729832
        306 :     500869   0.000732225
        307 :      45069   0.000734618
        308 :         37   0.000737011
...
        407 :         34   0.000973907
        408 :     353998   0.0009763
        409 :      43756   0.000978693
        410 :         40   0.000981085
...

From the scale factors it is clear that someone was "abusing" the scaling
mechanism of LAS to "preserve" an insane precision level that certainly
the data never had to start with. I am not suprised that the data set was
created by http://prologic-inc.com/ ... (-; Want more evidence? Here is
some output from the header contents with lasinfo.exe. I very much dislike
the way the offset that was chosen as well as how the scaling was set such
that the raw integers of x, y, and z range from 0 to 2147483647. bad idea!

>> lasinfo -i "Mount St Helens Oct 4 2004.las"
...
  generating_software:       'Lidar Explorer by ProLogic, Inc.'
...
  scale factor x y z         1.22154e-005 1.67604e-005 2.39289e-006
  offset x y z               1199921.7455906442 301025.6214073577 3191.5268554688
  min x y z                  1199921.7455906442 301025.6214073577 3191.5268554688
  max x y z                  1226154.1154316438 337018.3067043898 8330.221862793
...
  x 0 2147483647
  y 0 2147483647
  z 0 2147483647

If you look closely at the histogram of "X differences" you will notice
that there is one grouping around 804-818, the next around 1632-1633,
the next around 2446-2447 ... notice a pattern? If you multiply these
groupings with the "fake" x_scale_factor we get 0.01, 0.02, 0.03, ...
The same holds true for the "Y differences" where the groupings are
591-597, 1189-1190, 1783-1784, ... it is less obvious what those groupings
are for the "Z differences" but 0.001 should be enough (i.e. millimeter )
precision given the data presents elevations in meters. We also move the
offset a bit. Why? This is left as a exercise for the reader ... (-;

>>lasprecision -i "Mount St Helens Oct 4 2004.las" -o "Mount St Helens Oct 4 2004 rescaled.las" -rescale 0.01 0.01 0.001 -reoffset 1048576 262144 4096
new scale factors: 0.01 0.01 0.001

After rescaling the file we get a maximal floating point difference
of 0.005, 0.005, and 0.0005 for x, y, and z respectively.

>>lasdiff -i "Mount St Helens Oct 4 2004.las" -i "Mount St Helens Oct 4 2004 rescaled.las"
header is different
  WARNING: different x_scale_factor: 0.000012 0.01
  WARNING: different y_scale_factor: 0.000017 0.01
  WARNING: different z_scale_factor: 0.000002 0.001
  WARNING: different x_offset: 1199921.745591 1048576
  WARNING: different y_offset: 301025.621407 262144
  WARNING: different z_offset: 3191.526855 4096
  x: 493483974 15737385 scaled offset x 1.20595e+006 1.20595e+006
  y: 37431 3888225 scaled offset y 301026 301026
  z: 2307045 -898953 scaled offset z 3197.05 3197.05
  x: 494302434 15738385 scaled offset x 1.20596e+006 1.20596e+006
  y: 24954 3888204 scaled offset y 301026 301026
  z: 3244473 -896709 scaled offset z 3199.29 3199.29
  x: 495120893 15739384 scaled offset x 1.20597e+006 1.20597e+006
  y: 12477 3888183 scaled offset y 301026 301026
  z: 3296507 -896585 scaled offset z 3199.42 3199.42
scaled offset points are different (max diff: 0.005 0.005 0.0005).
both have 6743176 points.

Please note that laszip.exe compression works *much much much* better on a
properly scaled file because the compressor does not have to compress the
excessive and "fake" presision in the lower bits that is really just noise.

12/03/2010  06:28 AM       134,868,035 Mount St Helens Oct 4 2004 rescaled.las
12/03/2010  06:30 AM        12,984,744 Mount St Helens Oct 4 2004 rescaled.laz
12/03/2009  08:29 PM       134,868,035 Mount St Helens Oct 4 2004.las
12/03/2010  06:30 AM        20,337,000 Mount St Helens Oct 4 2004.laz

A few other files where I found excessive precision with lasprecision.exe:

>>lasprecision -i "Grass Lake Small.las"
original scale factors: 0.01 0.01 0.01
loading first 5000000 of 6190800 points
X differences
          0 :    4997360   0
         33 :       1760   0.33
         34 :        879   0.34
Y differences
          0 :    4998106   0
         33 :       1262   0.33
         34 :        631   0.34
Z differences
          0 :    4996046   0
          1 :       3905   0.01
          2 :         30   0.02
          3 :          8   0.03
          4 :          5   0.04
          5 :          1   0.05
          8 :          1   0.08
          9 :          1   0.09
         13 :          1   0.13
         17 :          1   0.17

>>lasprecision -i IowaDNR-CloudPeakSoft-1.0-UTM15N.las
original scale factors: 0.001 0.001 0.001
loading first 5000000 of 5847380 points
X differences
          0 :    4800004   0
         10 :     199991   0.01
         20 :          4   0.02
Y differences
          0 :    4800002   0
         10 :     199995   0.01
         20 :          2   0.02
Z differences
          0 :    4930616   0
          1 :      66129   0.001
          2 :       1167   0.002
          3 :        356   0.003

>>lasprecision -i LAS12_Sample_withRGB_QT_Modeler.las
original scale factors: 0.001 0.001 0.001
loading first 3837973 of 3837973 points
WARNING: end-of-file after 3813696 of 3837973 points
X differences
          0 :    3835738   0
          1 :         28   0.001
        333 :       1500   0.333
        334 :        706   0.334
Y differences
          0 :    3836244   0
        166 :          2   0.166
        333 :       1150   0.333
        334 :        576   0.334
Z differences
          0 :    3806893   0
          1 :      30238   0.001
          2 :        389   0.002
          3 :        155   0.003
          4 :         75   0.004
          5 :         60   0.005
          6 :         26   0.006

>>lasprecision -i Lincoln.las
original scale factors: 1.45584e-006 1.38473e-006 6.72668e-008
loading first 5000000 of 9278073 points
X differences
        128 :       1615   0.000186347
        129 :     512010   0.000187803
        392 :     723661   0.000570689
        393 :    1284377   0.000572145
        521 :     900990   0.000758492
        522 :    1577342   0.000759948
        914 :          3   0.00133064
        915 :          1   0.00133209
Y differences
        135 :    1022794   0.000186938
        136 :    1678978   0.000188323
        277 :     959022   0.000383569
        278 :     213386   0.000384954
        412 :     220537   0.000570507
        413 :     901455   0.000571892
        548 :       1623   0.00075883
        549 :       1205   0.000760215
Z differences
          0 :     744253   0
          1 :        381   6.72668e-008
          2 :        642   1.34534e-007
          3 :       2001   2.018e-007
          4 :       2262   2.69067e-007
          5 :        640   3.36334e-007
          6 :        440   4.03601e-007
          7 :      57413   4.70867e-007

>>lasprecision -i line_27007_dd.las
original scale factors: 1e-007 1e-007 1e-007
loading first 5000000 of 5380156 points
X differences
          0 :    4772386   0
          1 :     225217   1e-007
          2 :       1454   2e-007
          3 :        434   3e-007
          4 :        179   4e-007
          5 :         91   5e-007
          6 :         57   6e-007
          7 :         35   7e-007
Y differences
          0 :    4908448   0
          1 :      90572   1e-007
          2 :        679   2e-007
          3 :        155   3e-007
          4 :         64   4e-007
          5 :         26   5e-007
          6 :         15   6e-007
          7 :         11   7e-007
          8 :          9   8e-007
Z differences
          0 :    4995922   0
      90000 :        500   0.009
     100000 :       3058   0.01
     110000 :        502   0.011
     190000 :          2   0.019
     200000 :          6   0.02
     210000 :          1   0.021
     290000 :          1   0.029
     300000 :          1   0.03

>>lasprecision -i "Palm Beach Pre Hurricane.las"
original scale factors: 3.22228e-006 5.53148e-006 2.28007e-007
loading first 2580410 of 2580410 points
X differences
        250 :     592850   0.00080557
        251 :     837488   0.000808793
        501 :      65509   0.00161436
        502 :      13534   0.00161759
       1425 :     489084   0.00459175
       1426 :      35190   0.00459497
       1675 :     141943   0.00539732
       1676 :     266781   0.00540054
       1926 :      98603   0.00620611
       1927 :      30862   0.00620934
       2176 :        454   0.00701169
       2177 :       2087   0.00701491
       4103 :        851   0.013221
Y differences
        145 :      21380   0.000802064
        146 :     831975   0.000807596
        976 :     575719   0.00539872
        977 :      83759   0.00540425
       1122 :     909969   0.00620632
       1123 :     103297   0.00621185
       1268 :         71   0.00701391
       1269 :          5   0.00701945
       2098 :       3334   0.011605
       2099 :        982   0.0116106
       2244 :       4505   0.0124126
Z differences
          0 :      88603   0
          1 :       4279   2.28007e-007
          2 :       5345   4.56014e-007
          3 :       3967   6.84022e-007
          4 :      12520   9.12029e-007
          5 :       5730   1.14004e-006
          6 :       4621   1.36804e-006
          7 :       3917   1.59605e-006
          8 :      29901   1.82406e-006
          9 :      18615   2.05207e-006
         10 :       4138   2.28007e-006

for more info:

D:\lastools\bin>lasprecision.exe -h
Filter points based on their coordinates.
  -keep_tile 631000 4834000 1000 (ll_x ll_y size)
  -keep_circle 630250.00 4834750.00 100 (x y radius)
  -keep_xy 630000 4834000 631000 4836000 (min_x min_y max_x max_y)
  -drop_xy 630000 4834000 631000 4836000 (min_x min_y max_x max_y)
  -keep_x 631500.50 631501.00 (min_x max_x)
  -drop_x 631500.50 631501.00 (min_x max_x)
  -drop_x_below 630000.50 (min_x)
  -drop_x_above 630500.50 (max_x)
  -keep_y 4834500.25 4834550.25 (min_y max_y)
  -drop_y 4834500.25 4834550.25 (min_y max_y)
  -drop_y_below 4834500.25 (min_y)
  -drop_y_above 4836000.75 (max_y)
  -keep_z 11.125 130.725 (min_z max_z)
  -drop_z 11.125 130.725 (min_z max_z)
  -drop_z_below 11.125 (min_z)
  -drop_z_above 130.725 (max_z)
  -keep_xyz 620000 4830000 100 621000 4831000 200 (min_x min_y min_z max_x max_y max_z)
  -drop_xyz 620000 4830000 100 621000 4831000 200 (min_x min_y min_z max_x max_y max_z)
Filter points based on their return number.
  -first_only -keep_first -drop_first
  -last_only -keep_last -drop_last
  -keep_middle -drop_middle
  -keep_return 1 2 3
  -drop_return 3 4
  -keep_single -drop_single
  -keep_double -drop_double
  -keep_triple -drop_triple
  -keep_quadruple -drop_quadruple
  -keep_quintuple -drop_quintuple
Filter points based on the scanline flags.
  -drop_scan_direction 0
  -scan_direction_change_only
  -edge_of_flight_line_only
Filter points based on their intensity.
  -keep_intensity 20 380
  -drop_intensity_below 20
  -drop_intensity_above 380
  -drop_intensity_between 4000 5000
Filter points based on their classification.
  -keep_class 1 3 7
  -drop_class 4 2
  -drop_synthetic -keep_synthetic
  -drop_keypoint -keep_keypoint
  -drop_withheld -keep_withheld
Filter points based on their user data.
  -keep_user_data 1
  -drop_user_data 255
  -keep_user_data_between 10 20
  -drop_user_data_below 1
  -drop_user_data_above 100
  -drop_user_data_between 10 40
Filter points based on their point source ID.
  -keep_point_source 3
  -keep_point_source_between 2 6
  -drop_point_source 27
  -drop_point_source_below 6
  -drop_point_source_above 15
  -drop_point_source_between 17 21
Filter points based on their scan angle.
  -keep_scan_angle -15 15
  -drop_abs_scan_angle_above 15
  -drop_scan_angle_below -15
  -drop_scan_angle_above 15
  -drop_scan_angle_between -25 -23
Filter points based on their gps time.
  -keep_gps_time 11.125 130.725
  -drop_gps_time_below 11.125
  -drop_gps_time_above 130.725
  -drop_gps_time_between 22.0 48.0
Filter points based on their wavepacket.
  -keep_wavepacket 0
  -drop_wavepacket 3
Filter points with simple thinning.
  -keep_every_nth 2
  -keep_random_fraction 0.1
  -thin_with_grid 1.0
Transform coordinates.
  -translate_x -2.5
  -scale_z 0.3048
  -rotate_xy 15.0 620000 4100000 (angle + origin)
  -translate_xyz 0.5 0.5 0
  -translate_then_scale_y -0.5 1.001
  -clamp_z_below 70.5
  -clamp_z 70.5 72.5
Transform raw xyz integers.
  -translate_raw_z 20
  -translate_raw_xyz 1 1 0
  -clamp_raw_z 500 800
Transform intensity.
  -scale_intensity 2.5
  -translate_intensity 50
  -translate_then_scale_intensity 0.5 3.1
  -clamp_intensity 0 255
  -clamp_intensity_above 255
Transform scan_angle.
  -scale_scan_angle 1.944445
  -translate_scan_angle -5
  -translate_then_scale_scan_angle -0.5 2.1
Change the return number or return count of points.
  -repair_zero_returns
  -set_return_number 1
  -change_return_number_from_to 2 1
  -set_number_of_returns 2
  -change_number_of_returns_from_to 0 2
Modify the classification.
  -set_classification 2
  -change_classification_from_to 2 4
  -classify_z_below_as -5.0 7
  -classify_z_above_as 70.0 7
  -classify_z_between_as 2.0 5.0 4
  -classify_intensity_above_as 200 9
  -classify_intensity_below_as 30 11
Modify the user data.
  -set_user_data 0
  -change_user_data_from_to 23 26
Modify the point source ID.
  -set_point_source 500
  -change_point_source_from_to 1023 1024
  -quantize_Z_into_point_source 200
Transform gps_time.
  -translate_gps_time 40.50
  -adjusted_to_week
  -week_to_adjusted 1671
Transform RGB colors.
  -scale_rgb_down (by 256)
  -scale_rgb_up (by 256)
Supported LAS Inputs
  -i lidar.las
  -i lidar.laz
  -i lidar1.las lidar2.las lidar3.las -merged
  -i *.las - merged
  -i flight0??.laz flight1??.laz
  -i terrasolid.bin
  -i esri.shp
  -i nasa.qi
  -i lidar.txt -iparse xyzti -iskip 2 (on-the-fly from ASCII)
  -i lidar.txt -iparse xyzi -itranslate_intensity 1024
  -lof file_list.txt
  -stdin (pipe from stdin)
  -rescale 0.01 0.01 0.001
  -rescale_xy 0.01 0.01
  -rescale_z 0.01
  -reoffset 600000 4000000 0
Supported LAS Outputs
  -o lidar.las
  -o lidar.laz
  -o xyzta.txt -oparse xyzta (on-the-fly to ASCII)
  -o terrasolid.bin
  -o nasa.qi
  -odir C:\data\ground (specify output directory)
  -odix _classified (specify file name appendix)
  -ocut 2 (cut the last two characters from name)
  -olas -olaz -otxt -obin -oqfit (specify format)
  -stdout (pipe to stdout)
  -nil    (pipe to NULL)
LAStools (by martin@rapidlasso.com) version 140301
usage:
lasprecision -i in.las
lasprecision -i in.las -number 1000000
lasprecision -i in.las -all -gps -lines 50
lasprecision -i in.las -no_x -no_y -no_z -rgb
lasprecision -i in.las -diff_diff
lasprecision -i in.las -o out.las -rescale 0.01 0.01 0.001 -reoffset 300000 2000000 0
lasprecision -i in.las -o out.las -rescale 0.333333333 0.333333333 0.01
lasprecision -h

---------------

if you find bugs let me (martin.isenburg@rapidlasso.com) know.
