SuccessChanges

Summary

  1. Added S3 metadata reader to estimate partitions number basing on a desired partition size in bytes. (details)
  2. Add segment splits (details)
  3. Add segments reads (details)
  4. Fix singleband tile regression case (details)
  5. Fix by segment reads (details)
  6. TODO commit (details)
  7. Make multiband case better (details)
  8. Improve repartition strategy. (details)
  9. Optimise fromSegments funcs (details)
  10. Use crop instead of per segment reads (details)
  11. Use crop function istead of bySegments, added Iterator and Option variant to read by segments, code refactor. (details)
  12. Fix tests to use fromSegmentsIterator, looks like a cfor bug / feature (details)
  13. Remove useless method, rename fromSegments into fromWindows, make most of GeoTiffInfoReader function private (details)
  14. Satisfy Eugene comments (details)
  15. Fix tests (details)
Commit 18c3176d53a16e41ef747c8c900d064f7e0f5afb by Grisha Pomadchin
Added S3 metadata reader to estimate partitions number basing on a desired partition size in bytes.

Signed-off-by: Grisha Pomadchin <gr.pomadchin@gmail.com>
The file was modifieds3/src/main/scala/geotrellis/spark/io/s3/S3Client.scala (diff)
The file was modifieds3/src/main/scala/geotrellis/spark/io/s3/AmazonS3Client.scala (diff)
The file was addeds3/src/main/scala/geotrellis/spark/io/s3/S3GeoTiffMetadataReader.scala
The file was modifieds3/src/main/scala/geotrellis/spark/io/s3/S3GeoTiffRDD.scala (diff)
The file was modifieds3-testkit/src/main/scala/geotrellis/spark/io/s3/testkit/MockS3Client.scala (diff)
The file was addeds3/src/main/scala/geotrellis/spark/io/s3/S3GeoTiffInfoReader.scala
The file was modifiedraster/src/main/scala/geotrellis/raster/io/geotiff/reader/GeoTiffReader.scala (diff)
The file was modifieds3/src/main/scala/geotrellis/spark/io/s3/S3GeoTiffRDD.scala (diff)
Commit 1ed541d44b847c4bd90588b985d3f9aad6bdef85 by Grisha Pomadchin
Add segments reads

Signed-off-by: Grisha Pomadchin <gr.pomadchin@gmail.com>
The file was modifieds3/src/main/scala/geotrellis/spark/io/s3/S3GeoTiffInfoReader.scala (diff)
The file was modifiedraster/src/main/scala/geotrellis/raster/io/geotiff/GeoTiffMultibandTile.scala (diff)
The file was modifiedraster/src/main/scala/geotrellis/raster/io/geotiff/GeoTiffTile.scala (diff)
The file was modifieds3-test/src/test/scala/geotrellis/spark/io/s3/S3GeoTiffRDDSpec.scala (diff)
The file was modifieds3/src/main/scala/geotrellis/spark/io/s3/S3GeoTiffRDD.scala (diff)
The file was modifiedspark/src/main/scala/geotrellis/spark/io/RasterReader.scala (diff)
Commit 3ce61c6390e797b78648c37df6c0b2c6ec3436f8 by Grisha Pomadchin
Fix singleband tile regression case
The file was modifiedspark/src/main/scala/geotrellis/spark/io/RasterReader.scala (diff)
The file was modifiedraster/src/main/scala/geotrellis/raster/GridBounds.scala (diff)
The file was modifieds3/src/main/scala/geotrellis/spark/io/s3/S3GeoTiffInfoReader.scala (diff)
The file was modifieds3/src/main/scala/geotrellis/spark/io/s3/S3GeoTiffRDD.scala (diff)
The file was modifiedraster/src/main/scala/geotrellis/raster/io/geotiff/GeoTiffTile.scala (diff)
The file was modifiedraster/src/main/scala/geotrellis/raster/io/geotiff/reader/GeoTiffReader.scala (diff)
The file was modifiedspark/src/main/scala/geotrellis/spark/io/RasterReader.scala (diff)
The file was modifieds3-test/src/test/scala/geotrellis/spark/io/s3/S3GeoTiffRDDSpec.scala (diff)
The file was modifieds3/src/main/scala/geotrellis/spark/io/s3/S3GeoTiffInfoReader.scala (diff)
The file was modifiedraster/src/main/scala/geotrellis/raster/io/geotiff/GeoTiffMultibandTile.scala (diff)
The file was modifieds3/src/main/scala/geotrellis/spark/io/s3/S3GeoTiffInfoReader.scala (diff)
Commit 79c522dc1088621e6bc7d18d857e19963aeed337 by Grisha Pomadchin
Make multiband case better

Signed-off-by: Grisha Pomadchin <gr.pomadchin@gmail.com>
The file was modifieds3-test/src/test/scala/geotrellis/spark/io/s3/S3GeoTiffRDDSpec.scala (diff)
The file was modifieds3/src/main/scala/geotrellis/spark/io/s3/S3GeoTiffInfoReader.scala (diff)
Commit d95266521da63559c0be4f33f01a402ababe9368 by Grisha Pomadchin
Improve repartition strategy.

Signed-off-by: Grisha Pomadchin <gr.pomadchin@gmail.com>
The file was modifieds3-test/src/test/scala/geotrellis/spark/io/s3/S3GeoTiffRDDSpec.scala (diff)
The file was modifieds3/src/main/scala/geotrellis/spark/io/s3/S3GeoTiffInfoReader.scala (diff)
The file was removeds3/src/main/scala/geotrellis/spark/io/s3/S3GeoTiffMetadataReader.scala
The file was modifiedraster/src/main/scala/geotrellis/raster/io/geotiff/GeoTiffMultibandTile.scala (diff)
Commit a1ee681461e6665dcb11f32a9a407f1a9eb18938 by Grisha Pomadchin
Optimise fromSegments funcs
The file was modifiedraster/src/main/scala/geotrellis/raster/io/geotiff/GeoTiffTile.scala (diff)
The file was modifiedraster/src/main/scala/geotrellis/raster/io/geotiff/GeoTiffMultibandTile.scala (diff)
Commit a8c8464520e2ef825d33d0a7e567958dc5b12b95 by Grisha Pomadchin
Use crop instead of per segment reads
The file was modifieds3/src/main/scala/geotrellis/spark/io/s3/S3GeoTiffRDD.scala (diff)
The file was modifiedspark/src/main/scala/geotrellis/spark/io/RasterReader.scala (diff)
The file was modifieds3/src/main/scala/geotrellis/spark/io/s3/S3GeoTiffInfoReader.scala (diff)
The file was modifiedraster/src/main/scala/geotrellis/raster/io/geotiff/GeoTiffTile.scala (diff)
Commit a537d30cd4bc257c3016597ddf7c98a111e03566 by Grisha Pomadchin
Use crop function istead of bySegments, added Iterator and Option variant to read by segments, code refactor.

Signed-off-by: Grisha Pomadchin <gr.pomadchin@gmail.com>
The file was modifiedspark/src/main/scala/geotrellis/spark/io/RasterReader.scala (diff)
The file was modifiedraster/src/main/scala/geotrellis/raster/GridBounds.scala (diff)
The file was modifiedraster/src/main/scala/geotrellis/raster/io/geotiff/GeoTiffMultibandTile.scala (diff)
The file was modifieds3/src/main/scala/geotrellis/spark/io/s3/S3GeoTiffInfoReader.scala (diff)
The file was addedspark/src/main/scala/geotrellis/spark/io/GeoTiffInfoReader.scala
The file was modifiedspark/src/main/scala/geotrellis/spark/io/package.scala (diff)
The file was modifieds3/src/main/scala/geotrellis/spark/io/s3/S3GeoTiffRDD.scala (diff)
Commit 0f68feddfe1eef8f4847087ad979ab277a8bbb2e by Grisha Pomadchin
Fix tests to use fromSegmentsIterator, looks like a cfor bug / feature

Signed-off-by: Grisha Pomadchin <gr.pomadchin@gmail.com>
The file was modifieds3-test/src/test/scala/geotrellis/spark/io/s3/S3GeoTiffRDDSpec.scala (diff)
The file was modifieds3/src/main/scala/geotrellis/spark/io/s3/S3GeoTiffRDD.scala (diff)
Commit cbd9b608e8142101fb79462f14101ee6830e33bf by Grisha Pomadchin
Remove useless method, rename fromSegments into fromWindows, make most of GeoTiffInfoReader function private

Signed-off-by: Grisha Pomadchin <gr.pomadchin@gmail.com>
The file was modifieds3/src/main/scala/geotrellis/spark/io/s3/S3GeoTiffRDD.scala (diff)
The file was modifiedspark/src/main/scala/geotrellis/spark/io/GeoTiffInfoReader.scala (diff)
The file was modifiedspark/src/main/scala/geotrellis/spark/io/RasterReader.scala (diff)
The file was modifieds3/src/main/scala/geotrellis/spark/io/s3/S3GeoTiffRDD.scala (diff)
The file was modifieds3/src/main/resources/reference.conf (diff)
The file was modifieds3/src/main/scala/geotrellis/spark/io/s3/S3RDDReader.scala (diff)
The file was modifiedspark/src/main/scala/geotrellis/spark/io/GeoTiffInfoReader.scala (diff)
The file was modifieds3/src/main/scala/geotrellis/spark/io/s3/S3GeoTiffInfoReader.scala (diff)
The file was modifieds3-test/src/test/scala/geotrellis/spark/io/s3/S3GeoTiffRDDSpec.scala (diff)