storj/pkg/piecestore/pstore.go

128 lines
3.0 KiB
Go
Raw Normal View History

// Copyright (C) 2018 Storj Labs, Inc.
// See LICENSE for copying information.
package pstore
import (
"context"
"io"
"os"
"path"
"path/filepath"
"github.com/zeebo/errs"
"storj.io/storj/pkg/ranger"
)
Added rpc server (#40) * Added piecestore * gofmt * Added requested changes * Added cli * Removed ranger because I wanted something that can stand alone * Add example of http server using piece store * Changed piecestore code to make it more optial for error handelling * Merged with piecestore * Added missing package * Forgot io import * gofmt * gofmt * Forgot io * Make path by hash exported * updated to simplify again whoops * Updated server to work real good * Forgot ampersand * Updated to match FilePiece * Merged in cam's delete code * Remove unused io * Added RPC code * Give the download request a reader * Removed http server stuff; changed receive stream to say io.reader * Added expiration date to shardInfo * Change all instances of Shard to Piece; change protobuf name; moved client insance to outside functions * added ttl info request * Move scripts to http server pr; added close method for Retrieve api * added rpc server tests for getting piece meta data and retrieval routes * Resolved linter errors, moved to prc server to pkg, updated go.mod to use latest protobuf * Imported cams test * Bump gometalinter deadline * WIP adding tests * added tests for store and delete routes * Add changes as requested by Kaloyan, also cleaned up some code * Get the code actually working whoops * More cleanup * Separating database calls from api.go * need to rename expiration * Added some changes requested by JT * Fix read size * Fixed total amount to read * added tests * Simplify protobuf, add store tests, edited api to handle invalid stores properly, return errors instead of messages * Moved rpc client and server to piece store * Moved piecestore protobuf to the /protos folder * Cleaned up messages * Clean up linter errors * Added missing sqlite import * Add ability to do iterative reads and writes to pstore * Incrementally read data * Fix linter and import errors * Solve linter Error * Change return types * begin test refactor * refactored to implement only 1 db connection, moved SQLite row checking into separate function and removed defer on rows.Close(), fixed os.tempDir in rpc_test.go * Cleaning up tests * Added retrieve tests * refactored delete tests * Deleted old tests * Updated cmd/piecestore to reflect changes to piecestore * Refactored server tests and server/client store code * gofmt * WIP implementing TTL struct * Read 4k at a time when Retrieving * implemented ttl struct * Accidentally removed fpiece dependency? * Double resolve merge conflict * Moved client to the examples since it is an example * Change hash to id in protobuf. TODO: Change client and server to reflect these changes * Update client and protobuf * changed hash to id * Handle eof properly in retrieve * Id -> ID * forgot to change import for client after moving * Moved client and server main to examples * Made changes requested by JT * checkEntries is now private, created currentTime variable for checkEntries, added defer rows.Close() * Print not fatal * Handle overflow when reading from server * added const IDLength * removed types from comments * Add reader/writer for download data from and uploading data to server * goimports and comments * fixed nits, casts, added OK const, DBCleanup now exits program on error * Add stream reader and writer for server * Fix errors * i beforee except after c lol * customizable data dir * Forgot to rename variable * customizable data dir * Handle closing of server stream properly * linter * pass on inserting the same data twice * linter * linter * Do the easy things JT asked for * Handle overflow from reads properly; handle custom db path * Handle overflow for server stream read; TODO Combine server and client stream reads * Allow for TTL of 0 to stay forever * Make Client cleaner for user
2018-06-02 19:14:59 +01:00
// IDLength -- Minimum ID length
const IDLength = 20
// Errors
var (
ArgError = errs.Class("argError")
FSError = errs.Class("fsError")
)
Added rpc server (#40) * Added piecestore * gofmt * Added requested changes * Added cli * Removed ranger because I wanted something that can stand alone * Add example of http server using piece store * Changed piecestore code to make it more optial for error handelling * Merged with piecestore * Added missing package * Forgot io import * gofmt * gofmt * Forgot io * Make path by hash exported * updated to simplify again whoops * Updated server to work real good * Forgot ampersand * Updated to match FilePiece * Merged in cam's delete code * Remove unused io * Added RPC code * Give the download request a reader * Removed http server stuff; changed receive stream to say io.reader * Added expiration date to shardInfo * Change all instances of Shard to Piece; change protobuf name; moved client insance to outside functions * added ttl info request * Move scripts to http server pr; added close method for Retrieve api * added rpc server tests for getting piece meta data and retrieval routes * Resolved linter errors, moved to prc server to pkg, updated go.mod to use latest protobuf * Imported cams test * Bump gometalinter deadline * WIP adding tests * added tests for store and delete routes * Add changes as requested by Kaloyan, also cleaned up some code * Get the code actually working whoops * More cleanup * Separating database calls from api.go * need to rename expiration * Added some changes requested by JT * Fix read size * Fixed total amount to read * added tests * Simplify protobuf, add store tests, edited api to handle invalid stores properly, return errors instead of messages * Moved rpc client and server to piece store * Moved piecestore protobuf to the /protos folder * Cleaned up messages * Clean up linter errors * Added missing sqlite import * Add ability to do iterative reads and writes to pstore * Incrementally read data * Fix linter and import errors * Solve linter Error * Change return types * begin test refactor * refactored to implement only 1 db connection, moved SQLite row checking into separate function and removed defer on rows.Close(), fixed os.tempDir in rpc_test.go * Cleaning up tests * Added retrieve tests * refactored delete tests * Deleted old tests * Updated cmd/piecestore to reflect changes to piecestore * Refactored server tests and server/client store code * gofmt * WIP implementing TTL struct * Read 4k at a time when Retrieving * implemented ttl struct * Accidentally removed fpiece dependency? * Double resolve merge conflict * Moved client to the examples since it is an example * Change hash to id in protobuf. TODO: Change client and server to reflect these changes * Update client and protobuf * changed hash to id * Handle eof properly in retrieve * Id -> ID * forgot to change import for client after moving * Moved client and server main to examples * Made changes requested by JT * checkEntries is now private, created currentTime variable for checkEntries, added defer rows.Close() * Print not fatal * Handle overflow when reading from server * added const IDLength * removed types from comments * Add reader/writer for download data from and uploading data to server * goimports and comments * fixed nits, casts, added OK const, DBCleanup now exits program on error * Add stream reader and writer for server * Fix errors * i beforee except after c lol * customizable data dir * Forgot to rename variable * customizable data dir * Handle closing of server stream properly * linter * pass on inserting the same data twice * linter * linter * Do the easy things JT asked for * Handle overflow from reads properly; handle custom db path * Handle overflow for server stream read; TODO Combine server and client stream reads * Allow for TTL of 0 to stay forever * Make Client cleaner for user
2018-06-02 19:14:59 +01:00
// PathByID creates datapath from id and dir
func PathByID(id, dir string) (string, error) {
if len(id) < IDLength {
return "", ArgError.New("Invalid id length")
}
if dir == "" {
return "", ArgError.New("No path provided")
}
Added rpc server (#40) * Added piecestore * gofmt * Added requested changes * Added cli * Removed ranger because I wanted something that can stand alone * Add example of http server using piece store * Changed piecestore code to make it more optial for error handelling * Merged with piecestore * Added missing package * Forgot io import * gofmt * gofmt * Forgot io * Make path by hash exported * updated to simplify again whoops * Updated server to work real good * Forgot ampersand * Updated to match FilePiece * Merged in cam's delete code * Remove unused io * Added RPC code * Give the download request a reader * Removed http server stuff; changed receive stream to say io.reader * Added expiration date to shardInfo * Change all instances of Shard to Piece; change protobuf name; moved client insance to outside functions * added ttl info request * Move scripts to http server pr; added close method for Retrieve api * added rpc server tests for getting piece meta data and retrieval routes * Resolved linter errors, moved to prc server to pkg, updated go.mod to use latest protobuf * Imported cams test * Bump gometalinter deadline * WIP adding tests * added tests for store and delete routes * Add changes as requested by Kaloyan, also cleaned up some code * Get the code actually working whoops * More cleanup * Separating database calls from api.go * need to rename expiration * Added some changes requested by JT * Fix read size * Fixed total amount to read * added tests * Simplify protobuf, add store tests, edited api to handle invalid stores properly, return errors instead of messages * Moved rpc client and server to piece store * Moved piecestore protobuf to the /protos folder * Cleaned up messages * Clean up linter errors * Added missing sqlite import * Add ability to do iterative reads and writes to pstore * Incrementally read data * Fix linter and import errors * Solve linter Error * Change return types * begin test refactor * refactored to implement only 1 db connection, moved SQLite row checking into separate function and removed defer on rows.Close(), fixed os.tempDir in rpc_test.go * Cleaning up tests * Added retrieve tests * refactored delete tests * Deleted old tests * Updated cmd/piecestore to reflect changes to piecestore * Refactored server tests and server/client store code * gofmt * WIP implementing TTL struct * Read 4k at a time when Retrieving * implemented ttl struct * Accidentally removed fpiece dependency? * Double resolve merge conflict * Moved client to the examples since it is an example * Change hash to id in protobuf. TODO: Change client and server to reflect these changes * Update client and protobuf * changed hash to id * Handle eof properly in retrieve * Id -> ID * forgot to change import for client after moving * Moved client and server main to examples * Made changes requested by JT * checkEntries is now private, created currentTime variable for checkEntries, added defer rows.Close() * Print not fatal * Handle overflow when reading from server * added const IDLength * removed types from comments * Add reader/writer for download data from and uploading data to server * goimports and comments * fixed nits, casts, added OK const, DBCleanup now exits program on error * Add stream reader and writer for server * Fix errors * i beforee except after c lol * customizable data dir * Forgot to rename variable * customizable data dir * Handle closing of server stream properly * linter * pass on inserting the same data twice * linter * linter * Do the easy things JT asked for * Handle overflow from reads properly; handle custom db path * Handle overflow for server stream read; TODO Combine server and client stream reads * Allow for TTL of 0 to stay forever * Make Client cleaner for user
2018-06-02 19:14:59 +01:00
folder1 := string(id[0:2])
folder2 := string(id[2:4])
fileName := string(id[4:])
return path.Join(dir, folder1, folder2, fileName), nil
}
Added rpc server (#40) * Added piecestore * gofmt * Added requested changes * Added cli * Removed ranger because I wanted something that can stand alone * Add example of http server using piece store * Changed piecestore code to make it more optial for error handelling * Merged with piecestore * Added missing package * Forgot io import * gofmt * gofmt * Forgot io * Make path by hash exported * updated to simplify again whoops * Updated server to work real good * Forgot ampersand * Updated to match FilePiece * Merged in cam's delete code * Remove unused io * Added RPC code * Give the download request a reader * Removed http server stuff; changed receive stream to say io.reader * Added expiration date to shardInfo * Change all instances of Shard to Piece; change protobuf name; moved client insance to outside functions * added ttl info request * Move scripts to http server pr; added close method for Retrieve api * added rpc server tests for getting piece meta data and retrieval routes * Resolved linter errors, moved to prc server to pkg, updated go.mod to use latest protobuf * Imported cams test * Bump gometalinter deadline * WIP adding tests * added tests for store and delete routes * Add changes as requested by Kaloyan, also cleaned up some code * Get the code actually working whoops * More cleanup * Separating database calls from api.go * need to rename expiration * Added some changes requested by JT * Fix read size * Fixed total amount to read * added tests * Simplify protobuf, add store tests, edited api to handle invalid stores properly, return errors instead of messages * Moved rpc client and server to piece store * Moved piecestore protobuf to the /protos folder * Cleaned up messages * Clean up linter errors * Added missing sqlite import * Add ability to do iterative reads and writes to pstore * Incrementally read data * Fix linter and import errors * Solve linter Error * Change return types * begin test refactor * refactored to implement only 1 db connection, moved SQLite row checking into separate function and removed defer on rows.Close(), fixed os.tempDir in rpc_test.go * Cleaning up tests * Added retrieve tests * refactored delete tests * Deleted old tests * Updated cmd/piecestore to reflect changes to piecestore * Refactored server tests and server/client store code * gofmt * WIP implementing TTL struct * Read 4k at a time when Retrieving * implemented ttl struct * Accidentally removed fpiece dependency? * Double resolve merge conflict * Moved client to the examples since it is an example * Change hash to id in protobuf. TODO: Change client and server to reflect these changes * Update client and protobuf * changed hash to id * Handle eof properly in retrieve * Id -> ID * forgot to change import for client after moving * Moved client and server main to examples * Made changes requested by JT * checkEntries is now private, created currentTime variable for checkEntries, added defer rows.Close() * Print not fatal * Handle overflow when reading from server * added const IDLength * removed types from comments * Add reader/writer for download data from and uploading data to server * goimports and comments * fixed nits, casts, added OK const, DBCleanup now exits program on error * Add stream reader and writer for server * Fix errors * i beforee except after c lol * customizable data dir * Forgot to rename variable * customizable data dir * Handle closing of server stream properly * linter * pass on inserting the same data twice * linter * linter * Do the easy things JT asked for * Handle overflow from reads properly; handle custom db path * Handle overflow for server stream read; TODO Combine server and client stream reads * Allow for TTL of 0 to stay forever * Make Client cleaner for user
2018-06-02 19:14:59 +01:00
// StoreWriter stores data into piece store in multiple writes
// id is the id of the data to be stored
// dir is the pstore directory containing all other data stored
// returns error if failed and nil if successful
func StoreWriter(id string, dir string) (io.WriteCloser, error) {
Added rpc server (#40) * Added piecestore * gofmt * Added requested changes * Added cli * Removed ranger because I wanted something that can stand alone * Add example of http server using piece store * Changed piecestore code to make it more optial for error handelling * Merged with piecestore * Added missing package * Forgot io import * gofmt * gofmt * Forgot io * Make path by hash exported * updated to simplify again whoops * Updated server to work real good * Forgot ampersand * Updated to match FilePiece * Merged in cam's delete code * Remove unused io * Added RPC code * Give the download request a reader * Removed http server stuff; changed receive stream to say io.reader * Added expiration date to shardInfo * Change all instances of Shard to Piece; change protobuf name; moved client insance to outside functions * added ttl info request * Move scripts to http server pr; added close method for Retrieve api * added rpc server tests for getting piece meta data and retrieval routes * Resolved linter errors, moved to prc server to pkg, updated go.mod to use latest protobuf * Imported cams test * Bump gometalinter deadline * WIP adding tests * added tests for store and delete routes * Add changes as requested by Kaloyan, also cleaned up some code * Get the code actually working whoops * More cleanup * Separating database calls from api.go * need to rename expiration * Added some changes requested by JT * Fix read size * Fixed total amount to read * added tests * Simplify protobuf, add store tests, edited api to handle invalid stores properly, return errors instead of messages * Moved rpc client and server to piece store * Moved piecestore protobuf to the /protos folder * Cleaned up messages * Clean up linter errors * Added missing sqlite import * Add ability to do iterative reads and writes to pstore * Incrementally read data * Fix linter and import errors * Solve linter Error * Change return types * begin test refactor * refactored to implement only 1 db connection, moved SQLite row checking into separate function and removed defer on rows.Close(), fixed os.tempDir in rpc_test.go * Cleaning up tests * Added retrieve tests * refactored delete tests * Deleted old tests * Updated cmd/piecestore to reflect changes to piecestore * Refactored server tests and server/client store code * gofmt * WIP implementing TTL struct * Read 4k at a time when Retrieving * implemented ttl struct * Accidentally removed fpiece dependency? * Double resolve merge conflict * Moved client to the examples since it is an example * Change hash to id in protobuf. TODO: Change client and server to reflect these changes * Update client and protobuf * changed hash to id * Handle eof properly in retrieve * Id -> ID * forgot to change import for client after moving * Moved client and server main to examples * Made changes requested by JT * checkEntries is now private, created currentTime variable for checkEntries, added defer rows.Close() * Print not fatal * Handle overflow when reading from server * added const IDLength * removed types from comments * Add reader/writer for download data from and uploading data to server * goimports and comments * fixed nits, casts, added OK const, DBCleanup now exits program on error * Add stream reader and writer for server * Fix errors * i beforee except after c lol * customizable data dir * Forgot to rename variable * customizable data dir * Handle closing of server stream properly * linter * pass on inserting the same data twice * linter * linter * Do the easy things JT asked for * Handle overflow from reads properly; handle custom db path * Handle overflow for server stream read; TODO Combine server and client stream reads * Allow for TTL of 0 to stay forever * Make Client cleaner for user
2018-06-02 19:14:59 +01:00
dataPath, err := PathByID(id, dir)
if err != nil {
Added rpc server (#40) * Added piecestore * gofmt * Added requested changes * Added cli * Removed ranger because I wanted something that can stand alone * Add example of http server using piece store * Changed piecestore code to make it more optial for error handelling * Merged with piecestore * Added missing package * Forgot io import * gofmt * gofmt * Forgot io * Make path by hash exported * updated to simplify again whoops * Updated server to work real good * Forgot ampersand * Updated to match FilePiece * Merged in cam's delete code * Remove unused io * Added RPC code * Give the download request a reader * Removed http server stuff; changed receive stream to say io.reader * Added expiration date to shardInfo * Change all instances of Shard to Piece; change protobuf name; moved client insance to outside functions * added ttl info request * Move scripts to http server pr; added close method for Retrieve api * added rpc server tests for getting piece meta data and retrieval routes * Resolved linter errors, moved to prc server to pkg, updated go.mod to use latest protobuf * Imported cams test * Bump gometalinter deadline * WIP adding tests * added tests for store and delete routes * Add changes as requested by Kaloyan, also cleaned up some code * Get the code actually working whoops * More cleanup * Separating database calls from api.go * need to rename expiration * Added some changes requested by JT * Fix read size * Fixed total amount to read * added tests * Simplify protobuf, add store tests, edited api to handle invalid stores properly, return errors instead of messages * Moved rpc client and server to piece store * Moved piecestore protobuf to the /protos folder * Cleaned up messages * Clean up linter errors * Added missing sqlite import * Add ability to do iterative reads and writes to pstore * Incrementally read data * Fix linter and import errors * Solve linter Error * Change return types * begin test refactor * refactored to implement only 1 db connection, moved SQLite row checking into separate function and removed defer on rows.Close(), fixed os.tempDir in rpc_test.go * Cleaning up tests * Added retrieve tests * refactored delete tests * Deleted old tests * Updated cmd/piecestore to reflect changes to piecestore * Refactored server tests and server/client store code * gofmt * WIP implementing TTL struct * Read 4k at a time when Retrieving * implemented ttl struct * Accidentally removed fpiece dependency? * Double resolve merge conflict * Moved client to the examples since it is an example * Change hash to id in protobuf. TODO: Change client and server to reflect these changes * Update client and protobuf * changed hash to id * Handle eof properly in retrieve * Id -> ID * forgot to change import for client after moving * Moved client and server main to examples * Made changes requested by JT * checkEntries is now private, created currentTime variable for checkEntries, added defer rows.Close() * Print not fatal * Handle overflow when reading from server * added const IDLength * removed types from comments * Add reader/writer for download data from and uploading data to server * goimports and comments * fixed nits, casts, added OK const, DBCleanup now exits program on error * Add stream reader and writer for server * Fix errors * i beforee except after c lol * customizable data dir * Forgot to rename variable * customizable data dir * Handle closing of server stream properly * linter * pass on inserting the same data twice * linter * linter * Do the easy things JT asked for * Handle overflow from reads properly; handle custom db path * Handle overflow for server stream read; TODO Combine server and client stream reads * Allow for TTL of 0 to stay forever * Make Client cleaner for user
2018-06-02 19:14:59 +01:00
return nil, err
}
// Create directory path on file system
if err = os.MkdirAll(filepath.Dir(dataPath), 0700); err != nil {
Added rpc server (#40) * Added piecestore * gofmt * Added requested changes * Added cli * Removed ranger because I wanted something that can stand alone * Add example of http server using piece store * Changed piecestore code to make it more optial for error handelling * Merged with piecestore * Added missing package * Forgot io import * gofmt * gofmt * Forgot io * Make path by hash exported * updated to simplify again whoops * Updated server to work real good * Forgot ampersand * Updated to match FilePiece * Merged in cam's delete code * Remove unused io * Added RPC code * Give the download request a reader * Removed http server stuff; changed receive stream to say io.reader * Added expiration date to shardInfo * Change all instances of Shard to Piece; change protobuf name; moved client insance to outside functions * added ttl info request * Move scripts to http server pr; added close method for Retrieve api * added rpc server tests for getting piece meta data and retrieval routes * Resolved linter errors, moved to prc server to pkg, updated go.mod to use latest protobuf * Imported cams test * Bump gometalinter deadline * WIP adding tests * added tests for store and delete routes * Add changes as requested by Kaloyan, also cleaned up some code * Get the code actually working whoops * More cleanup * Separating database calls from api.go * need to rename expiration * Added some changes requested by JT * Fix read size * Fixed total amount to read * added tests * Simplify protobuf, add store tests, edited api to handle invalid stores properly, return errors instead of messages * Moved rpc client and server to piece store * Moved piecestore protobuf to the /protos folder * Cleaned up messages * Clean up linter errors * Added missing sqlite import * Add ability to do iterative reads and writes to pstore * Incrementally read data * Fix linter and import errors * Solve linter Error * Change return types * begin test refactor * refactored to implement only 1 db connection, moved SQLite row checking into separate function and removed defer on rows.Close(), fixed os.tempDir in rpc_test.go * Cleaning up tests * Added retrieve tests * refactored delete tests * Deleted old tests * Updated cmd/piecestore to reflect changes to piecestore * Refactored server tests and server/client store code * gofmt * WIP implementing TTL struct * Read 4k at a time when Retrieving * implemented ttl struct * Accidentally removed fpiece dependency? * Double resolve merge conflict * Moved client to the examples since it is an example * Change hash to id in protobuf. TODO: Change client and server to reflect these changes * Update client and protobuf * changed hash to id * Handle eof properly in retrieve * Id -> ID * forgot to change import for client after moving * Moved client and server main to examples * Made changes requested by JT * checkEntries is now private, created currentTime variable for checkEntries, added defer rows.Close() * Print not fatal * Handle overflow when reading from server * added const IDLength * removed types from comments * Add reader/writer for download data from and uploading data to server * goimports and comments * fixed nits, casts, added OK const, DBCleanup now exits program on error * Add stream reader and writer for server * Fix errors * i beforee except after c lol * customizable data dir * Forgot to rename variable * customizable data dir * Handle closing of server stream properly * linter * pass on inserting the same data twice * linter * linter * Do the easy things JT asked for * Handle overflow from reads properly; handle custom db path * Handle overflow for server stream read; TODO Combine server and client stream reads * Allow for TTL of 0 to stay forever * Make Client cleaner for user
2018-06-02 19:14:59 +01:00
return nil, err
}
// Create File on file system
return os.OpenFile(dataPath, os.O_RDWR|os.O_CREATE, 0755)
}
Added rpc server (#40) * Added piecestore * gofmt * Added requested changes * Added cli * Removed ranger because I wanted something that can stand alone * Add example of http server using piece store * Changed piecestore code to make it more optial for error handelling * Merged with piecestore * Added missing package * Forgot io import * gofmt * gofmt * Forgot io * Make path by hash exported * updated to simplify again whoops * Updated server to work real good * Forgot ampersand * Updated to match FilePiece * Merged in cam's delete code * Remove unused io * Added RPC code * Give the download request a reader * Removed http server stuff; changed receive stream to say io.reader * Added expiration date to shardInfo * Change all instances of Shard to Piece; change protobuf name; moved client insance to outside functions * added ttl info request * Move scripts to http server pr; added close method for Retrieve api * added rpc server tests for getting piece meta data and retrieval routes * Resolved linter errors, moved to prc server to pkg, updated go.mod to use latest protobuf * Imported cams test * Bump gometalinter deadline * WIP adding tests * added tests for store and delete routes * Add changes as requested by Kaloyan, also cleaned up some code * Get the code actually working whoops * More cleanup * Separating database calls from api.go * need to rename expiration * Added some changes requested by JT * Fix read size * Fixed total amount to read * added tests * Simplify protobuf, add store tests, edited api to handle invalid stores properly, return errors instead of messages * Moved rpc client and server to piece store * Moved piecestore protobuf to the /protos folder * Cleaned up messages * Clean up linter errors * Added missing sqlite import * Add ability to do iterative reads and writes to pstore * Incrementally read data * Fix linter and import errors * Solve linter Error * Change return types * begin test refactor * refactored to implement only 1 db connection, moved SQLite row checking into separate function and removed defer on rows.Close(), fixed os.tempDir in rpc_test.go * Cleaning up tests * Added retrieve tests * refactored delete tests * Deleted old tests * Updated cmd/piecestore to reflect changes to piecestore * Refactored server tests and server/client store code * gofmt * WIP implementing TTL struct * Read 4k at a time when Retrieving * implemented ttl struct * Accidentally removed fpiece dependency? * Double resolve merge conflict * Moved client to the examples since it is an example * Change hash to id in protobuf. TODO: Change client and server to reflect these changes * Update client and protobuf * changed hash to id * Handle eof properly in retrieve * Id -> ID * forgot to change import for client after moving * Moved client and server main to examples * Made changes requested by JT * checkEntries is now private, created currentTime variable for checkEntries, added defer rows.Close() * Print not fatal * Handle overflow when reading from server * added const IDLength * removed types from comments * Add reader/writer for download data from and uploading data to server * goimports and comments * fixed nits, casts, added OK const, DBCleanup now exits program on error * Add stream reader and writer for server * Fix errors * i beforee except after c lol * customizable data dir * Forgot to rename variable * customizable data dir * Handle closing of server stream properly * linter * pass on inserting the same data twice * linter * linter * Do the easy things JT asked for * Handle overflow from reads properly; handle custom db path * Handle overflow for server stream read; TODO Combine server and client stream reads * Allow for TTL of 0 to stay forever * Make Client cleaner for user
2018-06-02 19:14:59 +01:00
// RetrieveReader retrieves data from pstore directory
// id is the id of the stored data
// offset is the offset of the data that you are reading. Useful for multiple connections to split the data transfer
Added rpc server (#40) * Added piecestore * gofmt * Added requested changes * Added cli * Removed ranger because I wanted something that can stand alone * Add example of http server using piece store * Changed piecestore code to make it more optial for error handelling * Merged with piecestore * Added missing package * Forgot io import * gofmt * gofmt * Forgot io * Make path by hash exported * updated to simplify again whoops * Updated server to work real good * Forgot ampersand * Updated to match FilePiece * Merged in cam's delete code * Remove unused io * Added RPC code * Give the download request a reader * Removed http server stuff; changed receive stream to say io.reader * Added expiration date to shardInfo * Change all instances of Shard to Piece; change protobuf name; moved client insance to outside functions * added ttl info request * Move scripts to http server pr; added close method for Retrieve api * added rpc server tests for getting piece meta data and retrieval routes * Resolved linter errors, moved to prc server to pkg, updated go.mod to use latest protobuf * Imported cams test * Bump gometalinter deadline * WIP adding tests * added tests for store and delete routes * Add changes as requested by Kaloyan, also cleaned up some code * Get the code actually working whoops * More cleanup * Separating database calls from api.go * need to rename expiration * Added some changes requested by JT * Fix read size * Fixed total amount to read * added tests * Simplify protobuf, add store tests, edited api to handle invalid stores properly, return errors instead of messages * Moved rpc client and server to piece store * Moved piecestore protobuf to the /protos folder * Cleaned up messages * Clean up linter errors * Added missing sqlite import * Add ability to do iterative reads and writes to pstore * Incrementally read data * Fix linter and import errors * Solve linter Error * Change return types * begin test refactor * refactored to implement only 1 db connection, moved SQLite row checking into separate function and removed defer on rows.Close(), fixed os.tempDir in rpc_test.go * Cleaning up tests * Added retrieve tests * refactored delete tests * Deleted old tests * Updated cmd/piecestore to reflect changes to piecestore * Refactored server tests and server/client store code * gofmt * WIP implementing TTL struct * Read 4k at a time when Retrieving * implemented ttl struct * Accidentally removed fpiece dependency? * Double resolve merge conflict * Moved client to the examples since it is an example * Change hash to id in protobuf. TODO: Change client and server to reflect these changes * Update client and protobuf * changed hash to id * Handle eof properly in retrieve * Id -> ID * forgot to change import for client after moving * Moved client and server main to examples * Made changes requested by JT * checkEntries is now private, created currentTime variable for checkEntries, added defer rows.Close() * Print not fatal * Handle overflow when reading from server * added const IDLength * removed types from comments * Add reader/writer for download data from and uploading data to server * goimports and comments * fixed nits, casts, added OK const, DBCleanup now exits program on error * Add stream reader and writer for server * Fix errors * i beforee except after c lol * customizable data dir * Forgot to rename variable * customizable data dir * Handle closing of server stream properly * linter * pass on inserting the same data twice * linter * linter * Do the easy things JT asked for * Handle overflow from reads properly; handle custom db path * Handle overflow for server stream read; TODO Combine server and client stream reads * Allow for TTL of 0 to stay forever * Make Client cleaner for user
2018-06-02 19:14:59 +01:00
// length is the amount of data to read. Read all data if -1
// dir is the pstore directory containing all other data stored
// returns error if failed and nil if successful
func RetrieveReader(ctx context.Context, id string, offset int64, length int64, dir string) (io.ReadCloser, error) {
Added rpc server (#40) * Added piecestore * gofmt * Added requested changes * Added cli * Removed ranger because I wanted something that can stand alone * Add example of http server using piece store * Changed piecestore code to make it more optial for error handelling * Merged with piecestore * Added missing package * Forgot io import * gofmt * gofmt * Forgot io * Make path by hash exported * updated to simplify again whoops * Updated server to work real good * Forgot ampersand * Updated to match FilePiece * Merged in cam's delete code * Remove unused io * Added RPC code * Give the download request a reader * Removed http server stuff; changed receive stream to say io.reader * Added expiration date to shardInfo * Change all instances of Shard to Piece; change protobuf name; moved client insance to outside functions * added ttl info request * Move scripts to http server pr; added close method for Retrieve api * added rpc server tests for getting piece meta data and retrieval routes * Resolved linter errors, moved to prc server to pkg, updated go.mod to use latest protobuf * Imported cams test * Bump gometalinter deadline * WIP adding tests * added tests for store and delete routes * Add changes as requested by Kaloyan, also cleaned up some code * Get the code actually working whoops * More cleanup * Separating database calls from api.go * need to rename expiration * Added some changes requested by JT * Fix read size * Fixed total amount to read * added tests * Simplify protobuf, add store tests, edited api to handle invalid stores properly, return errors instead of messages * Moved rpc client and server to piece store * Moved piecestore protobuf to the /protos folder * Cleaned up messages * Clean up linter errors * Added missing sqlite import * Add ability to do iterative reads and writes to pstore * Incrementally read data * Fix linter and import errors * Solve linter Error * Change return types * begin test refactor * refactored to implement only 1 db connection, moved SQLite row checking into separate function and removed defer on rows.Close(), fixed os.tempDir in rpc_test.go * Cleaning up tests * Added retrieve tests * refactored delete tests * Deleted old tests * Updated cmd/piecestore to reflect changes to piecestore * Refactored server tests and server/client store code * gofmt * WIP implementing TTL struct * Read 4k at a time when Retrieving * implemented ttl struct * Accidentally removed fpiece dependency? * Double resolve merge conflict * Moved client to the examples since it is an example * Change hash to id in protobuf. TODO: Change client and server to reflect these changes * Update client and protobuf * changed hash to id * Handle eof properly in retrieve * Id -> ID * forgot to change import for client after moving * Moved client and server main to examples * Made changes requested by JT * checkEntries is now private, created currentTime variable for checkEntries, added defer rows.Close() * Print not fatal * Handle overflow when reading from server * added const IDLength * removed types from comments * Add reader/writer for download data from and uploading data to server * goimports and comments * fixed nits, casts, added OK const, DBCleanup now exits program on error * Add stream reader and writer for server * Fix errors * i beforee except after c lol * customizable data dir * Forgot to rename variable * customizable data dir * Handle closing of server stream properly * linter * pass on inserting the same data twice * linter * linter * Do the easy things JT asked for * Handle overflow from reads properly; handle custom db path * Handle overflow for server stream read; TODO Combine server and client stream reads * Allow for TTL of 0 to stay forever * Make Client cleaner for user
2018-06-02 19:14:59 +01:00
dataPath, err := PathByID(id, dir)
if err != nil {
Added rpc server (#40) * Added piecestore * gofmt * Added requested changes * Added cli * Removed ranger because I wanted something that can stand alone * Add example of http server using piece store * Changed piecestore code to make it more optial for error handelling * Merged with piecestore * Added missing package * Forgot io import * gofmt * gofmt * Forgot io * Make path by hash exported * updated to simplify again whoops * Updated server to work real good * Forgot ampersand * Updated to match FilePiece * Merged in cam's delete code * Remove unused io * Added RPC code * Give the download request a reader * Removed http server stuff; changed receive stream to say io.reader * Added expiration date to shardInfo * Change all instances of Shard to Piece; change protobuf name; moved client insance to outside functions * added ttl info request * Move scripts to http server pr; added close method for Retrieve api * added rpc server tests for getting piece meta data and retrieval routes * Resolved linter errors, moved to prc server to pkg, updated go.mod to use latest protobuf * Imported cams test * Bump gometalinter deadline * WIP adding tests * added tests for store and delete routes * Add changes as requested by Kaloyan, also cleaned up some code * Get the code actually working whoops * More cleanup * Separating database calls from api.go * need to rename expiration * Added some changes requested by JT * Fix read size * Fixed total amount to read * added tests * Simplify protobuf, add store tests, edited api to handle invalid stores properly, return errors instead of messages * Moved rpc client and server to piece store * Moved piecestore protobuf to the /protos folder * Cleaned up messages * Clean up linter errors * Added missing sqlite import * Add ability to do iterative reads and writes to pstore * Incrementally read data * Fix linter and import errors * Solve linter Error * Change return types * begin test refactor * refactored to implement only 1 db connection, moved SQLite row checking into separate function and removed defer on rows.Close(), fixed os.tempDir in rpc_test.go * Cleaning up tests * Added retrieve tests * refactored delete tests * Deleted old tests * Updated cmd/piecestore to reflect changes to piecestore * Refactored server tests and server/client store code * gofmt * WIP implementing TTL struct * Read 4k at a time when Retrieving * implemented ttl struct * Accidentally removed fpiece dependency? * Double resolve merge conflict * Moved client to the examples since it is an example * Change hash to id in protobuf. TODO: Change client and server to reflect these changes * Update client and protobuf * changed hash to id * Handle eof properly in retrieve * Id -> ID * forgot to change import for client after moving * Moved client and server main to examples * Made changes requested by JT * checkEntries is now private, created currentTime variable for checkEntries, added defer rows.Close() * Print not fatal * Handle overflow when reading from server * added const IDLength * removed types from comments * Add reader/writer for download data from and uploading data to server * goimports and comments * fixed nits, casts, added OK const, DBCleanup now exits program on error * Add stream reader and writer for server * Fix errors * i beforee except after c lol * customizable data dir * Forgot to rename variable * customizable data dir * Handle closing of server stream properly * linter * pass on inserting the same data twice * linter * linter * Do the easy things JT asked for * Handle overflow from reads properly; handle custom db path * Handle overflow for server stream read; TODO Combine server and client stream reads * Allow for TTL of 0 to stay forever * Make Client cleaner for user
2018-06-02 19:14:59 +01:00
return nil, err
}
fileInfo, err := os.Stat(dataPath)
if err != nil {
Added rpc server (#40) * Added piecestore * gofmt * Added requested changes * Added cli * Removed ranger because I wanted something that can stand alone * Add example of http server using piece store * Changed piecestore code to make it more optial for error handelling * Merged with piecestore * Added missing package * Forgot io import * gofmt * gofmt * Forgot io * Make path by hash exported * updated to simplify again whoops * Updated server to work real good * Forgot ampersand * Updated to match FilePiece * Merged in cam's delete code * Remove unused io * Added RPC code * Give the download request a reader * Removed http server stuff; changed receive stream to say io.reader * Added expiration date to shardInfo * Change all instances of Shard to Piece; change protobuf name; moved client insance to outside functions * added ttl info request * Move scripts to http server pr; added close method for Retrieve api * added rpc server tests for getting piece meta data and retrieval routes * Resolved linter errors, moved to prc server to pkg, updated go.mod to use latest protobuf * Imported cams test * Bump gometalinter deadline * WIP adding tests * added tests for store and delete routes * Add changes as requested by Kaloyan, also cleaned up some code * Get the code actually working whoops * More cleanup * Separating database calls from api.go * need to rename expiration * Added some changes requested by JT * Fix read size * Fixed total amount to read * added tests * Simplify protobuf, add store tests, edited api to handle invalid stores properly, return errors instead of messages * Moved rpc client and server to piece store * Moved piecestore protobuf to the /protos folder * Cleaned up messages * Clean up linter errors * Added missing sqlite import * Add ability to do iterative reads and writes to pstore * Incrementally read data * Fix linter and import errors * Solve linter Error * Change return types * begin test refactor * refactored to implement only 1 db connection, moved SQLite row checking into separate function and removed defer on rows.Close(), fixed os.tempDir in rpc_test.go * Cleaning up tests * Added retrieve tests * refactored delete tests * Deleted old tests * Updated cmd/piecestore to reflect changes to piecestore * Refactored server tests and server/client store code * gofmt * WIP implementing TTL struct * Read 4k at a time when Retrieving * implemented ttl struct * Accidentally removed fpiece dependency? * Double resolve merge conflict * Moved client to the examples since it is an example * Change hash to id in protobuf. TODO: Change client and server to reflect these changes * Update client and protobuf * changed hash to id * Handle eof properly in retrieve * Id -> ID * forgot to change import for client after moving * Moved client and server main to examples * Made changes requested by JT * checkEntries is now private, created currentTime variable for checkEntries, added defer rows.Close() * Print not fatal * Handle overflow when reading from server * added const IDLength * removed types from comments * Add reader/writer for download data from and uploading data to server * goimports and comments * fixed nits, casts, added OK const, DBCleanup now exits program on error * Add stream reader and writer for server * Fix errors * i beforee except after c lol * customizable data dir * Forgot to rename variable * customizable data dir * Handle closing of server stream properly * linter * pass on inserting the same data twice * linter * linter * Do the easy things JT asked for * Handle overflow from reads properly; handle custom db path * Handle overflow for server stream read; TODO Combine server and client stream reads * Allow for TTL of 0 to stay forever * Make Client cleaner for user
2018-06-02 19:14:59 +01:00
return nil, err
}
// If offset is greater than file size return
if offset >= fileInfo.Size() || offset < 0 {
return nil, ArgError.New("Invalid offset: %v", offset)
}
// If length less than 0 read the entire file
if length <= -1 {
length = fileInfo.Size()
}
// If trying to read past the end of the file, just read to the end
if fileInfo.Size() < offset+length {
length = fileInfo.Size() - offset
}
dataFile, err := os.OpenFile(dataPath, os.O_RDONLY, 0755)
if err != nil {
Added rpc server (#40) * Added piecestore * gofmt * Added requested changes * Added cli * Removed ranger because I wanted something that can stand alone * Add example of http server using piece store * Changed piecestore code to make it more optial for error handelling * Merged with piecestore * Added missing package * Forgot io import * gofmt * gofmt * Forgot io * Make path by hash exported * updated to simplify again whoops * Updated server to work real good * Forgot ampersand * Updated to match FilePiece * Merged in cam's delete code * Remove unused io * Added RPC code * Give the download request a reader * Removed http server stuff; changed receive stream to say io.reader * Added expiration date to shardInfo * Change all instances of Shard to Piece; change protobuf name; moved client insance to outside functions * added ttl info request * Move scripts to http server pr; added close method for Retrieve api * added rpc server tests for getting piece meta data and retrieval routes * Resolved linter errors, moved to prc server to pkg, updated go.mod to use latest protobuf * Imported cams test * Bump gometalinter deadline * WIP adding tests * added tests for store and delete routes * Add changes as requested by Kaloyan, also cleaned up some code * Get the code actually working whoops * More cleanup * Separating database calls from api.go * need to rename expiration * Added some changes requested by JT * Fix read size * Fixed total amount to read * added tests * Simplify protobuf, add store tests, edited api to handle invalid stores properly, return errors instead of messages * Moved rpc client and server to piece store * Moved piecestore protobuf to the /protos folder * Cleaned up messages * Clean up linter errors * Added missing sqlite import * Add ability to do iterative reads and writes to pstore * Incrementally read data * Fix linter and import errors * Solve linter Error * Change return types * begin test refactor * refactored to implement only 1 db connection, moved SQLite row checking into separate function and removed defer on rows.Close(), fixed os.tempDir in rpc_test.go * Cleaning up tests * Added retrieve tests * refactored delete tests * Deleted old tests * Updated cmd/piecestore to reflect changes to piecestore * Refactored server tests and server/client store code * gofmt * WIP implementing TTL struct * Read 4k at a time when Retrieving * implemented ttl struct * Accidentally removed fpiece dependency? * Double resolve merge conflict * Moved client to the examples since it is an example * Change hash to id in protobuf. TODO: Change client and server to reflect these changes * Update client and protobuf * changed hash to id * Handle eof properly in retrieve * Id -> ID * forgot to change import for client after moving * Moved client and server main to examples * Made changes requested by JT * checkEntries is now private, created currentTime variable for checkEntries, added defer rows.Close() * Print not fatal * Handle overflow when reading from server * added const IDLength * removed types from comments * Add reader/writer for download data from and uploading data to server * goimports and comments * fixed nits, casts, added OK const, DBCleanup now exits program on error * Add stream reader and writer for server * Fix errors * i beforee except after c lol * customizable data dir * Forgot to rename variable * customizable data dir * Handle closing of server stream properly * linter * pass on inserting the same data twice * linter * linter * Do the easy things JT asked for * Handle overflow from reads properly; handle custom db path * Handle overflow for server stream read; TODO Combine server and client stream reads * Allow for TTL of 0 to stay forever * Make Client cleaner for user
2018-06-02 19:14:59 +01:00
return nil, err
}
// Created a section reader so that we can concurrently retrieve the same file.
rr, err := ranger.FileHandleRanger(dataFile)
if err != nil {
return nil, err
}
return rr.Range(ctx, offset, length)
}
Added rpc server (#40) * Added piecestore * gofmt * Added requested changes * Added cli * Removed ranger because I wanted something that can stand alone * Add example of http server using piece store * Changed piecestore code to make it more optial for error handelling * Merged with piecestore * Added missing package * Forgot io import * gofmt * gofmt * Forgot io * Make path by hash exported * updated to simplify again whoops * Updated server to work real good * Forgot ampersand * Updated to match FilePiece * Merged in cam's delete code * Remove unused io * Added RPC code * Give the download request a reader * Removed http server stuff; changed receive stream to say io.reader * Added expiration date to shardInfo * Change all instances of Shard to Piece; change protobuf name; moved client insance to outside functions * added ttl info request * Move scripts to http server pr; added close method for Retrieve api * added rpc server tests for getting piece meta data and retrieval routes * Resolved linter errors, moved to prc server to pkg, updated go.mod to use latest protobuf * Imported cams test * Bump gometalinter deadline * WIP adding tests * added tests for store and delete routes * Add changes as requested by Kaloyan, also cleaned up some code * Get the code actually working whoops * More cleanup * Separating database calls from api.go * need to rename expiration * Added some changes requested by JT * Fix read size * Fixed total amount to read * added tests * Simplify protobuf, add store tests, edited api to handle invalid stores properly, return errors instead of messages * Moved rpc client and server to piece store * Moved piecestore protobuf to the /protos folder * Cleaned up messages * Clean up linter errors * Added missing sqlite import * Add ability to do iterative reads and writes to pstore * Incrementally read data * Fix linter and import errors * Solve linter Error * Change return types * begin test refactor * refactored to implement only 1 db connection, moved SQLite row checking into separate function and removed defer on rows.Close(), fixed os.tempDir in rpc_test.go * Cleaning up tests * Added retrieve tests * refactored delete tests * Deleted old tests * Updated cmd/piecestore to reflect changes to piecestore * Refactored server tests and server/client store code * gofmt * WIP implementing TTL struct * Read 4k at a time when Retrieving * implemented ttl struct * Accidentally removed fpiece dependency? * Double resolve merge conflict * Moved client to the examples since it is an example * Change hash to id in protobuf. TODO: Change client and server to reflect these changes * Update client and protobuf * changed hash to id * Handle eof properly in retrieve * Id -> ID * forgot to change import for client after moving * Moved client and server main to examples * Made changes requested by JT * checkEntries is now private, created currentTime variable for checkEntries, added defer rows.Close() * Print not fatal * Handle overflow when reading from server * added const IDLength * removed types from comments * Add reader/writer for download data from and uploading data to server * goimports and comments * fixed nits, casts, added OK const, DBCleanup now exits program on error * Add stream reader and writer for server * Fix errors * i beforee except after c lol * customizable data dir * Forgot to rename variable * customizable data dir * Handle closing of server stream properly * linter * pass on inserting the same data twice * linter * linter * Do the easy things JT asked for * Handle overflow from reads properly; handle custom db path * Handle overflow for server stream read; TODO Combine server and client stream reads * Allow for TTL of 0 to stay forever * Make Client cleaner for user
2018-06-02 19:14:59 +01:00
// Delete deletes data from farmer
// id is the id of the data to be stored
// dir is the pstore directory containing all other data stored
// returns error if failed and nil if successful
func Delete(id string, dir string) error {
dataPath, err := PathByID(id, dir)
if err != nil {
return err
}
if _, err = os.Stat(dataPath); os.IsNotExist(err) {
return nil
}
if err = os.Remove(dataPath); err != nil {
return err
}
return nil
}