-
Notifications
You must be signed in to change notification settings - Fork 2
Use Cases
This document outlines the different ways you should configure the input CSV in order to produce the desired output. For what each csv setting is, please see CSV Parameters
These are the general settings which apply to every form of time streaming you can do.
The input directory is denoted as SOURCE in the csv file. This will be the directory where we will look for images.
The output directory is denoted as DESTINATION in the csv file. This is where we will output our time streams to. It is best to think of the output as four parts. The output directory, then the time stream format, then the year / month / day folders, then finally the file name itself. With the exception of the year / month / day folders, each of these is fully customizable.
The Archive directory is denoted as ARCHIVE_DEST in the csv file. If the Archive mode is selected, then this will store all of the unprocessed images. This will be done before any resizing, or rotating.
These fields are useful if you only wish to look at images between specific dates, or times. All images which fall outside of EXPT_START and EXPT_END date range, will not be processed at all. The same can be said for times entered in SUNRISE and SUNSET.
Therefore, if we had the follwing values:
- Sunrise: 11:01
- Sunset: 11:03
- Expt_Start: 06/10/2015
- Expt_End: 06/10/2015
and the following 5 image times:
- 11:02 05/10/2015
- 11:00 06/10/2015
- 11:02 06/10/2015
- 11:04 06/10/2015
- 11:02 07/10/2015
Then only Image 3 would be processed.
It is also possible to filter images based on a string value. This is entered into the FN_PARSE value in the csv. This option is useful when images are not stored according to their camera, but rather by date, or folders based on when they were copied from a hard drive. This value works by ONLY processing images which have the string value entered in FN_PARSE somewhere in their Global file path.
If you had the following folder structure storing two cameras (Overstory_Nadir, and Understory Nadir):
-- Images -- Field_Photos -- 2015_06_06 -- Overstory_Nadir -- <Images>
\ \ Understory_Nadir -- <Images>
\
\ 2015_06_08 -- Overstory_Nadir -- <Images>
\- Understory_Nadir -- <Images>
Rather than having four values in the CSV, you can simply have one for each camera using the FN_PARSE parameter. The following values would be needed in overstory CSV entry:
- SOURCE: Images/Field_Photos
- FN_PARSE: Overstory
This would ignore all images in the Understory folders, as it doesn't have the FN_Parse value in their file path. However, if you had a FN_PARSE value of "erstory" Then all images would be used, as "erstory" is in all the global file path.
Sometimes its useful to shift all times and dates of images. This can be useful when images are recorded in UTC, but you wish to display them in local time. The csv parameter is called TIMESHIFT Simply enter the number of hours you wish to offset by. So if you wish to shift everything forward 2 hours, enter 2.
If you wanted to shift everything backwards by one day, then simply enter -24 as the value.
We can tell the program to only specifically time stream a particular image type. This is done by the IMAGE_TYPE csv parameter.
Each image type you want to process needs to be separated by a ~ character. So if you wanted to process only jpg images, then your IMAGE_TYPE value would be jpg. However if you wanted to process jpg and raw images, then the value should be jpg~raw
Sometimes images don't have the date taken stored in the image metadata, instead storing it in their filename. In this case, you will need to enter a FILENAME_DATE_MASK csv parameter.
This is a string value which stores how to get the date from the filename. Character meanings can be found here.
If the string was
20120901_122732 meaning the 12:27:32, 1st of September, 2012, then your FILENAME_DATE_MASK would be
%Y%m%d_%H%M%S
This section describes the different ways you can customise the output of the program.
When dealing with the output, its best to think of it as four separate parts. Destination, Timestream Strucutre, Date folders, and Filename.
/images/Timestreams/BVZ-0000/Original/Location-C01~fullres-orig/2015/2015_11/2015_11_01/BVZ00000-Location-C01~fullres-orig_2015_11_01_12_12_12.jpg
| DESTINATION | TS_STRUCTURE |COMPULSORY |COMPULSORY Date folders| FN_STRUCTURE | COMPULSORY |
The destination path is set by the DESTINATION csv parameter, however this will not be generated by the program. The compulsary parts are not editable. These are required by the timestream viewers. However, the TS_STRUCTURE and FN_STRUCTURE parts are editable.
This field determines everything from the destination, up to the ~{res}-{step} fields. This field CAN have folder separators, but CANNOT have underscores. This field is also a Replacable Parameters field. This means that fields in capitals will be replaced with their actual value.
If you wish to have a separate folder for original images, and resized images, then you will need to enter {folder} where you wish this split to occur.
The default value (or value entered if TS_STRUCTURE is blank) for TS_STRUCTURE in the CSV is EXPT/LOCATION/{folder}/EXPT-LOCATION-CCAM_NUM-DATASETID
No matter what you enter for this field, it will ALWAYS be appended with ~{res}-{step} automatically. This is not optional.
The FN_STRUCTURE csv parameter determines the editable portion of the filename. This is a Replacable Parameters field. This value determines everything up to the ~{res}-{step} portion. This portion is Always added.
The FN_STRUCTURE CANNOT contain any folder separators, or underscores.
This portion describes the methods used by the program, and the different times you would use these.
This is useful for when you wish to timestream images without deleting them from their original directory. This can be when you wish to experiment on a copy of the data, and then re-timestream it.
This can be achieved by setting the METHOD csv parameter to copy.
This is useful for when you wish to timestream images, and not continue storing the images in the source directory. This will timestream the images, and remove them from the source directory.
This can be achieved by setting the METHOD csv parameter to move
This is useful for when you wish to timestream the images, remove them from their original source directory, but keep a copy backed up somewhere. This is useful if you have a designated "Backup" folder / hard drive / server location.
This can be achieved by setting the METHOD csv paramater to archive.
This is a useful tool if you have a large number of full size timestreams and now wish to downsize all of the images. This can be useful for displaying images on the web. Note: This will not work on raw images.
If you have a fullres timestream of jpg images located at :
/images/timestreams/BVZ-0000/Understory-C01/original/BVZ-0000-Understory-C01~fullres-orig/
and you wanted to resize these images to 1920, and maintain a similar folder structure, then you would need the following CSV paramaters:
- SOURCE: /images/timestreams/BVZ-0000/Understory-C01/original/BVZ-0000-Understory-C01~fullres-orig/
- DESTINATION: /images/timestreams
- EXPT: BVZ-0000
- CAM_NUM : 1
- MODE: resize
- INTERVAL: 1
- EXPT_START: 1900_01_01
- EXPT_END: now
- IMAGE_TYPES: jpg
- LOCATION: Understory
- RESOLUTIONS:
original1920 - TS_STRUCTURE: EXPT/LOCATION-CCAM_NUM/{folder}/EXPT-LOCATION-CCAM_NUM
Sometimes Images are stored in an orientation which is not correct. Therefore, its useful to be able to rotate these images as you process them.
This is where the ORIENTATION field comes in handy. You need to enter the number of degrees you wish to rotate in a counter clockwise direction.
So if your images are upside down, you would need to enter "180". Or, if they need to be rotated Clockwise, enter "270".
Please note a couple of things when rotating:
- Rotating is a slow process. This is one of the more computationally intensive tasks. Expect processing time to slow down considerably.
- When rotating AND resizing images, the leading measurement that resized images will be stored by will be their height. So if you resize and rotate images so they are stored as 1080x1920, then the image name will be ..~1920-orig
Sometimes its useful to perform actions other than time streaming on already established time streams. These include things like: Listing images which fall outside a specific date range, or actually delete these. Another useful thing is to check for any gaps in the timestream.
To delete or list images by time and date, we first need to gain an understanding of the parameters we need to provide:
- USE: Whether or not we want to use this timestream (1 OR 0)
- TIMESTREAM_NAME: A Useful field for identifying this timestream in print statements
- ROOT_PATH: The Root Path of this particular timestream. The directory you want to look for images in.
- DELETE_DEST : If deleting images, then this stores where we move the images which we want to delete
- EXPT_END : The End cut off date
- EXPT_START : The start cut off date
- START_TIME : The start cut off time
- END_TIME : The end cut off time
- IMAGE_TYPES : Image types to process
- DATE_MASK : Date mask (If Required) for getting dates of images
This is selected when you want to filter out all images which fall outside the specified start and end date, OR, the start and end time. This is useful if you have photographs before / after your experiment ends, or if you have photographs before sunrise, or after sunset.
This mode is selected when you just want to move everything which fits the above category to a specified location. This can be your backup location if you don't want to delete the images. If you do wish to delete the images, then simply delete this location after processing. When Deleting Images by date and time, its important to note images are not actually deleted, but instead moved.
To operate this mode:
- Fill out your CSV as required.
- Run the process with the command :
python DeleteImagesByTime.py -l <log file location> -d -c <location of config csv> -t <number of threads>
This mode is selected when you simply want to get a list of all the images which would be moved by the above script. This is useful for checking to see which time streams have a lot of data which is not necessary, and you can then communicate to their respective owners that they may need to clean up their data.
This mode will generate a JSON for each experiment in the specified delete dest folder. This will contain a list of all the images which are outside the specified ranges.
To operate this mode:
- Fill out your CSV as required.
- Run the process with the command :
python ListImagesByTime.py -l <log file location> -d -c <location of config csv> -t <number of threads>
This mode is useful for checking multiple timestreams at once for missing images. Also this is used for graphing each timestreams missing images over time, and for listing all of the timestreams percentage of missing images.
To run this mode, you will only need to provide two things: a input directory, and an output directory. The input directory is the parent directory in which all of your time streams (That you wish to check) reside. The output directory is where you want the summary data (graph and csv) to be written to. The individual graph and csv for each timestream will be inside its parent folder.