dlight is a program that checks items in rss feeds and download those
items/links that are matched against a set of rules.
1350330dd2
A new datastructure is about to take dlhist place. dlhist is currently implemented as a mixture of an "process cache" that should record what rss items has been processed (that is why the url is used as a unique identifier), but right now it only stores an url if it has been downloaded. A new datastructure that should be "download history" shall be implemented, that will keep track of what title and where it has been downloaded to. this will make it possible to only download an rss title to a location once. Splitting this datastructure into two separated structures is trivial as a "process cache" will threat URL's as a unique identifier and a "download history" will threat the title in an rss item as a unique identifier (and also track it's destinations). This commit does not change any functionality, I just rename this to keep the "dlhist" prefix and source files clear for when implementing the real dlhist. |
||
|---|---|---|
| .gitignore | ||
| buffer.c | ||
| buffer.h | ||
| cconf.c | ||
| cconf.h | ||
| compile.c | ||
| config.sample | ||
| COPYING | ||
| dlight.c | ||
| env.c | ||
| env.h | ||
| error.c | ||
| error.h | ||
| filter-check.c | ||
| filter.c | ||
| filter.h | ||
| http.c | ||
| http.h | ||
| lockfile.c | ||
| lockfile.h | ||
| Makefile | ||
| proc-cache.c | ||
| proc-cache.h | ||
| read-config.c | ||
| README | ||
| rss.c | ||
| rss.h | ||
Dlight - automatic feed downloader -------------------------------------- dlight is a program that checks items in rss feeds and download those items/links that are matched against a set of rules. What this does different than other programs of this type is that configuration of the program should be easy and flexible. Not forcing users to write and maintain large lists of raw regular expressions. The best way to use dlight is by using time-based scheduling like cron. -------------------------------------------------------- # Make cron execute dlight every 15 minutes */15 * * * * /path/to/dlight >> /path/to/logs/dlight.log -------------------------------------------------------- dlight is divided into 3 major components: the dlight program, Configuration files and Compiler. * dlight The actual program that checks feeds and download items. The configuration data is read from the compiled config file "~/.dlight/config". The program first fetches the rss file (target), walks through all items applying all filters associated whit the current target. And if one matches, that item will be downloaded to the destination associated with the target. it does this for all rss files (targets) in the config. * Configuration Files A set of human-readable configuration files that the user should configure dlight through. This is where users defines their targets, destinations, filters and other types of information. (currently there is only one file with a similar structure that the compiled format use). * Compiler An compiler is provided that compiles configuration files down to a binary config file used by dlight, one can think of this step as publish/update the configuration used by the program. This design is used for two main reasons. One, if you edit your configuration structure and dlight would be executed by for example cron. if dlight would read directly from those files, it is possible that the configuration files are not in a desired state and making dlight do some weird things. The second reason is that processing all those files everytime dlight is invoked can be quite slow, the compiled format is designed to provide fast I/O reads. Also by using a source -> compiler -> output design, errors can be caught in the configuration files when the user invokes the compiler. which is a more natural way of notify the user on such errors then to have dlight abort and log the error. Because the program is supposed to be executed in an automatic manner, the error will not be seen right away.