diff --git a/.gitignore b/.gitignore index 52065156e3ac726b62fa0dc22c64f03ee791f2b6..566cf9944d46563554e6016cf46ef9669a369458 100644 --- a/.gitignore +++ b/.gitignore @@ -1,16 +1,16 @@ -# For checking that no file has been unduly ignored, run: -# $ git ls-files -i --exclude-per-directory=.gitignore -# Any file that is displayed should be removed from the ignore list -# (possibly by an exclusion rule beginning with an exclamation mark) - -# Generic ignore rules -*~ -*.o -*.a -*.fig -\#*\# -*.mat -*.asv - -# Test directory -m-unit-tests +# For checking that no file has been unduly ignored, run: +# $ git ls-files -i --exclude-per-directory=.gitignore +# Any file that is displayed should be removed from the ignore list +# (possibly by an exclusion rule beginning with an exclamation mark) + +# Generic ignore rules +*~ +*.o +*.a +*.fig +\#*\# +*.mat +*.asv + +# Test directory +m-unit-tests diff --git a/README.md b/README.md index 1443471f823188122a3779176c7983880b8d9716..215fddb2cc501fd834c6b6e80491404b5e6fee95 100644 --- a/README.md +++ b/README.md @@ -1,179 +1,179 @@ -[](https://git.dynare.org/DoraK/mdbnomics/commits/master) - -This MATLAB/Octave toolbox comes with routines to access DBnomics time series from MATLAB. - -The package is compatible with MATLAB 2019b and following versions, and (almost compatible with) -the latest Octave version. - -## Installation - -The toolbox can be installed by cloning the Git repository: - - ~$ git clone https://git.dynare.org/DoraK/mdbnomics.git - -or downloading a zip archive: - - ~$ wget https://git.dynare.org/DoraK/mdbnomics/-/archive/master/mdbnomics-master.zip - ~$ unsip mdbnomics-master.zip - -$ mv mdbnomics-master mdbnomics - -## Usage - -Add the `mdbnomics/src` folder to the MATLAB/Octave path, and run the following command (on MATLAB/Octave) prompt: - - >> initialize_mdbnomics() - -which, depending on your system, will add the necessary subfolders to -the MATLAB/Octave path. - -You are then ready to go. A full documentation will come soon. - -## Capabilities - -### Fetch one time series by ID -First, let's assume that we know which series we want to download. -A series identifier (ID) is defined by three values, formatted like this: `provider_code/dataset_code/series_code`. -The `fetch_series` function is used to construct the cell array. -For example, to fetch the time series `EA19.1.0.0.0.ZUTN` from the -[\"Unemployment rate\" [ZUTN] dataset](https://db.nomics.world/AMECO/ZUTN) -belonging to the [AMECO provider](https://db.nomics.world/AMECO). - -Example: - - >> df_id = fetch_series('series_ids', 'AMECO/ZUTN/EA19.1.0.0.0.ZUTN'); - -The returned data is stored in the `df_id` variable. Its type is a cell array. To display the first 3 rows of the array -(including the column headers), type: - - >> df_id(1:4,:) - - 6×16 cell array - - Columns 1 through 5 - - {'x_frequency'} {'provider_code'} {'dataset_code'} {'dataset_name' } {'series_code' } - {'annual' } {'AMECO' } {'ZUTN' } {'Unemployment ra…'} {'EA19.1.0.0.0.ZUTN'} - {'annual' } {'AMECO' } {'ZUTN' } {'Unemployment ra…'} {'EA19.1.0.0.0.ZUTN'} - {'annual' } {'AMECO' } {'ZUTN' } {'Unemployment ra…'} {'EA19.1.0.0.0.ZUTN'} - - Columns 6 through 11 - - {'series_name' } {'original_period'} {'period' } {'original_value'} {'value'} {'freq'} - {'Annually – (Per…'} {'1960' } {'1960-01-01'} {'NA' } {[ NaN]} {'a' } - {'Annually – (Per…'} {'1961' } {'1961-01-01'} {'NA' } {[ NaN]} {'a' } - {'Annually – (Per…'} {'1962' } {'1962-01-01'} {'NA' } {[ NaN]} {'a' } - - Columns 12 through 16 - - {'unit' } {'geo' } {'Frequency'} {'Unit' } {'Country' } - {'percentage-of-a…'} {'ea19'} {'Annually' } {'(Percentage of …'} {'Euro area'} - {'percentage-of-a…'} {'ea19'} {'Annually' } {'(Percentage of …'} {'Euro area'} - {'percentage-of-a…'} {'ea19'} {'Annually' } {'(Percentage of …'} {'Euro area'} - - >> - -In such cell array, you will always find at least those columns: -* `x_frequency`: (harmonized frequency generated by DBnomics) -* `provider_code` -* `dataset_code` -* `dataset_name` -* `series_code` -* `series_name` -* `original_period`: the `period` as returned by DBnomics -* `period`: the first day of `original_period` -* `original_value` (`str` or `float`): the observation value as returned by DBnomics, where not available values are represented by `'NA'` -* `value` (`float` or `NaN`): the observation value as returned by DBnomics, where not available values are represented by `NaN` - -Followed by dimensions columns, corresponding to the dimensions of the dataset: -* dimensions labels: `freq`, `unit`, `geo` -* dimensions values labels: `Frequency`, `Unit`, `Country` - -### Fetch two time series by ID -Again, let's assume that we know which series we want to download. -We can reuse the `fetch_series` function, this time with two series codes. -For example, to fetch the time series `EA19.1.0.0.0.ZUTN` and `DNK.1.0.0.0.ZUTN` from the -[\"Unemployment rate\" [ZUTN] dataset](https://db.nomics.world/AMECO/ZUTN) -belonging to the [AMECO provider](https://db.nomics.world/AMECO). - -Example: - - >> df_ids = fetch_series('series_ids', {'AMECO/ZUTN/EA19.1.0.0.0.ZUTN', 'AMECO/ZUTN/DNK.1.0.0.0.ZUTN'}); - -### Fetch time series by code mask -The code mask notation is a very concise way to select one or many time series at once. -It is not compatible with all the providers. In particular, only the providers from the following list accept code mask: -* BIS -* ECB -* Eurostat -* FED -* IMF -* IMF-WEO -* INSEE -* OECD -* WTO - -Given 3 dimensions 'frequency', 'country' and 'indicator', the user can select: -* one time series by giving its code: `'M.FR.PCPIEC_IX'` -* many series by enumerating dimensions codes: `'M.FR+DE.PCPIEC_IX'` is equivalent to `{'M.FR.PCPIEC_IX', 'M.DE.PCPIEC_IX'}` -* many series by skipping a dimension, repeating '.' in the code mask: `'M..PCPIEC_IX'` is equivalent to `{'M.country1.PCPIEC_IX', 'M.country2.PCPIEC_IX', ..., 'M.countryN.PCPIEC_IX'}` - -Examples: - - >> df_code_mask1 = fetch_series('provider_code', 'IMF', 'dataset_code', 'CPI', 'series_code', 'M.FR+DE.PCPIEC_IX+PCPIA_IX'); - >> df_code_mask2 = fetch_series('provider_code', 'IMF', 'dataset_code', 'CPI', 'series_code', '.FR.PCPIEC_WT'); - >> df_code_mask3 = fetch_series('provider_code', 'IMF', 'dataset_code', 'CPI', 'series_code', 'M..PCPIEC_IX+PCPIA_IX', 'max_nb_series', 400); - -### Fetch time series by dimension -Searching by dimension is a less concise way to select time series than using the code mask, but it's universal: -some fetchers are not compatible with the code mask notation. The following example fetches many series from the -["Doing Business" [DB]](https://db.nomics.world/WB/DB) dataset of the [World Bank](https://db.nomics.world/WB) provider, selecting for time series about France, Italy and Spain (`country` dimension), -and the indicator "Procedures required to start a business - Women (number)" (`indicator` dimension). - -Example: - - >> df_dims = fetch_series('provider_code', 'WB', 'dataset_code', 'DB', 'dimensions', '{"country":["ES","FR","IT"],"indicator":["IC.REG.COST.PC.FE.ZS.DRFN"]}'); - -### Fetch time series by API link -When the dimensions, provider, dataset or series codes are unknown, the user can: -* go to the page of a dataset on DBnomics website (eg: [Doing Business](https://db.nomics.world/WB/DB)) -* select some dimensions by using the input widgets of the left column -* click on `Copy API link` in the menu of the `Download` button -* use the `fetch_series_by_api_link` function - -Example: - - >> df_link = fetch_series_by_api_link('https://api.db.nomics.world/v22/series/WB/DB/ENF.CONT.COEN.ATDR-AE?observations=1'); - -### Fetch time series from the cart -On the [cart page](https://db.nomics.world/cart) of the DBnomics website, click on "Copy API link" and copy-paste it as an argument of the fetch_series_by_api_link function. -Please note that when you update your cart, you have to copy this link again, because the link itself contains the IDs of the series in the cart. - -Example: - - >> df_cart = fetch_series_by_api_link('https://api.db.nomics.world/v22/series?series_ids=AMECO%2FZUTN%2FEA19.1.0.0.0.ZUTN&observations=1'); - -### Fetch time series with different frequencies - -Example: - - >> df_multi_freq = fetch_series('series_ids', {'BEA/NIUnderlyingDetail-U001BC/S315-A',... - 'BEA/NIUnderlyingDetail-U001BC/S315-Q',... - 'BEA/NIUnderlyingDetail-U001BC/S315-M'}); - -### Transform time series -The routines can interact with the [Time Series Editor](https://editor.nomics.world/) to transform time series by applying filters to them. -Available filters are listed on the [filters page](https://editor.nomics.world/filters). -The Time Series Editor is usable via a web interface ([example with AMECO/ZUTN/EA19.1.0.0.0.ZUTN](https://editor.nomics.world/series?source=dbnomics&series_id=AMECO/ZUTN/EA19.1.0.0.0.ZUTN)) -but you can call it directly from MATLAB. The user is also able to chain many filters. -Here is an example of how to interpolate two annual time series with a monthly frequency, using a spline interpolation. - -Example: - - >> filters_ = '[{"code": "interpolate", "parameters": {"frequency": "monthly", "method": "spline"}}]'; - >> df_filter = fetch_series('series_ids', 'AMECO/ZUTN/EA19.1.0.0.0.ZUTN', 'dbnomics_filters', filters_); - -The first row of the final cell array changes when filters are used: -* `period_middle_day`: the middle day of `original_period` (can be useful when you compare graphically interpolated series and original ones) -* `filtered` (`bool`): True if the series is filtered -* `series_code`: same as before for original series, but the suffix `_filtered` is added for filtered series -* `series_name`: same as before for original series, but the suffix `(filtered)` is added for filtered series \ No newline at end of file +[](https://git.dynare.org/DoraK/mdbnomics/commits/master) + +This MATLAB/Octave toolbox comes with routines to access DBnomics time series from MATLAB. + +The package is compatible with MATLAB 2019b and following versions, and (almost compatible with) +the latest Octave version. + +## Installation + +The toolbox can be installed by cloning the Git repository: + + ~$ git clone https://git.dynare.org/DoraK/mdbnomics.git + +or downloading a zip archive: + + ~$ wget https://git.dynare.org/DoraK/mdbnomics/-/archive/master/mdbnomics-master.zip + ~$ unsip mdbnomics-master.zip + -$ mv mdbnomics-master mdbnomics + +## Usage + +Add the `mdbnomics/src` folder to the MATLAB/Octave path, and run the following command (on MATLAB/Octave) prompt: + + >> initialize_mdbnomics() + +which, depending on your system, will add the necessary subfolders to +the MATLAB/Octave path. + +You are then ready to go. A full documentation will come soon. + +## Capabilities + +### Fetch one time series by ID +First, let's assume that we know which series we want to download. +A series identifier (ID) is defined by three values, formatted like this: `provider_code/dataset_code/series_code`. +The `fetch_series` function is used to construct the cell array. +For example, to fetch the time series `EA19.1.0.0.0.ZUTN` from the +[\"Unemployment rate\" [ZUTN] dataset](https://db.nomics.world/AMECO/ZUTN) +belonging to the [AMECO provider](https://db.nomics.world/AMECO). + +Example: + + >> df_id = fetch_series('series_ids', 'AMECO/ZUTN/EA19.1.0.0.0.ZUTN'); + +The returned data is stored in the `df_id` variable. Its type is a cell array. To display the first 3 rows of the array +(including the column headers), type: + + >> df_id(1:4,:) + + 6�16 cell array + + Columns 1 through 5 + + {'x_frequency'} {'provider_code'} {'dataset_code'} {'dataset_name' } {'series_code' } + {'annual' } {'AMECO' } {'ZUTN' } {'Unemployment ra�'} {'EA19.1.0.0.0.ZUTN'} + {'annual' } {'AMECO' } {'ZUTN' } {'Unemployment ra�'} {'EA19.1.0.0.0.ZUTN'} + {'annual' } {'AMECO' } {'ZUTN' } {'Unemployment ra�'} {'EA19.1.0.0.0.ZUTN'} + + Columns 6 through 11 + + {'series_name' } {'original_period'} {'period' } {'original_value'} {'value'} {'freq'} + {'Annually � (Per�'} {'1960' } {'1960-01-01'} {'NA' } {[ NaN]} {'a' } + {'Annually � (Per�'} {'1961' } {'1961-01-01'} {'NA' } {[ NaN]} {'a' } + {'Annually � (Per�'} {'1962' } {'1962-01-01'} {'NA' } {[ NaN]} {'a' } + + Columns 12 through 16 + + {'unit' } {'geo' } {'Frequency'} {'Unit' } {'Country' } + {'percentage-of-a�'} {'ea19'} {'Annually' } {'(Percentage of �'} {'Euro area'} + {'percentage-of-a�'} {'ea19'} {'Annually' } {'(Percentage of �'} {'Euro area'} + {'percentage-of-a�'} {'ea19'} {'Annually' } {'(Percentage of �'} {'Euro area'} + + >> + +In such cell array, you will always find at least those columns: +* `x_frequency`: (harmonized frequency generated by DBnomics) +* `provider_code` +* `dataset_code` +* `dataset_name` +* `series_code` +* `series_name` +* `original_period`: the `period` as returned by DBnomics +* `period`: the first day of `original_period` +* `original_value` (`str` or `float`): the observation value as returned by DBnomics, where not available values are represented by `'NA'` +* `value` (`float` or `NaN`): the observation value as returned by DBnomics, where not available values are represented by `NaN` + +Followed by dimensions columns, corresponding to the dimensions of the dataset: +* dimensions labels: `freq`, `unit`, `geo` +* dimensions values labels: `Frequency`, `Unit`, `Country` + +### Fetch two time series by ID +Again, let's assume that we know which series we want to download. +We can reuse the `fetch_series` function, this time with two series codes. +For example, to fetch the time series `EA19.1.0.0.0.ZUTN` and `DNK.1.0.0.0.ZUTN` from the +[\"Unemployment rate\" [ZUTN] dataset](https://db.nomics.world/AMECO/ZUTN) +belonging to the [AMECO provider](https://db.nomics.world/AMECO). + +Example: + + >> df_ids = fetch_series('series_ids', {'AMECO/ZUTN/EA19.1.0.0.0.ZUTN', 'AMECO/ZUTN/DNK.1.0.0.0.ZUTN'}); + +### Fetch time series by code mask +The code mask notation is a very concise way to select one or many time series at once. +It is not compatible with all the providers. In particular, only the providers from the following list accept code mask: +* BIS +* ECB +* Eurostat +* FED +* IMF +* IMF-WEO +* INSEE +* OECD +* WTO + +Given 3 dimensions 'frequency', 'country' and 'indicator', the user can select: +* one time series by giving its code: `'M.FR.PCPIEC_IX'` +* many series by enumerating dimensions codes: `'M.FR+DE.PCPIEC_IX'` is equivalent to `{'M.FR.PCPIEC_IX', 'M.DE.PCPIEC_IX'}` +* many series by skipping a dimension, repeating '.' in the code mask: `'M..PCPIEC_IX'` is equivalent to `{'M.country1.PCPIEC_IX', 'M.country2.PCPIEC_IX', ..., 'M.countryN.PCPIEC_IX'}` + +Examples: + + >> df_code_mask1 = fetch_series('provider_code', 'IMF', 'dataset_code', 'CPI', 'series_code', 'M.FR+DE.PCPIEC_IX+PCPIA_IX'); + >> df_code_mask2 = fetch_series('provider_code', 'IMF', 'dataset_code', 'CPI', 'series_code', '.FR.PCPIEC_WT'); + >> df_code_mask3 = fetch_series('provider_code', 'IMF', 'dataset_code', 'CPI', 'series_code', 'M..PCPIEC_IX+PCPIA_IX', 'max_nb_series', 400); + +### Fetch time series by dimension +Searching by dimension is a less concise way to select time series than using the code mask, but it's universal: +some fetchers are not compatible with the code mask notation. The following example fetches many series from the +["Doing Business" [DB]](https://db.nomics.world/WB/DB) dataset of the [World Bank](https://db.nomics.world/WB) provider, selecting for time series about France, Italy and Spain (`country` dimension), +and the indicator "Procedures required to start a business - Women (number)" (`indicator` dimension). + +Example: + + >> df_dims = fetch_series('provider_code', 'WB', 'dataset_code', 'DB', 'dimensions', '{"country":["ES","FR","IT"],"indicator":["IC.REG.COST.PC.FE.ZS.DRFN"]}'); + +### Fetch time series by API link +When the dimensions, provider, dataset or series codes are unknown, the user can: +* go to the page of a dataset on DBnomics website (eg: [Doing Business](https://db.nomics.world/WB/DB)) +* select some dimensions by using the input widgets of the left column +* click on `Copy API link` in the menu of the `Download` button +* use the `fetch_series_by_api_link` function + +Example: + + >> df_link = fetch_series_by_api_link('https://api.db.nomics.world/v22/series/WB/DB/ENF.CONT.COEN.ATDR-AE?observations=1'); + +### Fetch time series from the cart +On the [cart page](https://db.nomics.world/cart) of the DBnomics website, click on "Copy API link" and copy-paste it as an argument of the fetch_series_by_api_link function. +Please note that when you update your cart, you have to copy this link again, because the link itself contains the IDs of the series in the cart. + +Example: + + >> df_cart = fetch_series_by_api_link('https://api.db.nomics.world/v22/series?series_ids=AMECO%2FZUTN%2FEA19.1.0.0.0.ZUTN&observations=1'); + +### Fetch time series with different frequencies + +Example: + + >> df_multi_freq = fetch_series('series_ids', {'BEA/NIUnderlyingDetail-U001BC/S315-A',... + 'BEA/NIUnderlyingDetail-U001BC/S315-Q',... + 'BEA/NIUnderlyingDetail-U001BC/S315-M'}); + +### Transform time series +The routines can interact with the [Time Series Editor](https://editor.nomics.world/) to transform time series by applying filters to them. +Available filters are listed on the [filters page](https://editor.nomics.world/filters). +The Time Series Editor is usable via a web interface ([example with AMECO/ZUTN/EA19.1.0.0.0.ZUTN](https://editor.nomics.world/series?source=dbnomics&series_id=AMECO/ZUTN/EA19.1.0.0.0.ZUTN)) +but you can call it directly from MATLAB. The user is also able to chain many filters. +Here is an example of how to interpolate two annual time series with a monthly frequency, using a spline interpolation. + +Example: + + >> filters_ = '[{"code": "interpolate", "parameters": {"frequency": "monthly", "method": "spline"}}]'; + >> df_filter = fetch_series('series_ids', 'AMECO/ZUTN/EA19.1.0.0.0.ZUTN', 'dbnomics_filters', filters_); + +The first row of the final cell array changes when filters are used: +* `period_middle_day`: the middle day of `original_period` (can be useful when you compare graphically interpolated series and original ones) +* `filtered` (`bool`): True if the series is filtered +* `series_code`: same as before for original series, but the suffix `_filtered` is added for filtered series +* `series_name`: same as before for original series, but the suffix `(filtered)` is added for filtered series