Skip to content
Snippets Groups Projects

Compare revisions

Changes are shown as if the source revision was being merged into the target revision. Learn more about comparing revisions.

Source

Select target project
No results found

Target

Select target project
  • dbnomics/mdbnomics
  • DoraK/mdbnomics
  • giovanma/mdbnomics
3 results
Show changes
Commits on Source (43)
Expand all Hide whitespace changes
Inline Side-by-side
Showing
with 2006 additions and 377 deletions
.gitignore 0 → 100644
View file @ 009fadab
# For checking that no file has been unduly ignored, run:
# $ git ls-files -i --exclude-per-directory=.gitignore
# Any file that is displayed should be removed from the ignore list
# (possibly by an exclusion rule beginning with an exclamation mark)
# Generic ignore rules
*~
*.o
*.a
*.fig
\#*\#
*.mat
*.asv
# Test directory
m-unit-tests
.gitlab-ci.yml 0 → 100644
View file @ 009fadab
variables:
GIT_SUBMODULE_STRATEGY: normal
TERM: linux
MATLAB_VERSION: R2023a
OLD_MATLAB_VERSION: R2015a
test_matlab:
stage: test
script:
- make check-matlab MATLAB=/opt/MATLAB/$MATLAB_VERSION/bin/matlab
test_old_matlab:
stage: test
script:
- make check-matlab MATLAB=/opt/MATLAB/$OLD_MATLAB_VERSION/bin/matlab
.gitmodules 0 → 100644
View file @ 009fadab
[submodule "contrib/jsonlab"]
path = contrib/jsonlab
url = https://github.com/fangq/jsonlab.git
COPYING.txt 0 → 100644
View file @ 009fadab
This diff is collapsed.
Makefile 0 → 100644
View file @ 009fadab
MATLAB ?= $(shell which matlab)
all: check-matlab
m-unit-tests/src/mtest.m:
git clone https://git.dynare.org/Dynare/m-unit-tests && cd m-unit-tests && git checkout 343d9e13654a07c348fad863f7f79a03896c6bc9
check-matlab: m-unit-tests/src/mtest.m
@$(MATLAB) -nosplash -nodisplay -r "addpath([pwd '/m-unit-tests/src']); cd tests; runalltests; quit" && [ ! -f ./tests/failed ] && [ -f ./tests/pass ]
check-clean:
rm -f tests/*_test_*.m tests/*.csv tests/*.xls tests/*.xlsx tests/*.mat tests/failed tests/datafile_for_test
rm -f git.info git.last-commit-hash tests/pass tests/failed tests/*.spc
rm -rf m-unit-tests
README.md 0 → 100644
View file @ 009fadab
[![pipeline status](https://git.dynare.org/DoraK/mdbnomics/badges/master/pipeline.svg)](https://git.dynare.org/DoraK/mdbnomics/commits/master)
This MATLAB toolbox comes with routines to access DBnomics time series from MATLAB.
The package is compatible with MATLAB 2015a and following versions.
Octave compatibility will follow after the release of Octave 6 (an implementation to webread/webwrite/jsondecode is required).
## Installation
The toolbox can be installed by cloning the Git repository:
~$ git clone https://git.dynare.org/DoraK/mdbnomics.git
or downloading a zip archive:
~$ wget https://git.dynare.org/DoraK/mdbnomics/-/archive/master/mdbnomics-master.zip
~$ unsip mdbnomics-master.zip
-$ mv mdbnomics-master mdbnomics
## Usage
Add the `mdbnomics/src` folder to the MATLAB path, and run the following command (on MATLAB) prompt:
>> initialize_mdbnomics()
which, depending on your system, will add the necessary subfolders to
the MATLAB path.
You are then ready to go. A full documentation will come soon.
## Capabilities
### Fetch one time series by ID
First, let's assume that we know which series we want to download.
A series identifier (ID) is defined by three values, formatted like this: `provider_code/dataset_code/series_code`.
The `mdbnomics` function is used to construct the cell array.
For example, to fetch the time series `EA19.1.0.0.0.ZUTN` from the
[\"Unemployment rate\" [ZUTN] dataset](https://db.nomics.world/AMECO/ZUTN)
belonging to the [AMECO provider](https://db.nomics.world/AMECO).
Example:
>> df_id = mdbnomics('series_ids', 'AMECO/ZUTN/EA19.1.0.0.0.ZUTN');
The returned data is stored in the `df_id` variable. Its type is a cell array. To display the first 3 rows of the array as a table
(including the column headers), type:
>> tab_id = cell2table(df_id(2:end,:), 'VariableNames', df_id(1,:));
>> tab_id(1:3,:)
3�16 table
x_frequency provider_code dataset_code dataset_name series_code series_name original_period period original_value value freq unit geo Frequency Unit Country
___________ _____________ ____________ __________________________________________________________________ _____________________ ____________________________________________________________ _______________ ______________ ______________ _____ _____ ___________________________________ ________ ____________ _____________________________________ _____________
{'annual'} {'AMECO'} {'ZUTN'} {'Unemployment rate: total :- Member States: definition EUROSTAT'} {'EA19.1.0.0.0.ZUTN'} {'Annually � (Percentage of active population) � Euro area'} {'1960'} {'1960-01-01'} {'NA'} NaN {'a'} {'percentage-of-active-population'} {'ea19'} {'Annually'} {'(Percentage of active population)'} {'Euro area'}
{'annual'} {'AMECO'} {'ZUTN'} {'Unemployment rate: total :- Member States: definition EUROSTAT'} {'EA19.1.0.0.0.ZUTN'} {'Annually � (Percentage of active population) � Euro area'} {'1961'} {'1961-01-01'} {'NA'} NaN {'a'} {'percentage-of-active-population'} {'ea19'} {'Annually'} {'(Percentage of active population)'} {'Euro area'}
{'annual'} {'AMECO'} {'ZUTN'} {'Unemployment rate: total :- Member States: definition EUROSTAT'} {'EA19.1.0.0.0.ZUTN'} {'Annually � (Percentage of active population) � Euro area'} {'1962'} {'1962-01-01'} {'NA'} NaN {'a'} {'percentage-of-active-population'} {'ea19'} {'Annually'} {'(Percentage of active population)'} {'Euro area'}
>>
In such cell array, you will always find at least those columns:
* `x_frequency`: (harmonized frequency generated by DBnomics)
* `provider_code`
* `dataset_code`
* `dataset_name`
* `series_code`
* `series_name`
* `original_period`: the `period` as returned by DBnomics
* `period`: the first day of `original_period`
* `original_value` (`str` or `float`): the observation value as returned by DBnomics, where not available values are represented by `'NA'`
* `value` (`float` or `NaN`): the observation value as returned by DBnomics, where not available values are represented by `NaN`
Followed by dimensions columns, corresponding to the dimensions of the dataset:
* dimensions labels: `freq`, `unit`, `geo`
* dimensions values labels: `Frequency`, `Unit`, `Country`
### Fetch two time series by ID
Again, let's assume that we know which series we want to download.
We can reuse the `mdbnomics` function, this time with two series codes.
For example, to fetch the time series `EA19.1.0.0.0.ZUTN` and `DNK.1.0.0.0.ZUTN` from the
[\"Unemployment rate\" [ZUTN] dataset](https://db.nomics.world/AMECO/ZUTN)
belonging to the [AMECO provider](https://db.nomics.world/AMECO).
Example:
>> df_ids = mdbnomics('series_ids', {'AMECO/ZUTN/EA19.1.0.0.0.ZUTN', 'AMECO/ZUTN/DNK.1.0.0.0.ZUTN'});
### Fetch time series by code mask
The code mask notation is a very concise way to select one or many time series at once.
It is not compatible with all the providers. In particular, only the providers from the following list accept code mask:
* BIS
* ECB
* Eurostat
* FED
* IMF
* IMF-WEO
* INSEE
* OECD
* WTO
Given 3 dimensions 'frequency', 'country' and 'indicator', the user can select:
* one time series by giving its code: `'M.FR.PCPIEC_IX'`
* many series by enumerating dimensions codes: `'M.FR+DE.PCPIEC_IX'` is equivalent to `{'M.FR.PCPIEC_IX', 'M.DE.PCPIEC_IX'}`
* many series by skipping a dimension, repeating '.' in the code mask: `'M..PCPIEC_IX'` is equivalent to `{'M.country1.PCPIEC_IX', 'M.country2.PCPIEC_IX', ..., 'M.countryN.PCPIEC_IX'}`
Examples:
>> df_code_mask1 = mdbnomics('provider_code', 'IMF', 'dataset_code', 'CPI', 'series_code', 'M.FR+DE.PCPIEC_IX+PCPIA_IX');
>> df_code_mask2 = mdbnomics('provider_code', 'IMF', 'dataset_code', 'CPI', 'series_code', '.FR.PCPIEC_WT');
>> df_code_mask3 = mdbnomics('provider_code', 'IMF', 'dataset_code', 'CPI', 'series_code', 'M..PCPIEC_IX+PCPIA_IX', 'max_nb_series', 400);
### Fetch time series by dimension
Searching by dimension is a less concise way to select time series than using the code mask, but it's universal:
some fetchers are not compatible with the code mask notation. The following example fetches many series from the
["Doing Business" [DB]](https://db.nomics.world/WB/DB) dataset of the [World Bank](https://db.nomics.world/WB) provider, selecting for time series about France, Italy and Spain (`country` dimension),
and the indicator "Procedures required to start a business - Women (number)" (`indicator` dimension).
Example:
>> df_dims = mdbnomics('provider_code', 'WB', 'dataset_code', 'DB', 'dimensions', '{"country":["ES","FR","IT"],"indicator":["IC.REG.COST.PC.FE.ZS.DRFN"]}');
### Fetch time series by API link
When the dimensions, provider, dataset or series codes are unknown, the user can:
* go to the page of a dataset on DBnomics website (eg: [Doing Business](https://db.nomics.world/WB/DB))
* select some dimensions by using the input widgets of the left column
* click on `Copy API link` in the menu of the `Download` button
* use the `mdbnomics` function with the `api_link` parameter.
Example:
>> df_link = mdbnomics('api_link', 'https://api.db.nomics.world/v22/series/WB/DB/ENF.CONT.COEN.ATDR-AE?observations=1');
### Fetch time series from the cart
On the [cart page](https://db.nomics.world/cart) of the DBnomics website, click on "Copy API link" and copy-paste it as an argument of the `mdbnomics` function.
Please note that when you update your cart, you have to copy this link again, because the link itself contains the IDs of the series in the cart.
Example:
>> df_cart = mdbnomics('api_link', 'https://api.db.nomics.world/v22/series?series_ids=AMECO%2FZUTN%2FEA19.1.0.0.0.ZUTN&observations=1');
### Fetch time series with different frequencies
Example:
>> df_multi_freq = mdbnomics('series_ids', {'BEA/NIUnderlyingDetail-U001BC/S315-A',...
'BEA/NIUnderlyingDetail-U001BC/S315-Q',...
'BEA/NIUnderlyingDetail-U001BC/S315-M'});
### Fetch the available datasets of a provider
When fetching series from DBnomics, the user needs to give a provider and a dataset before specifying correct dimensions.
With the function `mdbnomics_datasets`, the user can download the list of the available datasets for a provider.
If no `provider_code` was supplied, an array of all datasets for every provider is returned.
Example:
>> datasets = mdbnomics_datasets('provider_code', 'IMF');
The result is a structure with with a cell array containing the dataset codes and names of the requested providers.
With the same function, if the user wants to fetch the available datasets for multiple providers, a cell array of providers has to be given.
Example:
>> datasets = mdbnomics_datasets('provider_code', {'IMF', 'BDF'});
In the event that the user only requests the datasets for one provider, if `simplify` is defined as `true`, then the result will be a simple cell array, not a structure.
Example:
>> datasets = mdbnomics_datasets('provider_code', 'IMF', 'simplify', true);
### Fetch the possible dimensions of available datasets of a provider
When fetching series from DBnomics, it can be interesting and especially useful to specify dimensions for a particular dataset to download only the series you want to analyse. With the function `mdbnomics_dimensions`,
the user can download these dimensions and their meanings.
Example:
>> datasets = mdbnomics_dimensions('provider_code', 'IMF', 'dataset_code', 'WEO');
The result is a nested structure (its names are IMF_WEO and the dimensions names) with a structure at the end of each branch.
In the event that the user only requests the dimensions for one dataset for one provider, if `simplify` is defined as `true`, then the result will be a simple structure, not a nested one.
Example:
>> datasets = mdbnomics_dimensions('provider_code', 'IMF', 'dataset_code', 'WEO', 'simplify', true);
To download the dimensions of every dataset gathered by DBnomics, the user does not have to set any arguments.
Example:
>> datasets = mdbnomics_dimensions();
### Fetch the series codes and names of available datasets of a provider
The user can download the list of series, and especially their codes, of a dataset’s provider by using the function `mdbnomics_series`. The result is a structure with a cell array at the end of each branch. If `simplify` is defined as `true`,
then the result will be a simple cell array.
Example:
>> series = mdbnomics_series('provider_code', 'IMF', 'dataset_code', 'WEO', 'simplify', true);
Like the function `mdbnomics()`, features can be added to `mdbnomics_series()`. The user can ask for the series with specific dimensions:
Example:
>> series = mdbnomics_series('provider_code', 'IMF', 'dataset_code', 'WEO', 'dimensions', '{"weo-subject":["NGDP_RPCH"]}', 'simplify', true);
or with a query:
Example:
>> series = mdbnomics_series('provider_code', 'IMF', 'dataset_code', 'WEO', 'query', 'NGDP_RPCH');
> :warning: **We ask the user to use this function parsimoniously because there are a huge amount of series per dataset. Please only fetch for one dataset if you need it or visit the DBnomics website.**
### Transform time series
The routines can interact with the [Time Series Editor](https://editor.nomics.world/) to transform time series by applying filters to them.
Available filters are listed on the [filters page](https://editor.nomics.world/filters).
The Time Series Editor is usable via a web interface ([example with AMECO/ZUTN/EA19.1.0.0.0.ZUTN](https://editor.nomics.world/series?source=dbnomics&series_id=AMECO/ZUTN/EA19.1.0.0.0.ZUTN))
but you can call it directly from MATLAB. The user is also able to chain many filters.
Here is an example of how to interpolate two annual time series with a monthly frequency, using a spline interpolation.
Example:
>> filters_ = '[{"code": "interpolate", "parameters": {"frequency": "monthly", "method": "spline"}}]';
>> df_filter = mdbnomics('series_ids', 'AMECO/ZUTN/EA19.1.0.0.0.ZUTN', 'dbnomics_filters', filters_);
The first row of the final cell array changes when filters are used:
* `period_middle_day`: the middle day of `original_period` (can be useful when you compare graphically interpolated series and original ones)
* `filtered` (`bool`): True if the series is filtered
* `series_code`: same as before for original series, but the suffix `_filtered` is added for filtered series
* `series_name`: same as before for original series, but the suffix `(filtered)` is added for filtered series
jsonlab @ 7499bd8f
Subproject commit 7499bd8fae221d3018d31b359a74422237a8b8ec
get_dataset.m deleted 100644 → 0
View file @ 7e85f512
function [dd, dat] = get_dataset(provider,dataset, varargin)
if numel(varargin) > 0
options = parse_options(varargin);
else
options = ['?limit=1000&offset=0&q=&observations=1&align_periods=1&' ...
'dimensions={}'];
end
default_api_base_url = 'https://api.db.nomics.world/v22/series/';
url = join([default_api_base_url, provider, '/', dataset, options]);
dat = webread(join(url));
docs = dat.series.docs;
nseries = dat.dataset.nb_series;
nrow = 1;
for id = 1:nseries
nrow = nrow + length(docs(id).period);
end
dimensions_codes = dat.dataset.dimensions_codes_order;
ndim = length(dimensions_codes);
ncol = ndim + 10;
dd = cell(nrow,ncol);
dd{1,1} = '@frequency';
dd{1,2} = 'dataset_code';
dd{1,3} = 'dataset_name';
dd{1,4} = 'indexed_at';
dd{1,5} = 'original_period';
dd{1,6} = 'period';
dd{1,7} = 'provider_code';
dd{1,8} = 'series_code';
dd{1,9} = 'series name';
dd{1,10} = 'value';
for i = 1:ndim
dd{1,10+i} = dimensions_codes{i};
end
row = 2;
for i = 1:nseries
doc = docs(i);
x_frequency = doc.x_frequency;
dataset_code = doc.dataset_code;
dataset_name = doc.dataset_name;
series_code = doc.series_code;
series_name = doc.series_name;
indexed_at = doc.indexed_at;
provider_code = doc.provider_code;
dimensions = doc.dimensions;
period = doc.period;
period_start_day = doc.period_start_day;
value = doc.value;
for j = 1:length(period)
dd{row,1} = x_frequency;
dd{row,2} = dataset_code;
dd{row,3} = dataset_name;
dd{row,4} = indexed_at;
dd{row,5} = period{j};
dd{row,6} = period_start_day{j};
dd{row,7} = provider_code;
dd{row,8} = series_code;
dd{row,9} = series_name;
if iscell(value)
dd{row,10} = value{j};
else
dd{row,10} = value(j);
end
for k = 1:ndim
dim_code = dimensions.(dimensions_codes{k});
dd{row,10+k} = dim_code;
end
row = row + 1;
end
end
function options_str = parse_options(arguments)
i = 1;
options = struct();
options.limit = 1000;
options.dimensions = '{}';
options.mask = '';
while i <= length(arguments)
if arguments{1} == 'limit'
options.limit = arguments{2};
i = i + 2;
elseif arguments{1} == 'dimensions'
if ~isempty(options.mask)
error('rdb: you can''t use both option dimension and option mask');
end
elseif arguments =='mask'
if options.dimensions ~= '{}'
error('rdb: you can''t use both option dimension and option mask');
end
options.mask = arguments{2};
i = i+2;
else
disp_usage();
end
end
options_str = sprintf('?limit=%d0&q=&observations=1&align_periods=1&dimensions=%s',...
options.limit, options.dimensions);
if options.mask
options_str = options_str + '&' + options.mask;
end
\ No newline at end of file
get_dseries.m deleted 100644 → 0
View file @ 7e85f512
function [dd, dat] = get_dseries(provider,dataset,series_id)
%url = join(['https://api.db.nomics.world/v22/series/', provider, '/',...
% dataset, '/', series_id, '?observations=true'])
%json = urlread(join(url))
%save test json
f = fopen("test.json");
json = fgets(f);
fclose(f);
dat = jsondecode(json);
dd = ''
docs = dat.series.docs;
first_period = docs.period{1};
if docs.x_frequency == 'annual'
first_period = join([first_period, 'A']);
end
dd = dseries(docs.value, first_period, docs.series_code, docs.series_name)
get_series.m deleted 100644 → 0
View file @ 7e85f512
function [dd, dat] = get_series(provider,dataset,series_id)
%url = join(['https://api.db.nomics.world/v22/series/', provider, '/',...
% dataset, '/', series_id, '?observations=true'])
%json = urlread(join(url))
%save test json
f = fopen("test.json");
json = fgets(f);
fclose(f);
dat = jsondecode(json);
dd = ''
docs = dat.series.docs;
first_period = docs.period{1};
if docs.x_frequency == 'annual'
first_period = join([first_period, 'A']);
end
dd = struct()
dd.values = docs.value
dd.periods = docs.period
dd.code = docs.series_code
dd.name = docs.series_name
rdb.m deleted 100644 → 0
View file @ 7e85f512
function data = rdb(str, varargin)
if nargin == 0
[d, dd] = get_series(str)
elseif nargin == 1
[d, dd] = get_dataset(str, varargin{1})
elseif nargin == 2
[d, dd] = get_series(str, varargin{1}, varargin{2})
elseif nargin > 1
[d, dd] = get_dataset(str, vararin{1}, varargin{2:end})
end
src/fetch_series.m deleted 100644 → 0
View file @ 7e85f512
function df = fetch_series(varargin)
% Download time series from DBnomics.
%
% If not `None`, `dimensions` parameter must be a `dict` of dimensions (`list` of `str`), like so:
% `{"freq": ["A", "M"], "country": ["FR"]}`.
%
% If not `None`, `series_code` must be a `str`. It can be a series code (one series), or a "mask" (many series):
% - remove a constraint on a dimension, for example `M..PCPIEC_WT`;
% - enumerate many values for a dimension, separated by a '+', for example `M.FR+DE.PCPIEC_WT`;
% - combine these possibilities many times in the same SDMX filter.
%
% If the rightmost dimension value code is removed, then the final '.' can be removed too: `A.FR.` = `A.FR`.
%
% If not `None`, `series_ids` parameter must be a non-empty `list` of series IDs.
% A series ID is a string formatted like `provider_code/dataset_code/series_code`.
%
% If `max_nb_series` is `None`, a default value of 50 series will be used.
%
% Return a cell array.
%
% Examples:
%
% - fetch one series:
% fetch_series('provider_code', "IMF", 'dataset_code', "CPI", 'series_code', "M.FR+DE.PCPIEC_IX+PCPIA_IX")
% fetch_series('provider_code', "IMF", 'dataset_code', "CPI", 'series_code', ".FR.PCPIEC_WT")
% fetch_series('provider_code', "IMF", 'dataset_code', "CPI", 'series_code', "M..PCPIEC_IX+PCPIA_IX")
%
% - fetch all the series of a dataset:
% fetch_series('provider_code', "AMECO", 'dataset_code', "UVGD", 'max_nb_series', 500);
%
% - fetch many series from different datasets:
% fetch_series('series_ids', ["AMECO/ZUTN/EA19.1.0.0.0.ZUTN", "AMECO/ZUTN/DNK.1.0.0.0.ZUTN", "IMF/CPI/A.AT.PCPIT_IX"])
%
% - fetch many series from the same dataset, searching by dimension:
% fetch_series('provider_code', "AMECO", 'dataset_code', "ZUTN", dimensions={"geo": ["dnk"]})
%
default_api_base_url = 'https://api.db.nomics.world/v22/';
p = inputParser;
validStringInput = @(x) isstring(x) || ischar(x) || iscellstr(x);
p.addParameter('provider_code', '', validStringInput);
p.addParameter('dataset_code', '', validStringInput);
p.addParameter('series_code', '', validStringInput);
p.addParameter('dimensions', @ischar); %if no dimensions specified, provider_code & dataset_code MUST BE GIVEN
p.addParameter('series_ids', '',validStringInput);
p.addParameter('max_nb_series', NaN, @isnumeric);
p.addParameter('api_base_url', default_api_base_url, validStringInput);
p.addParameter('dbnomics_filters', '', @ischar);
p.KeepUnmatched = false;
p.parse(varargin{:});
if strcmp(p.Results.api_base_url(end),'/') == 0
p.Results.api_base_url = [p.Results.api_base_url '/'];
end
if isempty(p.Results.dataset_code)
if iscell(p.Results.provider_code)
p.Results.series_ids = p.Results.provider_code;
p.Results.provider_code = '';
elseif ~isempty(p.Results.provider_code)
p.Results.series_ids = {p.Results.provider_code};
p.Results.provider_code = '';
end
end
series_base_url = [p.Results.api_base_url 'series'];
if isa(p.Results.dimensions, 'function_handle') && isempty(p.Results.series_code) && isempty(p.Results.series_ids)
if isempty(p.Results.provider_code) || isempty(p.Results.dataset_code)
error('When you don''t use dimensions, you must specifiy provider_code and dataset_code.');
end
api_link = sprintf('%s/%s/%s?observations=1', series_base_url, p.Results.provider_code, p.Results.dataset_code);
end
if ~isa(p.Results.dimensions, 'function_handle')
if isempty(p.Results.provider_code) || isempty(p.Results.dataset_code)
error('When you use dimensions, you must specifiy provider_code and dataset_code.');
end
api_link = sprintf('%s/%s/%s?observations=1&dimensions=%s', series_base_url, p.Results.provider_code, p.Results.dataset_code, p.Results.dimensions); %jsonencode(
end
if ~isempty(p.Results.series_code)
if isempty(p.Results.provider_code) || isempty(p.Results.dataset_code)
error('When you use series_code, you must specifiy provider_code and dataset_code.');
end
api_link = sprintf('%s/%s/%s/%s?observations=1', series_base_url, p.Results.provider_code, p.Results.dataset_code, p.Results.series_code);
end
if ~isempty(p.Results.series_ids)
if ~isempty(p.Results.provider_code) || ~isempty(p.Results.dataset_code)
error('When you use series_ids, you must not specifiy provider_code nor dataset_code.');
end
api_link = sprintf('%s?observations=1&series_ids=%s', series_base_url, strjoin(p.Results.series_ids,','));
end
df = fetch_series_by_api_link(api_link, p.Results.dbnomics_filters, p.Results.max_nb_series);
end
src/fetch_series_by_api_link.m deleted 100644 → 0
View file @ 7e85f512
function df = fetch_series_by_api_link(api_link, varargin)
% Fetch series given an "API link" URL.
% "API link" URLs can be found on DBnomics web site (https://db.nomics.world/) on dataset or series pages using "Download" buttons.
%
% Example:
% fetch_series_by_api_link("https://api.db.nomics.world/v22/series?provider_code=AMECO&dataset_code=ZUTN")
p = inputParser;
p.addRequired('api_link');
p.addOptional('dbnomics_filters', '',@ischar);
p.addOptional('max_nb_series', NaN, @isnumeric);
p.KeepUnmatched = false;
p.parse(api_link, varargin{:});
[datasets_dimensions, series_dims_by_dataset_code, series_list]= iter_series_info(api_link, p.Results.max_nb_series);
if isempty(series_list)
df = {};
else
common_columns = ["x_frequency", "provider_code", "dataset_code", "dataset_name", "series_code", "series_name","original_period", "period", "original_value", "value"];
% Compute dimensions_labels_columns_names and dimensions_codes_columns_names
dimensions_labels_columns_names = [];
dimensions_codes_columns_names = [];
dataset_codes = fieldnames(datasets_dimensions);
for ii = 1:length(fieldnames(datasets_dimensions))
dataset_dimensions = datasets_dimensions.(dataset_codes{ii});
dimensions_codes_columns_names = [dimensions_codes_columns_names, string(dataset_dimensions.dimensions_codes_order')];
for jj = 1:length(dataset_dimensions.dimensions_codes_order)
if isfield(dataset_dimensions, 'dimensions_labels') && isfield(dataset_dimensions, 'dimensions_values_labels')
dimensions_labels_columns_names_dataset{jj} = dataset_dimensions.dimensions_labels.(dataset_dimensions.dimensions_codes_order{jj});
else
if isfield(dataset_dimensions, 'dimensions_values_labels')
dimensions_labels_columns_names_dataset{jj} = [dataset_dimensions.dimensions_codes_order{jj} '_label'];
end
end
end
dimensions_labels_columns_names = [dimensions_labels_columns_names, string(dimensions_labels_columns_names_dataset)];
end
ordered_columns_names = [common_columns, dimensions_codes_columns_names, dimensions_labels_columns_names];
df = cell(length(series_list{1}.value)*length(series_list)+1,length(ordered_columns_names));
for col = 1:length(ordered_columns_names)
col_ = ordered_columns_names{col};
df{1,col} = col_;
end
series_length=0;
% Flatten series received from the API (rename some keys of JSON result to match DataFrame organization)
for ii = 1:length(series_list)
flat_series = flatten_dbnomics_series(series_list{ii});
% Add dimensions labels to flat_series
complete_dataset_code = [flat_series.provider_code '_' flat_series.dataset_code];
dataset_dimensions = datasets_dimensions.(complete_dataset_code);
if isfield(dataset_dimensions, 'dimensions_labels')
dataset_dimensions_labels = dataset_dimensions.dimensions_labels;
else
dataset_dimensions_labels = struct();
for jj = 1:length(dataset_dimensions.dimensions_codes_order)
dataset_dimensions_labels.(dataset_dimensions.dimensions_codes_order{jj}) = [dataset_dimensions.dimensions_codes_order{jj} '_label'];
end
end
% Add dimensions values labels to current series
if isfield(dataset_dimensions, 'dimensions_values_labels')
dimension_codes = intersect(dimensions_codes_columns_names, string(fieldnames(dataset_dimensions_labels)')); %string(fieldnames(dataset_dimensions_labels)');
for jj = 1:length(dimension_codes)
series_code = regexprep(flat_series.series_code,'[^a-zA-Z0-9]','_');
dimension_label = dataset_dimensions_labels.(dimension_codes{jj});
flat_series.labels{jj, 1} = dimension_label;
dimension_value_code = regexprep(series_dims_by_dataset_code.(complete_dataset_code).(series_code).(dimension_codes{jj}),'[^a-zA-Z0-9]','_');
if isstrprop(dimension_value_code(1), 'digit') %MATLAB doesn't allow struct fieldnames to start with a digit
dimension_value_code = ['x' dimension_value_code];
end
try
flat_series.labels{jj, 2} = dataset_dimensions.dimensions_values_labels.(dimension_codes{jj}).(dimension_value_code);
catch
for it = 1:size(dataset_dimensions.dimensions_values_labels.(dimension_codes{jj}), 1)
tmp = regexprep(dataset_dimensions.dimensions_values_labels.(dimension_codes{jj}){it}{1}, '[^a-zA-Z0-9]', '_');
if strcmp(tmp, dimension_value_code)
flat_series.labels{jj, 2} = dataset_dimensions.dimensions_values_labels.(dimension_codes{jj}){it}{2};
end
end
end
end
end
% Create final cell array
for col = 1:length(ordered_columns_names)
col_ = ordered_columns_names{col};
for jj = 1:length(flat_series.value)
if strcmp(col_, 'original_value') || strcmp(col_,'value') || strcmp(col_, 'original_period') || strcmp(col_, 'period')
df{series_length+jj+1,col} = flat_series.(col_){jj};
elseif any(strcmp(col_,dimensions_labels_columns_names))
if ~any(strcmp(col_,flat_series.labels))
df{series_length+jj+1,col} = NaN;
else
idx = find(strcmp(flat_series.labels, col_));
df{series_length+jj+1,col} = flat_series.labels{idx,2};
end
elseif any(strcmp(col_, dimensions_codes_columns_names)) && ~any(strcmp(col_,string(fieldnames(flat_series)')))
df{series_length+jj+1,col} = NaN;
else
df{series_length+jj+1,col} = flat_series.(col_);
end
end
end
series_length=series_length+length(flat_series.value);
end
end
end
\ No newline at end of file
src/fetch_series_page.m deleted 100644 → 0
View file @ 7e85f512
function response_json = fetch_series_page(series_endpoint_url,offset)
% Adapt series_endpoint_url and make API request
if contains(series_endpoint_url, '?')
series_page_url = sprintf('%s%soffset=%i', series_endpoint_url, '&', offset);
else
series_page_url = sprintf('%s%soffset=%i', series_endpoint_url, '?', offset);
end
options = weboptions('ContentType','json');
try
response_json = webread(series_page_url, options);
catch ME
error_message = ['Could not fetch data from URL: ' series_page_url ' because: ' ME.identifier];
error(error_message);
end
series_page = response_json.series;
if ~isempty(series_page)
assert(series_page.offset == offset);
end
end
src/initialize_mdbnomics.m 0 → 100644
View file @ 009fadab
function initialize_mdbnomics()
% Copyright (C) 2020 Dynare Team
%
% This file is part of Dynare.
%
% Dynare is free software: you can redistribute it and/or modify
% it under the terms of the GNU General Public License as published by
% the Free Software Foundation, either version 3 of the License, or
% (at your option) any later version.
%
% Dynare is distributed in the hope that it will be useful,
% but WITHOUT ANY WARRANTY; without even the implied warranty of
% MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
% GNU General Public License for more details.
%
% You should have received a copy of the GNU General Public License
% along with Dynare. If not, see <http://www.gnu.org/licenses/>.
% Get the path to the mdbnomics toolbox.
global mdb_options
mdbnomics_src_root = strrep(which('initialize_mdbnomics'), 'initialize_mdbnomics.m', '');
% Set the subfolders to be added in the path.
p = {'utils'};
% Add missing routines if dynare is not in the path
if ~exist('OCTAVE_VERSION', 'builtin') || ~exist('contains','builtin')
p{end+1} = 'missing/contains';
end
if ~exist('isoctave','file')
p{end+1} = 'missing/isoctave';
end
if ~exist('matlab_ver_less_than','file')
p{end+1} = 'missing/matlab_ver_less_than';
end
if ~exist('octave_ver_less_than','file')
p{end+1} = 'missing/octave_ver_less_than';
end
if exist('OCTAVE_VERSION', 'builtin') && ~exist('user_has_octave_forge_package','file')
p{end+1} = 'missing/user_has_octave_forge_package';
end
% Set path
P = cellfun(@(c)[mdbnomics_src_root c], p, 'uni', false);
addpath(P{:});
if ~isoctave()
% Check minimal MATLAB requirements.
if matlab_ver_less_than('8.5')
error('The minimum MATLAB requirement of this package is R2015a.');
end
% Add jsonlab if MATLAB version < R2016b
if matlab_ver_less_than('9.1')
addpath([mdbnomics_src_root '/../contrib/jsonlab']);
end
end
mdb_options.api_base_url = 'https://api.db.nomics.world';
mdb_options.editor_base_url = 'https://editor.nomics.world';
mdb_options.api_version = 22;
mdb_options.editor_version = 1;
assignin('caller', 'mdb_options', mdb_options);
assignin('caller', 'mdbnomics_src_root', mdbnomics_src_root);
src/mdbnomics.m 0 → 100644
View file @ 009fadab
function df = mdbnomics(varargin) % --*-- Unitary tests --*--
% function mdbnomics(varargin)
% Download time series from DBnomics.
% Returns a cell array.
%
% Examples:
% Fetch one series:
% mdbnomics('provider_code', 'IMF', 'dataset_code', 'CPI', 'series_code', 'M.FR+DE.PCPIEC_IX+PCPIA_IX');
% mdbnomics('provider_code', 'IMF', 'dataset_code', 'CPI', 'series_code', '.FR.PCPIEC_WT');
%
% Fetch all the series of a dataset:
% mdbnomics('provider_code', 'AMECO', 'dataset_code', 'UVGD', 'max_nb_series', 500);
%
% Fetch many series from different datasets:
% mdbnomics('series_ids', {'AMECO/ZUTN/EA19.1.0.0.0.ZUTN', 'AMECO/ZUTN/DNK.1.0.0.0.ZUTN', 'IMF/CPI/A.AT.PCPIT_IX'});
%
% Fetch many series from the same dataset, searching by dimension:
% mdbnomics('provider_code','AMECO', 'dataset_code', 'ZUTN', 'dimensions', '{"geo":["dnk"]}');
%
% Fetch series given an "API link" URL.
% "API link" URLs can be found on DBnomics web site (https://db.nomics.world/) on dataset or series pages using "Download" buttons.
% mdbnomics('api_link', 'https://api.db.nomics.world/v22/series?series_ids=AMECO%2FZUTN%2FEA19.1.0.0.0.ZUTN&observations=1');
%
% POSSIBLE PARAMETERS
% provider_code [string] the code of the dataset provider.
% dataset_code [string] the code of the dataset.
% series_code [string] the code mask of the series. If provided, the provider_code and the dataset_code must specified.
% dimensions [char] dataset dimension codes. If provided it must be a string formatted like: '{"country":["ES","FR","IT"],"indicator":["IC.REG.COST.PC.FE.ZS.DRFN"]}'.
% series_ids [string] list of series IDs. It is string formatted like `provider_code/dataset_code/series_code`. If provided, the provider_code and the dataset_code shouldn't be specified.
% max_nb_series [integer] maximum number of series requested by the API. If not provided, a default value of 50 series will be used.
% api_base_url [string] the base URL used for API requests. If not provided, a default value of: 'https://api.db.nomics.world/v22/' will be used.
% dbnomics_filters [char] filters to apply on the requested series. If provided it must be a string formatted like: '[{"code": "interpolate", "parameters": {"frequency": "monthly", "method": "spline"}}]'.
% api_link [char] fetch series given an "API link" URL.
%
% OUTPUTS
% df
%
% SPECIAL REQUIREMENTS
% When you (don't) use dimensions, you must specifiy provider_code and dataset_code.
% When you use series_code, you must specifiy provider_code and dataset_code.
% When you use series_ids, you must not specifiy provider_code nor dataset_code.
% Copyright (C) 2020 Dynare Team
%
% This file is part of Dynare.
%
% Dynare is free software: you can redistribute it and/or modify
% it under the terms of the GNU General Public License as published by
% the Free Software Foundation, either version 3 of the License, or
% (at your option) any later version.
%
% Dynare is distributed in the hope that it will be useful,
% but WITHOUT ANY WARRANTY; without even the implied warranty of
% MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
% GNU General Public License for more details.
%
% You should have received a copy of the GNU General Public License
% along with Dynare. If not, see <http://www.gnu.org/licenses/>.
global mdb_options
default_api_base_url = sprintf('%s/v%d/', mdb_options.api_base_url, mdb_options.api_version);
default_editor_base_url = sprintf('%s/api/v%d/', mdb_options.editor_base_url, mdb_options.editor_version);
p = inputParser;
validStringInput = @(x) ischar(x) || iscellstr(x);
p.addParameter('provider_code', '', validStringInput);
p.addParameter('dataset_code', '', validStringInput);
p.addParameter('series_code', '', validStringInput);
p.addParameter('dimensions', @ischar);
p.addParameter('series_ids', '',validStringInput);
p.addParameter('max_nb_series', NaN, @isnumeric);
p.addParameter('api_base_url', default_api_base_url, validStringInput);
p.addParameter('dbnomics_filters', '', @ischar);
p.addParameter('api_link', '', @ischar);
p.KeepUnmatched = false;
p.parse(varargin{:});
if strcmp(p.Results.api_base_url(end),'/') == 0
p.Results.api_base_url = [p.Results.api_base_url '/'];
end
if isempty(p.Results.dataset_code)
if iscell(p.Results.provider_code)
p.Results.series_ids = p.Results.provider_code;
p.Results.provider_code = '';
elseif ~isempty(p.Results.provider_code)
p.Results.series_ids = {p.Results.provider_code};
p.Results.provider_code = '';
end
end
series_base_url = [p.Results.api_base_url 'series'];
if isa(p.Results.dimensions, 'function_handle') && isempty(p.Results.series_code) && isempty(p.Results.series_ids)
if (isempty(p.Results.provider_code) || isempty(p.Results.dataset_code)) && isempty(p.Results.api_link)
error('When you don''t use dimensions, you must specifiy provider_code and dataset_code.');
end
api_link = sprintf('%s/%s/%s?observations=1', series_base_url, p.Results.provider_code, p.Results.dataset_code);
end
if ~isa(p.Results.dimensions, 'function_handle')
if isempty(p.Results.provider_code) || isempty(p.Results.dataset_code)
error('When you use dimensions, you must specifiy provider_code and dataset_code.');
end
api_link = sprintf('%s/%s/%s?observations=1&dimensions=%s', series_base_url, p.Results.provider_code, p.Results.dataset_code, p.Results.dimensions);
end
if ~isempty(p.Results.series_code)
if isempty(p.Results.provider_code) || isempty(p.Results.dataset_code)
error('When you use series_code, you must specifiy provider_code and dataset_code.');
end
api_link = sprintf('%s/%s/%s/%s?observations=1', series_base_url, p.Results.provider_code, p.Results.dataset_code, p.Results.series_code);
end
if ~isempty(p.Results.series_ids)
if ~isempty(p.Results.provider_code) || ~isempty(p.Results.dataset_code)
error('When you use series_ids, you must not specifiy provider_code nor dataset_code.');
end
if iscellstr(p.Results.series_ids)
series_ids = strjoin(p.Results.series_ids,',');
else
series_ids = p.Results.series_ids;
end
api_link = sprintf('%s?observations=1&series_ids=%s', series_base_url, series_ids);
end
if ~isempty(p.Results.api_link)
api_link = p.Results.api_link;
end
df = fetch_series_by_api_link(api_link, p.Results.dbnomics_filters, p.Results.max_nb_series, default_editor_base_url);
end
%@test:1 % test_fetch_series_by_code
%$ try
%$ df = mdbnomics('provider_code', 'AMECO', 'dataset_code', 'ZUTN', 'series_code', 'EA19.1.0.0.0.ZUTN');
%$ t(1) = 1;
%$ catch
%$ t = 0;
%$ end
%$
%$ if t(1)
%$ t(2) = dassert(length(unique(df(2:end,2))),1);
%$ t(3) = dassert(df(2,2), {'AMECO'});
%$ t(4) = dassert(length(unique(df(2:end,3))),1);
%$ t(5) = dassert(df(2,3), {'ZUTN'});
%$ t(6) = dassert(length(unique(df(2:end,5))),1);
%$ t(7) = dassert(df(2,5), {'EA19.1.0.0.0.ZUTN'});
%$ end
%$
%$ T = all(t);
%@eof:1
%@test:2 % test_fetch_series_by_code_mask
%$ try
%$ df = mdbnomics('provider_code', 'IMF', 'dataset_code', 'CPI', 'series_code', 'M.FR+DE.PCPIEC_IX+PCPIA_IX');
%$ t(1) = 1;
%$ catch
%$ t = 0;
%$ end
%$
%$ if t(1)
%$ t(2) = dassert(length(unique(df(2:end,2))),1);
%$ t(3) = dassert(df(2,2), {'IMF'});
%$ t(4) = dassert(length(unique(df(2:end,3))),1);
%$ t(5) = dassert(df(2,3), {'CPI'});
%$ t(6) = dassert(length(unique(df(2:end,5))),4);
%$ end
%$
%$ T = all(t);
%@eof:2
%@test:3 % test_fetch_series_by_code_mask_with_plus
%$ try
%$ df = mdbnomics('provider_code', 'SCB', 'dataset_code', 'AKIAM', 'series_code', '"J+K"+"G+H".AM0301C1');
%$ t(1) = 1;
%$ catch
%$ t = 0;
%$ end
%$
%$ if t(1)
%$ t(2) = dassert(length(unique(df(2:end,2))),1);
%$ t(3) = dassert(df(2,2), {'SCB'});
%$ t(4) = dassert(length(unique(df(2:end,3))),1);
%$ t(5) = dassert(df(2,3), {'AKIAM'});
%$ t(6) = dassert(length(unique(df(2:end,5))),2);
%$ end
%$
%$ T = all(t);
%@eof:3
%@test:4 % test_fetch_series_by_dimension
%$ try
%$ df = mdbnomics('provider_code','WB','dataset_code','DB', 'dimensions', '{"country":["ES","FR","IT"],"indicator":["IC.REG.COST.PC.FE.ZS.DRFN"]}');
%$ t(1) = 1;
%$ catch
%$ t = 0;
%$ end
%$
%$ if t(1)
%$ t(2) = dassert(length(unique(df(2:end,2))),1);
%$ t(3) = dassert(df(2,2), {'WB'});
%$ t(4) = dassert(length(unique(df(2:end,3))),1);
%$ t(5) = dassert(df(2,3), {'DB'});
%$ t(6) = dassert(length(unique(df(2:end,5))),3);
%$ end
%$
%$ T = all(t);
%@eof:4
%@test:5 % test_fetch_series_by_id
%$ try
%$ df = mdbnomics('series_ids','AMECO/ZUTN/EA19.1.0.0.0.ZUTN');
%$ t(1) = 1;
%$ catch
%$ t = 0;
%$ end
%$
%$ if t(1)
%$ t(2) = dassert(length(unique(df(2:end,2))),1);
%$ t(3) = dassert(df(2,2), {'AMECO'});
%$ t(4) = dassert(length(unique(df(2:end,3))),1);
%$ t(5) = dassert(df(2,3), {'ZUTN'});
%$ t(6) = dassert(length(unique(df(2:end,5))),1);
%$ t(7) = dassert(df(2,5), {'EA19.1.0.0.0.ZUTN'});
%$ end
%$
%$ T = all(t);
%@eof:5
%@test:6 % test_fetch_series_by_ids_in_different_datasets
%$ try
%$ df = mdbnomics('series_ids', {'AMECO/ZUTN/EA19.1.0.0.0.ZUTN', 'BIS/cbs/Q.S.5A.4B.F.B.A.A.LC1.A.1C'});
%$ t(1) = 1;
%$ catch
%$ t = 0;
%$ end
%$
%$ if t(1)
%$ provider_codes = unique(df(2:end,2));
%$ t(2) = dassert(length(provider_codes),2);
%$ t(3) = dassert(provider_codes{1}, 'AMECO');
%$ t(4) = dassert(provider_codes{2}, 'BIS');
%$ dataset_codes = unique(df(2:end,3));
%$ t(5) = dassert(length(dataset_codes),2);
%$ t(6) = dassert(dataset_codes{1}, 'ZUTN');
%$ t(7) = dassert(dataset_codes{2}, 'cbs');
%$ series_codes = unique(df(2:end,5));
%$ t(8) = dassert(length(series_codes),2);
%$ t(9) = dassert(series_codes{1}, 'EA19.1.0.0.0.ZUTN');
%$ t(10) = dassert(series_codes{2}, 'Q.S.5A.4B.F.B.A.A.LC1.A.1C');
%$ end
%$
%$ T = all(t);
%@eof:6
%@test:7 % test_fetch_series_by_ids_in_same_dataset
%$ try
%$ df = mdbnomics('series_ids', {'AMECO/ZUTN/EA19.1.0.0.0.ZUTN',...
%$ 'AMECO/ZUTN/DNK.1.0.0.0.ZUTN'});
%$ t(1) = 1;
%$ catch
%$ t = 0;
%$ end
%$
%$ if t(1)
%$ t(2) = dassert(length(unique(df(2:end,2))),1);
%$ t(3) = dassert(df(2,2), {'AMECO'});
%$ t(4) = dassert(length(unique(df(2:end,3))),1);
%$ t(5) = dassert(df(2,3), {'ZUTN'});
%$ series_codes = unique(df(2:end,5));
%$ t(6) = dassert(length(series_codes),2);
%$ t(7) = dassert(series_codes{1}, 'DNK.1.0.0.0.ZUTN');
%$ t(8) = dassert(series_codes{2}, 'EA19.1.0.0.0.ZUTN');
%$ end
%$
%$ T = all(t);
%@eof:7
%@test:8 % test_fetch_series_of_dataset
%$ try
%$ df = mdbnomics('provider_code', 'AMECO', 'dataset_code', 'ZUTN');
%$ t(1) = 1;
%$ catch
%$ t = 0;
%$ end
%$
%$ if t(1)
%$ t(2) = dassert(length(unique(df(2:end,2))),1);
%$ t(3) = dassert(df(2,2), {'AMECO'});
%$ t(4) = dassert(length(unique(df(2:end,3))),1);
%$ t(5) = dassert(df(2,3), {'ZUTN'});
%$ t(6) = dassert(length(unique(df(2:end,5))), 48);
%$ end
%$
%$ T = all(t);
%@eof:8
%@test:9 % test_fetch_series_with_filter_on_one_series
%$ try
%$ filters_ = '[{"code": "interpolate", "parameters": {"frequency": "monthly", "method": "spline"}}]';
%$ df = mdbnomics('provider_code', 'AMECO', 'dataset_code', 'ZUTN', 'series_code', 'DEU.1.0.0.0.ZUTN', 'dbnomics_filters', filters_);
%$ t(1) = 1;
%$ catch
%$ t = 0;
%$ end
%$
%$ if t(1)
%$ t(2) = dassert(length(unique(df(2:end,2))),1);
%$ t(3) = dassert(df(2,2), {'AMECO'});
%$ t(4) = dassert(length(unique(df(2:end,3))),1);
%$ t(5) = dassert(df(2,3), {'ZUTN'});
%$ series_codes = unique(df(2:end,5));
%$ t(6) = dassert(length(unique(df(2:end,5))),2);
%$ t(7) = dassert(df(2,5), {'DEU.1.0.0.0.ZUTN'});
%$ t(8) = dassert(series_codes{2}, 'DEU.1.0.0.0.ZUTN_filtered');
%$ freq_ = unique(df(2:end, 1));
%$ t(9) = dassert(freq_{2}, 'monthly');
%$ t(10) = dassert(df(1,12), {'filtered'});
%$ end
%$
%$ T = all(t);
%@eof:9
%@test:10 % test_fetch_series_with_max_nb_series
%$ try
%$ df = mdbnomics('provider_code', 'AMECO', 'dataset_code', 'ZUTN', 'max_nb_series',20);
%$ t(1) = 1;
%$ catch
%$ t = 0;
%$ end
%$
%$ if t(1)
%$ t(2) = dassert(length(unique(df(2:end,2))),1);
%$ t(3) = dassert(df(2,2), {'AMECO'});
%$ t(4) = dassert(length(unique(df(2:end,3))),1);
%$ t(5) = dassert(df(2,3), {'ZUTN'});
%$ series_codes = unique(df(2:end,5));
%$ assert(length(series_codes) <= 20);
%$ end
%$
%$ T = all(t);
%@eof:10
%@test:11 % test_fetch_series_with_na_values
%$ try
%$ df = mdbnomics('provider_code', 'AMECO', 'dataset_code', 'ZUTN', 'series_code', 'DEU.1.0.0.0.ZUTN');
%$ t(1) = 1;
%$ catch
%$ t = 0;
%$ end
%$
%$ if t(1)
%$ t(2) = dassert(length(unique(df(2:end,2))),1);
%$ t(3) = dassert(df(2,2), {'AMECO'});
%$ t(4) = dassert(length(unique(df(2:end,3))),1);
%$ t(5) = dassert(df(2,3), {'ZUTN'});
%$ t(6) = dassert(length(unique(df(2:end,5))),1);
%$ t(7) = dassert(df(2,5), {'DEU.1.0.0.0.ZUTN'});
%$ assert(any(strcmp('NA', df(2:end,9))) == true);
%$ assert(any(isnan(cell2mat(df(2:end,10)))) == true);
%$ end
%$
%$ T = all(t);
%@eof:11
%@test:12 % test_fetch_series_by_api_link
%$ try
%$ df = mdbnomics('api_link', 'https://api.db.nomics.world/v22/series/BIS/long_pp?limit=1000&offset=0&q=&observations=1&align_periods=1&dimensions=%7B%7D');
%$ t(1) = 1;
%$ catch
%$ t = 0;
%$ end
%$
%$ if t(1)
%$ t(2) = dassert(length(unique(df(2:end,2))),1);
%$ t(3) = dassert(df(2,2), {'BIS'});
%$ t(4) = dassert(length(unique(df(2:end,3))),1);
%$ t(5) = dassert(df(2,3), {'long_pp'});
%$ t(6) = dassert(length(unique(df(2:end,5))),23);
%$ end
%$
%$ T = all(t);
%@eof:12
src/mdbnomics_datasets.m 0 → 100644
View file @ 009fadab
function datasets = mdbnomics_datasets(varargin) % --*-- Unitary tests --*--
% function mdbnomics_datasets(varargin)
% Downloads the list of available datasets for a selection of providers (or all of them) from https://db.nomics.world/.
% By default, the function returns a structure with a cell array containing the dataset codes and names of the requested providers.
%
% POSSIBLE PARAMETERS
% provider_code [char] DBnomics code of one or multiple providers. If empty, the providers are firstly
% dowloaded with the function mdbnomics_providers and then the available datasets are requested.
% simplify [logical] If true, when the datasets are requested for only one provider then a cell array is returned, not a structure.
% If not provided, the default value is false.
%
% OUTPUTS
% datasets
%
% SPECIAL REQUIREMENTS
% none
% Copyright (C) 2020 Dynare Team
%
% This file is part of Dynare.
%
% Dynare is free software: you can redistribute it and/or modify
% it under the terms of the GNU General Public License as published by
% the Free Software Foundation, either version 3 of the License, or
% (at your option) any later version.
%
% Dynare is distributed in the hope that it will be useful,
% but WITHOUT ANY WARRANTY; without even the implied warranty of
% MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
% GNU General Public License for more details.
%
% You should have received a copy of the GNU General Public License
% along with Dynare. If not, see <http://www.gnu.org/licenses/>.
global mdb_options
p = inputParser;
validStringInput = @(x) ischar(x) || iscellstr(x);
p.addParameter('provider_code', '', validStringInput);
p.addParameter('simplify', false, @islogical);
p.KeepUnmatched = false;
p.parse(varargin{:});
if isempty(p.Results.provider_code)
provider_code = mdbnomics_providers('code', true);
else
if ischar(p.Results.provider_code)
provider_code = {p.Results.provider_code};
else
provider_code = p.Results.provider_code;
end
end
datasets = struct();
for i = 1:numel(provider_code)
pc = provider_code{i};
provider_page = sprintf('%s/v%d/providers/%s', mdb_options.api_base_url, mdb_options.api_version, pc);
provider_info = webread(provider_page);
provider_info = provider_info.category_tree;
code = [];
name = [];
if isfield(provider_info, 'children')
unpack_children(provider_info, code, name);
else
try
for n = 1:numel(provider_info)
code = [code, {provider_info{n}.code}];
name = [name, {provider_info{n}.name}];
end
catch
for n = 1:numel(provider_info)
code = [code, {provider_info(n).code}];
name = [name, {provider_info(n).name}];
end
end
end
datasets.(pc) = horzcat(code', name');
end
if p.Results.simplify
if length(fieldnames(datasets)) == 1
datasets = datasets.(pc);
else
error('Your query corresponds to multiple providers, not possible to simplify');
end
end
end
%@test:1
%$ try
%$ datasets = mdbnomics_datasets('provider_code', 'IMF', 'simplify', true);
%$ t(1) = 1;
%$ catch
%$ t = 0;
%$ end
%$
%$ if t(1)
%$ t(2) = dassert(length(unique(datasets(:,1))), 43);
%$ t(3) = dassert(size(datasets, 2), 2);
%$ end
%$
%$ T = all(t);
%@eof:1
%@test:2
%$ try
%$ datasets = mdbnomics_datasets('provider_code', 'IMF');
%$ t(1) = 1;
%$ catch
%$ t = 0;
%$ end
%$
%$ if t(1)
%$ t(2) = dassert(fieldnames(datasets), {'IMF'});
%$ t(3) = dassert(size(datasets.IMF,1), 43);
%$ end
%$
%$ T = all(t);
%@eof:2
%@test:3
%$ try
%$ datasets = mdbnomics_datasets('provider_code', {'IMF', 'AMECO'});
%$ t(1) = 1;
%$ catch
%$ t = 0;
%$ end
%$
%$ if t(1)
%$ t(2) = dassert(fieldnames(datasets), {'IMF'; 'AMECO'});
%$ t(3) = dassert(size(datasets.IMF,1), 43);
%$ t(4) = dassert(size(datasets.AMECO,1), 473);
%$ end
%$
%$ T = all(t);
%@eof:3
\ No newline at end of file
src/mdbnomics_dimensions.m 0 → 100644
View file @ 009fadab
function dimensions = mdbnomics_dimensions(varargin) % --*-- Unitary tests --*--
% function mdbnomics_dimensions(varargin)
% Downloads the list of dimensions (if they exist) for available datasets of a selection of providers from https://db.nomics.world/.
% By default, the function returns a structure containing the dimensions of datasets for DBnomics providers.
%
% POSSIBLE PARAMETERS
% provider_code [char] DBnomics code of one or multiple providers. If empty, the providers are firstly
% dowloaded with the function mdbnomics_providers and then the available datasets are requested.
% dataset_code [char] DBnomics code of one or multiple datasets of a provider. If empty, the datasets codes are dowloaded
% with the function mdbnomics_datasets and then the dimensions are requested.
% simplify [logical] If true, when the dimensions are requested for only one provider and one dataset then only the dimension names and their values are provided.
% If not provided, the default value is false.
%
% OUTPUTS
% dimensions
%
% SPECIAL REQUIREMENTS
% none
% Copyright (C) 2020 Dynare Team
%
% This file is part of Dynare.
%
% Dynare is free software: you can redistribute it and/or modify
% it under the terms of the GNU General Public License as published by
% the Free Software Foundation, either version 3 of the License, or
% (at your option) any later version.
%
% Dynare is distributed in the hope that it will be useful,
% but WITHOUT ANY WARRANTY; without even the implied warranty of
% MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
% GNU General Public License for more details.
%
% You should have received a copy of the GNU General Public License
% along with Dynare. If not, see <http://www.gnu.org/licenses/>.
global mdb_options
p = inputParser;
validStringInput = @(x) ischar(x) || iscellstr(x);
p.addParameter('provider_code', '', validStringInput);
p.addParameter('dataset_code', '', validStringInput);
p.addParameter('simplify', false, @islogical);
p.KeepUnmatched = false;
p.parse(varargin{:});
if isempty(p.Results.provider_code) && ~isempty(p.Results.dataset_code)
error('When you use dataset_code, you must specify provider_code as well.');
end
if iscell(p.Results.provider_code) || iscell(p.Results.dataset_code)
if ~isempty(p.Results.provider_code) && ~isempty(p.Results.dataset_code) && length(p.Results.provider_code) ~= length(p.Results.dataset_code)
error('Please specify as many provider codes as dataset codes.')
end
end
if isempty(p.Results.provider_code)
provider_code = mdbnomics_providers('code', true);
else
if ischar(p.Results.provider_code)
provider_code = {p.Results.provider_code};
else
provider_code = p.Results.provider_code;
end
end
if isempty(p.Results.dataset_code)
dataset_code = mdbnomics_datasets('provider_code', provider_code);
else
if ischar(p.Results.dataset_code)
dataset_code = {p.Results.dataset_code};
else
dataset_code = p.Results.dataset_code;
end
end
dimensions = struct();
for i = 1:numel(provider_code)
pc = provider_code{i};
dc = dataset_code{i};
dataset_page = sprintf('%s/v%d/datasets/%s/%s', mdb_options.api_base_url, mdb_options.api_version, pc, dc);
dataset_info = webread(dataset_page);
dataset_name = sprintf('%s_%s', pc, dc);
try
tmp1 = dataset_info.datasets.docs.dimensions_labels;
catch
try
tmp1 = dataset_info.datasets.(dataset_name).dimensions_labels;
catch
tmp1 = {};
end
end
try
tmp2 = dataset_info.datasets.docs.dimensions_values_labels;
catch
try
tmp2 = dataset_info.datasets.(dataset_name).dimensions_values_labels;
catch
tmp2 = {};
end
end
dataset_dimensions = fieldnames(tmp1);
for d = 1:numel(dataset_dimensions)
dimensions.(dataset_name).(dataset_dimensions{d}) = tmp2.(dataset_dimensions{d});
end
end
if p.Results.simplify
if length(fieldnames(dimensions)) == 1
dimensions = dimensions.(dataset_name);
else
error('Your query corresponds to multiple datasets, not possible to simplify');
end
end
end
%@test:1
%$ try
%$ dimensions = mdbnomics_dimensions('provider_code', 'IMF', 'dataset_code', 'WEO');
%$ t(1) = 1;
%$ catch
%$ t = 0;
%$ end
%$
%$ if t(1)
%$ t(2) = dassert(fieldnames(dimensions), {'IMF_WEO'});
%$ t(3) = dassert(isfield(dimensions.IMF_WEO, 'unit'), true);
%$ t(4) = dassert(length(fieldnames(dimensions.IMF_WEO.unit)), 13);
%$ end
%$
%$ T = all(t);
%@eof:1
%@test:2
%$ try
%$ dimensions = mdbnomics_dimensions('provider_code', 'IMF', 'dataset_code', 'WEO', 'simplify', true);
%$ t(1) = 1;
%$ catch
%$ t = 0;
%$ end
%$
%$ if t(1)
%$ t(2) = dassert(isfield(dimensions, 'IMF_WEO'), false);
%$ t(3) = dassert(isfield(dimensions, 'unit'), true);
%$ t(4) = dassert(length(fieldnames(dimensions.unit)), 13);
%$ end
%$
%$ T = all(t);
%@eof:2
src/mdbnomics_series.m 0 → 100644
View file @ 009fadab
function series = mdbnomics_series(varargin) % --*-- Unitary tests --*--
% function mdbnomics_series(varargin)
% Downloads the list of series for available datasets of a selection of providers from https://db.nomics.world/.
% We warn the user that this function can be (very) long to execute!
% We remind that DBnomics requests data from 63 providers to retrieve 21675 datasets for a total of approximately 720 millions series.
% By default, the function returns a structure with a cell array at the end of each branch containing the series codes and names of datasets for DBnomics providers.
%
% POSSIBLE PARAMETERS
% provider_code [char] DBnomics code of one or multiple providers. If empty, the providers are firstly
% dowloaded with the function mdbnomics_providers and then the available datasets are requested.
% dataset_code [char] DBnomics code of one or multiple datasets of a provider. If empty, the datasets codes are dowloaded
% with the function mdbnomics_datasets and then the dimensions are requested.
% dimensions [char] DBnomics code of one or several dimensions in the specified provider and dataset.
% If provided it must be a string formatted like: '{"country":["ES","FR","IT"],"indicator":["IC.REG.COST.PC.FE.ZS.DRFN"]}'.
% query [char] A query to filter/select series from a provider's dataset.
% only_number_of_series [logical] If true, only the number of series for the given query will be printed in the command window.
% If not provided, the default value is false.
% simplify [logical] If true, when the datasets are requested for only one provider then a cell array is returned, not a structure.
% If not provided, the default value is false.
%
% OUTPUTS
% series
%
% SPECIAL REQUIREMENTS
% none
% Copyright (C) 2020 Dynare Team
%
% This file is part of Dynare.
%
% Dynare is free software: you can redistribute it and/or modify
% it under the terms of the GNU General Public License as published by
% the Free Software Foundation, either version 3 of the License, or
% (at your option) any later version.
%
% Dynare is distributed in the hope that it will be useful,
% but WITHOUT ANY WARRANTY; without even the implied warranty of
% MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
% GNU General Public License for more details.
%
% You should have received a copy of the GNU General Public License
% along with Dynare. If not, see <http://www.gnu.org/licenses/>.
global mdb_options
p = inputParser;
validStringInput = @(x) ischar(x) || iscellstr(x);
p.addParameter('provider_code', '', validStringInput);
p.addParameter('dataset_code', '', validStringInput);
p.addParameter('dimensions', '', validStringInput);
p.addParameter('query', '', validStringInput);
p.addParameter('only_number_of_series', false, @islogical);
p.addParameter('simplify', false, @islogical);
p.KeepUnmatched = false;
p.parse(varargin{:});
if iscell(p.Results.provider_code) || iscell(p.Results.dataset_code)
if ~isempty(p.Results.provider_code) && ~isempty(p.Results.dataset_code) && length(p.Results.provider_code) ~= length(p.Results.dataset_code)
error('Please specify as many provider codes as dataset codes.')
end
end
if isempty(p.Results.provider_code)
provider_code = mdbnomics_providers('code', true);
else
if ischar(p.Results.provider_code)
provider_code = {p.Results.provider_code};
else
provider_code = p.Results.provider_code;
end
end
if isempty(p.Results.dataset_code)
dataset_code = mdbnomics_datasets('provider_code', provider_code);
else
if ischar(p.Results.dataset_code)
dataset_code = {p.Results.dataset_code};
else
dataset_code = p.Results.dataset_code;
end
end
if ~isempty(p.Results.query)
if ischar(p.Results.query)
db_query = {p.Results.query};
else
db_query = p.Results.query;
end
end
if ~isempty(p.Results.dimensions)
if ischar(p.Results.dimensions)
dimensions = {p.Results.dimensions};
else
dimensions = p.Results.dimensions;
end
end
series = struct();
for i = 1:numel(provider_code)
pc = provider_code{i};
dc = dataset_code{i};
dataset_page = sprintf('%s/v%d/series/%s/%s', mdb_options.api_base_url, mdb_options.api_version, pc, dc);
if exist('db_query', 'var')
dataset_page = sprintf('%s?q=%s', dataset_page, db_query{i});
end
if exist('dimensions', 'var')
if contains(dimensions{i}, '\\?')
spec = '&';
else
spec = '?';
end
dataset_page = sprintf('%s%sdimensions=%s', dataset_page, spec, dimensions{i});
end
dataset_info = webread(dataset_page);
dataset_name = sprintf('%s_%s', pc, dc);
limit = dataset_info.series.limit;
num_found = dataset_info.series.num_found;
if p.Results.only_number_of_series
sprintf('Number of series = %d', num_found)
return
else
sprintf('The dataset %s from provider %s contains %d series.', dc, pc, num_found)
series_code = [];
series_name = [];
if num_found > limit
sequence = 0:1:floor(num_found/limit);
if contains(dataset_page, 'offset=')
dataset_page = regexprep(dataset_page, '\\&offset=[0-9]+', '');
dataset_page = regexprep(dataset_page, '\\?offset=[0-9]+', '');
end
if contains(dataset_page, '\\?')
sep = '&';
else
sep = '?';
end
for j = 1:numel(sequence)
tmp_api_link = sprintf('%s%soffset=%d', dataset_page, sep, sequence(j)*limit);
dataset_info = webread(tmp_api_link);
series_info = dataset_info.series.docs;
for s = 1:numel(series_info)
series_code = [series_code, {series_info(s).series_code}];
series_name = [series_name, {series_info(s).series_name}];
end
end
series.(dataset_name) = horzcat(series_code', series_name');
else
series_info = dataset_info.series.docs;
for s = 1:numel(series_info)
series_code = [series_code, {series_info(s).series_code}];
series_name = [series_name, {series_info(s).series_name}];
end
series.(dataset_name) = horzcat(series_code', series_name');
end
end
end
if p.Results.simplify
if length(fieldnames(series)) == 1
series = series.(dataset_name);
else
error('Your query corresponds to multiple datasets, not possible to simplify');
end
end
end
%@test:1
%$ try
%$ series = mdbnomics_series('provider_code', 'IMF', 'dataset_code', 'WEO', 'simplify', true);
%$ t(1) = 1;
%$ catch
%$ t = 0;
%$ end
%$
%$ if t(1)
%$ t(2) = dassert(length(unique(series(:,1))), 8924);
%$ t(3) = dassert(size(series, 2), 2);
%$ end
%$
%$ T = all(t);
%@eof:1
%@test:2
%$ try
%$ series = mdbnomics_series('provider_code', 'IMF', 'dataset_code', 'WEO');
%$ t(1) = 1;
%$ catch
%$ t = 0;
%$ end
%$
%$ if t(1)
%$ t(2) = dassert(fieldnames(series), {'IMF_WEO'});
%$ t(3) = dassert(length(unique(series.IMF_WEO(:,1))), 8924);
%$ t(4) = dassert(size(series.IMF_WEO, 2), 2);
%$ end
%$
%$ T = all(t);
%@eof:2
%@test:3
%$ try
%$ series = mdbnomics_series('provider_code', 'IMF', 'dataset_code', 'WEO', 'dimensions', '{"weo-subject":["NGDP_RPCH"]}', 'simplify', true);
%$ t(1) = 1;
%$ catch
%$ t = 0;
%$ end
%$
%$ if t(1)
%$ t(2) = dassert(length(unique(series(:,1))), 194);
%$ t(3) = dassert(size(series, 2), 2);
%$ end
%$
%$ T = all(t);
%@eof:3
%@test:4
%$ try
%$ series = mdbnomics_series('provider_code', 'IMF', 'dataset_code', 'WEO', 'query', 'NGDP_RPCH');
%$ t(1) = 1;
%$ catch
%$ t = 0;
%$ end
%$
%$ if t(1)
%$ t(2) = dassert(length(unique(series.IMF_WEO(:,1))), 194);
%$ t(3) = dassert(size(series.IMF_WEO, 2), 2);
%$ end
%$
%$ T = all(t);
%@eof:4
\ No newline at end of file
src/missing/contains/contains.m 0 → 100644
View file @ 009fadab
function tf = contains(string, pattern, varargin)
% CONTAINS Returns 1 if the pattern is found in string, and 0 otherwise.
%
% INPUTS
% - string [string, char, cell(str)] String to be searhced.
% - pattern [string, char, cell(str)] The searched pattern.
%
% If 'IgnoreCase',IGNORE is provided, the function ignores case if IGNORE is true.
% The default value is false.
%
% OUTPUT
% - tf [logical]
%
% Copyright (C) 2019 Dynare Team
%
% This file is part of Dynare.
%
% Dynare is free software: you can redistribute it and/or modify
% it under the terms of the GNU General Public License as published by
% the Free Software Foundation, either version 3 of the License, or
% (at your option) any later version.
%
% Dynare is distributed in the hope that it will be useful,
% but WITHOUT ANY WARRANTY; without even the implied warranty of
% MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
% GNU General Public License for more details.
%
% You should have received a copy of the GNU General Public License
% along with Dynare. If not, see <http://www.gnu.org/licenses/>.
if mod(nargin,2) ~= 0
error('contains: inputs should be a text, followed by the queried patterns, then Name-Value pair arguments.');
end
if ~((ischar(string) || isstring(string) || iscellstr(string)) && (ischar(pattern) || isstring(pattern) || iscellstr(pattern)))
error('contains: first and second input arguments must be a string array, char array or cell array');
end
if ~isempty(varargin)
if ~ischar(varargin{1})
error('The #%d and #%d inputs must be a Name-Value pair.', nargin-1, nargin);
end
parameters = {'ignorecase', 'caseignore','insensitivecase', 'caseinsensitive','insensitive','ignore',...
'casesensitive', 'sensitive', 'sensitivecase'};
if ~cellfun('isempty',(cellfun(@(s)strcmpi(varargin{1}, s), parameters, 'uni', 0)))
case_ignore = varargin{end};
else
error('Unsupported parameter "%s".', varargin{1});
end
else
case_ignore = false;
end
string = cellstr(string);
pattern = cellstr(pattern);
if case_ignore
string = lower(string);
pattern = lower(pattern);
end
tf = false(size(string));
for ii = 1:numel(pattern)
idx = regexp(string, pattern{ii});
for jj = 1:numel(string)
tf(jj) = tf(jj) || ~isempty(idx{jj});
end
end
end
\ No newline at end of file