Rework Code Base for Single Instance by JVickery-TBS · Pull Request #1583 · open-data/ckanext-canada

JVickery-TBS · 2025-06-18T20:14:52Z

No description provided.

- Tearing out all unnecessary code.

# Conflicts: # ckanext/canada/tests/test_trim_package.py ### RESOLVED.

codecov · 2025-10-07T16:53:50Z

❌ 1 Tests Failed:

Tests completed	Failed	Passed	Skipped
315	1	314	0

View the top 1 failed test(s) by shortest run time

ckanext/canada/tests/test_domain_map.py::TestDomainMap::test_registry_datastore_urls

Stack Traces | 11.1s run time

self = <ckanext.canada.tests.test_domain_map.TestDomainMap object at 0x7f336555e430>
app = <ckan.tests.helpers.CKANTestApp object at 0x7f33638482b0>
mock_uploads = None

    @mock.patch.object(h, 'is_registry_domain', mock_is_registry_domain)
    @pytest.mark.usefixtures('mock_uploads')
    def test_registry_datastore_urls(self, app, mock_uploads):  # noqa: F811
        """
        Creating XLoader resources should only store relative URIs
        in the database and in SOLR. datastore_search should prepend the
        requesting hostname to the paging URIs.
        """
        queue = get_queue('test_queue')
        queue.empty()
        pkg_id = make_uuid()
    
        offset = h.url_for('dataset.new', locale='en')
        response = app.post(offset,
                            data=self._filled_dataset_form(pkg_id),
                            extra_environ=self.extra_environ_tester_registry,
                            environ_overrides=self.environ_overrides_tester,
                            status=302,
                            follow_redirects=False)  # catch redirect
    
        offset, _host = get_relative_offset_from_response(response)
        assert offset == f'/en/dataset/{pkg_id}/resource/new'
    
        res_form_fields = self._filled_resource_form(pkg_id)
        sample_filepath = get_sample_filepath('sample.csv')
        fake_file_obj = io.BytesIO()
        with open(sample_filepath, 'rb') as f:
            file_data_r1 = f.read()
            fake_file_obj.write(file_data_r1)
            mock_field_store_r1 = MockFieldStorage(fake_file_obj, 'sample.csv')
    
        res_form_fields['url'] = 'sample.csv'
        res_form_fields['url_type'] = 'upload'
        res_form_fields['format'] = 'CSV'
        res_form_fields['upload'] = mock_field_store_r1
    
        response = app.post(offset,
                            data=res_form_fields,
                            extra_environ=self.extra_environ_tester_registry,
                            environ_overrides=self.environ_overrides_tester,
                            follow_redirects=False)  # catch redirect
    
        # load file into the datastore
        pkg_dict = self.sysadmin_action.package_show(id=pkg_id)
    
        def _fake_open(*args, **kwargs):
            return io.BufferedReader(io.BytesIO(file_data_r1))
    
        # should have a ckanext-validation job queued right now
        jobs = queue.jobs
        assert len(jobs) == 1
        job = jobs[0]
        assert job.func_name == 'ckanext.validation.jobs.run_validation_job'
        assert job.meta['title'] == 'Validate Resource'
        with mock.patch('io.open', _fake_open), mock.patch('builtins.open', _fake_open):
            model.Session.commit()
            model.Session.remove()
            _session = model.Session  # noqa: F841
            job.func(*job.args, **job.kwargs)
        queue.remove(job)
        job.delete()
    
        # check resource_validation_show in english
        offset = h.url_for('api.action', ver=3, logic_function='resource_validation_show',
                           resource_id=pkg_dict['resources'][0]['id'], locale='en')
        response = app.get(offset, extra_environ=self.extra_environ_tester_registry,
                           environ_overrides=self.environ_overrides_tester,
                           status=200,
                           follow_redirects=False)  # no need for redirects
        validation_report = response.json['result']
    
        assert validation_report['error'] is None
        assert validation_report['language'] == 'en'
        assert validation_report['resource_id'] == pkg_dict['resources'][0]['id']
        assert validation_report['status'] == 'success'
    
        # check resource_validation_show in french
        offset = h.url_for('api.action', ver=3, logic_function='resource_validation_show',
                           resource_id=pkg_dict['resources'][0]['id'], locale='fr')
        response = app.get(offset, extra_environ=self.extra_environ_tester_registry,
                           environ_overrides=self.environ_overrides_tester,
                           status=200,
                           follow_redirects=False)  # no need for redirects
        validation_report = response.json['result']
    
        assert validation_report['error'] is None
        assert validation_report['language'] == 'fr'
        assert validation_report['resource_id'] == pkg_dict['resources'][0]['id']
        assert validation_report['status'] == 'success'
    
        # submit it to XLoader as the xloader_submit action is what adds ckan_url and original_url
        offset = h.url_for('api.action', ver=3, logic_function='xloader_submit', ignore_hash=True,
                           resource_id=pkg_dict['resources'][0]['id'], locale='en')
        response = app.post(offset,
                            data={
                                'resource_id': pkg_dict['resources'][0]['id'],
                                'ignore_hash': True
                            },
                            extra_environ=self.extra_environ_tester_registry,
                            environ_overrides=self.environ_overrides_tester,
                            status=200,
                            follow_redirects=False)  # no need for redirects
        response = response.json
    
        task = self.sysadmin_action.xloader_status(resource_id=pkg_dict['resources'][0]['id'])
        assert task['error'] == {}
        assert task['status'] == 'pending'
        assert response['help'] == 'http://%.../3/action/help_show?name=xloader_submit' % (
            self.test_domain_map['registry']['en'])
        assert response['success'] is True
    
        # should have a ckanext-xloader job queued right now
        jobs = queue.jobs
        assert len(jobs) == 1
        job = jobs[0]
        assert job.func_name == 'ckanext.xloader.jobs.xloader_data_into_datastore'
        assert job.meta['title'] == 'Upload to DataStore'
        assert job.args[0]['metadata']['ckan_url'] is None
        assert job.args[0]['metadata']['resource_id'] == pkg_dict['resources'][0]['id']
        assert job.args[0]['metadata']['original_url'] == '/dataset/%s/resource/%s/download/sample.csv' % (
            pkg_id, pkg_dict['resources'][0]['id'])
    
        rq_job = mock.Mock()
        rq_job.meta = {}
        rq_job.id = task['job_id']
    
        file_response = mock.Mock()
        file_response.status_code = 200
        file_response.raw = io.BytesIO(file_data_r1)
        file_response.iter_content.return_value = [
            file_data_r1,
        ]
        file_response.headers = {}
        file_response.url = pkg_dict['resources'][0]['url']
    
        with mock.patch('ckanext.xloader.jobs.get_current_job', return_value=rq_job), \
             mock.patch('ckanext.xloader.jobs.requests.get', return_value=file_response):
            # NOTE: because XLoader is coded in a specific way which expects to be inside
            #       of the Redis Job worker process, we need to patch get_current_job.
            # NOTE: because XLoader uses requests.get we have to patch it, as it will always
            #       be outside of the app/test request context.
            model.Session.commit()
            model.Session.remove()
            _session = model.Session  # noqa: F841
            job.func(*job.args, **job.kwargs)
        queue.remove(job)
        job.delete()
    
        # check task_status_show in english
        offset = h.url_for('api.action', ver=3, logic_function='task_status_show',
                           entity_id=pkg_dict['resources'][0]['id'], task_type='xloader',
                           key='xloader', locale='en')
        response = app.get(offset, extra_environ=self.extra_environ_tester_registry,
                           environ_overrides=self.environ_overrides_tester,
                           status=200,
                           follow_redirects=False)  # no need for redirects
        task_status = response.json['result']
    
        assert task_status['error'] == r'{}'
        assert task_status['state'] == 'complete'
    
        # check task_status_show in french
        offset = h.url_for('api.action', ver=3, logic_function='task_status_show',
                           entity_id=pkg_dict['resources'][0]['id'], task_type='xloader',
                           key='xloader', locale='fr')
        response = app.get(offset, extra_environ=self.extra_environ_tester_registry,
                           environ_overrides=self.environ_overrides_tester,
                           status=200,
                           follow_redirects=False)  # no need for redirects
        task_status = response.json['result']
    
        assert task_status['error'] == r'{}'
        assert task_status['state'] == 'complete'
    
        # check xloader_status in english
        offset = h.url_for('api.action', ver=3, logic_function='xloader_status',
                           resource_id=pkg_dict['resources'][0]['id'], locale='en')
        response = app.get(offset, extra_environ=self.extra_environ_tester_registry,
                           environ_overrides=self.environ_overrides_tester,
                           status=200,
                           follow_redirects=False)  # no need for redirects
        xloader_job = response.json['result']
    
        assert xloader_job['error'] == {}
        assert xloader_job['job_id'] == rq_job.id
        assert xloader_job['status'] == 'complete'
        assert xloader_job['task_info']['data']['metadata']['ckan_url'] is None
        assert xloader_job['task_info']['data']['metadata']['resource_id'] == pkg_dict['resources'][0]['id']
        assert xloader_job['task_info']['data']['metadata']['original_url'] == '/dataset/%s/resource/%s/download/sample.csv' % (
            pkg_id, pkg_dict['resources'][0]['id'])
        assert xloader_job['task_info']['metadata']['ckan_url'] is None
        assert xloader_job['task_info']['metadata']['resource_id'] == pkg_dict['resources'][0]['id']
        assert xloader_job['task_info']['metadata']['original_url'] == '/dataset/%s/resource/%s/download/sample.csv' % (
            pkg_id, pkg_dict['resources'][0]['id'])
        assert xloader_job['task_info']['error'] is None
        assert xloader_job['task_info']['job_id'] == rq_job.id
        assert xloader_job['task_info']['status'] == 'complete'
    
        # check xloader_status in french
        offset = h.url_for('api.action', ver=3, logic_function='xloader_status',
                           resource_id=pkg_dict['resources'][0]['id'], locale='fr')
        response = app.get(offset, extra_environ=self.extra_environ_tester_registry,
                           environ_overrides=self.environ_overrides_tester,
                           status=200,
                           follow_redirects=False)  # no need for redirects
        xloader_job = response.json['result']
    
        assert xloader_job['error'] == {}
        assert xloader_job['job_id'] == rq_job.id
        assert xloader_job['status'] == 'complete'
        assert xloader_job['task_info']['data']['metadata']['ckan_url'] is None
        assert xloader_job['task_info']['data']['metadata']['resource_id'] == pkg_dict['resources'][0]['id']
        assert xloader_job['task_info']['data']['metadata']['original_url'] == '/dataset/%s/resource/%s/download/sample.csv' % (
            pkg_id, pkg_dict['resources'][0]['id'])
        assert xloader_job['task_info']['metadata']['ckan_url'] is None
        assert xloader_job['task_info']['metadata']['resource_id'] == pkg_dict['resources'][0]['id']
        assert xloader_job['task_info']['metadata']['original_url'] == '/dataset/%s/resource/%s/download/sample.csv' % (
            pkg_id, pkg_dict['resources'][0]['id'])
        assert xloader_job['task_info']['error'] is None
        assert xloader_job['task_info']['job_id'] == rq_job.id
        assert xloader_job['task_info']['status'] == 'complete'
    
        # check package_show in english
        offset = h.url_for('api.action', ver=3, logic_function='package_show', id=pkg_id, locale='en')
        response = app.get(offset, extra_environ=self.extra_environ_tester_registry,
                           environ_overrides=self.environ_overrides_tester,
                           status=200,
                           follow_redirects=False)  # no need for redirects
        pkg_dict = response.json['result']
        res_dict = pkg_dict['resources'][0]
    
        assert res_dict['url'] == 'http://%s/en/dataset/%s/resource/%s/download/sample.csv' % (
            self.test_domain_map['registry']['en'], pkg_id, pkg_dict['resources'][0]['id'])
        assert res_dict['original_url'] == 'http://%s/en/dataset/%s/resource/%s/download/sample.csv' % (
            self.test_domain_map['registry']['en'], pkg_id, pkg_dict['resources'][0]['id'])
    
        # check package_show in french
        offset = h.url_for('api.action', ver=3, logic_function='package_show', id=pkg_id, locale='fr')
        response = app.get(offset, extra_environ=self.extra_environ_tester_registry,
                           environ_overrides=self.environ_overrides_tester,
                           status=200,
                           follow_redirects=False)  # no need for redirects
        pkg_dict = response.json['result']
        res_dict = pkg_dict['resources'][0]
    
        assert res_dict['url'] == 'http://%s/fr/dataset/%s/resource/%s/download/sample.csv' % (
            self.test_domain_map['registry']['fr'], pkg_id, pkg_dict['resources'][0]['id'])
>       assert res_dict['original_url'] == 'http://%s/fr/dataset/%s/resource/%s/download/sample.csv' % (
            self.test_domain_map['registry']['fr'], pkg_id, pkg_dict['resources'][0]['id'])
E       AssertionError: assert 'http://regis...ad/sample.csv' == 'http://regis...ad/sample.csv'
E         - http://registry-test.ckan..../98f0ada6-53a6-4f1e-afa1-e5db6a71de16/download/sample.csv
E         ?                               ^^
E         + http://registry-test.ckan..../98f0ada6-53a6-4f1e-afa1-e5db6a71de16/download/sample.csv
E         ?                               ^^

.../canada/tests/test_domain_map.py:1237: AssertionError

To view more test analytics, go to the Test Analytics Dashboard
_{📋 Got 3 mins? Take this short survey to help us improve Test Analytics.}

- Syntax fixes. - Typing fixes. - Started working on the dataset visibility validator. - Added more validation to portal release date field. - Changed the published search facet to capacity. - Set capacity=private for non standard dataset types.

- After validator for dataset visibility. - Added more sysadmin UI for user publishing perms.

- Side effect free.

- Removed geozoom.

- Fix different logic for single instance.

- Force True for non portal package types. - Override activity stuff for logged out users to show a dummy user. - Minor template UI fixes. - Added UI for private/public packages. - Removed old logic methods. - Removed anon actions. - Use fq_list instead of complicated fq for package search. - Handled remaining search issues between registry and portal. - Added helper cli subcommand to set non portal package types to private. - Added more bootstrap4+ classes.

- Improved some code.

- Language switcher for language domains. - Minor remplate fixes. - Pinned required dependency versions.

- Try/catch for dataset search.

- Flake8 fix.

wardi

looks good, just a few minor suggestions

- Have to specify HTTP_HOST for test app contexts.

# Conflicts: # ckanext/canada/plugin/theme_plugin.py ### RESOLVED.

# Conflicts: # ckanext/canada/assets/internal/canada_internal.css # ckanext/canada/assets/public/canada_public.css # ckanext/canada/templates/admin/portal_sync.html ### RESOLVED.

JVickery-TBS added 2 commits June 18, 2025 18:33

removal(dev): tear it out;

9dc480b

- Tearing out all unnecessary code.

removal(dev): tear it out;

41c979d

- Tearing out all unnecessary code.

JVickery-TBS requested a review from wardi June 18, 2025 20:14

JVickery-TBS assigned wardi and JVickery-TBS Jun 18, 2025

JVickery-TBS added enhancement Business Requirement labels Jun 18, 2025

JVickery-TBS added 3 commits October 7, 2025 16:27

Merge branch 'refactor/is-registry' into feature/single-instance

380699a

# Conflicts: # ckanext/canada/tests/test_trim_package.py ### RESOLVED.

Merge branch 'refactor/is-registry' into feature/single-instance

3c052ea

# Conflicts: # ckanext/canada/tests/test_trim_package.py ### RESOLVED.

Merge branch 'refactor/is-registry' into feature/single-instance

db9fabb

JVickery-TBS added 17 commits October 7, 2025 20:29

feat(dev): validators, facets, syntax;

cdc4f75

- Syntax fixes. - Typing fixes. - Started working on the dataset visibility validator. - Added more validation to portal release date field. - Changed the published search facet to capacity. - Set capacity=private for non standard dataset types.

feat(dev): validators;

6bf2a1b

- After validator for dataset visibility. - Added more sysadmin UI for user publishing perms.

Merge branch 'master' into feature/single-instance

978bb86

fix(logic): user show;

b996589

- Side effect free.

removal(js): webassets;

3c210e9

- Removed geozoom.

fix(dev): misc logic;

1e815eb

- Fix different logic for single instance.

fix(dev): misc logic;

2d3fd89

- Fix different logic for single instance.

feat(dev): some TODOs;

0cc412e

- Improved some code.

Merge branch 'master' into feature/single-instance

5243dd8

Merge branch 'master' into feature/single-instance

9502402

Merge branch 'master' into feature/single-instance

11187cd

Merge branch 'master' into feature/single-instance

8334f52

Merge branch 'master' into feature/single-instance

91d6ef4

fix(dev): language switcher;

4033abe

- Language switcher for language domains. - Minor remplate fixes. - Pinned required dependency versions.

fix(logic): dataset search;

44e80c4

- Try/catch for dataset search.

fix(syntax): lint;

b4dd6d1

- Flake8 fix.

JVickery-TBS marked this pull request as ready for review October 17, 2025 14:54

wardi reviewed Oct 17, 2025

View reviewed changes

JVickery-TBS added 30 commits July 15, 2026 15:46

debug github wokflow;

e25741a

debug github wokflow;

620d456

fix(tests): test app host;

9db2009

- Have to specify HTTP_HOST for test app contexts.

fix(tests): test app host;

9305418

- Have to specify HTTP_HOST for test app contexts.

fix(tests): test app host;

b720e12

- Have to specify HTTP_HOST for test app contexts.

fix(tests): test app host;

06dde4f

- Have to specify HTTP_HOST for test app contexts.

fix(tests): test app host;

b926b69

- Have to specify HTTP_HOST for test app contexts.

fix(tests): test app host;

10059f3

- Have to specify HTTP_HOST for test app contexts.

fix(tests): test app host;

7440fe9

- Have to specify HTTP_HOST for test app contexts.

fix(tests): test app host;

20fb440

- Have to specify HTTP_HOST for test app contexts.

fix(tests): test app host;

1fa3b08

- Have to specify HTTP_HOST for test app contexts.

fix(tests): test app host;

292074f

- Have to specify HTTP_HOST for test app contexts.

fix(tests): test app host;

369b533

- Have to specify HTTP_HOST for test app contexts.

fix(tests): test app host;

f46a31c

- Have to specify HTTP_HOST for test app contexts.

fix(tests): test app host;

b80891a

- Have to specify HTTP_HOST for test app contexts.

fix(tests): test app host;

81ef725

- Have to specify HTTP_HOST for test app contexts.

fix(tests): test app host;

6239e1e

- Have to specify HTTP_HOST for test app contexts.

fix(tests): test app host;

1f9a83e

- Have to specify HTTP_HOST for test app contexts.

fix(tests): test app host;

e9a1995

- Have to specify HTTP_HOST for test app contexts.

debug github wokflow;

52cd8d3

debug github wokflow;

1e6d8cb

debug github wokflow;

6b4ca04

debug github wokflow;

0469d7f

debug github wokflow;

4f72efb

debug github wokflow;

3077045

debug github wokflow;

087afd2

Merge branch 'master' into feature/single-instance

355a5c8

Merge branch 'master' into feature/single-instance

4cbccf5

# Conflicts: # ckanext/canada/plugin/theme_plugin.py ### RESOLVED.

Merge branch 'master' into feature/single-instance

2ecece6

Merge branch 'master' into feature/single-instance

9f48e74

# Conflicts: # ckanext/canada/assets/internal/canada_internal.css # ckanext/canada/assets/public/canada_public.css # ckanext/canada/templates/admin/portal_sync.html ### RESOLVED.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Rework Code Base for Single Instance - #1583

Rework Code Base for Single Instance#1583
JVickery-TBS wants to merge 136 commits into
masterfrom
feature/single-instance

JVickery-TBS commented Jun 18, 2025

Uh oh!

codecov Bot commented Oct 7, 2025 •

edited

Loading

Uh oh!

wardi left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

JVickery-TBS commented Jun 18, 2025

Uh oh!

codecov Bot commented Oct 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

❌ 1 Tests Failed:

Uh oh!

wardi left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

codecov Bot commented Oct 7, 2025 •

edited

Loading